fio.git
21 months agoFio 3.2 fio-3.2
Jens Axboe [Fri, 3 Nov 2017 21:23:49 +0000 (15:23 -0600)]
Fio 3.2

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoMerge branch 'json_bw_bytes' of https://github.com/sitsofe/fio
Jens Axboe [Fri, 3 Nov 2017 18:37:03 +0000 (12:37 -0600)]
Merge branch 'json_bw_bytes' of https://github.com/sitsofe/fio

21 months agosolaris: add get_fs_free_size() implementation
Tomohiro Kusumi [Fri, 3 Nov 2017 16:46:03 +0000 (18:46 +0200)]
solaris: add get_fs_free_size() implementation

Copied from os/os-freebsd.h. POSIX statvfs works on Solaris
(actually compiled/tested on illumos) as well.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agosolaris: add os_phys_mem() implementation
Tomohiro Kusumi [Fri, 3 Nov 2017 16:46:02 +0000 (18:46 +0200)]
solaris: add os_phys_mem() implementation

Copied from os/os-linux.h. The standard sysconf(3C) names work on
Solaris (actually compiled/tested on illumos) as well.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agosolaris: #include <pthread.h>
Tomohiro Kusumi [Fri, 3 Nov 2017 16:46:01 +0000 (18:46 +0200)]
solaris: #include <pthread.h>

os/os-solaris.h header uses pthread, but doesn't include <pthread.h>
which should be to be able to include this header alone.
Needed for unittesting of OS specific stuff.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agostat: add bw_bytes JSON key
Sitsofe Wheeler [Fri, 3 Nov 2017 07:23:55 +0000 (07:23 +0000)]
stat: add bw_bytes JSON key

Fixes: https://github.com/axboe/fio/issues/422
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agoio_u_queue: convert rings to bool
Jens Axboe [Thu, 2 Nov 2017 18:26:39 +0000 (12:26 -0600)]
io_u_queue: convert rings to bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofilesetup: allocate 'r' locally in fallocate_file()
Jens Axboe [Thu, 2 Nov 2017 15:09:31 +0000 (09:09 -0600)]
filesetup: allocate 'r' locally in fallocate_file()

For platforms that don't have posix or linux fallocate, we trigger
a warning on 'r' being unused.

Fixes: c0aabe37db1b ("filesetup: don't print non-debug error on native fallocate failure")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofilesetup: recurse_dir() can use bool
Jens Axboe [Thu, 2 Nov 2017 15:07:11 +0000 (09:07 -0600)]
filesetup: recurse_dir() can use bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofilesetup: create_work_dirs() can return bool
Jens Axboe [Thu, 2 Nov 2017 15:06:14 +0000 (09:06 -0600)]
filesetup: create_work_dirs() can return bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofilesetup: change random file init to be bool based
Jens Axboe [Thu, 2 Nov 2017 15:04:56 +0000 (09:04 -0600)]
filesetup: change random file init to be bool based

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofilesetup: __init_rand_distribution() can be void
Jens Axboe [Thu, 2 Nov 2017 15:00:04 +0000 (09:00 -0600)]
filesetup: __init_rand_distribution() can be void

It can no longer fail, just make it void.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofilesetup: pre_read_files() can use a bool
Jens Axboe [Thu, 2 Nov 2017 14:59:07 +0000 (08:59 -0600)]
filesetup: pre_read_files() can use a bool

This flips the success return from 0 to true.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofilesetup: don't inline native_fallocate()
Jens Axboe [Thu, 2 Nov 2017 14:55:44 +0000 (08:55 -0600)]
filesetup: don't inline native_fallocate()

The compiler will figure this out, and it isn't a performance
hot path.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofilesetup: don't print non-debug error on native fallocate failure
Jens Axboe [Thu, 2 Nov 2017 14:53:54 +0000 (08:53 -0600)]
filesetup: don't print non-debug error on native fallocate failure

We expect this to fail in cases where it isn't available, so don't
clutter the standard/error output with it.

Also ensure that native_fallocate() returns true/false, since it's
a bool function. Just a cleanup, no functional change there.

Fixes: 2c3e17be4c7c ("filesetup: add native fallocate")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoio_u: reset file to initial offset
Jens Axboe [Wed, 1 Nov 2017 20:51:03 +0000 (14:51 -0600)]
io_u: reset file to initial offset

Don't assume that initial offset is 0, we should use the set
f->file_offset when resetting.

Fixes: 17373ce2f38a ("io_u: wrap to beginning when end-of-file is reached for time_based")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoio_u: wrap to beginning when end-of-file is reached for time_based
Jens Axboe [Wed, 1 Nov 2017 17:02:30 +0000 (11:02 -0600)]
io_u: wrap to beginning when end-of-file is reached for time_based

The logic around using io_size isn't correct.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoengines/windowsaio: style fixups
Jens Axboe [Tue, 31 Oct 2017 20:01:16 +0000 (14:01 -0600)]
engines/windowsaio: style fixups

No functional changes in this patch.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoDefault buffer_compress_chunk to 512
Jens Axboe [Tue, 31 Oct 2017 16:50:03 +0000 (10:50 -0600)]
Default buffer_compress_chunk to 512

This makes us behave better out of the box, for the cases where
people use bigger block sizes. Without this change and if the
user set buffer_compress_percentage=50 and bs=128k, then we'd
have 64k of zeroes at the end. This is probably not what the
user expects.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoUpdate compression documentation
Jens Axboe [Tue, 31 Oct 2017 16:18:54 +0000 (10:18 -0600)]
Update compression documentation

Make it clearer (hopefully) that buffer_compress_chunk should be
used if the user wants a finer granularity than the write block
size.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoinit: make sure that compression enables refill buffers
Jens Axboe [Tue, 31 Oct 2017 14:55:02 +0000 (08:55 -0600)]
init: make sure that compression enables refill buffers

Otherwise the results aren't going to be great.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoMerge branch 'misc' of https://github.com/sitsofe/fio
Jens Axboe [Mon, 30 Oct 2017 15:05:43 +0000 (09:05 -0600)]
Merge branch 'misc' of https://github.com/sitsofe/fio

Pull misc packaging fixes from Sitsofe:

"Fixes to appease stricter packaging guidelines and some documentation
changes.  The big one is updating the COPYING file in fio to use the one
from https://www.gnu.org/licenses/old-licenses/gpl-2.0.txt . It's worth
double checking that it was a like-for-like replacement given the
significance of the file..."

21 months agodoc: rewrite write_*_log sections
Sitsofe Wheeler [Sun, 29 Oct 2017 16:08:51 +0000 (16:08 +0000)]
doc: rewrite write_*_log sections

- Put the big example of how log files work in write_bw_log and make the
  other sections reference that option so there's less repetition.
- Add a note that the write_hist_log depends on log_hist_msec.
- Make a link from log_hist_coarseness to write_hist_log.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agodoc: minor formatting fixes
Sitsofe Wheeler [Sun, 29 Oct 2017 09:29:38 +0000 (09:29 +0000)]
doc: minor formatting fixes

- Fix broken HOWTO formatting in serialize_overlap.
- Make a link to (and correct the name of) latency_target option within
  Interpreting the output, IO latency.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agodoc: IO -> I/O, sync filecreate documentation
Sitsofe Wheeler [Sun, 29 Oct 2017 09:08:14 +0000 (09:08 +0000)]
doc: IO -> I/O, sync filecreate documentation

Change IO -> I/O and copy the filecreate description from the HOWTO to
the man page.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agofio: update FSF address
Sitsofe Wheeler [Sun, 29 Oct 2017 09:04:24 +0000 (09:04 +0000)]
fio: update FSF address

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agoCOPYING: update license file
Sitsofe Wheeler [Sun, 29 Oct 2017 08:45:48 +0000 (08:45 +0000)]
COPYING: update license file

Change COPYING to use the most recent version of the GPLv2 license file
making the following changes:
- Updates the FSF address.
- Changes some formatting.
- Changes "GNU Library General Public License" to "GNU Lesser General
  Public License".

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agodoc: fix groff line that started with a dot
Sitsofe Wheeler [Sun, 29 Oct 2017 08:29:12 +0000 (08:29 +0000)]
doc: fix groff line that started with a dot

It's illegal to start a groff line with a dot if it's not part of a
macro so use the workaround described on
https://stackoverflow.com/questions/11469341/escaping-dots-in-groff/11566859#11566859
to solve the problem.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agofio: fix interpreter lines
Sitsofe Wheeler [Sun, 29 Oct 2017 16:24:24 +0000 (16:24 +0000)]
fio: fix interpreter lines

Minor changes to adhere to various distro packaging guidelines (e.g.
https://fedoraproject.org/wiki/Packaging:Guidelines#Shebang_lines ):

- doc/conf.py doesn't need an interpreter line because it's not
  executable.
- Make the genfio interpreter line use an absolute path rather than env.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agotools: use /usr/bin/python2.7 as the interpreter
Sitsofe Wheeler [Sun, 29 Oct 2017 08:05:47 +0000 (08:05 +0000)]
tools: use /usr/bin/python2.7 as the interpreter

Distros such as Fedora
(https://fedoraproject.org/wiki/Packaging:Python#Multiple_Python_Runtimes
) are clamping down on python interpreter paths due to the desire to
switch to python3 so /usr/bin/python or /usr/bin/env python are no
longer liked. Unfortunately macOS doesn't ship a /usr/bin/python2 link
so switch all python interpreter paths to /usr/bin/python2.7 as a
compromise.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agosteadystate_tests.py: fix up usage comment
Sitsofe Wheeler [Sun, 29 Oct 2017 07:47:36 +0000 (07:47 +0000)]
steadystate_tests.py: fix up usage comment

Make the comment about usage match the argparse line.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
21 months agoFix 'nice' parameter range: should be -20 to 19, not -19 to 20.
Rebecca Cran [Thu, 26 Oct 2017 21:28:23 +0000 (15:28 -0600)]
Fix 'nice' parameter range: should be -20 to 19, not -19 to 20.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoio_u: re-invalidate cache when looping around without file open/close
Jens Axboe [Thu, 26 Oct 2017 18:08:20 +0000 (12:08 -0600)]
io_u: re-invalidate cache when looping around without file open/close

If we're doing buffered IO and we end up wrapping around for a
time based run, then ensure that we re-invalidate the kernel
cache for the file.

Reported-by: Paolo Valente <paolo.valente@linaro.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoAdd offset_align option
Jeff Furlong [Thu, 26 Oct 2017 17:47:03 +0000 (11:47 -0600)]
Add offset_align option

This allows us to align the start offsets, if percentage offsets
are used.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoparse: minimum options values are signed
Jens Axboe [Wed, 25 Oct 2017 19:06:40 +0000 (13:06 -0600)]
parse: minimum options values are signed

Rebecca reports that using an invalid value for the nice
parameter yields:

min value out of range: 18446744073709551596 (4294967277 min)

where it really should be:

min value out of range: -20 (-19 min)

Print the minimum values as signed, if we hit the out-of-range
error.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agodoc: minor formatting fixes
Sitsofe Wheeler [Fri, 20 Oct 2017 06:08:37 +0000 (07:08 +0100)]
doc: minor formatting fixes

- Sphinx doesn't like non-punctuation characters directly after closing
  backticks unless they are escaped so add some backslashes
  appropriately.
- Markup /proc/sys/kernel/shmmax in --max-jobs like other /proc/
  references.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
22 months agoMerge branch 'cpuclock-test'
Jens Axboe [Tue, 17 Oct 2017 18:59:40 +0000 (12:59 -0600)]
Merge branch 'cpuclock-test'

Pulling in this fix, as Sitsofe tested it and didn't see any failures.
I wasn't able to trigger the failure here myself, either before or
after the fix...

22 months agogettime: improve cpu clock test
Jens Axboe [Tue, 17 Oct 2017 18:11:44 +0000 (12:11 -0600)]
gettime: improve cpu clock test

We're missing a synchronization before reading and comparing
with the current *t->seq in the loop. Rewrite the loop to
use compare-and-exchange for the increment, and ensure that
we have the proper synchronization inserted. This should both
be faster and more reliable.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoflow: fix bad overflowing math
Jens Axboe [Thu, 12 Oct 2017 16:54:27 +0000 (10:54 -0600)]
flow: fix bad overflowing math

No point in multiplying with a 1/-1 integer, just assign a local
variable appropriately.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoMerge branch 'master' of https://github.com/Venutiwa/fio
Jens Axboe [Thu, 12 Oct 2017 16:31:17 +0000 (10:31 -0600)]
Merge branch 'master' of https://github.com/Venutiwa/fio

22 months agoMerge branch 'overflow_fix' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 12 Oct 2017 16:22:55 +0000 (10:22 -0600)]
Merge branch 'overflow_fix' of https://github.com/sitsofe/fio

22 months agogettime: fix cycles_per_msec overflow when using 32 bit longs
Sitsofe Wheeler [Thu, 12 Oct 2017 16:09:48 +0000 (17:09 +0100)]
gettime: fix cycles_per_msec overflow when using 32 bit longs

Compiling fio with clang's undefined behaviour sanitizer and unsigned
wraparound detection enabled on a 32 bit Linux build turned up the
following:

gettime.c:313:28: runtime error: unsigned integer overflow: 3600 * 2600730 cannot be represented in type 'unsigned long'

Fix this by making cycles_per_msec a long long.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
22 months agoFix more overflows in percentile calculation for Windows
Andrzej Jakowski [Thu, 12 Oct 2017 14:31:06 +0000 (08:31 -0600)]
Fix more overflows in percentile calculation for Windows

The previous patch caught one place where we could potentially
overflow for large number of IOs, this patches up a few more.

Signed-off-by: Andrzej Jakowski <andrzej.jakowski@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoFix overflow in percentile calculation for Windows
Andrzej Jakowski [Wed, 11 Oct 2017 22:26:00 +0000 (16:26 -0600)]
Fix overflow in percentile calculation for Windows

This patch fixes percentile calculation for Windows where overflow
on 32 bit type happened while calculating actual percentiles.

Signed-off-by: Andrzej Jakowski <andrzej.jakowski@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agofio: kill td_ioengine_flags()
Jens Axboe [Wed, 11 Oct 2017 20:52:32 +0000 (14:52 -0600)]
fio: kill td_ioengine_flags()

It's unused.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoWindows mkdir() fix
Jens Axboe [Wed, 11 Oct 2017 20:25:09 +0000 (14:25 -0600)]
Windows mkdir() fix

Apparently mingw mkdir() only takes the path as an argument,
it doesn't include a permission mode...

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoadd documentation about filename_format directory behavior
Josef Bacik [Wed, 11 Oct 2017 20:15:19 +0000 (16:15 -0400)]
add documentation about filename_format directory behavior

We've changed the behavior of filename_format, document that fact.

Signed-off-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agouse mkdir instead of mkdirat
Josef Bacik [Wed, 11 Oct 2017 20:15:18 +0000 (16:15 -0400)]
use mkdir instead of mkdirat

Windows doesn't have a way to do mkdirat, so convert to mkdir instead.

Signed-off-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoFix broken path separator definition on Windows
Jens Axboe [Wed, 11 Oct 2017 19:07:34 +0000 (13:07 -0600)]
Fix broken path separator definition on Windows

Fixes: 53a7af851836 ("convert FIO_OS_PATH_SEPARATOR to a character")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agocreate subdirs if specified in the filename_format
Josef Bacik [Wed, 11 Oct 2017 18:58:12 +0000 (14:58 -0400)]
create subdirs if specified in the filename_format

Currently there's no way for a user to specify a directory that doesn't
exist yet for a job.  Fix this by looking for paths in the file name
provided by the job and creating any subdirectories that may be
required.

Signed-off-by: Josef Bacik <jbacik@fb.com>
Removed TD_ENG_FLAG_SHIFT change, not needed.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoconvert FIO_OS_PATH_SEPARATOR to a character
Josef Bacik [Wed, 11 Oct 2017 18:58:11 +0000 (14:58 -0400)]
convert FIO_OS_PATH_SEPARATOR to a character

Instead of a string let's use this as a character so we can search for
it in path names.

Signed-off-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoError if td flags overlap with engine flags
Jens Axboe [Wed, 11 Oct 2017 18:28:50 +0000 (12:28 -0600)]
Error if td flags overlap with engine flags

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agofio: rearrange TD_F_ flag logic
Jens Axboe [Wed, 11 Oct 2017 18:26:37 +0000 (12:26 -0600)]
fio: rearrange TD_F_ flag logic

In preparation for checking if we need to bump TD_ENG_FLAG_SHIFT.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agofio: kill unused TD_F_ flag
Jens Axboe [Wed, 11 Oct 2017 18:23:28 +0000 (12:23 -0600)]
fio: kill unused TD_F_ flag

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoMerge branch 'fgp_fixes' of https://github.com/sitsofe/fio
Jens Axboe [Wed, 11 Oct 2017 14:26:28 +0000 (08:26 -0600)]
Merge branch 'fgp_fixes' of https://github.com/sitsofe/fio

22 months agoengines/windowsaio: style
Jens Axboe [Wed, 11 Oct 2017 14:25:10 +0000 (08:25 -0600)]
engines/windowsaio: style

Don't use braces for a single line block.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoMerge branch 'windowaio_invalidate' of https://github.com/sitsofe/fio
Jens Axboe [Wed, 11 Oct 2017 14:24:54 +0000 (08:24 -0600)]
Merge branch 'windowaio_invalidate' of https://github.com/sitsofe/fio

22 months agoMerge branch 'ci_and_configure' of https://github.com/sitsofe/fio
Jens Axboe [Wed, 11 Oct 2017 14:23:41 +0000 (08:23 -0600)]
Merge branch 'ci_and_configure' of https://github.com/sitsofe/fio

22 months agobackend: don't dereference ->io_ops in reap_threads()
Justin Eno [Tue, 10 Oct 2017 17:54:54 +0000 (11:54 -0600)]
backend: don't dereference ->io_ops in reap_threads()

Without some type of mutex, td->io_ps cannot be dereferenced safely in
reap_threads().

Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoblktrace: use for_each_file()
Jens Axboe [Tue, 10 Oct 2017 16:46:07 +0000 (10:46 -0600)]
blktrace: use for_each_file()

Instead of open-coding its own version.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoengines/filecreate: set data direction for stats
Jens Axboe [Tue, 10 Oct 2017 14:27:38 +0000 (08:27 -0600)]
engines/filecreate: set data direction for stats

When we init the engine, use whatever data direction the job has
for the latency we log. This means that we're no longer dependent
on the job file setting rw=read/randread or not setting it at all.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoHOWTO: include note about increasing shared memory limits
Jens Axboe [Tue, 10 Oct 2017 14:12:59 +0000 (08:12 -0600)]
HOWTO: include note about increasing shared memory limits

Similarly to commit 7f4811bb36d3, but for the HOWTO.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoMerge branch 'pr/note-for-shmmax' of https://github.com/taghos/fio
Jens Axboe [Tue, 10 Oct 2017 14:12:05 +0000 (08:12 -0600)]
Merge branch 'pr/note-for-shmmax' of https://github.com/taghos/fio

22 months agoAdd note for increasing shmmax if necessary
Ricardo Nabinger Sanchez [Mon, 9 Oct 2017 23:57:27 +0000 (20:57 -0300)]
Add note for increasing shmmax if necessary

When launching a large job, fio can attempt to grab more shared memory
than allowed by the kernel.  Hint the user about increasing such limit
on Linux systems.

Signed-off-by: Ricardo Nabinger Sanchez <rnsanchez@taghos.com.br>
22 months agoUpdate file creation example
Jens Axboe [Mon, 9 Oct 2017 20:42:45 +0000 (14:42 -0600)]
Update file creation example

Don't use a write data direction, use read or just don't set it
as fio defaults to read.

This ensures that all stats are printed.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoadd an filecreate example file to examples/
Josef Bacik [Mon, 9 Oct 2017 20:35:51 +0000 (16:35 -0400)]
add an filecreate example file to examples/

This adds an example with some documentation on how to use
ioengine=filecreate.

Signed-off-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoadd FIO_FILENOHASH ioengine flag
Josef Bacik [Mon, 9 Oct 2017 20:31:02 +0000 (16:31 -0400)]
add FIO_FILENOHASH ioengine flag

filecreate doesn't use the file hash table, so don't bother doing shared
memory allocations for the file objects, ye olde malloc will work just
fine.  This way we can create millions of files without jacking up
--alloc-size.

Signed-off-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoengine/filecreate: use clat and reads for stats
Jens Axboe [Mon, 9 Oct 2017 19:13:13 +0000 (13:13 -0600)]
engine/filecreate: use clat and reads for stats

Fio defaults to reads, we should use the same direction as the job
though. Additionally, we must use clat latencies for tracking,
as this is a sync engine and whole latencies are logged as
completions.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agotime: add ntime_since_now()
Jens Axboe [Mon, 9 Oct 2017 18:58:26 +0000 (12:58 -0600)]
time: add ntime_since_now()

Convert filecreate engine to using it as well.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agofilesetup: don't track file allocation for jobs == 1
Jens Axboe [Mon, 9 Oct 2017 18:37:14 +0000 (12:37 -0600)]
filesetup: don't track file allocation for jobs == 1

We only really need this for multiple jobs, and we only check it
if we are in job > 1. But we always mark the file as allocated.
For jobs with lots of files (millions), this takes a long time
to do at setup time.

Running this sample job:

./fio --alloc-size=131072 --name=filecreate --ioengine=filecreate --size=1g --nr_files=1m --openfiles=256

takes ~16s before, and about ~7s afterwards.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoengines/filecreate: set FIO_NOSTATS flag
Jens Axboe [Mon, 9 Oct 2017 16:11:19 +0000 (10:11 -0600)]
engines/filecreate: set FIO_NOSTATS flag

Before this change, we bundle the fake IO latencies with the file
open latencies. That's not intended. Add a flag for IO engines to
tell the core to ignore any IO latencies.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoengines/filecreate: don't use file hash
Jens Axboe [Mon, 9 Oct 2017 16:10:39 +0000 (10:10 -0600)]
engines/filecreate: don't use file hash

We just care about the file open latencies, so pointless to
utilize the hash as regular engines do.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoengines/filecreate: a few fixes
Jens Axboe [Mon, 9 Oct 2017 15:13:52 +0000 (09:13 -0600)]
engines/filecreate: a few fixes

- Ensure that it shows up in the engine list
- Ensure that we actually do anything, if the user configures
  the size too small.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoadd a filecreate engine
Josef Bacik [Fri, 6 Oct 2017 19:43:14 +0000 (15:43 -0400)]
add a filecreate engine

This is for doing empty file creation jobs much like what we do with
fs_mark.  You still need to specify a filesize so that fio will pretend
to do something, but the IO won't actually be done on the file.

Signed-off-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agostat: update json+ output format for --lat_percentiles option
Vincent Fu [Mon, 9 Oct 2017 14:47:12 +0000 (10:47 -0400)]
stat: update json+ output format for --lat_percentiles option

If --lat_percentiles is set to 1, change the json+ output format
so that the bins are inside the 'lat_ns' object.

Also update fio_jsonplus_clat2csv to look in the two possible
locations.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agostat: update description of clat accounting in stat.h
Vincent Fu [Mon, 9 Oct 2017 14:47:11 +0000 (10:47 -0400)]
stat: update description of clat accounting in stat.h

The description in stat.h of the data structure used for
storing completion latency percentiles had not been updated
to reflect the transition to nanosecond timing. This patch
updates the description to be consistent with the nanosecond
changes. There have been a few instances on github and the
mailing list where this had caused confusion.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoAdding support for multiple jobs for fio test
Venu [Mon, 9 Oct 2017 05:57:28 +0000 (00:57 -0500)]
Adding support for multiple jobs for fio test

Adding support for multiple jobs for fio test, adding 6 job files.

Signed-off-by: Venu <venutiwa@in.ibm.com>
22 months agowindowsaio: add best effort cache invalidation
Sitsofe Wheeler [Tue, 26 Sep 2017 22:06:12 +0000 (23:06 +0100)]
windowsaio: add best effort cache invalidation

https://stackoverflow.com/questions/478340/clear-file-cache-to-repeat-performance-testing/7113153#7113153
mentions that opening a file without buffering on Windows has the side
effect of flushing cached parts of the file out of RAM. Use this side
effect to do cache invalidation on non-direct jobs that are using the
windowsaio engine.

This change may make default bandwidth speeds on Windows look lower
compared to older versions of fio but this matches the behaviour of fio
on other platforms with invalidation (such as Linux) because we are
trying to avoid measuring cache reuse (unless invalidate=0 is set).

Note that cache invalidation will fail to happen if the file is also
open elsewhere. Unfortunately, we can't check whether we have exclusive
access without introducing a race where other jobs/programs fail during
open because we temporarily have exclusive access to the file.

Invalidation is done as a call from within fio_windowsaio_open_file
because invalidation fails with multiple file handles (see above) and
it's simpler than trying to use ReOpenFile (which is only available from
Windows 2003/Windows Vista onwards).

The impact of invalidation is demonstrated by the bandwidths achieved by
the following jobs running on a hard disk of an otherwise idle system
with 4GBytes of RAM:
./fio --stonewall --thread --size=128M --filename=fio.tmp --iodepth=64 \
  --bs=128k --invalidate=0 --direct=0 \
  --name=create --rw=write \
  --name=cached --rw=read --loops=2 \
  --name=invalidated --rw=read --loops=2 --invalidate=1

[...]
cached: (groupid=1, jobs=1): err= 0: pid=3888: Sat Oct 7 08:05:39 2017
   read: IOPS=11.7k, BW=1463MiB/s (1534MB/s)(256MiB/175msec)
[...]
invalidated: (groupid=2, jobs=1): err= 0: pid=828: Sat Oct 7 08:05:39 2017
   read: IOPS=2089, BW=261MiB/s (274MB/s)(256MiB/980msec)

v2:
- Move invalidation into a helper function.
- Add more detail and rationale to the commit message.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
22 months agofio_generate_plots: cope with per_job_logs filenames
Sitsofe Wheeler [Sun, 1 Oct 2017 08:07:15 +0000 (09:07 +0100)]
fio_generate_plots: cope with per_job_logs filenames

- Teach fio_generate_plots how to find log files that are generated when
  per_job_logs=1 (which has been the fio default for a while).
- Make fio_generate_plots spit out an error message when no log files
  are found.

Fixes: https://github.com/axboe/fio/issues/43
Fixes: https://github.com/axboe/fio/issues/323

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
22 months agobackend: don't complain about no IO done for create_only=1
Jens Axboe [Fri, 6 Oct 2017 17:41:47 +0000 (11:41 -0600)]
backend: don't complain about no IO done for create_only=1

It's expected that we didn't actually run any IO, if the user
asked for just file creation.

Fixes: 48366f3a92cb ("Don't silently terminate td when no I/O performed due to error")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoMerge branch 'master' of https://github.com/dyniusz/fio
Jens Axboe [Tue, 3 Oct 2017 17:19:26 +0000 (11:19 -0600)]
Merge branch 'master' of https://github.com/dyniusz/fio

22 months agoclient: fix pointer vs uint8_t comparison
Jens Axboe [Tue, 3 Oct 2017 15:30:37 +0000 (09:30 -0600)]
client: fix pointer vs uint8_t comparison

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoMerge branch 'evelu/cleanup' of https://github.com/ErwanAliasr1/fio
Jens Axboe [Tue, 3 Oct 2017 15:29:35 +0000 (09:29 -0600)]
Merge branch 'evelu/cleanup' of https://github.com/ErwanAliasr1/fio

22 months agoclient: Fixing invalid use after free()
Erwan Velu [Tue, 3 Oct 2017 08:56:09 +0000 (10:56 +0200)]
client: Fixing invalid use after free()

As per commit be6fa258, reply->opcode is now called after freeing the structure.

This patch move the check of reply->opcode before freeing reply.

Signed-off-by: Erwan Velu <erwan@redhat.com>
22 months agoclient: Avoid memory leak in fio_client_handle_iolog()
Erwan Velu [Tue, 3 Oct 2017 08:43:11 +0000 (10:43 +0200)]
client:  Avoid memory leak in fio_client_handle_iolog()

The actual code was returning in different places without freeing some pointers in some cases.

We have two places where memory was allocated :
 - pdu = convert_iolog
 - log_pathname = malloc ()

This patch creates a label to manage the return code while checking if
those two pointers needs to be freed.

Signed-off-by: Erwan Velu <erwan@redhat.com>
22 months agot/gen-rand: Avoid memleak of buckets()
Erwan Velu [Tue, 3 Oct 2017 09:33:55 +0000 (11:33 +0200)]
t/gen-rand: Avoid memleak of buckets()

When exiting, buckets structure should be freed to avoid the memleak.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agooslib/libmtd: Removing useless err assigment
Erwan Velu [Tue, 3 Oct 2017 09:26:44 +0000 (11:26 +0200)]
oslib/libmtd: Removing useless err assigment

At this point, setting up err to 0 is useless since it is never read again.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agobackend: Removing memory leak in run_threads()
Erwan Velu [Tue, 3 Oct 2017 09:19:57 +0000 (11:19 +0200)]
backend: Removing memory leak in run_threads()

The fd structure must be freed before breaking the foreach() unless it
leaks its memory

This patch add a free() call in the last break statement.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agobackend: Removing double definition of the same variable
Erwan Velu [Tue, 3 Oct 2017 09:06:37 +0000 (11:06 +0200)]
backend: Removing double definition of the same variable

While f is defined as "struct fio_file *f = io_u->file;", it's pretty
useless to define it just before its usage.

This patch simply remove the 2nd definition of f.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoio_u: Converting usec from long to uint64_t
Erwan Velu [Tue, 3 Oct 2017 07:49:24 +0000 (09:49 +0200)]
io_u: Converting usec from long to uint64_t

'rate_next_io_time' structure and usec_sleep() are manipulating uint64_t.
It's more consistent having usec being expressed in the same stdint unit.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agobackend: fix a case where we complain about no IO being done
Jens Axboe [Thu, 28 Sep 2017 12:58:23 +0000 (06:58 -0600)]
backend: fix a case where we complain about no IO being done

If we're just doing a read phase verify, then we clear
bytes_done before we can check it at the end. Ensure that we
always notice if we did IO, so we don't spew this warning:

job_0: No I/O performed by libaio, perhaps try --debug=io option for details?

when we actually did do exactly what we had to.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoFio 3.1 fio-3.1
Jens Axboe [Thu, 28 Sep 2017 10:23:20 +0000 (04:23 -0600)]
Fio 3.1

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoRemove old exp/README.md file
Jens Axboe [Tue, 26 Sep 2017 19:38:38 +0000 (13:38 -0600)]
Remove old exp/README.md file

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoconfigure: update compiler probing
Sitsofe Wheeler [Sat, 23 Sep 2017 21:48:12 +0000 (22:48 +0100)]
configure: update compiler probing

- Allow clang to be always guessable as a fallback compiler after gcc.
  This increases the likelihood we have a compiler when one wasn't
  explicitly set before we search through #define definitions to guess
  what platform we're targeting. The fallback can be overridden later
  (e.g. to prefer clang above other fallbacks on specific platforms).
- Add comments to explain the preferred compiler search (inspired by
  comments in QEMU's ./configure).
- Change from only checking $CC to also checking $cross_prefix before
  overriding.
- Switch from setting CC to setting cc when overriding. This fixes the
  case where cc is "gcc" but CC is "i686-w64-mingw32-gcc" on Cygwin64
  when building a 32 bit fio.
- Because it is now possible for no compiler to be set, bail out with an
  explicit error message if we find this is the case.
- Add macOS/Darwin to the platforms that prefer clang as the fallback
  compiler.
- Minor refactoring.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
22 months agoappveyor: install zlib and minor clean ups
Sitsofe Wheeler [Sat, 23 Sep 2017 12:38:11 +0000 (13:38 +0100)]
appveyor: install zlib and minor clean ups

- Install the appropriate mingw64 version of zlib so fio compile finds
  and uses it.
- Always use Cygwin64. Appveyor always provides a 64 bit environment and
  Cygwin64 comes with a 32 bit cross-compiler and 32 bit versions of
  libraries.
- Reduce the clone depth to 1 since we aren't making use of the git
  history and it can be a touch faster.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoinit: typo in help output
Jens Axboe [Thu, 21 Sep 2017 04:21:35 +0000 (22:21 -0600)]
init: typo in help output

Reported-by: Rebecca Cran <rebecca@bluestop.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoadd fio_set_directio() error message for platforms without direct I/O
Tomohiro Kusumi [Mon, 18 Sep 2017 18:42:50 +0000 (21:42 +0300)]
add fio_set_directio() error message for platforms without direct I/O

This lets platforms without dio (e.g. OpenBSD) fail with an explicit
message, otherwise they fail without a clue about dio even with
--debug=all enabled.

The message modified based on comments from Jens.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoadd "invalid block size" to "first direct IO errored." message
Tomohiro Kusumi [Mon, 18 Sep 2017 18:23:54 +0000 (21:23 +0300)]
add "invalid block size" to "first direct IO errored." message

It could fail due to bs not multiple of sector size.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agofix strncpy(3) copy length
Tomohiro Kusumi [Mon, 18 Sep 2017 17:53:50 +0000 (20:53 +0300)]
fix strncpy(3) copy length

0 terminate the last byte, and copy at most size-1.
(or 0 terminate the last byte after copying upto size bytes)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agolib/memalign: don't malloc size twice
Tomohiro Kusumi [Mon, 18 Sep 2017 17:53:49 +0000 (20:53 +0300)]
lib/memalign: don't malloc size twice

The footer offset (diff between malloc'd address and aligned address)
is (alignment-1) at most, and footer appears `size' bytes after the
aligned address, thus required size is (alignment-1 + size + sizeof(*f)).

The existing code seems to allocate extra unused `size' bytes.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>