fio.git
2 years agocompiler: set minimum compiler version to GCC 4.1.0
Sitsofe Wheeler [Wed, 21 Mar 2018 22:35:21 +0000 (22:35 +0000)]
compiler: set minimum compiler version to GCC 4.1.0

fio hasn't compiled under GCC 3 for a while due to unguarded calls to
GCC's __sync intrinsics in critical paths. Since those intrinsics didn't
come along until GCC 4.1.0 change fio to explicitly error out on older
compilers.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agosmalloc: oom cleanups
Jens Axboe [Thu, 22 Mar 2018 14:34:48 +0000 (08:34 -0600)]
smalloc: oom cleanups

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agosg: fix sign extension
Jens Axboe [Thu, 22 Mar 2018 02:09:36 +0000 (20:09 -0600)]
sg: fix sign extension

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoserver: fix error handling for shared memory handling
Jens Axboe [Thu, 22 Mar 2018 01:19:58 +0000 (19:19 -0600)]
server: fix error handling for shared memory handling

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'include_refactor' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 22 Mar 2018 01:01:33 +0000 (19:01 -0600)]
Merge branch 'include_refactor' of https://github.com/sitsofe/fio

* 'include_refactor' of https://github.com/sitsofe/fio:
  Add include-what-you-use pragmas
  oslib: make str* compat functions more uniform
  Use POSIX path for poll.h and fcntl.h headers
  Refactor #includes and headers

2 years agoserver: handle shared mem pool allocation failures
Jens Axboe [Thu, 22 Mar 2018 00:58:22 +0000 (18:58 -0600)]
server: handle shared mem pool allocation failures

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoAdd include-what-you-use pragmas
Sitsofe Wheeler [Tue, 20 Mar 2018 15:18:43 +0000 (15:18 +0000)]
Add include-what-you-use pragmas

Add IWYU pragmas to improve its analysis.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agooslib: make str* compat functions more uniform
Sitsofe Wheeler [Tue, 20 Mar 2018 10:46:32 +0000 (10:46 +0000)]
oslib: make str* compat functions more uniform

- Ensure there are CONFIG_ guards in the headers and .c files because
  sometime utilities include oslib headers directly and reference oslib
  .c files in the Makefile
- Ensure all the oslib/str*.h files have redefinition guards
- Shuffle some #include lines from the .h files to the .c ones
- Ensure the <name>.c files have #include "<name>.h" in them so the
  compiler can check the declaration and definition match

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoUse POSIX path for poll.h and fcntl.h headers
Sitsofe Wheeler [Mon, 19 Mar 2018 05:43:41 +0000 (05:43 +0000)]
Use POSIX path for poll.h and fcntl.h headers

musl complains when using include paths like <sys/poll.h> rather than
POSIX's <poll.h> so change the poll.h include path and update the
windows poll.h compatibility shim to match.

Allegedly sys/poll.h was needed for GLIBCs that were older than 2.3
according to
https://www.winehq.org/pipermail/wine-patches/2014-June/132877.html but
I can still see that path in an ancient Red Hat 5.0
glibc-devel-2.0.7-32.i386.rpm ...

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoRefactor #includes and headers
Sitsofe Wheeler [Mon, 19 Mar 2018 05:33:13 +0000 (05:33 +0000)]
Refactor #includes and headers

- Try and remove unneeded #include lines
- Try and add #include lines that would allow the files to be built in a
  more standalone manner

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoserver: process STOP/QUIT commands out-of-line
Jens Axboe [Wed, 21 Mar 2018 21:48:17 +0000 (15:48 -0600)]
server: process STOP/QUIT commands out-of-line

At the end of a job, the stats entries are queued up for
out-of-line transmission, but we immediately queue a client
STOP+QUIT command right after that. If the command handling
loop doesn't wakeup and process the stats commands before we
process the STOP+QUIT inline, then we lose the stats output
from those commands. The result is that the client won't see
the job output from that job, or the "All clients" output.

We should impose proper ordering for all commands, but this
is a quick fix to ensure that we don't hit the above mentioned
but. For other commands we don't care about ordering between
inline and out-of-line commands, so it's likely good enough
to carry us forward.

Reported-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'asprintf' of https://github.com/bvanassche/fio
Jens Axboe [Wed, 21 Mar 2018 14:26:28 +0000 (08:26 -0600)]
Merge branch 'asprintf' of https://github.com/bvanassche/fio

* 'asprintf' of https://github.com/bvanassche/fio:
  verify: Simplify dump_buf()
  log: Modify the implementation such that it uses asprintf()
  Add an asprintf() implementation

2 years agodebug: remove extra parens
Jens Axboe [Wed, 21 Mar 2018 14:23:50 +0000 (08:23 -0600)]
debug: remove extra parens

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooptgroup: move debug code into function
Jens Axboe [Tue, 20 Mar 2018 17:19:19 +0000 (11:19 -0600)]
optgroup: move debug code into function

The current global construct breaks on some compilers:

optgroup.c:208: error: expected identifier or '(' before 'do'
optgroup.c:208: error: expected identifier or '(' before 'while'

Reported-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Fixes: 96344ff00349 ("optgroup: add check for optgroup bit numbers being within range")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoFix whitespace issues in previous commit
Jens Axboe [Tue, 20 Mar 2018 14:27:43 +0000 (08:27 -0600)]
Fix whitespace issues in previous commit

Lots of trailing whitespace.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'expand_fiohistparser' of https://github.com/shimrot/fio
Jens Axboe [Tue, 20 Mar 2018 14:24:20 +0000 (08:24 -0600)]
Merge branch 'expand_fiohistparser' of https://github.com/shimrot/fio

* 'expand_fiohistparser' of https://github.com/shimrot/fio:
  Expand fiologparser_hist operations with new options

2 years agoExpand fiologparser_hist operations with new options
krisd [Tue, 20 Mar 2018 05:42:26 +0000 (00:42 -0500)]
Expand fiologparser_hist operations with new options

New options are:
* noweight: don't weight samples across intervals
* percentiles: allow print of more percentiles than just 50,90,95%s
* Fix so works with newer 'ns' results, but add --usbin option
  so can still be used with older results.
* directions=rwtm option to allow print of independent direction results

2 years agooptgroup: add check for optgroup bit numbers being within range
Jens Axboe [Mon, 19 Mar 2018 21:56:20 +0000 (15:56 -0600)]
optgroup: add check for optgroup bit numbers being within range

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoverify: Simplify dump_buf()
Bart Van Assche [Mon, 12 Mar 2018 22:29:34 +0000 (15:29 -0700)]
verify: Simplify dump_buf()

Use asprintf() to build the output file path instead of using a fixed
size buffer for that path.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agolog: Modify the implementation such that it uses asprintf()
Bart Van Assche [Tue, 13 Mar 2018 22:28:17 +0000 (15:28 -0700)]
log: Modify the implementation such that it uses asprintf()

This makes the logging implementation simpler and easier to read.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoAdd an asprintf() implementation
Bart Van Assche [Mon, 12 Mar 2018 22:38:28 +0000 (15:38 -0700)]
Add an asprintf() implementation

Since I would like to use the asprintf() function in the ZBC code and
since that function is not available on every platform supported by
fio, add an asprintf() implementation.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Mon, 19 Mar 2018 16:24:10 +0000 (10:24 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Suppress uninteresting data race reports
  gettime: Rework the clock thread starting mechanism
  Improve Valgrind instrumentation of memory allocations
  Rename fio_mutex into fio_sem
  Split mutex.c and .h each into three files

2 years agosg: add read/write FUA options
Kris Davis [Mon, 19 Mar 2018 16:19:44 +0000 (10:19 -0600)]
sg: add read/write FUA options

Enable the user to ask for the FUA bits to be set on read and
write operations.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'pthread-cond' of https://github.com/bvanassche/fio
Jens Axboe [Sat, 17 Mar 2018 20:43:16 +0000 (14:43 -0600)]
Merge branch 'pthread-cond' of https://github.com/bvanassche/fio

* 'pthread-cond' of https://github.com/bvanassche/fio:
  Signal td->free_cond with the associated mutex held
  Make sure that assert() expressions do not have side effects

2 years agoSignal td->free_cond with the associated mutex held
Bart Van Assche [Fri, 16 Mar 2018 15:38:34 +0000 (08:38 -0700)]
Signal td->free_cond with the associated mutex held

Calling pthread_cond_signal() or pthread_cond_broadcast() without
holding the associated mutex can lead to missed wakeups. Hence ensure
that td->io_u_lock is held around pthread_cond_signal(&td->free_cond)
calls. A quote from the POSIX spec
(http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_cond_broadcast.html):

"The pthread_cond_broadcast() or pthread_cond_signal() functions may be
called by a thread whether or not it currently owns the mutex that
threads calling pthread_cond_wait() or pthread_cond_timedwait() have
associated with the condition variable during their waits; however, if
predictable scheduling behavior is required, then that mutex shall be
locked by the thread calling pthread_cond_broadcast() or
pthread_cond_signal()."

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoMake sure that assert() expressions do not have side effects
Bart Van Assche [Fri, 16 Mar 2018 15:44:35 +0000 (08:44 -0700)]
Make sure that assert() expressions do not have side effects

Assert statements are compiled out if NDEBUG is defined. Hence make sure
that the expressions passed to assert do not have side effects.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoSuppress uninteresting data race reports
Bart Van Assche [Tue, 13 Mar 2018 21:44:51 +0000 (14:44 -0700)]
Suppress uninteresting data race reports

Thread checkers like Helgrind and DRD report all concurrent
modifications of variables that are serialized by a recognized
synchronization primitive. Suppress reports about intentionally
unsynchronized accesses.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agogettime: Rework the clock thread starting mechanism
Bart Van Assche [Tue, 13 Mar 2018 17:43:02 +0000 (10:43 -0700)]
gettime: Rework the clock thread starting mechanism

Replace two mutexes in struct clock_thread that are intended to make
the clock threads start simultaneously by a fio_mutex.

This patch avoids that drd reports the following:

==23831== Thread 2:
==23831== Mutex not locked by calling thread: mutex 0x6cb7bf8, recursion count 1, owner 1.
==23831==    at 0x4C38EC3: pthread_mutex_unlock_intercept (drd_pthread_intercepts.c:978)
==23831==    by 0x4C38EC3: pthread_mutex_unlock (drd_pthread_intercepts.c:991)
==23831==    by 0x420974: clock_thread_fn (gettime.c:604)
==23831==    by 0x4C34C44: vgDrd_thread_wrapper (drd_pthread_intercepts.c:444)
==23831==    by 0x5E4159A: start_thread (in /lib64/libpthread-2.27.so)
==23831==    by 0x6356A1E: clone (in /lib64/libc-2.27.so)

==23924== Mutex still locked at thread exit: mutex 0x6cb7bd0, recursion count 1, owner 2.
==23924==    at 0x4C35BF2: pthread_join_intercept (drd_pthread_intercepts.c:711)
==23924==    by 0x4C35BF2: pthread_join (drd_pthread_intercepts.c:718)
==23924==    by 0x420D7F: fio_monotonic_clocktest (gettime.c:722)
==23924==    by 0x420441: fio_clock_init (gettime.c:424)
==23924==    by 0x437526: fio_time_init (time.c:141)
==23924==    by 0x49BF41: main (fio.c:56)

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoImprove Valgrind instrumentation of memory allocations
Bart Van Assche [Thu, 8 Mar 2018 21:41:36 +0000 (13:41 -0800)]
Improve Valgrind instrumentation of memory allocations

In smalloc, let Valgrind do detection of red zone modifications. In
synchr.c, let Valgrind perform use-after-free detection of
synchronization objects.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoRename fio_mutex into fio_sem
Bart Van Assche [Wed, 14 Mar 2018 15:31:37 +0000 (08:31 -0700)]
Rename fio_mutex into fio_sem

Since fio_mutex implements a semaphore, rename fio_mutex into fio_sem.
Rename those fio_sem instances that have 'mutex' embedded in their
name. Rename the mutex.h and .c source files into fio_sem.h and .c
respectively.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoSplit mutex.c and .h each into three files
Bart Van Assche [Wed, 14 Mar 2018 15:26:15 +0000 (08:26 -0700)]
Split mutex.c and .h each into three files

Create separate header and source files for the struct fio_mutex
functions, for the struct fio_rwlock functions and for the helper
functions for initializing process shared mutexes and condition
variables. Replace an #include directive by a forward declaration
in fio.h. Minimize the #include directives in mutex.c, rwlock.c
and pshared.c. This patch does not change any functionality.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoio_u: only rewind file position if it's non-zero
Jens Axboe [Tue, 13 Mar 2018 17:49:55 +0000 (11:49 -0600)]
io_u: only rewind file position if it's non-zero

This:

bs=8k
rw=read:-4k

is supposed to read 0..8k, then add 8k and subtract 4k, ending
up with a next read at 4k..12k and so forth. But we rewind too
quickly, and the first IO fails as being out-of-bounds.

Fixes: c22825bb537af ("Fix backwards reads with --size smaller than the file size")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoconfigure: don't disable lex on FreeBSD
Jens Axboe [Tue, 13 Mar 2018 14:41:37 +0000 (08:41 -0600)]
configure: don't disable lex on FreeBSD

Previous commit turned it off by mistake.

Fixes: 89556808ef14 ("configure: Disable lex on NetBSD")
Reported-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 13 Mar 2018 00:13:10 +0000 (18:13 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  parse: Fix two compiler warnings
  stat: Fix a compiler warning in __show_run_stats()
  Rename struct rb_node into struct fio_rb_node
  configure: Disable lex on NetBSD

2 years agoparse: Fix two compiler warnings
Bart Van Assche [Mon, 12 Mar 2018 21:24:27 +0000 (14:24 -0700)]
parse: Fix two compiler warnings

Avoid that gcc on NetBSD reports the following compiler warnings:

    CC parse.o
parse.c: In function 'get_mult_time':
parse.c:175: warning: array subscript has type 'char'
parse.c: In function '__get_mult_bytes':
parse.c:221: warning: array subscript has type 'char'

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agostat: Fix a compiler warning in __show_run_stats()
Bart Van Assche [Mon, 12 Mar 2018 21:16:56 +0000 (14:16 -0700)]
stat: Fix a compiler warning in __show_run_stats()

Avoid that gcc reports the following warning on NetBSD systems:

    CC stat.o
stat.c: In function '__show_run_stats':
stat.c:1868: warning: dereferencing type-punned pointer will break strict-aliasing rules

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoRename struct rb_node into struct fio_rb_node
Bart Van Assche [Mon, 12 Mar 2018 21:13:38 +0000 (14:13 -0700)]
Rename struct rb_node into struct fio_rb_node

This patch avoids that building fio fails as follows on NetBSD systems:

    CC crc/test.o
In file included from crc/../iolog.h:4,
                 from crc/../stat.h:4,
                 from crc/../thread_options.h:7,
                 from crc/../fio.h:18,
                 from crc/test.c:5:
crc/../lib/rbtree.h:101: error: redefinition of 'struct rb_node'
Makefile:336: recipe for target 'crc/test.o' failed
gmake: *** [crc/test.o] Error 1

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agoconfigure: Disable lex on NetBSD
Bart Van Assche [Mon, 12 Mar 2018 21:37:37 +0000 (14:37 -0700)]
configure: Disable lex on NetBSD

Since the NetBSD lex software does not support all features (yylval)
used by fio, disable lex on NetBSD.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2 years agofilesetup: Initialize all members of struct fio_file
Bart Van Assche [Sat, 10 Mar 2018 00:46:36 +0000 (16:46 -0800)]
filesetup: Initialize all members of struct fio_file

This patch avoids that Valgrind reports complaints like the following:

==10492== Conditional jump or move depends on uninitialised value(s)
==10492==    at 0x43BF19: fio_file_reset (filesetup.c:1872)
==10492==    by 0x43AB7F: alloc_new_file (filesetup.c:1496)
==10492==    by 0x43AED4: add_file (filesetup.c:1581)
==10492==    by 0x4528D3: str_filename_cb (options.c:1276)
==10492==    by 0x44D83A: __handle_option (parse.c:722)
==10492==    by 0x44E303: handle_option (parse.c:941)
==10492==    by 0x44E68B: parse_cmd_option (parse.c:1061)
==10492==    by 0x4541DD: fio_cmd_option_parse (options.c:4966)
==10492==    by 0x427C6D: parse_cmd_line (init.c:2652)
==10492==    by 0x428685: parse_options (init.c:2924)
==10492==    by 0x4920AC: main (fio.c:47)

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agohelper_thread: Initialize all helper_data members before using it
Bart Van Assche [Sat, 10 Mar 2018 00:46:22 +0000 (16:46 -0800)]
helper_thread: Initialize all helper_data members before using it

This patch avoids that Valgrind reports the following:

==10492== Thread 2:
==10492== Conditional jump or move depends on uninitialised value(s)
==10492==    at 0x480295: helper_thread_main (helper_thread.c:89)
==10492==    by 0x57B961A: start_thread (pthread_create.c:465)
==10492==    by 0x5CEC98E: clone (clone.S:95)

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoDeclare debug_levels[] const
Bart Van Assche [Sat, 10 Mar 2018 00:46:02 +0000 (16:46 -0800)]
Declare debug_levels[] const

Since debug_levels[] is not modified by any code, declare it const.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofilesetup: don't round/adjust size for size_percent == 100
Jens Axboe [Fri, 9 Mar 2018 19:55:21 +0000 (12:55 -0700)]
filesetup: don't round/adjust size for size_percent == 100

For the full device, don't apply any rounding or bs based
adjustment. Just treat it like size_percent wasn't even set.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMakefile: Rerun the configure script if it has been modified
Bart Van Assche [Fri, 9 Mar 2018 16:51:04 +0000 (08:51 -0800)]
Makefile: Rerun the configure script if it has been modified

Because of a typo in the Makefile ("config-host-mak") the configure
script is not rerun if the configure script is modified after
config-host.mak has been generated. Merge the two make goals for
rerunning configure into a single goal. This patch relies on the
following make behavior (see also https://www.gnu.org/software/make/manual/make.html#Remaking-Makefiles):

  Sometimes makefiles can be remade from other files, such as RCS or SCCS
  files. If a makefile can be remade from other files, you probably want
  make to get an up-to-date version of the makefile to read in.

  To this end, after reading in all makefiles, make will consider each as a
  goal target and attempt to update it. If a makefile has a rule which says
  how to update it (found either in that very makefile or in another one) or
  if an implicit rule applies to it (see Using Implicit Rules), it will be
  updated if necessary. After all makefiles have been checked, if any have
  actually been changed, make starts with a clean slate and reads all the
  makefiles over again. (It will also attempt to update each of them over
  again, but normally this will not change them again, since they are
  already up to date.)

This patch has been tested by verifying that the commands run by make
were as expected for the following invocations:

rm -f config-host.mak && make
make && touch configure && make
make

Note: fio already requires GNU make because of the use of "ifeq" etc.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: 'is_random' can be a boolean
Jens Axboe [Fri, 9 Mar 2018 17:23:53 +0000 (10:23 -0700)]
io_u: 'is_random' can be a boolean

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: kill get_next_{offset,buflen} wrappers
Jens Axboe [Fri, 9 Mar 2018 17:22:01 +0000 (10:22 -0700)]
io_u: kill get_next_{offset,buflen} wrappers

After the previous commit, they are just wrappers. Kill them.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoRemove prof_io_ops.fill_io_u_off(), .fill_io_u_size() and .get_next_file()
Bart Van Assche [Fri, 9 Mar 2018 17:16:23 +0000 (09:16 -0800)]
Remove prof_io_ops.fill_io_u_off(), .fill_io_u_size() and .get_next_file()

Remove the struct prof_io_ops methods for which no implementations
have been defined. This patch does not change any functionality.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agomutex: fix other locations where we are not waking within the lock
Jens Axboe [Fri, 9 Mar 2018 14:59:03 +0000 (07:59 -0700)]
mutex: fix other locations where we are not waking within the lock

Our workqueue implementation had a few cases where it did not wake
while holding the lock, fix those up. Ditto for verify, where
async IO completion also needed the ordering switched.

Also see commit e4ccf13bad6d.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agomutex: ensure that fio_mutex_up() holds mutex lock during wakeup
Jens Axboe [Fri, 9 Mar 2018 14:55:40 +0000 (07:55 -0700)]
mutex: ensure that fio_mutex_up() holds mutex lock during wakeup

Some implementations mandate that you hold the lock while
issuing the wakeup, as not to lose a wakeup. This seems to
be the case for Windows. See the lengthy thread on the topic
here:

https://www.spinics.net/lists/fio/msg06772.html

Thanks to Sitsofe Wheeler for heroic debugging.

Reported-by: Rebecca Cran <rebecca@bluestop.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'disable_opt' of https://github.com/sitsofe/fio
Jens Axboe [Wed, 7 Mar 2018 17:22:28 +0000 (10:22 -0700)]
Merge branch 'disable_opt' of https://github.com/sitsofe/fio

* 'disable_opt' of https://github.com/sitsofe/fio:
  appveyor: disable setting compiler march
  configure: make --disable-optimizations disable march=native

2 years agoappveyor: disable setting compiler march
Sitsofe Wheeler [Wed, 7 Mar 2018 16:36:32 +0000 (16:36 +0000)]
appveyor: disable setting compiler march

Commit a817dc3b3b5a0efc95aaca366875eac67607cd5b ("Default to building
native code") tunes binaries to only run on the build system's
architecture by default. However as AppVeyor builds Windows MSIs for
public consumption, let's turn that feature off here to prevent people
accidentally ending up with binaries that contain instructions their
system can't run.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoconfigure: make --disable-optimizations disable march=native
Sitsofe Wheeler [Wed, 7 Mar 2018 09:54:45 +0000 (09:54 +0000)]
configure: make --disable-optimizations disable march=native

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoconfigure: don't override march if already set
Jens Axboe [Wed, 7 Mar 2018 15:32:05 +0000 (08:32 -0700)]
configure: don't override march if already set

Some archs (like arm or s390) already set an appropriate architecture,
so don't override it for those cases.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoReduce LOG_MSEC_SLACK
Jeff Furlong [Wed, 7 Mar 2018 00:53:20 +0000 (17:53 -0700)]
Reduce LOG_MSEC_SLACK

Starting in fio 2.99, there is an odd problem when --ramp_time is used
with --write_iops_log.  The reported IOPs for the first ~200s or so are
artificially inflated by fio, always monotonically increasing until
about 200s, then resets to normal.  With fio 2.99, disabling --ramp_time
avoids the issue.  And, with fio 2.21, using --ramp_time and
--write_iops_log does not create the problem.  Consider a job:

fio --name=test --ioengine=libaio --direct=1 --rw=randread --iodepth=1 \\
--size=100% --bs=4k --filename=/dev/nvme1n1 --runtime=300s \\
--ramp_time=10s --write_iops_log=test --log_avg_msec=1000

See the attached plot for fio induced IOPs during the first ~200s.
Using a longer runtime makes identifying the issue easier if the DUT is
consistently performing.

One simple method to resolve this inflation is by reducing the
LOG_MSEC_SLACK from 10 milliseconds to 1 millisecond, given that we now
operate some time structures to the nanosecond level (so 10ms of slack
seems excessive).  Below I also fixup some data type issues to match
existing types (avoids typecasts or potential compiler warnings).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'wip-rbd-engine' of https://github.com/dillaman/fio
Jens Axboe [Wed, 7 Mar 2018 00:44:17 +0000 (17:44 -0700)]
Merge branch 'wip-rbd-engine' of https://github.com/dillaman/fio

* 'wip-rbd-engine' of https://github.com/dillaman/fio:
  rbd: remove support for blkin tracing
  rbd: fixed busy-loop when using eventfd polling

2 years agorbd: remove support for blkin tracing
Jason Dillaman [Wed, 7 Mar 2018 00:05:01 +0000 (19:05 -0500)]
rbd: remove support for blkin tracing

The referenced APIs for blkin tracing were not merged into librbd.
Instead, blkin tracing can be enabled for all IO by setting
'rbd blkin trace all = true' in the config.

2 years agorbd: fixed busy-loop when using eventfd polling
Jason Dillaman [Wed, 7 Mar 2018 00:01:18 +0000 (19:01 -0500)]
rbd: fixed busy-loop when using eventfd polling

The eventfd poll would continuously fire since the fd was never
read. Switched to a semaphore eventfd and now drain the events
as completions are handled.

2 years agoDefault to building native code
Jens Axboe [Tue, 6 Mar 2018 21:28:16 +0000 (14:28 -0700)]
Default to building native code

If the local compiler supports it, use -march=native to ensure we
take full advantage of a local build.

If building distro packages, or otherwise for a different platform
than the host, then use --disable-native with configure to remove
this optimization.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoDon't make fadvise failure fatal
Jens Axboe [Tue, 6 Mar 2018 19:42:57 +0000 (12:42 -0700)]
Don't make fadvise failure fatal

It's just an advisory hint. Warn once that it failed, then
march on.

Fixes: https://github.com/axboe/fio/issues/554
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'howto_typos' of https://github.com/dirtyharrycallahan/fio
Jens Axboe [Mon, 5 Mar 2018 17:29:39 +0000 (10:29 -0700)]
Merge branch 'howto_typos' of https://github.com/dirtyharrycallahan/fio

* 'howto_typos' of https://github.com/dirtyharrycallahan/fio:
  doc: fix typos in HOWTO

2 years agodoc: fix typos in HOWTO
Patrick Callahan [Mon, 5 Mar 2018 17:23:51 +0000 (12:23 -0500)]
doc: fix typos in HOWTO

2 years agoMerge branch 'manpage_typos' of https://github.com/dirtyharrycallahan/fio
Jens Axboe [Mon, 5 Mar 2018 15:47:51 +0000 (08:47 -0700)]
Merge branch 'manpage_typos' of https://github.com/dirtyharrycallahan/fio

* 'manpage_typos' of https://github.com/dirtyharrycallahan/fio:
  doc: fix typos in fio man page

2 years agodoc: fix typos in fio man page
Patrick Callahan [Mon, 5 Mar 2018 14:23:39 +0000 (09:23 -0500)]
doc: fix typos in fio man page

2 years agoMerge branch 'mpath_nvme_diskutil' of https://github.com/sitsofe/fio
Jens Axboe [Fri, 2 Mar 2018 19:02:08 +0000 (12:02 -0700)]
Merge branch 'mpath_nvme_diskutil' of https://github.com/sitsofe/fio

* 'mpath_nvme_diskutil' of https://github.com/sitsofe/fio:
  diskutil: minor style cleanup
  diskutil: try additional slave device path if first fails

2 years agodiskutil: minor style cleanup
Sitsofe Wheeler [Fri, 2 Mar 2018 15:51:01 +0000 (15:51 +0000)]
diskutil: minor style cleanup

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agodiskutil: try additional slave device path if first fails
Potnuri Bharat Teja [Fri, 2 Mar 2018 15:26:01 +0000 (15:26 +0000)]
diskutil: try additional slave device path if first fails

Running fio on an NVMEoF device with multipath slaves fails with a "No
such file or directory" error when displaying disk stats. Correct this
by using a different path to the sysfs slave device.

[sitsofe: change new path to be a fallback when regular path fails]

Fixes: https://github.com/axboe/fio/issues/525
Closes: https://github.com/axboe/fio/pull/526

Signed-off-by: Potnuri Bharat Teja <bharat@chelsio.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoMerge branch 'hotfix_counter_overflow' of https://github.com/ifke/fio
Jens Axboe [Thu, 1 Mar 2018 16:18:43 +0000 (09:18 -0700)]
Merge branch 'hotfix_counter_overflow' of https://github.com/ifke/fio

* 'hotfix_counter_overflow' of https://github.com/ifke/fio:
  Fix overflow of counters incremented on each I/O operation

2 years agoFix overflow of counters incremented on each I/O operation
Alexander Larin [Wed, 28 Feb 2018 11:25:42 +0000 (14:25 +0300)]
Fix overflow of counters incremented on each I/O operation

 - In the thread_stat struct: uint32_t counters are updated to uint64_t.
 - In the io_u_plat_entry struct: unsigned int counters are updated to
   uint64_t.
It fixes overflow of these counters.

Signed-off-by: Alexander Larin <znahar@yandex-team.ru>
2 years agoMerge branch 'win_build' of https://github.com/sitsofe/fio
Jens Axboe [Wed, 28 Feb 2018 16:44:08 +0000 (09:44 -0700)]
Merge branch 'win_build' of https://github.com/sitsofe/fio

* 'win_build' of https://github.com/sitsofe/fio:
  windows: document MinGW zlib install and remove custom zlib search
  windows: minor windows installer improvements
  appveyor: minor refactoring, clarifications

2 years agoconfigure: improve static zlib package warning
Jens Axboe [Wed, 28 Feb 2018 15:43:22 +0000 (08:43 -0700)]
configure: improve static zlib package warning

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'master' of https://github.com/brycepg/fio
Jens Axboe [Tue, 27 Feb 2018 23:15:22 +0000 (16:15 -0700)]
Merge branch 'master' of https://github.com/brycepg/fio

* 'master' of https://github.com/brycepg/fio:
  Refer to zlib-static package in zlib static build warning

2 years agoRefer to zlib-static package in zlib static build warning
Bryce Guinta [Tue, 27 Feb 2018 23:09:35 +0000 (16:09 -0700)]
Refer to zlib-static package in zlib static build warning

Signed-off-by: Bryce Guinta <bryce.paul.guinta@gmail.com>
2 years agoMerge branch 'wip-ifed-howto-update' of https://github.com/ifed01/fio
Jens Axboe [Mon, 26 Feb 2018 18:59:54 +0000 (11:59 -0700)]
Merge branch 'wip-ifed-howto-update' of https://github.com/ifed01/fio

* 'wip-ifed-howto-update' of https://github.com/ifed01/fio:
  Update HOWTO with RADOS information

2 years agoUpdate HOWTO with RADOS information
Igor Fedotov [Mon, 26 Feb 2018 17:53:52 +0000 (20:53 +0300)]
Update HOWTO with RADOS information

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2 years agowindows: document MinGW zlib install and remove custom zlib search
Sitsofe Wheeler [Wed, 21 Feb 2018 18:45:55 +0000 (18:45 +0000)]
windows: document MinGW zlib install and remove custom zlib search

In 5580d200d7ae49cdb4f30913e3ef2d74e431ff60 ("appveyor: install zlib and minor
clean ups") I found AppVeyor builds could use a pre-built, statically linkable
zlib from Cygwin/MinGW.

Document the MinGW zlib packages in the README and remove the custom Windows
zlib searching from the configure script.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agowindows: minor windows installer improvements
Sitsofe Wheeler [Wed, 21 Feb 2018 14:41:14 +0000 (14:41 +0000)]
windows: minor windows installer improvements

- Stop hard coding fio version in the Windows installer build file and try and
  use the generated version instead
- Make installer build use sys.BUILDARCH rather than env.FIO_ARCH
- Change IO to I/O in descriptions to match other packages

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoappveyor: minor refactoring, clarifications
Sitsofe Wheeler [Wed, 21 Feb 2018 09:56:38 +0000 (09:56 +0000)]
appveyor: minor refactoring, clarifications

- Set platform directly to x64 or x86 and use PLATFORM instead of BUILD_ARCH
- Set the PATH once (via the install section) because environment variables
  persist to later sections
- Add a note that shallow cloning means FIO-VERSION-GEN is unable to build a
  fancy version number (because it won't find any tags)

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoFio 3.5 fio-3.5
Jens Axboe [Tue, 20 Feb 2018 22:30:28 +0000 (15:30 -0700)]
Fio 3.5

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'wip-ifed-rados' of https://github.com/ifed01/fio
Jens Axboe [Wed, 14 Feb 2018 19:43:04 +0000 (12:43 -0700)]
Merge branch 'wip-ifed-rados' of https://github.com/ifed01/fio

* 'wip-ifed-rados' of https://github.com/ifed01/fio:
  Add support for Ceph Rados benchmarking.

2 years agoAdd support for Ceph Rados benchmarking.
Igor Fedotov [Fri, 9 Feb 2018 14:39:28 +0000 (17:39 +0300)]
Add support for Ceph Rados benchmarking.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2 years agodebug: fix inverted logic in fio_did_warn() fio-3.4
Jens Axboe [Mon, 12 Feb 2018 17:55:07 +0000 (10:55 -0700)]
debug: fix inverted logic in fio_did_warn()

Fixes: 5ec3bc5401f3 ("init: add global 'warned' state")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoiolog: convert drop warning to fio_did_warn()
Jens Axboe [Mon, 12 Feb 2018 17:19:18 +0000 (10:19 -0700)]
iolog: convert drop warning to fio_did_warn()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: convert zoned bug warning to fio_did_warn()
Jens Axboe [Mon, 12 Feb 2018 17:18:32 +0000 (10:18 -0700)]
io_u: convert zoned bug warning to fio_did_warn()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoverify: convert verify buf too small warning to fio_did_warn()
Jens Axboe [Mon, 12 Feb 2018 17:14:35 +0000 (10:14 -0700)]
verify: convert verify buf too small warning to fio_did_warn()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofilesetup: convert root flush warning to fio_did_warn()
Jens Axboe [Mon, 12 Feb 2018 17:13:04 +0000 (10:13 -0700)]
filesetup: convert root flush warning to fio_did_warn()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoinit: add global 'warned' state
Jens Axboe [Mon, 12 Feb 2018 17:11:31 +0000 (10:11 -0700)]
init: add global 'warned' state

In various places we spew a warning if some static variable
isn't set, but this still means it can happen once per job.
Add some global state to allow to track this globally instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'master' of https://github.com/brycepg/fio
Jens Axboe [Mon, 12 Feb 2018 15:24:21 +0000 (08:24 -0700)]
Merge branch 'master' of https://github.com/brycepg/fio

* 'master' of https://github.com/brycepg/fio:
  Make fiologparser_hist compatible with python3

2 years agoinit: fixup some bad style in previous commit
Jens Axboe [Sat, 10 Feb 2018 21:44:49 +0000 (14:44 -0700)]
init: fixup some bad style in previous commit

Make it follow the fio coding style.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoinit: fix broken verify_interval
Damian Yurzola [Fri, 9 Feb 2018 01:29:38 +0000 (17:29 -0800)]
init: fix broken verify_interval

The operands seems to have been inverted which in turn
created the situation whereby the interval was always
changed to match the min_bs

Fixes: https://github.com/axboe/fio/issues/522
Signed-off-by: Damian Yurzola <damian@yurzola.net>
2 years agoInclude 'numjobs' in global options output
Jens Axboe [Thu, 8 Feb 2018 22:46:46 +0000 (15:46 -0700)]
Include 'numjobs' in global options output

Fixes: https://github.com/axboe/fio/issues/237
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agommap: don't include MADV_FREE in fadvise_hint check
Jens Axboe [Wed, 7 Feb 2018 18:30:59 +0000 (11:30 -0700)]
mmap: don't include MADV_FREE in fadvise_hint check

This really should be based on invalidate, leave it as it was
for now.

Fixes: c712c97ab871 ("Let fadvise_hint also apply too mmap engine and madvise")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoLet fadvise_hint also apply too mmap engine and madvise
Jens Axboe [Wed, 7 Feb 2018 18:13:05 +0000 (11:13 -0700)]
Let fadvise_hint also apply too mmap engine and madvise

Fixes: https://github.com/axboe/fio/issues/528
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMake fiologparser_hist compatible with python3
Bryce Guinta [Tue, 6 Feb 2018 00:36:33 +0000 (17:36 -0700)]
Make fiologparser_hist compatible with python3

In python3, map functions return generators, which caused numpy to break
in this script.

Use list comprehensions instead of map (more pythonic)
They also have the added benefit of behaving the same
between python2 and python3.

2 years agostat: add total fsync ios to json output
Jens Axboe [Thu, 25 Jan 2018 23:04:20 +0000 (16:04 -0700)]
stat: add total fsync ios to json output

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_ddir: revert separate ddir count change
Jens Axboe [Thu, 25 Jan 2018 17:28:23 +0000 (10:28 -0700)]
io_ddir: revert separate ddir count change

This essentially reverts the previous commit. We need more
careful changes for looping and indexing, the easiest is just
to move the count values at the end as not to skew the values
of the actual ddir entries.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_ddir: move count values out of the enum fio_ddir
Jens Axboe [Thu, 25 Jan 2018 17:16:51 +0000 (10:16 -0700)]
io_ddir: move count values out of the enum fio_ddir

They are not directions, just counts.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agostat: ensure that we align ts->sync_stat appropriately
Jens Axboe [Thu, 25 Jan 2018 16:51:07 +0000 (09:51 -0700)]
stat: ensure that we align ts->sync_stat appropriately

clang correctly complains that we don't know if the alignment is
correct when the structure is packed, so ensure that we align it
to an 8-byte boundary.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoTrack fsync/fdatasync/sync_file_range issue count
Jens Axboe [Thu, 25 Jan 2018 16:27:14 +0000 (09:27 -0700)]
Track fsync/fdatasync/sync_file_range issue count

Also fix an issue where we didn't initialize the sync minimum
latency correctly, so it was always reported as 0.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoSwitch last_was_sync and terminate to bool and pack better
Jens Axboe [Thu, 25 Jan 2018 03:22:50 +0000 (20:22 -0700)]
Switch last_was_sync and terminate to bool and pack better

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoAdd suppor for logging fsync (and friends) latencies
Jens Axboe [Wed, 24 Jan 2018 22:32:07 +0000 (15:32 -0700)]
Add suppor for logging fsync (and friends) latencies

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agonull: drop unneeded casts from void* to non-void*
Tomohiro Kusumi [Mon, 15 Jan 2018 19:52:12 +0000 (21:52 +0200)]
null: drop unneeded casts from void* to non-void*

Some functions in null ioengine are used by C/C++ (e.g. null_init()
which calls malloc(3)), but C specific ones (not __cplusplus) don't
need explicit cast from void* to non-void*, and one usually doesn't.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>