Bart Van Assche [Tue, 3 Apr 2018 18:23:11 +0000 (11:23 -0700)]
Ensure that .minfp and .maxfp are respected for FIO_OPT_FLOAT_LIST
option_init() overwrites the .minfp and .maxfp members defined in
fio_options[] which is wrong. Remove the code that overwrites these two
members and change the convention for no FIO_OPT_FLOAT_LIST limits from
(minfp == DBL_MIN && maxfp == DBL_MAX) into (minfp != 0 || maxfp != 0).
Fixes:
833491908a1a ("stats: Add a function to report completion latency percentiles")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 3 Apr 2018 20:04:02 +0000 (13:04 -0700)]
option parsing: Mark arguments that are not modified as 'const'
This patch does not modify any functionality but makes the option
parsing code slightly easier to read by marking arguments that are
not modified explicitly as 'const'. A new function has been added,
find_option_c(), that behaves identically to find_option(). Only
the function signature differs between these two functions.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 3 Apr 2018 20:04:25 +0000 (13:04 -0700)]
parse.h: Remove a superfluous cast
Adding an offset to a void pointer has the same effect as adding an
offset to a char pointer. Hence removing the (char *) cast from
td_var() does not change its behavior.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 26 Mar 2018 16:00:28 +0000 (09:00 -0700)]
Only populate the write buffer if necessary
This patch moves the populate_verify_io_u() call from inside
get_io_u() into all its callers except do_dry_run() and thereby skips
write buffer population for dry runs. This patch does not change the
behavior of fio but is necessary because the ZBC patch will insert
code in fill_io_u() after the get_io_u() call that may modify the
write offset.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 26 Mar 2018 15:50:09 +0000 (08:50 -0700)]
Rename TD_F_VER_NONE into TD_F_DO_VERIFY
Rename TD_F_VER_NONE into TD_F_DO_VERIFY to make it clear that this flag means
that data verification has to be performed. See also commit
d72be5454c8c ("Cache
layout improvements").
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 4 Apr 2018 20:45:54 +0000 (13:45 -0700)]
engines/sg: Make I/O error messages more informative
Ensure that if an I/O error is encountered that not only the error
number but also a textual error message and the engine name are
reported. Without this patch, SG error messages look like this:
fio: pid=12803, err=5/
With this patch applied SG I/O error messages change into:
fio: pid=12803, err=5/file:engines/sg.c:290, func=fio_sgio_doio, error=Input/output error
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Tue, 3 Apr 2018 14:19:38 +0000 (08:19 -0600)]
Merge branch 'fixbug_glfs' of https://github.com/simon-rock/fio
* 'fixbug_glfs' of https://github.com/simon-rock/fio:
glusterfs: always allocate io_u->engine_data
simon [Tue, 3 Apr 2018 12:16:36 +0000 (20:16 +0800)]
glusterfs: always allocate io_u->engine_data
When the iodepth is greater then 1 some io_us in fio_gf_io_u_init() can
be fooled into thinking their .engine_data has already been set when it
hasn't. This happens because .index and .engine_data are part of the
same union and .index can be set to non-zero values:
static int init_io_u(struct thread_data *td)
...
for (i = 0; i < max_units; i++) {
...
io_u->index = i;
...
if (td->io_ops->io_u_init) {
int ret = td->io_ops->io_u_init(td, io_u);
which means io_u->engine_data != 0 in fio_gf_io_u_init().
Fix the issue by removing the initial check of io_u->engine_data.
Signed-off-by: Simon Gao <simon29rock@gmail.com>
Jens Axboe [Fri, 30 Mar 2018 16:16:27 +0000 (10:16 -0600)]
Merge branch 'aarch64-crc32c' of https://github.com/sitsofe/fio
* 'aarch64-crc32c' of https://github.com/sitsofe/fio:
Minor style changes
aarch64: refactor HW ARM CRC32c detection
Sitsofe Wheeler [Wed, 28 Mar 2018 15:09:22 +0000 (16:09 +0100)]
Minor style changes
Change some #endif comments to exactly match the starting #ifdef and
in crc files and make the second posixaio configure test more
descriptive.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Wed, 28 Mar 2018 14:49:14 +0000 (15:49 +0100)]
aarch64: refactor HW ARM CRC32c detection
- Tweak how fio does the configure probe for hardware accelerated ARM
CRC32c
- Move the Linux specific runtime detection into os-linux.h making it
cleaner to implement runtime probing for OSes other than Linux in the
future
- Make the ARCH_HAVE_CRC_CRYPTO define go into the config-host.* files
the same way as most other configure defines
- Set/use only the ARCH_HAVE_CRC_CRYPTO define rather than also
creating/using ARCH_HAVE_ARM64_CRC_CRYPTO
This commit should silence the "Fails to compile: error: sys/auxv.h file
not found" compilation message that was seen on FreeBSD 12/arm64 (but
note a future commit will be required to implement runtime probing and
actually enable the accelerated path on that platform).
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Thu, 29 Mar 2018 16:02:25 +0000 (10:02 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
switch_ioscheduler(): only remove the last character if it's a newline
Make it clear to Coverity that the tmp buffer in switch_ioscheduler() is \0-terminated
Bart Van Assche [Thu, 29 Mar 2018 15:30:38 +0000 (08:30 -0700)]
switch_ioscheduler(): only remove the last character if it's a newline
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Thu, 29 Mar 2018 15:24:23 +0000 (08:24 -0700)]
Make it clear to Coverity that the tmp buffer in switch_ioscheduler() is \0-terminated
The 'tmp' buffer used by switch_ioscheduler() will be '\0'-terminated
because the I/O scheduler name read from sysfs has much fewer characters
than the buffer size (256). Make it clear to Coverity that it is
guaranteed that the 'tmp' buffer is '\0'-terminated. Additionally, fix
the order of the fread() arguments. This patch fixes Coverity ID #24132.
Fixes:
b44b9e45cf38 ("Clear sysfs path before reading current ioscheduler from sysfs")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Fri, 23 Mar 2018 15:58:51 +0000 (09:58 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
smalloc: Remove Valgrind instrumentation
Jens Axboe [Thu, 22 Mar 2018 17:29:25 +0000 (11:29 -0600)]
smalloc: Remove Valgrind instrumentation
Since it can happen that memory allocated with smalloc() is freed by
a child process and since Valgrind's memory tracking is per process
the smalloc()/sfree() instrumentation leads to incorrect Valgrind
reports. Hence remove that instrumentation again.
Fixes:
0ffccc21fcd6 ("Improve Valgrind instrumentation of memory allocations")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Thu, 22 Mar 2018 17:29:25 +0000 (11:29 -0600)]
server: use scalloc() for sk_out allocation
This informs valgrind that it's properly initialized.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 22 Mar 2018 17:02:52 +0000 (11:02 -0600)]
Merge branch 'gcc' of https://github.com/sitsofe/fio
* 'gcc' of https://github.com/sitsofe/fio:
compiler: set minimum compiler version to GCC 4.1.0
Sitsofe Wheeler [Wed, 21 Mar 2018 22:35:21 +0000 (22:35 +0000)]
compiler: set minimum compiler version to GCC 4.1.0
fio hasn't compiled under GCC 3 for a while due to unguarded calls to
GCC's __sync intrinsics in critical paths. Since those intrinsics didn't
come along until GCC 4.1.0 change fio to explicitly error out on older
compilers.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Thu, 22 Mar 2018 14:34:48 +0000 (08:34 -0600)]
smalloc: oom cleanups
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 22 Mar 2018 02:09:36 +0000 (20:09 -0600)]
sg: fix sign extension
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 22 Mar 2018 01:19:58 +0000 (19:19 -0600)]
server: fix error handling for shared memory handling
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 22 Mar 2018 01:01:33 +0000 (19:01 -0600)]
Merge branch 'include_refactor' of https://github.com/sitsofe/fio
* 'include_refactor' of https://github.com/sitsofe/fio:
Add include-what-you-use pragmas
oslib: make str* compat functions more uniform
Use POSIX path for poll.h and fcntl.h headers
Refactor #includes and headers
Jens Axboe [Thu, 22 Mar 2018 00:58:22 +0000 (18:58 -0600)]
server: handle shared mem pool allocation failures
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Sitsofe Wheeler [Tue, 20 Mar 2018 15:18:43 +0000 (15:18 +0000)]
Add include-what-you-use pragmas
Add IWYU pragmas to improve its analysis.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Tue, 20 Mar 2018 10:46:32 +0000 (10:46 +0000)]
oslib: make str* compat functions more uniform
- Ensure there are CONFIG_ guards in the headers and .c files because
sometime utilities include oslib headers directly and reference oslib
.c files in the Makefile
- Ensure all the oslib/str*.h files have redefinition guards
- Shuffle some #include lines from the .h files to the .c ones
- Ensure the <name>.c files have #include "<name>.h" in them so the
compiler can check the declaration and definition match
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Mon, 19 Mar 2018 05:43:41 +0000 (05:43 +0000)]
Use POSIX path for poll.h and fcntl.h headers
musl complains when using include paths like <sys/poll.h> rather than
POSIX's <poll.h> so change the poll.h include path and update the
windows poll.h compatibility shim to match.
Allegedly sys/poll.h was needed for GLIBCs that were older than 2.3
according to
https://www.winehq.org/pipermail/wine-patches/2014-June/132877.html but
I can still see that path in an ancient Red Hat 5.0
glibc-devel-2.0.7-32.i386.rpm ...
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Mon, 19 Mar 2018 05:33:13 +0000 (05:33 +0000)]
Refactor #includes and headers
- Try and remove unneeded #include lines
- Try and add #include lines that would allow the files to be built in a
more standalone manner
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Wed, 21 Mar 2018 21:48:17 +0000 (15:48 -0600)]
server: process STOP/QUIT commands out-of-line
At the end of a job, the stats entries are queued up for
out-of-line transmission, but we immediately queue a client
STOP+QUIT command right after that. If the command handling
loop doesn't wakeup and process the stats commands before we
process the STOP+QUIT inline, then we lose the stats output
from those commands. The result is that the client won't see
the job output from that job, or the "All clients" output.
We should impose proper ordering for all commands, but this
is a quick fix to ensure that we don't hit the above mentioned
but. For other commands we don't care about ordering between
inline and out-of-line commands, so it's likely good enough
to carry us forward.
Reported-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 21 Mar 2018 14:26:28 +0000 (08:26 -0600)]
Merge branch 'asprintf' of https://github.com/bvanassche/fio
* 'asprintf' of https://github.com/bvanassche/fio:
verify: Simplify dump_buf()
log: Modify the implementation such that it uses asprintf()
Add an asprintf() implementation
Jens Axboe [Wed, 21 Mar 2018 14:23:50 +0000 (08:23 -0600)]
debug: remove extra parens
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 20 Mar 2018 17:19:19 +0000 (11:19 -0600)]
optgroup: move debug code into function
The current global construct breaks on some compilers:
optgroup.c:208: error: expected identifier or '(' before 'do'
optgroup.c:208: error: expected identifier or '(' before 'while'
Reported-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Fixes:
96344ff00349 ("optgroup: add check for optgroup bit numbers being within range")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 20 Mar 2018 14:27:43 +0000 (08:27 -0600)]
Fix whitespace issues in previous commit
Lots of trailing whitespace.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 20 Mar 2018 14:24:20 +0000 (08:24 -0600)]
Merge branch 'expand_fiohistparser' of https://github.com/shimrot/fio
* 'expand_fiohistparser' of https://github.com/shimrot/fio:
Expand fiologparser_hist operations with new options
krisd [Tue, 20 Mar 2018 05:42:26 +0000 (00:42 -0500)]
Expand fiologparser_hist operations with new options
New options are:
* noweight: don't weight samples across intervals
* percentiles: allow print of more percentiles than just 50,90,95%s
* Fix so works with newer 'ns' results, but add --usbin option
so can still be used with older results.
* directions=rwtm option to allow print of independent direction results
Jens Axboe [Mon, 19 Mar 2018 21:56:20 +0000 (15:56 -0600)]
optgroup: add check for optgroup bit numbers being within range
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Mon, 12 Mar 2018 22:29:34 +0000 (15:29 -0700)]
verify: Simplify dump_buf()
Use asprintf() to build the output file path instead of using a fixed
size buffer for that path.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 13 Mar 2018 22:28:17 +0000 (15:28 -0700)]
log: Modify the implementation such that it uses asprintf()
This makes the logging implementation simpler and easier to read.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 22:38:28 +0000 (15:38 -0700)]
Add an asprintf() implementation
Since I would like to use the asprintf() function in the ZBC code and
since that function is not available on every platform supported by
fio, add an asprintf() implementation.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Mon, 19 Mar 2018 16:24:10 +0000 (10:24 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Suppress uninteresting data race reports
gettime: Rework the clock thread starting mechanism
Improve Valgrind instrumentation of memory allocations
Rename fio_mutex into fio_sem
Split mutex.c and .h each into three files
Kris Davis [Mon, 19 Mar 2018 16:19:44 +0000 (10:19 -0600)]
sg: add read/write FUA options
Enable the user to ask for the FUA bits to be set on read and
write operations.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 17 Mar 2018 20:43:16 +0000 (14:43 -0600)]
Merge branch 'pthread-cond' of https://github.com/bvanassche/fio
* 'pthread-cond' of https://github.com/bvanassche/fio:
Signal td->free_cond with the associated mutex held
Make sure that assert() expressions do not have side effects
Bart Van Assche [Fri, 16 Mar 2018 15:38:34 +0000 (08:38 -0700)]
Signal td->free_cond with the associated mutex held
Calling pthread_cond_signal() or pthread_cond_broadcast() without
holding the associated mutex can lead to missed wakeups. Hence ensure
that td->io_u_lock is held around pthread_cond_signal(&td->free_cond)
calls. A quote from the POSIX spec
(http://pubs.opengroup.org/onlinepubs/
9699919799/functions/pthread_cond_broadcast.html):
"The pthread_cond_broadcast() or pthread_cond_signal() functions may be
called by a thread whether or not it currently owns the mutex that
threads calling pthread_cond_wait() or pthread_cond_timedwait() have
associated with the condition variable during their waits; however, if
predictable scheduling behavior is required, then that mutex shall be
locked by the thread calling pthread_cond_broadcast() or
pthread_cond_signal()."
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Fri, 16 Mar 2018 15:44:35 +0000 (08:44 -0700)]
Make sure that assert() expressions do not have side effects
Assert statements are compiled out if NDEBUG is defined. Hence make sure
that the expressions passed to assert do not have side effects.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 13 Mar 2018 21:44:51 +0000 (14:44 -0700)]
Suppress uninteresting data race reports
Thread checkers like Helgrind and DRD report all concurrent
modifications of variables that are serialized by a recognized
synchronization primitive. Suppress reports about intentionally
unsynchronized accesses.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 13 Mar 2018 17:43:02 +0000 (10:43 -0700)]
gettime: Rework the clock thread starting mechanism
Replace two mutexes in struct clock_thread that are intended to make
the clock threads start simultaneously by a fio_mutex.
This patch avoids that drd reports the following:
==23831== Thread 2:
==23831== Mutex not locked by calling thread: mutex 0x6cb7bf8, recursion count 1, owner 1.
==23831== at 0x4C38EC3: pthread_mutex_unlock_intercept (drd_pthread_intercepts.c:978)
==23831== by 0x4C38EC3: pthread_mutex_unlock (drd_pthread_intercepts.c:991)
==23831== by 0x420974: clock_thread_fn (gettime.c:604)
==23831== by 0x4C34C44: vgDrd_thread_wrapper (drd_pthread_intercepts.c:444)
==23831== by 0x5E4159A: start_thread (in /lib64/libpthread-2.27.so)
==23831== by 0x6356A1E: clone (in /lib64/libc-2.27.so)
==23924== Mutex still locked at thread exit: mutex 0x6cb7bd0, recursion count 1, owner 2.
==23924== at 0x4C35BF2: pthread_join_intercept (drd_pthread_intercepts.c:711)
==23924== by 0x4C35BF2: pthread_join (drd_pthread_intercepts.c:718)
==23924== by 0x420D7F: fio_monotonic_clocktest (gettime.c:722)
==23924== by 0x420441: fio_clock_init (gettime.c:424)
==23924== by 0x437526: fio_time_init (time.c:141)
==23924== by 0x49BF41: main (fio.c:56)
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Thu, 8 Mar 2018 21:41:36 +0000 (13:41 -0800)]
Improve Valgrind instrumentation of memory allocations
In smalloc, let Valgrind do detection of red zone modifications. In
synchr.c, let Valgrind perform use-after-free detection of
synchronization objects.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 14 Mar 2018 15:31:37 +0000 (08:31 -0700)]
Rename fio_mutex into fio_sem
Since fio_mutex implements a semaphore, rename fio_mutex into fio_sem.
Rename those fio_sem instances that have 'mutex' embedded in their
name. Rename the mutex.h and .c source files into fio_sem.h and .c
respectively.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 14 Mar 2018 15:26:15 +0000 (08:26 -0700)]
Split mutex.c and .h each into three files
Create separate header and source files for the struct fio_mutex
functions, for the struct fio_rwlock functions and for the helper
functions for initializing process shared mutexes and condition
variables. Replace an #include directive by a forward declaration
in fio.h. Minimize the #include directives in mutex.c, rwlock.c
and pshared.c. This patch does not change any functionality.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Tue, 13 Mar 2018 17:49:55 +0000 (11:49 -0600)]
io_u: only rewind file position if it's non-zero
This:
bs=8k
rw=read:-4k
is supposed to read 0..8k, then add 8k and subtract 4k, ending
up with a next read at 4k..12k and so forth. But we rewind too
quickly, and the first IO fails as being out-of-bounds.
Fixes:
c22825bb537af ("Fix backwards reads with --size smaller than the file size")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 13 Mar 2018 14:41:37 +0000 (08:41 -0600)]
configure: don't disable lex on FreeBSD
Previous commit turned it off by mistake.
Fixes:
89556808ef14 ("configure: Disable lex on NetBSD")
Reported-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 13 Mar 2018 00:13:10 +0000 (18:13 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
parse: Fix two compiler warnings
stat: Fix a compiler warning in __show_run_stats()
Rename struct rb_node into struct fio_rb_node
configure: Disable lex on NetBSD
Bart Van Assche [Mon, 12 Mar 2018 21:24:27 +0000 (14:24 -0700)]
parse: Fix two compiler warnings
Avoid that gcc on NetBSD reports the following compiler warnings:
CC parse.o
parse.c: In function 'get_mult_time':
parse.c:175: warning: array subscript has type 'char'
parse.c: In function '__get_mult_bytes':
parse.c:221: warning: array subscript has type 'char'
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 21:16:56 +0000 (14:16 -0700)]
stat: Fix a compiler warning in __show_run_stats()
Avoid that gcc reports the following warning on NetBSD systems:
CC stat.o
stat.c: In function '__show_run_stats':
stat.c:1868: warning: dereferencing type-punned pointer will break strict-aliasing rules
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 21:13:38 +0000 (14:13 -0700)]
Rename struct rb_node into struct fio_rb_node
This patch avoids that building fio fails as follows on NetBSD systems:
CC crc/test.o
In file included from crc/../iolog.h:4,
from crc/../stat.h:4,
from crc/../thread_options.h:7,
from crc/../fio.h:18,
from crc/test.c:5:
crc/../lib/rbtree.h:101: error: redefinition of 'struct rb_node'
Makefile:336: recipe for target 'crc/test.o' failed
gmake: *** [crc/test.o] Error 1
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 21:37:37 +0000 (14:37 -0700)]
configure: Disable lex on NetBSD
Since the NetBSD lex software does not support all features (yylval)
used by fio, disable lex on NetBSD.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Sat, 10 Mar 2018 00:46:36 +0000 (16:46 -0800)]
filesetup: Initialize all members of struct fio_file
This patch avoids that Valgrind reports complaints like the following:
==10492== Conditional jump or move depends on uninitialised value(s)
==10492== at 0x43BF19: fio_file_reset (filesetup.c:1872)
==10492== by 0x43AB7F: alloc_new_file (filesetup.c:1496)
==10492== by 0x43AED4: add_file (filesetup.c:1581)
==10492== by 0x4528D3: str_filename_cb (options.c:1276)
==10492== by 0x44D83A: __handle_option (parse.c:722)
==10492== by 0x44E303: handle_option (parse.c:941)
==10492== by 0x44E68B: parse_cmd_option (parse.c:1061)
==10492== by 0x4541DD: fio_cmd_option_parse (options.c:4966)
==10492== by 0x427C6D: parse_cmd_line (init.c:2652)
==10492== by 0x428685: parse_options (init.c:2924)
==10492== by 0x4920AC: main (fio.c:47)
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Sat, 10 Mar 2018 00:46:22 +0000 (16:46 -0800)]
helper_thread: Initialize all helper_data members before using it
This patch avoids that Valgrind reports the following:
==10492== Thread 2:
==10492== Conditional jump or move depends on uninitialised value(s)
==10492== at 0x480295: helper_thread_main (helper_thread.c:89)
==10492== by 0x57B961A: start_thread (pthread_create.c:465)
==10492== by 0x5CEC98E: clone (clone.S:95)
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Sat, 10 Mar 2018 00:46:02 +0000 (16:46 -0800)]
Declare debug_levels[] const
Since debug_levels[] is not modified by any code, declare it const.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 19:55:21 +0000 (12:55 -0700)]
filesetup: don't round/adjust size for size_percent == 100
For the full device, don't apply any rounding or bs based
adjustment. Just treat it like size_percent wasn't even set.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 9 Mar 2018 16:51:04 +0000 (08:51 -0800)]
Makefile: Rerun the configure script if it has been modified
Because of a typo in the Makefile ("config-host-mak") the configure
script is not rerun if the configure script is modified after
config-host.mak has been generated. Merge the two make goals for
rerunning configure into a single goal. This patch relies on the
following make behavior (see also https://www.gnu.org/software/make/manual/make.html#Remaking-Makefiles):
Sometimes makefiles can be remade from other files, such as RCS or SCCS
files. If a makefile can be remade from other files, you probably want
make to get an up-to-date version of the makefile to read in.
To this end, after reading in all makefiles, make will consider each as a
goal target and attempt to update it. If a makefile has a rule which says
how to update it (found either in that very makefile or in another one) or
if an implicit rule applies to it (see Using Implicit Rules), it will be
updated if necessary. After all makefiles have been checked, if any have
actually been changed, make starts with a clean slate and reads all the
makefiles over again. (It will also attempt to update each of them over
again, but normally this will not change them again, since they are
already up to date.)
This patch has been tested by verifying that the commands run by make
were as expected for the following invocations:
rm -f config-host.mak && make
make && touch configure && make
make
Note: fio already requires GNU make because of the use of "ifeq" etc.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 17:23:53 +0000 (10:23 -0700)]
io_u: 'is_random' can be a boolean
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 17:22:01 +0000 (10:22 -0700)]
io_u: kill get_next_{offset,buflen} wrappers
After the previous commit, they are just wrappers. Kill them.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 9 Mar 2018 17:16:23 +0000 (09:16 -0800)]
Remove prof_io_ops.fill_io_u_off(), .fill_io_u_size() and .get_next_file()
Remove the struct prof_io_ops methods for which no implementations
have been defined. This patch does not change any functionality.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 14:59:03 +0000 (07:59 -0700)]
mutex: fix other locations where we are not waking within the lock
Our workqueue implementation had a few cases where it did not wake
while holding the lock, fix those up. Ditto for verify, where
async IO completion also needed the ordering switched.
Also see commit
e4ccf13bad6d.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 14:55:40 +0000 (07:55 -0700)]
mutex: ensure that fio_mutex_up() holds mutex lock during wakeup
Some implementations mandate that you hold the lock while
issuing the wakeup, as not to lose a wakeup. This seems to
be the case for Windows. See the lengthy thread on the topic
here:
https://www.spinics.net/lists/fio/msg06772.html
Thanks to Sitsofe Wheeler for heroic debugging.
Reported-by: Rebecca Cran <rebecca@bluestop.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 7 Mar 2018 17:22:28 +0000 (10:22 -0700)]
Merge branch 'disable_opt' of https://github.com/sitsofe/fio
* 'disable_opt' of https://github.com/sitsofe/fio:
appveyor: disable setting compiler march
configure: make --disable-optimizations disable march=native
Sitsofe Wheeler [Wed, 7 Mar 2018 16:36:32 +0000 (16:36 +0000)]
appveyor: disable setting compiler march
Commit
a817dc3b3b5a0efc95aaca366875eac67607cd5b ("Default to building
native code") tunes binaries to only run on the build system's
architecture by default. However as AppVeyor builds Windows MSIs for
public consumption, let's turn that feature off here to prevent people
accidentally ending up with binaries that contain instructions their
system can't run.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Wed, 7 Mar 2018 09:54:45 +0000 (09:54 +0000)]
configure: make --disable-optimizations disable march=native
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Wed, 7 Mar 2018 15:32:05 +0000 (08:32 -0700)]
configure: don't override march if already set
Some archs (like arm or s390) already set an appropriate architecture,
so don't override it for those cases.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jeff Furlong [Wed, 7 Mar 2018 00:53:20 +0000 (17:53 -0700)]
Reduce LOG_MSEC_SLACK
Starting in fio 2.99, there is an odd problem when --ramp_time is used
with --write_iops_log. The reported IOPs for the first ~200s or so are
artificially inflated by fio, always monotonically increasing until
about 200s, then resets to normal. With fio 2.99, disabling --ramp_time
avoids the issue. And, with fio 2.21, using --ramp_time and
--write_iops_log does not create the problem. Consider a job:
fio --name=test --ioengine=libaio --direct=1 --rw=randread --iodepth=1 \\
--size=100% --bs=4k --filename=/dev/nvme1n1 --runtime=300s \\
--ramp_time=10s --write_iops_log=test --log_avg_msec=1000
See the attached plot for fio induced IOPs during the first ~200s.
Using a longer runtime makes identifying the issue easier if the DUT is
consistently performing.
One simple method to resolve this inflation is by reducing the
LOG_MSEC_SLACK from 10 milliseconds to 1 millisecond, given that we now
operate some time structures to the nanosecond level (so 10ms of slack
seems excessive). Below I also fixup some data type issues to match
existing types (avoids typecasts or potential compiler warnings).
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 7 Mar 2018 00:44:17 +0000 (17:44 -0700)]
Merge branch 'wip-rbd-engine' of https://github.com/dillaman/fio
* 'wip-rbd-engine' of https://github.com/dillaman/fio:
rbd: remove support for blkin tracing
rbd: fixed busy-loop when using eventfd polling
Jason Dillaman [Wed, 7 Mar 2018 00:05:01 +0000 (19:05 -0500)]
rbd: remove support for blkin tracing
The referenced APIs for blkin tracing were not merged into librbd.
Instead, blkin tracing can be enabled for all IO by setting
'rbd blkin trace all = true' in the config.
Jason Dillaman [Wed, 7 Mar 2018 00:01:18 +0000 (19:01 -0500)]
rbd: fixed busy-loop when using eventfd polling
The eventfd poll would continuously fire since the fd was never
read. Switched to a semaphore eventfd and now drain the events
as completions are handled.
Jens Axboe [Tue, 6 Mar 2018 21:28:16 +0000 (14:28 -0700)]
Default to building native code
If the local compiler supports it, use -march=native to ensure we
take full advantage of a local build.
If building distro packages, or otherwise for a different platform
than the host, then use --disable-native with configure to remove
this optimization.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 6 Mar 2018 19:42:57 +0000 (12:42 -0700)]
Don't make fadvise failure fatal
It's just an advisory hint. Warn once that it failed, then
march on.
Fixes: https://github.com/axboe/fio/issues/554
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 5 Mar 2018 17:29:39 +0000 (10:29 -0700)]
Merge branch 'howto_typos' of https://github.com/dirtyharrycallahan/fio
* 'howto_typos' of https://github.com/dirtyharrycallahan/fio:
doc: fix typos in HOWTO
Patrick Callahan [Mon, 5 Mar 2018 17:23:51 +0000 (12:23 -0500)]
doc: fix typos in HOWTO
Jens Axboe [Mon, 5 Mar 2018 15:47:51 +0000 (08:47 -0700)]
Merge branch 'manpage_typos' of https://github.com/dirtyharrycallahan/fio
* 'manpage_typos' of https://github.com/dirtyharrycallahan/fio:
doc: fix typos in fio man page
Patrick Callahan [Mon, 5 Mar 2018 14:23:39 +0000 (09:23 -0500)]
doc: fix typos in fio man page
Jens Axboe [Fri, 2 Mar 2018 19:02:08 +0000 (12:02 -0700)]
Merge branch 'mpath_nvme_diskutil' of https://github.com/sitsofe/fio
* 'mpath_nvme_diskutil' of https://github.com/sitsofe/fio:
diskutil: minor style cleanup
diskutil: try additional slave device path if first fails
Sitsofe Wheeler [Fri, 2 Mar 2018 15:51:01 +0000 (15:51 +0000)]
diskutil: minor style cleanup
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Potnuri Bharat Teja [Fri, 2 Mar 2018 15:26:01 +0000 (15:26 +0000)]
diskutil: try additional slave device path if first fails
Running fio on an NVMEoF device with multipath slaves fails with a "No
such file or directory" error when displaying disk stats. Correct this
by using a different path to the sysfs slave device.
[sitsofe: change new path to be a fallback when regular path fails]
Fixes: https://github.com/axboe/fio/issues/525
Closes: https://github.com/axboe/fio/pull/526
Signed-off-by: Potnuri Bharat Teja <bharat@chelsio.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Thu, 1 Mar 2018 16:18:43 +0000 (09:18 -0700)]
Merge branch 'hotfix_counter_overflow' of https://github.com/ifke/fio
* 'hotfix_counter_overflow' of https://github.com/ifke/fio:
Fix overflow of counters incremented on each I/O operation
Alexander Larin [Wed, 28 Feb 2018 11:25:42 +0000 (14:25 +0300)]
Fix overflow of counters incremented on each I/O operation
- In the thread_stat struct: uint32_t counters are updated to uint64_t.
- In the io_u_plat_entry struct: unsigned int counters are updated to
uint64_t.
It fixes overflow of these counters.
Signed-off-by: Alexander Larin <znahar@yandex-team.ru>
Jens Axboe [Wed, 28 Feb 2018 16:44:08 +0000 (09:44 -0700)]
Merge branch 'win_build' of https://github.com/sitsofe/fio
* 'win_build' of https://github.com/sitsofe/fio:
windows: document MinGW zlib install and remove custom zlib search
windows: minor windows installer improvements
appveyor: minor refactoring, clarifications
Jens Axboe [Wed, 28 Feb 2018 15:43:22 +0000 (08:43 -0700)]
configure: improve static zlib package warning
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 27 Feb 2018 23:15:22 +0000 (16:15 -0700)]
Merge branch 'master' of https://github.com/brycepg/fio
* 'master' of https://github.com/brycepg/fio:
Refer to zlib-static package in zlib static build warning
Bryce Guinta [Tue, 27 Feb 2018 23:09:35 +0000 (16:09 -0700)]
Refer to zlib-static package in zlib static build warning
Signed-off-by: Bryce Guinta <bryce.paul.guinta@gmail.com>
Jens Axboe [Mon, 26 Feb 2018 18:59:54 +0000 (11:59 -0700)]
Merge branch 'wip-ifed-howto-update' of https://github.com/ifed01/fio
* 'wip-ifed-howto-update' of https://github.com/ifed01/fio:
Update HOWTO with RADOS information
Igor Fedotov [Mon, 26 Feb 2018 17:53:52 +0000 (20:53 +0300)]
Update HOWTO with RADOS information
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
Sitsofe Wheeler [Wed, 21 Feb 2018 18:45:55 +0000 (18:45 +0000)]
windows: document MinGW zlib install and remove custom zlib search
In
5580d200d7ae49cdb4f30913e3ef2d74e431ff60 ("appveyor: install zlib and minor
clean ups") I found AppVeyor builds could use a pre-built, statically linkable
zlib from Cygwin/MinGW.
Document the MinGW zlib packages in the README and remove the custom Windows
zlib searching from the configure script.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Wed, 21 Feb 2018 14:41:14 +0000 (14:41 +0000)]
windows: minor windows installer improvements
- Stop hard coding fio version in the Windows installer build file and try and
use the generated version instead
- Make installer build use sys.BUILDARCH rather than env.FIO_ARCH
- Change IO to I/O in descriptions to match other packages
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Wed, 21 Feb 2018 09:56:38 +0000 (09:56 +0000)]
appveyor: minor refactoring, clarifications
- Set platform directly to x64 or x86 and use PLATFORM instead of BUILD_ARCH
- Set the PATH once (via the install section) because environment variables
persist to later sections
- Add a note that shallow cloning means FIO-VERSION-GEN is unable to build a
fancy version number (because it won't find any tags)
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Tue, 20 Feb 2018 22:30:28 +0000 (15:30 -0700)]
Fio 3.5
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 14 Feb 2018 19:43:04 +0000 (12:43 -0700)]
Merge branch 'wip-ifed-rados' of https://github.com/ifed01/fio
* 'wip-ifed-rados' of https://github.com/ifed01/fio:
Add support for Ceph Rados benchmarking.
Igor Fedotov [Fri, 9 Feb 2018 14:39:28 +0000 (17:39 +0300)]
Add support for Ceph Rados benchmarking.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
Jens Axboe [Mon, 12 Feb 2018 17:55:07 +0000 (10:55 -0700)]
debug: fix inverted logic in fio_did_warn()
Fixes:
5ec3bc5401f3 ("init: add global 'warned' state")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 12 Feb 2018 17:19:18 +0000 (10:19 -0700)]
iolog: convert drop warning to fio_did_warn()
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 12 Feb 2018 17:18:32 +0000 (10:18 -0700)]
io_u: convert zoned bug warning to fio_did_warn()
Signed-off-by: Jens Axboe <axboe@kernel.dk>