Jens Axboe [Mon, 19 Mar 2018 16:24:10 +0000 (10:24 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Suppress uninteresting data race reports
gettime: Rework the clock thread starting mechanism
Improve Valgrind instrumentation of memory allocations
Rename fio_mutex into fio_sem
Split mutex.c and .h each into three files
Kris Davis [Mon, 19 Mar 2018 16:19:44 +0000 (10:19 -0600)]
sg: add read/write FUA options
Enable the user to ask for the FUA bits to be set on read and
write operations.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 17 Mar 2018 20:43:16 +0000 (14:43 -0600)]
Merge branch 'pthread-cond' of https://github.com/bvanassche/fio
* 'pthread-cond' of https://github.com/bvanassche/fio:
Signal td->free_cond with the associated mutex held
Make sure that assert() expressions do not have side effects
Bart Van Assche [Fri, 16 Mar 2018 15:38:34 +0000 (08:38 -0700)]
Signal td->free_cond with the associated mutex held
Calling pthread_cond_signal() or pthread_cond_broadcast() without
holding the associated mutex can lead to missed wakeups. Hence ensure
that td->io_u_lock is held around pthread_cond_signal(&td->free_cond)
calls. A quote from the POSIX spec
(http://pubs.opengroup.org/onlinepubs/
9699919799/functions/pthread_cond_broadcast.html):
"The pthread_cond_broadcast() or pthread_cond_signal() functions may be
called by a thread whether or not it currently owns the mutex that
threads calling pthread_cond_wait() or pthread_cond_timedwait() have
associated with the condition variable during their waits; however, if
predictable scheduling behavior is required, then that mutex shall be
locked by the thread calling pthread_cond_broadcast() or
pthread_cond_signal()."
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Fri, 16 Mar 2018 15:44:35 +0000 (08:44 -0700)]
Make sure that assert() expressions do not have side effects
Assert statements are compiled out if NDEBUG is defined. Hence make sure
that the expressions passed to assert do not have side effects.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 13 Mar 2018 21:44:51 +0000 (14:44 -0700)]
Suppress uninteresting data race reports
Thread checkers like Helgrind and DRD report all concurrent
modifications of variables that are serialized by a recognized
synchronization primitive. Suppress reports about intentionally
unsynchronized accesses.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 13 Mar 2018 17:43:02 +0000 (10:43 -0700)]
gettime: Rework the clock thread starting mechanism
Replace two mutexes in struct clock_thread that are intended to make
the clock threads start simultaneously by a fio_mutex.
This patch avoids that drd reports the following:
==23831== Thread 2:
==23831== Mutex not locked by calling thread: mutex 0x6cb7bf8, recursion count 1, owner 1.
==23831== at 0x4C38EC3: pthread_mutex_unlock_intercept (drd_pthread_intercepts.c:978)
==23831== by 0x4C38EC3: pthread_mutex_unlock (drd_pthread_intercepts.c:991)
==23831== by 0x420974: clock_thread_fn (gettime.c:604)
==23831== by 0x4C34C44: vgDrd_thread_wrapper (drd_pthread_intercepts.c:444)
==23831== by 0x5E4159A: start_thread (in /lib64/libpthread-2.27.so)
==23831== by 0x6356A1E: clone (in /lib64/libc-2.27.so)
==23924== Mutex still locked at thread exit: mutex 0x6cb7bd0, recursion count 1, owner 2.
==23924== at 0x4C35BF2: pthread_join_intercept (drd_pthread_intercepts.c:711)
==23924== by 0x4C35BF2: pthread_join (drd_pthread_intercepts.c:718)
==23924== by 0x420D7F: fio_monotonic_clocktest (gettime.c:722)
==23924== by 0x420441: fio_clock_init (gettime.c:424)
==23924== by 0x437526: fio_time_init (time.c:141)
==23924== by 0x49BF41: main (fio.c:56)
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Thu, 8 Mar 2018 21:41:36 +0000 (13:41 -0800)]
Improve Valgrind instrumentation of memory allocations
In smalloc, let Valgrind do detection of red zone modifications. In
synchr.c, let Valgrind perform use-after-free detection of
synchronization objects.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 14 Mar 2018 15:31:37 +0000 (08:31 -0700)]
Rename fio_mutex into fio_sem
Since fio_mutex implements a semaphore, rename fio_mutex into fio_sem.
Rename those fio_sem instances that have 'mutex' embedded in their
name. Rename the mutex.h and .c source files into fio_sem.h and .c
respectively.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 14 Mar 2018 15:26:15 +0000 (08:26 -0700)]
Split mutex.c and .h each into three files
Create separate header and source files for the struct fio_mutex
functions, for the struct fio_rwlock functions and for the helper
functions for initializing process shared mutexes and condition
variables. Replace an #include directive by a forward declaration
in fio.h. Minimize the #include directives in mutex.c, rwlock.c
and pshared.c. This patch does not change any functionality.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Tue, 13 Mar 2018 17:49:55 +0000 (11:49 -0600)]
io_u: only rewind file position if it's non-zero
This:
bs=8k
rw=read:-4k
is supposed to read 0..8k, then add 8k and subtract 4k, ending
up with a next read at 4k..12k and so forth. But we rewind too
quickly, and the first IO fails as being out-of-bounds.
Fixes:
c22825bb537af ("Fix backwards reads with --size smaller than the file size")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 13 Mar 2018 14:41:37 +0000 (08:41 -0600)]
configure: don't disable lex on FreeBSD
Previous commit turned it off by mistake.
Fixes:
89556808ef14 ("configure: Disable lex on NetBSD")
Reported-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 13 Mar 2018 00:13:10 +0000 (18:13 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
parse: Fix two compiler warnings
stat: Fix a compiler warning in __show_run_stats()
Rename struct rb_node into struct fio_rb_node
configure: Disable lex on NetBSD
Bart Van Assche [Mon, 12 Mar 2018 21:24:27 +0000 (14:24 -0700)]
parse: Fix two compiler warnings
Avoid that gcc on NetBSD reports the following compiler warnings:
CC parse.o
parse.c: In function 'get_mult_time':
parse.c:175: warning: array subscript has type 'char'
parse.c: In function '__get_mult_bytes':
parse.c:221: warning: array subscript has type 'char'
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 21:16:56 +0000 (14:16 -0700)]
stat: Fix a compiler warning in __show_run_stats()
Avoid that gcc reports the following warning on NetBSD systems:
CC stat.o
stat.c: In function '__show_run_stats':
stat.c:1868: warning: dereferencing type-punned pointer will break strict-aliasing rules
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 21:13:38 +0000 (14:13 -0700)]
Rename struct rb_node into struct fio_rb_node
This patch avoids that building fio fails as follows on NetBSD systems:
CC crc/test.o
In file included from crc/../iolog.h:4,
from crc/../stat.h:4,
from crc/../thread_options.h:7,
from crc/../fio.h:18,
from crc/test.c:5:
crc/../lib/rbtree.h:101: error: redefinition of 'struct rb_node'
Makefile:336: recipe for target 'crc/test.o' failed
gmake: *** [crc/test.o] Error 1
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 21:37:37 +0000 (14:37 -0700)]
configure: Disable lex on NetBSD
Since the NetBSD lex software does not support all features (yylval)
used by fio, disable lex on NetBSD.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Sat, 10 Mar 2018 00:46:36 +0000 (16:46 -0800)]
filesetup: Initialize all members of struct fio_file
This patch avoids that Valgrind reports complaints like the following:
==10492== Conditional jump or move depends on uninitialised value(s)
==10492== at 0x43BF19: fio_file_reset (filesetup.c:1872)
==10492== by 0x43AB7F: alloc_new_file (filesetup.c:1496)
==10492== by 0x43AED4: add_file (filesetup.c:1581)
==10492== by 0x4528D3: str_filename_cb (options.c:1276)
==10492== by 0x44D83A: __handle_option (parse.c:722)
==10492== by 0x44E303: handle_option (parse.c:941)
==10492== by 0x44E68B: parse_cmd_option (parse.c:1061)
==10492== by 0x4541DD: fio_cmd_option_parse (options.c:4966)
==10492== by 0x427C6D: parse_cmd_line (init.c:2652)
==10492== by 0x428685: parse_options (init.c:2924)
==10492== by 0x4920AC: main (fio.c:47)
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Sat, 10 Mar 2018 00:46:22 +0000 (16:46 -0800)]
helper_thread: Initialize all helper_data members before using it
This patch avoids that Valgrind reports the following:
==10492== Thread 2:
==10492== Conditional jump or move depends on uninitialised value(s)
==10492== at 0x480295: helper_thread_main (helper_thread.c:89)
==10492== by 0x57B961A: start_thread (pthread_create.c:465)
==10492== by 0x5CEC98E: clone (clone.S:95)
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Sat, 10 Mar 2018 00:46:02 +0000 (16:46 -0800)]
Declare debug_levels[] const
Since debug_levels[] is not modified by any code, declare it const.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 19:55:21 +0000 (12:55 -0700)]
filesetup: don't round/adjust size for size_percent == 100
For the full device, don't apply any rounding or bs based
adjustment. Just treat it like size_percent wasn't even set.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 9 Mar 2018 16:51:04 +0000 (08:51 -0800)]
Makefile: Rerun the configure script if it has been modified
Because of a typo in the Makefile ("config-host-mak") the configure
script is not rerun if the configure script is modified after
config-host.mak has been generated. Merge the two make goals for
rerunning configure into a single goal. This patch relies on the
following make behavior (see also https://www.gnu.org/software/make/manual/make.html#Remaking-Makefiles):
Sometimes makefiles can be remade from other files, such as RCS or SCCS
files. If a makefile can be remade from other files, you probably want
make to get an up-to-date version of the makefile to read in.
To this end, after reading in all makefiles, make will consider each as a
goal target and attempt to update it. If a makefile has a rule which says
how to update it (found either in that very makefile or in another one) or
if an implicit rule applies to it (see Using Implicit Rules), it will be
updated if necessary. After all makefiles have been checked, if any have
actually been changed, make starts with a clean slate and reads all the
makefiles over again. (It will also attempt to update each of them over
again, but normally this will not change them again, since they are
already up to date.)
This patch has been tested by verifying that the commands run by make
were as expected for the following invocations:
rm -f config-host.mak && make
make && touch configure && make
make
Note: fio already requires GNU make because of the use of "ifeq" etc.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 17:23:53 +0000 (10:23 -0700)]
io_u: 'is_random' can be a boolean
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 17:22:01 +0000 (10:22 -0700)]
io_u: kill get_next_{offset,buflen} wrappers
After the previous commit, they are just wrappers. Kill them.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 9 Mar 2018 17:16:23 +0000 (09:16 -0800)]
Remove prof_io_ops.fill_io_u_off(), .fill_io_u_size() and .get_next_file()
Remove the struct prof_io_ops methods for which no implementations
have been defined. This patch does not change any functionality.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 14:59:03 +0000 (07:59 -0700)]
mutex: fix other locations where we are not waking within the lock
Our workqueue implementation had a few cases where it did not wake
while holding the lock, fix those up. Ditto for verify, where
async IO completion also needed the ordering switched.
Also see commit
e4ccf13bad6d.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 14:55:40 +0000 (07:55 -0700)]
mutex: ensure that fio_mutex_up() holds mutex lock during wakeup
Some implementations mandate that you hold the lock while
issuing the wakeup, as not to lose a wakeup. This seems to
be the case for Windows. See the lengthy thread on the topic
here:
https://www.spinics.net/lists/fio/msg06772.html
Thanks to Sitsofe Wheeler for heroic debugging.
Reported-by: Rebecca Cran <rebecca@bluestop.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 7 Mar 2018 17:22:28 +0000 (10:22 -0700)]
Merge branch 'disable_opt' of https://github.com/sitsofe/fio
* 'disable_opt' of https://github.com/sitsofe/fio:
appveyor: disable setting compiler march
configure: make --disable-optimizations disable march=native
Sitsofe Wheeler [Wed, 7 Mar 2018 16:36:32 +0000 (16:36 +0000)]
appveyor: disable setting compiler march
Commit
a817dc3b3b5a0efc95aaca366875eac67607cd5b ("Default to building
native code") tunes binaries to only run on the build system's
architecture by default. However as AppVeyor builds Windows MSIs for
public consumption, let's turn that feature off here to prevent people
accidentally ending up with binaries that contain instructions their
system can't run.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Wed, 7 Mar 2018 09:54:45 +0000 (09:54 +0000)]
configure: make --disable-optimizations disable march=native
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Wed, 7 Mar 2018 15:32:05 +0000 (08:32 -0700)]
configure: don't override march if already set
Some archs (like arm or s390) already set an appropriate architecture,
so don't override it for those cases.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jeff Furlong [Wed, 7 Mar 2018 00:53:20 +0000 (17:53 -0700)]
Reduce LOG_MSEC_SLACK
Starting in fio 2.99, there is an odd problem when --ramp_time is used
with --write_iops_log. The reported IOPs for the first ~200s or so are
artificially inflated by fio, always monotonically increasing until
about 200s, then resets to normal. With fio 2.99, disabling --ramp_time
avoids the issue. And, with fio 2.21, using --ramp_time and
--write_iops_log does not create the problem. Consider a job:
fio --name=test --ioengine=libaio --direct=1 --rw=randread --iodepth=1 \\
--size=100% --bs=4k --filename=/dev/nvme1n1 --runtime=300s \\
--ramp_time=10s --write_iops_log=test --log_avg_msec=1000
See the attached plot for fio induced IOPs during the first ~200s.
Using a longer runtime makes identifying the issue easier if the DUT is
consistently performing.
One simple method to resolve this inflation is by reducing the
LOG_MSEC_SLACK from 10 milliseconds to 1 millisecond, given that we now
operate some time structures to the nanosecond level (so 10ms of slack
seems excessive). Below I also fixup some data type issues to match
existing types (avoids typecasts or potential compiler warnings).
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 7 Mar 2018 00:44:17 +0000 (17:44 -0700)]
Merge branch 'wip-rbd-engine' of https://github.com/dillaman/fio
* 'wip-rbd-engine' of https://github.com/dillaman/fio:
rbd: remove support for blkin tracing
rbd: fixed busy-loop when using eventfd polling
Jason Dillaman [Wed, 7 Mar 2018 00:05:01 +0000 (19:05 -0500)]
rbd: remove support for blkin tracing
The referenced APIs for blkin tracing were not merged into librbd.
Instead, blkin tracing can be enabled for all IO by setting
'rbd blkin trace all = true' in the config.
Jason Dillaman [Wed, 7 Mar 2018 00:01:18 +0000 (19:01 -0500)]
rbd: fixed busy-loop when using eventfd polling
The eventfd poll would continuously fire since the fd was never
read. Switched to a semaphore eventfd and now drain the events
as completions are handled.
Jens Axboe [Tue, 6 Mar 2018 21:28:16 +0000 (14:28 -0700)]
Default to building native code
If the local compiler supports it, use -march=native to ensure we
take full advantage of a local build.
If building distro packages, or otherwise for a different platform
than the host, then use --disable-native with configure to remove
this optimization.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 6 Mar 2018 19:42:57 +0000 (12:42 -0700)]
Don't make fadvise failure fatal
It's just an advisory hint. Warn once that it failed, then
march on.
Fixes: https://github.com/axboe/fio/issues/554
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 5 Mar 2018 17:29:39 +0000 (10:29 -0700)]
Merge branch 'howto_typos' of https://github.com/dirtyharrycallahan/fio
* 'howto_typos' of https://github.com/dirtyharrycallahan/fio:
doc: fix typos in HOWTO
Patrick Callahan [Mon, 5 Mar 2018 17:23:51 +0000 (12:23 -0500)]
doc: fix typos in HOWTO
Jens Axboe [Mon, 5 Mar 2018 15:47:51 +0000 (08:47 -0700)]
Merge branch 'manpage_typos' of https://github.com/dirtyharrycallahan/fio
* 'manpage_typos' of https://github.com/dirtyharrycallahan/fio:
doc: fix typos in fio man page
Patrick Callahan [Mon, 5 Mar 2018 14:23:39 +0000 (09:23 -0500)]
doc: fix typos in fio man page
Jens Axboe [Fri, 2 Mar 2018 19:02:08 +0000 (12:02 -0700)]
Merge branch 'mpath_nvme_diskutil' of https://github.com/sitsofe/fio
* 'mpath_nvme_diskutil' of https://github.com/sitsofe/fio:
diskutil: minor style cleanup
diskutil: try additional slave device path if first fails
Sitsofe Wheeler [Fri, 2 Mar 2018 15:51:01 +0000 (15:51 +0000)]
diskutil: minor style cleanup
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Potnuri Bharat Teja [Fri, 2 Mar 2018 15:26:01 +0000 (15:26 +0000)]
diskutil: try additional slave device path if first fails
Running fio on an NVMEoF device with multipath slaves fails with a "No
such file or directory" error when displaying disk stats. Correct this
by using a different path to the sysfs slave device.
[sitsofe: change new path to be a fallback when regular path fails]
Fixes: https://github.com/axboe/fio/issues/525
Closes: https://github.com/axboe/fio/pull/526
Signed-off-by: Potnuri Bharat Teja <bharat@chelsio.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Thu, 1 Mar 2018 16:18:43 +0000 (09:18 -0700)]
Merge branch 'hotfix_counter_overflow' of https://github.com/ifke/fio
* 'hotfix_counter_overflow' of https://github.com/ifke/fio:
Fix overflow of counters incremented on each I/O operation
Alexander Larin [Wed, 28 Feb 2018 11:25:42 +0000 (14:25 +0300)]
Fix overflow of counters incremented on each I/O operation
- In the thread_stat struct: uint32_t counters are updated to uint64_t.
- In the io_u_plat_entry struct: unsigned int counters are updated to
uint64_t.
It fixes overflow of these counters.
Signed-off-by: Alexander Larin <znahar@yandex-team.ru>
Jens Axboe [Wed, 28 Feb 2018 16:44:08 +0000 (09:44 -0700)]
Merge branch 'win_build' of https://github.com/sitsofe/fio
* 'win_build' of https://github.com/sitsofe/fio:
windows: document MinGW zlib install and remove custom zlib search
windows: minor windows installer improvements
appveyor: minor refactoring, clarifications
Jens Axboe [Wed, 28 Feb 2018 15:43:22 +0000 (08:43 -0700)]
configure: improve static zlib package warning
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 27 Feb 2018 23:15:22 +0000 (16:15 -0700)]
Merge branch 'master' of https://github.com/brycepg/fio
* 'master' of https://github.com/brycepg/fio:
Refer to zlib-static package in zlib static build warning
Bryce Guinta [Tue, 27 Feb 2018 23:09:35 +0000 (16:09 -0700)]
Refer to zlib-static package in zlib static build warning
Signed-off-by: Bryce Guinta <bryce.paul.guinta@gmail.com>
Jens Axboe [Mon, 26 Feb 2018 18:59:54 +0000 (11:59 -0700)]
Merge branch 'wip-ifed-howto-update' of https://github.com/ifed01/fio
* 'wip-ifed-howto-update' of https://github.com/ifed01/fio:
Update HOWTO with RADOS information
Igor Fedotov [Mon, 26 Feb 2018 17:53:52 +0000 (20:53 +0300)]
Update HOWTO with RADOS information
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
Sitsofe Wheeler [Wed, 21 Feb 2018 18:45:55 +0000 (18:45 +0000)]
windows: document MinGW zlib install and remove custom zlib search
In
5580d200d7ae49cdb4f30913e3ef2d74e431ff60 ("appveyor: install zlib and minor
clean ups") I found AppVeyor builds could use a pre-built, statically linkable
zlib from Cygwin/MinGW.
Document the MinGW zlib packages in the README and remove the custom Windows
zlib searching from the configure script.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Wed, 21 Feb 2018 14:41:14 +0000 (14:41 +0000)]
windows: minor windows installer improvements
- Stop hard coding fio version in the Windows installer build file and try and
use the generated version instead
- Make installer build use sys.BUILDARCH rather than env.FIO_ARCH
- Change IO to I/O in descriptions to match other packages
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Wed, 21 Feb 2018 09:56:38 +0000 (09:56 +0000)]
appveyor: minor refactoring, clarifications
- Set platform directly to x64 or x86 and use PLATFORM instead of BUILD_ARCH
- Set the PATH once (via the install section) because environment variables
persist to later sections
- Add a note that shallow cloning means FIO-VERSION-GEN is unable to build a
fancy version number (because it won't find any tags)
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Tue, 20 Feb 2018 22:30:28 +0000 (15:30 -0700)]
Fio 3.5
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 14 Feb 2018 19:43:04 +0000 (12:43 -0700)]
Merge branch 'wip-ifed-rados' of https://github.com/ifed01/fio
* 'wip-ifed-rados' of https://github.com/ifed01/fio:
Add support for Ceph Rados benchmarking.
Igor Fedotov [Fri, 9 Feb 2018 14:39:28 +0000 (17:39 +0300)]
Add support for Ceph Rados benchmarking.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
Jens Axboe [Mon, 12 Feb 2018 17:55:07 +0000 (10:55 -0700)]
debug: fix inverted logic in fio_did_warn()
Fixes:
5ec3bc5401f3 ("init: add global 'warned' state")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 12 Feb 2018 17:19:18 +0000 (10:19 -0700)]
iolog: convert drop warning to fio_did_warn()
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 12 Feb 2018 17:18:32 +0000 (10:18 -0700)]
io_u: convert zoned bug warning to fio_did_warn()
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 12 Feb 2018 17:14:35 +0000 (10:14 -0700)]
verify: convert verify buf too small warning to fio_did_warn()
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 12 Feb 2018 17:13:04 +0000 (10:13 -0700)]
filesetup: convert root flush warning to fio_did_warn()
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 12 Feb 2018 17:11:31 +0000 (10:11 -0700)]
init: add global 'warned' state
In various places we spew a warning if some static variable
isn't set, but this still means it can happen once per job.
Add some global state to allow to track this globally instead.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 12 Feb 2018 15:24:21 +0000 (08:24 -0700)]
Merge branch 'master' of https://github.com/brycepg/fio
* 'master' of https://github.com/brycepg/fio:
Make fiologparser_hist compatible with python3
Jens Axboe [Sat, 10 Feb 2018 21:44:49 +0000 (14:44 -0700)]
init: fixup some bad style in previous commit
Make it follow the fio coding style.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Damian Yurzola [Fri, 9 Feb 2018 01:29:38 +0000 (17:29 -0800)]
init: fix broken verify_interval
The operands seems to have been inverted which in turn
created the situation whereby the interval was always
changed to match the min_bs
Fixes: https://github.com/axboe/fio/issues/522
Signed-off-by: Damian Yurzola <damian@yurzola.net>
Jens Axboe [Thu, 8 Feb 2018 22:46:46 +0000 (15:46 -0700)]
Include 'numjobs' in global options output
Fixes: https://github.com/axboe/fio/issues/237
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 7 Feb 2018 18:30:59 +0000 (11:30 -0700)]
mmap: don't include MADV_FREE in fadvise_hint check
This really should be based on invalidate, leave it as it was
for now.
Fixes:
c712c97ab871 ("Let fadvise_hint also apply too mmap engine and madvise")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 7 Feb 2018 18:13:05 +0000 (11:13 -0700)]
Let fadvise_hint also apply too mmap engine and madvise
Fixes: https://github.com/axboe/fio/issues/528
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bryce Guinta [Tue, 6 Feb 2018 00:36:33 +0000 (17:36 -0700)]
Make fiologparser_hist compatible with python3
In python3, map functions return generators, which caused numpy to break
in this script.
Use list comprehensions instead of map (more pythonic)
They also have the added benefit of behaving the same
between python2 and python3.
Jens Axboe [Thu, 25 Jan 2018 23:04:20 +0000 (16:04 -0700)]
stat: add total fsync ios to json output
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 25 Jan 2018 17:28:23 +0000 (10:28 -0700)]
io_ddir: revert separate ddir count change
This essentially reverts the previous commit. We need more
careful changes for looping and indexing, the easiest is just
to move the count values at the end as not to skew the values
of the actual ddir entries.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 25 Jan 2018 17:16:51 +0000 (10:16 -0700)]
io_ddir: move count values out of the enum fio_ddir
They are not directions, just counts.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 25 Jan 2018 16:51:07 +0000 (09:51 -0700)]
stat: ensure that we align ts->sync_stat appropriately
clang correctly complains that we don't know if the alignment is
correct when the structure is packed, so ensure that we align it
to an 8-byte boundary.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 25 Jan 2018 16:27:14 +0000 (09:27 -0700)]
Track fsync/fdatasync/sync_file_range issue count
Also fix an issue where we didn't initialize the sync minimum
latency correctly, so it was always reported as 0.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 25 Jan 2018 03:22:50 +0000 (20:22 -0700)]
Switch last_was_sync and terminate to bool and pack better
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 24 Jan 2018 22:32:07 +0000 (15:32 -0700)]
Add suppor for logging fsync (and friends) latencies
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Tomohiro Kusumi [Mon, 15 Jan 2018 19:52:12 +0000 (21:52 +0200)]
null: drop unneeded casts from void* to non-void*
Some functions in null ioengine are used by C/C++ (e.g. null_init()
which calls malloc(3)), but C specific ones (not __cplusplus) don't
need explicit cast from void* to non-void*, and one usually doesn't.
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Tomohiro Kusumi [Mon, 15 Jan 2018 19:52:11 +0000 (21:52 +0200)]
null: make *impl_ private
This can/should be private, given struct is default public in C++.
The null ioengine only needs access to new'd NullData which
encapsulates malloc'd null_data *impl_.
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Tomohiro Kusumi [Mon, 15 Jan 2018 19:52:10 +0000 (21:52 +0200)]
null: fix compile time warning on OpenBSD
Fix the following warning on OpenBSD 6.2 (gcc 4.2.1).
This can probably be silenced with void** argument and memcpy() too,
but returning a non-void pointer is clearer IMO when the code also
targets C++.
Compiles with regular make/gmake as well as
# g++ -O2 -g -shared -rdynamic -fPIC -o cpp_null null.c -DFIO_EXTERNAL_ENGINE
--
CC engines/null.o
engines/null.c: In function 'fio_null_init':
engines/null.c:142: warning: dereferencing type-punned pointer will break strict-aliasing rules
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 12 Jan 2018 17:59:08 +0000 (10:59 -0700)]
Merge branch 'diskless_invalidate' of https://github.com/sitsofe/fio
Jens Axboe [Fri, 12 Jan 2018 15:22:41 +0000 (08:22 -0700)]
Merge branch 'fio-issue-450' of https://github.com/gvkovai/fio
Sitsofe Wheeler [Sat, 30 Dec 2017 20:58:59 +0000 (20:58 +0000)]
filesetup: skip fallback invalidation with diskless ioengines
Since commit
0bcf41cdc22dfee6b3f3b2ba9a533b4b103c70c2 ("io_u:
re-invalidate cache when looping around without file open/close") the
job:
./fio --ioengine=null --size=4k --bs=4k --rate=4k \
--time_based --runtime=2s --name=badinvalidate
repeatedly produces the following warning while running:
fio: cache invalidation of badinvalidate.0.0 failed: Bad file descriptor
Fix this by skipping cache invalidation and logging a debug message for
diskless ioengines that don't define an explicit invalidation op.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Robert Elliott [Wed, 10 Jan 2018 15:50:01 +0000 (09:50 -0600)]
ioengines: don't call munmap unless size boundary is exceeded
In ioengines using mmap() and munmap(), don't call munmap() when doing an
IO that accesses the last byte of a file. This affects the dev-dax,
libpmem, and mmap ioengines.
Example:
io 12987 DEBUG fio_libpmem_prep
io 12987 io_u->offset
7516061696 : fdd->libpmem_off 0 : io_u->buflen 131072 : fdd->libpmem_sz
7516192768
io 12987 munmap addr=0x7f5600000000 size=0x1c0000000
That accesses the last 128 KiB of a 7 GiB file, triggering an munmap
because the comparison to 7 GiB fails:
7516061696 + 131072 <
7516192768
Change < to <= so that doesn't occur unless an overflow really
happens.
In one test in linux, this increases performance from 5 GB/s to 20 GB/s
as fio stops spending time in mmap() page table mapping spinlocks and
instead spends its time in memmove(). The perf top results improve from:
36.81% [kernel] [k] queued_spin_lock_slowpath
24.26% libc-2.26.so [.] __memmove_avx_unaligned_erms
4.09% [kernel] [k] __radix_tree_lookup
3.85% [kernel] [k] find_next_iomem_res
3.70% [kernel] [k] _raw_spin_lock_irq
3.48% [kernel] [k] down_read
to:
79.76% libc-2.26.so [.] __memmove_avx_unaligned_erms
3.23% libc-2.26.so [.] vfprintf
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 10 Jan 2018 15:30:07 +0000 (08:30 -0700)]
io/examples/fio-seq-read: use direct=1
It's confusing to use direct=0 with libaio, since that isn't
queued.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 6 Jan 2018 21:47:01 +0000 (14:47 -0700)]
stat: don't add duplicate clat entries for json
For some reason the initial commit always filled all
entries with zeroes, when it should just stop printing.
I suspect this was carried over from the terse/csv output,
where we do the same. But there it's actually valid, as we
have to have a fixed number of fields. For json we both
don't need it, and it's causing us to create invalid json
when we have duplicate keys.
Fixes: https://github.com/axboe/fio/issues/511
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 5 Jan 2018 20:38:40 +0000 (13:38 -0700)]
verify: don't adjust verification length based on interval when unaligned
It's perfectly fine to have a verification length that exceeds the
default verify interval when we have unaligned block sizes and
using block size ranges, since the verify interval just defaults to
the smallest blocksize.
Fixes: https://github.com/axboe/fio/issues/509
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jeff Furlong [Wed, 3 Jan 2018 23:20:38 +0000 (23:20 +0000)]
Fix client/server "all clients" reporting
In commit
e883cb3501de0f99297282c18842d50aaa7fa531, the default significant figures was added. The default value was preserved for local jobs and when client/server was used if jobs=1, but not when client/server was used if jobs > 1. This patch restores some reporting outputs ("all clients" output for IOPS and BW) when client/server is used if > 1 jobs, for example:
OLD:
All clients: (groupid=0, jobs=2): err= 0: pid=0: Wed Jan 3 14:30:13 2018
read: IOPS=0, BW=0 (0)(0B/5001msec) <--ZERO VALUES
slat (nsec): min=762, max=108579, avg=1058.97, stdev=221.30
clat (nsec): min=254, max=4544.2k, avg=11297.64, stdev=8284.14
lat (usec): min=9, max=4545, avg=12.41, stdev= 8.30
bw ( KiB/s): min=290192, max=347160, per=49.35%, avg=303613.33, stdev=11582.87, samples=18
iops : min=72548, max=86790, avg=75903.33, stdev=2895.72, samples=18
NEW:
All clients: (groupid=0, jobs=2): err= 0: pid=0: Wed Jan 3 15:08:33 2018
read: IOPS=147k, BW=575Mi (603M)(2876MiB/5001msec) <--CORRECT VALUES
slat (nsec): min=841, max=61945, avg=1304.83, stdev=232.75
clat (nsec): min=544, max=4487.2k, avg=11736.68, stdev=9123.64
lat (usec): min=10, max=4489, avg=13.10, stdev= 9.13
bw ( KiB/s): min=260945, max=305976, per=49.99%, avg=294350.22, stdev=9192.53, samples=18
iops : min=65236, max=76494, avg=73587.56, stdev=2298.13, samples=18
Regards,
Jeff
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 5 Jan 2018 15:30:32 +0000 (08:30 -0700)]
Change bluestop link to be https
Doesn't answer on http:// anymore.
Also include link to AppVeyor builds for Windows, as they always build
the latest and greatest.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 2 Jan 2018 16:05:44 +0000 (09:05 -0700)]
Merge branch 'percentiles' of https://github.com/sitsofe/fio
Sitsofe Wheeler [Sun, 31 Dec 2017 12:42:59 +0000 (12:42 +0000)]
init: disable percentiles when latency gathering is disabled
When disable_clat/disable_lat is set also disable the corresponding
clat_percentiles/lat_percentiles option.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 31 Dec 2017 11:41:55 +0000 (11:41 +0000)]
stat: make lat_percentiles=1 use sample count from lat_stat
The following job
./fio --name=nolatencies --ioengine=posixaio --direct=1 --rw=randread \
--iodepth=1 --bs=4k --filename=/tmp/fio.tmp --runtime=2s --size=50m \
--disable_lat=0 --disable_clat=1 --disable_slat=1 --lat_percentiles=1
outputs the following:
fio-3.3
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=7316: Sun Dec 31 11:55:46 2017
read: IOPS=41.0k, BW=164MiB/s (172MB/s)(50.0MiB/305msec)
lat (usec): min=8, max=132, avg=17.58, stdev= 4.36
lat percentiles (nsec):
| 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0],
| 30.00th=[ 0], 40.00th=[ 0], 50.00th=[ 0], 60.00th=[ 0],
| 70.00th=[ 0], 80.00th=[ 0], 90.00th=[ 0], 95.00th=[ 0],
| 99.00th=[ 0], 99.50th=[ 0], 99.90th=[ 0], 99.95th=[ 0],
| 99.99th=[ 0]
cpu : usr=56.39%, sys=29.18%, ctx=12043, majf=0, minf=85
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=12800,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=164MiB/s (172MB/s), 164MiB/s-164MiB/s (172MB/s-172MB/s), io=50.0MiB (52.4MB), run=305-305msec
which isn't showing total latency percentiles (lat percentiles) when it
should. Additionally there is a segfault when running the following job:
./fio --name=segfault --ioengine=posixaio --direct=1 --rw=randread \
--iodepth=1 --bs=4k --filename=/tmp/fio.tmp --runtime=2s --size=50m \
--disable_lat=1 --disable_clat=0 --disable_slat=1 --lat_percentiles=1
Fix these by making latency printing use the number of samples from the
correct latency type (clat_stat for lat_percentiles=0/clat_percentiles=1
and lat_stat for lat_percentiles=1/clat_percentiles=0).
Fixes https://www.spinics.net/lists/fio/msg06628.html ("fio lat reporting
").
Reported-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Fri, 29 Dec 2017 15:45:22 +0000 (08:45 -0700)]
log: fix bad < 0 check for unsigned
Fixes:
e5f9a813 ("debug: make debug=io readable with multiple threads")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Robert Elliott [Fri, 29 Dec 2017 01:13:06 +0000 (01:13 +0000)]
debug: make debug=io readable with multiple threads
When multiple threads are active, debug=io prints are unreadable as the output
is interleaved. Multiple log_info_buf() calls are used to construct each line,
but there is no mutual exclusion covering multiple calls.
Change the dprint call tree to construct the entire line before passing it to
log_info_buf(), rather than make several calls.
Other nits:
* print the thread ID rather than the process ID
* change offset and length from decimal to hex
* separate offset, length, ddir, and file with , rather than / since the
filename on the same line likely has / of its own
* change "fill_io_u" to "fill" to match the others
* change "io complete" to "complete" to match the others
* change "->prep()=%d" to "prep: io_u %p: ret=%d" to resemble the others
* change offset/buflen in an error print to better resemble the normal prints
* add "file=" prefix for the filename
* check the calloc() return values inside the valist_to_buf functions
Old:
fill_io_u: io_u 0x7feeac010b80: off=720896/len=65536/ddir=1
io 50692io 50692//dev/dax0.0io 50692fill_io_u: io_u 0x7fee98010b80: off=196608/len=65536/ddir=1io 50692io 50692
io 50692->prep(0x7fef10010b80)=0
//dev/dax0.0//dev/dax1.0io 50692io 50692io 50692io 50692->prep(0x7feeec010b80)=0
io 50692prep: io_u 0x7feec4010b80: off=
1966080/len=65536/ddir=1io 50692io complete: io_u 0x7feedc010b80: off=393216/len=65536/ddir=1io
50692io 50692prep: io_u 0x7feef4010b80: off=720896/len=65536/ddir=1io 50692io 50692
io 50692io 50692prep: io_u 0x7feef0010b80: off=851968/len=65536/ddir=1//dev/dax0.0io 50692//dev/dax0.0
//dev/dax0.0
io 50692io 50692//dev/dax0.0//dev/dax0.0io 50692
New:
io 71400 queue: io_u 0x7fd0f0010b80: off=0x2f0000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71395 fill: io_u 0x7fd0fc010b80: off=0x80000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71395 prep: io_u 0x7fd0fc010b80: off=0x80000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71395 prep: io_u 0x7fd0fc010b80: ret=0
io 71395 queue: io_u 0x7fd0fc010b80: off=0x80000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71430 complete: io_u 0x7fd05c010b80: off=0x180000,len=0x10000,ddir=1,file=/dev/dax1.0
io 71400 complete: io_u 0x7fd0f0010b80: off=0x2f0000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71430 fill: io_u 0x7fd05c010b80: off=0x190000,len=0x10000,ddir=1,file=/dev/dax1.0
io 71400 fill: io_u 0x7fd0f0010b80: off=0x300000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71400 prep: io_u 0x7fd0f0010b80: off=0x300000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71400 prep: io_u 0x7fd0f0010b80: ret=0
io 71400 queue: io_u 0x7fd0f0010b80: off=0x300000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71430 prep: io_u 0x7fd05c010b80: off=0x190000,len=0x10000,ddir=1,file=/dev/dax1.0
io 71395 complete: io_u 0x7fd0fc010b80: off=0x80000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71430 prep: io_u 0x7fd05c010b80: ret=0
io 71430 queue: io_u 0x7fd05c010b80: off=0x190000,len=0x10000,ddir=1,file=/dev/dax1.0
io 71421 complete: io_u 0x7fd090010b80: off=0x320000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71419 complete: io_u 0x7fd098010b80: off=0x320000,len=0x10000,ddir=1,file=/dev/dax0.0
io 71404 complete: io_u 0x7fd0d0010b80: off=0x0,len=0x10000,ddir=1,file=/dev/dax0.0
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Tomohiro Kusumi [Fri, 29 Dec 2017 13:38:23 +0000 (15:38 +0200)]
lib/memcpy: fix warning on FreeBSD
Include struct timeval via sys/time.h.
http://pubs.opengroup.org/onlinepubs/
009604499/basedefs/sys/time.h.html
--
In file included from lib/memcpy.c:7:
lib/../fio_time.h:14:45: warning: declaration of 'struct timeval' will not be visible outside of this function [-Wvisibility]
extern uint64_t mtime_since_tv(const struct timeval *, const struct timeval *);
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 28 Dec 2017 15:35:11 +0000 (08:35 -0700)]
Merge branch 'master' of https://github.com/yashi/fio
Yasushi SHOJI [Thu, 28 Dec 2017 01:53:47 +0000 (10:53 +0900)]
mutex: down_timeout: check against the base time
When the commit
8b6a404cdd2c40715885e562416c3db039912773 changed
timeval to timespec, it accsidentally picked timeout time `t' instead
of the current `base' time for checking timewait() really timed out.
t is base + msecs as follows:
base = gettimeofday()
t = base + msecs
pthread_cond_timedwait()
while (!done)
if (timedout)
if (mutex_timed_out(t, msecs))
done = true
Thus, after a good timedout, it burns the cpu for the given msecs
seconds doing tight loop in the while shown above. This causes 30 to
40 % CPU load on my i5-3360M 2.80GHz while fio server is running.
Jens Axboe [Wed, 27 Dec 2017 21:05:46 +0000 (14:05 -0700)]
Merge branch 'eta_display' of https://github.com/sitsofe/fio
Jens Axboe [Wed, 27 Dec 2017 21:05:27 +0000 (14:05 -0700)]
Merge branch 'barak/mmap_verify_only' of https://github.com/barakp/fio