Jens Axboe [Sat, 4 Jan 2020 00:15:31 +0000 (17:15 -0700)]
Merge branch 'master' of https://github.com/kenbarr1/fio
* 'master' of https://github.com/kenbarr1/fio:
io_u: fix rate limiting to handle file wrap-around
Ken Barr [Fri, 3 Jan 2020 20:11:44 +0000 (15:11 -0500)]
io_u: fix rate limiting to handle file wrap-around
When using "rate" with "time_based", the amount of time to sleep should be based
on the invocation time (epoch) rather than the start time of a given loop
(start) because no other statistics are reset when a file is re-opened.
Prior to this change, a wrap-around would cause the rate_ddir code to compute
that ~0 time had elapsed since the last I/O (i.e., a near-infinite rate), and
trigger a very long sleep in an attempt to adjust to the given rate.
Testing done:
Using configuration below...
Before: starts at 100 MB/sec and drops to 0 after ~10 sec (upon wrap) before
recovering at t ~ 20.
After: maintains 100 MB/sec for the duration of the test
[repro]
# 1 GiB + 1 Byte, to force a fill_io_u failure, the file to close and
# reopen, and the start timestamp to reset.
size=
1073741825
bs=
2097152
direct=1
ioengine=psync
readwrite=write
runtime=30s
time_based
rate=100m
filename=data_file
Jens Axboe [Fri, 3 Jan 2020 19:32:07 +0000 (12:32 -0700)]
Revert "Fix a potential deadlock in helper_do_stat()"
This reverts commit
31eca641ad91634e5ffcf369cd756b0506a700c1.
Killing this for now as it breaks the Windows build.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 3 Jan 2020 19:20:41 +0000 (12:20 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Fix a potential deadlock in helper_do_stat()
t/read-to-pipe-async: Use the monotonic clock when measuring time intervals
Use CLOCK_MONOTONIC for condition variables used by pthread_cond_timedwait()
Makefile: Build more test code
Suppress a Coverity taint warning in check_status_file()
Micro-optimize __load_ioengine()
Jens Axboe [Fri, 3 Jan 2020 19:19:10 +0000 (12:19 -0700)]
Merge branch 'unit1' of https://github.com/kusumi/fio
* 'unit1' of https://github.com/kusumi/fio:
unittests: add unittest suite for oslib/strsep.c
unittests: add unittest suite for oslib/strcasestr.c
Bart Van Assche [Tue, 31 Dec 2019 17:37:55 +0000 (09:37 -0800)]
Fix a potential deadlock in helper_do_stat()
pthread_mutex_lock(), pthread_cond_signal() and pthread_mutex_unlock() are
not async-signal-safe and hence must not be used inside a singal handler
implementation. Rework the code for communication with the helper thread
such that it becomes async-signal-safe.
Fixes:
a47591e4923f ("Improve logging accuracy")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Wed, 1 Jan 2020 23:01:53 +0000 (15:01 -0800)]
t/read-to-pipe-async: Use the monotonic clock when measuring time intervals
Make the output independent of wall clock changes applied by e.g. ntpd.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Wed, 1 Jan 2020 00:23:32 +0000 (16:23 -0800)]
Use CLOCK_MONOTONIC for condition variables used by pthread_cond_timedwait()
This patch ensures that the time during which pthread_cond_timedwait()
waits is not affected by wall clock adjustments by e.g. ntpd.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Wed, 1 Jan 2020 22:23:53 +0000 (14:23 -0800)]
Makefile: Build more test code
Build the t/io_uring, t/memlock and t/read-to-pipe-async executables if
'make all' is executed. Do not include t/time-test because it uses a
non-portable 128-bit integer type.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Tue, 31 Dec 2019 17:35:10 +0000 (09:35 -0800)]
Suppress a Coverity taint warning in check_status_file()
Suppress the following Coverity complaint:
CID 24143 (#1 of 1): Use of untrusted string value (TAINTED_STRING)
tainted_string: Passing tainted string fio_status_file_path to unlink, which
cannot accept tainted data.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Wed, 1 Jan 2020 00:31:26 +0000 (16:31 -0800)]
Micro-optimize __load_ioengine()
Instead of copying the I/O engine name, use a pointer to the I/O engine
name. This patch suppresses the following false positive Valgrind complaint:
Conditional jump or move depends on uninitialised value(s)
at 0x41D451: __load_ioengine (ioengines.c:133)
by 0x41D451: load_ioengine (ioengines.c:161)
by 0x41FA59: ioengine_load (init.c:1126)
by 0x423B64: parse_cmd_line (init.c:2670)
by 0x4241A3: parse_options (init.c:2965)
by 0x40F587: main (fio.c:42)
The code for copying I/O engine names was introduced by commit
2866c82d598e
("[PATCH] Separate io engines into separate loadable objects").
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Zhenyu Ye [Wed, 25 Dec 2019 15:02:57 +0000 (08:02 -0700)]
io_uring: Enable io_uring ioengine on aarch64 arch
This patch enables io_uring ioengine on aarch64 arch.
Signed-off-by: Zhenyu Ye <yezhenyu2@huawei.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 23 Dec 2019 15:54:25 +0000 (08:54 -0700)]
io_uring: add option for non-vectored read/write commands
Also syncs io_uring.h with the kernel 5.6 version.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Tomohiro Kusumi [Sun, 22 Dec 2019 05:26:25 +0000 (14:26 +0900)]
unittests: add unittest suite for oslib/strsep.c
See
b8b0e1eea7780a02ff67f0caeba446cc403f1b37
("unittests: add CUnit based unittest framework") for usage.
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Tomohiro Kusumi [Sat, 21 Dec 2019 16:46:28 +0000 (01:46 +0900)]
unittests: add unittest suite for oslib/strcasestr.c
See
b8b0e1eea7780a02ff67f0caeba446cc403f1b37
("unittests: add CUnit based unittest framework") for usage.
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Jens Axboe [Sat, 21 Dec 2019 14:04:58 +0000 (07:04 -0700)]
Merge branch 'doc_fixes' of https://github.com/sitsofe/fio
* 'doc_fixes' of https://github.com/sitsofe/fio:
doc: fix up sphinx warnings
HOWTO: fix up broken formatting in registerfiles section
doc: stop saying backslashes need escaping
Jens Axboe [Sat, 21 Dec 2019 14:04:34 +0000 (07:04 -0700)]
Merge branch 'travis' of https://github.com/sitsofe/fio
* 'travis' of https://github.com/sitsofe/fio:
travis: switch to ubuntu 18.04 and install more libraries
travis: remove duplicate xcode image and add comments
Sitsofe Wheeler [Sat, 21 Dec 2019 09:53:19 +0000 (09:53 +0000)]
doc: fix up sphinx warnings
- Reference to merge-blktrace-only needs the leading dashes to work
(seemingly because it takes no parameters?)
- Nothing seems to be html_static_path, we don't have an _static dir and
it generates a warning "html_static_path entry '_static' does not
exist" so disable it for now
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sat, 21 Dec 2019 09:17:50 +0000 (09:17 +0000)]
HOWTO: fix up broken formatting in registerfiles section
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Wed, 30 Oct 2019 06:14:12 +0000 (06:14 +0000)]
doc: stop saying backslashes need escaping
In
02dd2689885c6d64d9d01a92e67d40f73c0afc72 ("HOWTO: update directory
and filename option descriptions") I added a change that stated that
backslashes needed escaping too but this is not (and never has been) the
case. Fix this thinko by removing all mentions of backslash escaping
being needed from the documentation.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sat, 5 Oct 2019 14:18:38 +0000 (15:18 +0100)]
travis: switch to ubuntu 18.04 and install more libraries
Also sort the package dependencies into alphabetical order to make it
easier to see what has changed in the future.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sat, 5 Oct 2019 13:36:48 +0000 (14:36 +0100)]
travis: remove duplicate xcode image and add comments
We already build on whatever the default Travis xcode image is
which is currently 9.4 so remove the duplicate build targeting 9.4 and
add some comments explaining the purpose of the different builds.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Thu, 19 Dec 2019 12:38:30 +0000 (05:38 -0700)]
Merge branch 'mine/patch-2' of https://github.com/hannesweisbach/fio
* 'mine/patch-2' of https://github.com/hannesweisbach/fio:
Add example job file for exit_what
Expand choices for exitall
Hannes Weisbach [Thu, 19 Dec 2019 09:33:24 +0000 (10:33 +0100)]
Add example job file for exit_what
Hannes Weisbach [Wed, 27 Nov 2019 09:57:49 +0000 (10:57 +0100)]
Expand choices for exitall
Add exit_what thread_option with the following choices:
- "group" (default) exit all jobs of the same group (equal to "exitall=1")
- "stonewall", quit all jobs until the next stonewall
- "all", quit *all* jobs
Signed-off-by: Hannes Weisbach <hannes.weisbach@gmail.com>
Jens Axboe [Thu, 19 Dec 2019 03:13:58 +0000 (20:13 -0700)]
Merge branch 'windows_mkdir' of https://github.com/sitsofe/fio
* 'windows_mkdir' of https://github.com/sitsofe/fio:
filesetup: fix directory creation issues
Sitsofe Wheeler [Sun, 8 Dec 2019 18:03:57 +0000 (18:03 +0000)]
filesetup: fix directory creation issues
- Fix double directory prefix concatenation that was happening after
95af8dd57a7a9a564f99492bb1e04bb6d13b95b1 ("filesetup: don't call
create_work_dirs() for ioengine with FIO_DISKLESSIO") refactored some
code
- Stop always using '/' and switch to using FIO_OS_PATH_SEPARATOR when
concatenating the directory value to the filename in set_name_idx()
- Change the permissions directories are created with to have the
execute bit so they can be entered(!)
- Fix bug where filenames that were absolute unix paths would prevent
the creation of intermediate directories
- Remove the configure mkdir test and introduce fio_mkdir to hide mkdir
platform differences
- Make Windows mkdir emulation more robust against multiple slashes
after the drive letter and when dealing dealing with with paths
containing drive letters in general
This commit makes the following work on Windows:
./fio --thread --name=doubleslashtest \
--filename='C\:\\Windows\Temp\fio.tmp' --size=4k --bs=4k --rw=read
The next example now works when D:\1\ exists but D:\1\2\ doesn't:
./fio --thread --name=mkdirtest --directory='D\:\1' \
--filename='2\fio.tmp' --size=4k --bs=4k --rw=read
The following now works on macOS/Linux when /tmp/1/ exists but /tmp/1/2/
doesn't:
./fio --name=mkdirtest --directory='/tmp/1' \
--filename='2/fio.tmp' --size=4k --bs=4k --rw=read
Fixes https://github.com/axboe/fio/issues/784 ,
https://github.com/axboe/fio/issues/864 and
https://github.com/axboe/fio/issues/870
Tested-by: Astolfo Rueda <astolfor@microsoft.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Wed, 18 Dec 2019 16:53:48 +0000 (09:53 -0700)]
Merge branch 'issue-878' of https://github.com/vincentkfu/fio
* 'issue-878' of https://github.com/vincentkfu/fio:
client/server: add missing fsync data structures
Vincent Fu [Mon, 16 Dec 2019 15:15:11 +0000 (10:15 -0500)]
client/server: add missing fsync data structures
In client/server mode, fsync latencies were missing because a couple
data structures were missed in the client/server send/receive thread
stats code. This patch adds those data structures and now we are able to
see the fsync latencies in client/server mode.
Fixes: https://github.com/axboe/fio/issues/878
Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
Jens Axboe [Mon, 16 Dec 2019 23:14:40 +0000 (16:14 -0700)]
Merge branch 'cygwin-build-error' of https://github.com/vincentkfu/fio
* 'cygwin-build-error' of https://github.com/vincentkfu/fio:
Makefile: add libssp for Windows
Jens Axboe [Mon, 16 Dec 2019 23:08:35 +0000 (16:08 -0700)]
Merge branch 'travis-xcode11.2-python' of https://github.com/vincentkfu/fio
* 'travis-xcode11.2-python' of https://github.com/vincentkfu/fio:
.travis.yml: xcode11.2 scipy issue
Jens Axboe [Mon, 16 Dec 2019 22:48:43 +0000 (15:48 -0700)]
Fio 3.17
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Thu, 12 Dec 2019 22:22:06 +0000 (15:22 -0700)]
Makefile: add libssp for Windows
Recent Cygwin mingw releases require linking against libssp to avoid
undefined symbols.
See https://github.com/msys2/MINGW-packages/issues/5868 for discussion.
https://github.com/msys2/MINGW-packages/issues/5803 also contains
discussion of possibly disabling the FORTIFY_SOURCE linker flag.
Fixes: https://github.com/axboe/fio/issues/881
Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
Vincent Fu [Fri, 13 Dec 2019 19:58:23 +0000 (12:58 -0700)]
.travis.yml: xcode11.2 scipy issue
The previous commit that added support for xcode11.2 resolved the
steadystate_tests scipy dependency with 'pip3 install scipy'. However,
the travis xcode11.2 image changed and now we need 'pip install scipy'
to provide the scipy dependency. To forestall future issues, install
scipy for both python2 and python3.
Also enable debug output in run-fio-tests.py to provide more information
in case of failure.
Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
Jens Axboe [Thu, 12 Dec 2019 19:34:50 +0000 (12:34 -0700)]
t/io_uring: check for CONFIG_HAVE_GETTID
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 10 Dec 2019 17:54:42 +0000 (12:54 -0500)]
t/run-fio-tests: relax acceptance criterion for t0011
I saw a handful of benign failures for t0011 when run on
travis/appveyor. This patch allows the test to still pass with a
measured IOPS of 998 to 1002 and adds a debug print for the value
actually observed.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 10 Dec 2019 17:54:41 +0000 (12:54 -0500)]
.appveyor.yml: run run-fio-tests.py
- Add CUnit support
- install SciPy
- skip test 5 because Windows does not support direct I/O with sync
ioengines
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 10 Dec 2019 17:54:40 +0000 (12:54 -0500)]
.travis.yml: run t/run-fio.tests.py as part of build
- install SciPy and CUnit support
- skip test 6 because of timeout
- skip zbd tests 1007, 1008 because installing kernel modules is not supported
- replace xcode8.3 (lacks python3) with latest xcode11.2
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 10 Dec 2019 17:54:39 +0000 (12:54 -0500)]
t/steadystate_tests: use null ioengine for tests
Instead of reading from /dev/zero just use the null ioengine. This
enables the script to run on Windows.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 10 Dec 2019 17:54:38 +0000 (12:54 -0500)]
t/run-fio-tests: identify test id for debug messages
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 10 Dec 2019 17:54:37 +0000 (12:54 -0500)]
t/run-fio-tests: improve Windows support
- add .exe extension for fio and unittest
- for python scripts use 'python.exe script.py' instead of ./script.py
- make JSON decoding more resilient by lopping off up to four of the
first lines when encountering decoding errors. This helps Windows
because for some jobs fio prints an informational message about
requiring threads
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 10 Dec 2019 17:54:36 +0000 (12:54 -0500)]
t/run-fio-tests: detect requirements and skip tests accordingly
Also provide an option to skip requirement checking
Suggested-by: <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 10 Dec 2019 17:54:35 +0000 (12:54 -0500)]
t/run-fio-tests: a few small improvements
- add debug option that changes logging level
- improve error handling
- more output on test failures
- Python 3.5 compatibility with pathlib.Path and os.path.join
- warn if fio executable not found but don't abort because some tests
can run without the fio executable
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 10 Dec 2019 17:54:34 +0000 (12:54 -0500)]
.gitignore: ignore zbd test output files
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 7 Dec 2019 05:03:04 +0000 (22:03 -0700)]
filesetup: ensure to setup random generator properly
If we have ZONED or ZONED_ABS set, we need to go through the same
path as RANDOM. If not, we fail miserably later.
Fixes: https://github.com/axboe/fio/issues/873
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 6 Dec 2019 19:36:53 +0000 (12:36 -0700)]
Renumber RWF_UNCACHED
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 6 Dec 2019 16:08:29 +0000 (09:08 -0700)]
pvsync2: add support for RWF_UNCACHED
See commit
4a87b5840d35 ("io_uring: add support for RWF_UNCACHED")
for details on this feature.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 6 Dec 2019 15:35:28 +0000 (08:35 -0700)]
io_uring: add support for RWF_UNCACHED
If this is set, and the kernel supports it, buffered IO will be
uncached. This means that reads are dropped from the cache if we
are the ones instantiating the pages, and writes are sync and
dropped from the page cache on IO completion.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 27 Nov 2019 17:23:12 +0000 (10:23 -0700)]
Merge branch 'engine-rados-fix-first' of https://github.com/aclamk/fio
* 'engine-rados-fix-first' of https://github.com/aclamk/fio:
engines/rados: fix error with getting last instead of first element from list
Adam Kupczyk [Wed, 27 Nov 2019 15:18:32 +0000 (10:18 -0500)]
engines/rados: fix error with getting last instead of first element from list
A thinko has popped in implementation. Consumer function `getevent()` did pick last (newest) instead of first (oldest).
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Jens Axboe [Tue, 26 Nov 2019 18:57:52 +0000 (11:57 -0700)]
Merge branch 'fallocate-truncate' of https://github.com/tripped/fio
* 'fallocate-truncate' of https://github.com/tripped/fio:
filesetup: add fallocate=truncate option.
Jens Axboe [Tue, 26 Nov 2019 04:40:38 +0000 (21:40 -0700)]
Merge branch 'rados-now-use-completion-callbacks' of https://github.com/aclamk/fio
* 'rados-now-use-completion-callbacks' of https://github.com/aclamk/fio:
engines/rados: changed polling to completion callbacks methodology
Jens Axboe [Tue, 26 Nov 2019 04:40:19 +0000 (21:40 -0700)]
Merge branch 'const1' of https://github.com/kusumi/fio
* 'const1' of https://github.com/kusumi/fio:
parse: Silence discard-const warning on OpenBSD
Adam Kupczyk [Mon, 25 Nov 2019 17:55:07 +0000 (12:55 -0500)]
engines/rados: changed polling to completion callbacks methodology
Previously, getevent() function polled operations that have been submitted to check if some of them have already finished.
Now, each rados operation invokes completion callback. This callback adds to list of completed operations.
In this version getevent() only peeks if completed list has some operations.
There are two benefits:
1) small one - this works faster then previous version
2) big one - when there is a huge amount of operations in fly, getevent() might be more overloaded then producer. In this scenario some old operations are finished, but not picked up and properly noticed. This caused absurdely long execution times, when actually everything was working properly.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Tomohiro Kusumi [Mon, 25 Nov 2019 15:13:48 +0000 (00:13 +0900)]
parse: Silence discard-const warning on OpenBSD
This might be overkill just to silence the warnings, but having
a common function for const and non-const version makes it discard
const pointer at some point.
--
CC parse.o
parse.c: In function 'find_option_c':
parse.c:1051: warning: passing argument 1 of 'find_option' discards qualifiers from pointer target type
parse.c: In function 'get_option':
parse.c:1051: warning: passing argument 1 of 'find_option' discards qualifiers from pointer target type
parse.c:1051: warning: passing argument 1 of 'find_option' discards qualifiers from pointer target type
parse.c: In function 'parse_cmd_option':
parse.c:1051: warning: passing argument 1 of 'find_option' discards qualifiers from pointer target type
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Vincent Fu [Mon, 11 Nov 2019 16:30:55 +0000 (11:30 -0500)]
t/run-fio-tests: improve error handling
More gracefully handle errors when running a test. Also change the
EXAMPLE to refer to fio's canonical git repository.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Mon, 11 Nov 2019 16:30:54 +0000 (11:30 -0500)]
io_u: move to next zone even if zoneskip is unset
This patch makes fio's behavior under zonemode=strided conform to the
documentation:
I/O happens in a single zone until zonesize bytes have been
transferred. After that number of bytes has been transferred
processing of the next zone starts.
With zonemode=strided, before commit
35f561eb, fio would only move to
the next zone when zoneskip > 0 and zonesize bytes were written. There
would always be zoneskip bytes between the end of one zone and the
beginning of the next zone. If zoneskip was not set or set to 0, all IO
would happen in the first zone.
Commit
35f561eb changed this so that fio would move to the next zone
upon writing zonesize bytes if zoneskip was explicitly set to a value >=
0. This option made it possible for zones to be contiguous. The
documentation was not updated to reflect the new behavior.
I originally intended to submit a patch to update fio's documentation,
but upon further reflection it seems better to change fio's behavior and
have a clean user interface than to change the documentation to note
that zoneskip must be explciitly set in order for fio to move to the
next zone.
This patch also updates t/strided.py to reflect the new behavior.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Mon, 11 Nov 2019 16:30:53 +0000 (11:30 -0500)]
filesetup: improve LFSR init failure error message
Especially with small sample spaces, the LFSR random generator
occasionally fails to initialize successfully. When this occurs, the
error message refers to problems allocating a random map. Change the
error message to explicitly mention the LFSR failure.
OLD BEHAVIOR
$ ./fio --name=test --ioengine=null --size=4k --random_generator=lfsr --rw=randread --randrepeat=0
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.16-13-g998b-dirty
Starting 1 process
fio: failed allocating random map. If running a large number of jobs, try the 'norandommap' option or set 'softrandommap'. Or give a larger --alloc-size to fio.
NEW BEHAVIOR
$ ./fio --name=test --ioengine=null --size=4k --random_generator=lfsr --rw=randread --randrepeat=0
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.16-37-g65ee
Starting 1 process
fio: failed initializing LFSR
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Trip Volpe [Thu, 7 Nov 2019 23:13:48 +0000 (15:13 -0800)]
filesetup: add fallocate=truncate option.
Fixes #833. Provides the ability to initially "layout" the file
by ftruncating it to the desired size before performing IO. This
is mainly useful on Windows, which serializes all writes that
extend the size of a file. Using this option with a suitable
iodepth allows fio to emulate the behavior of Windows Explorer
file copy, which always truncates to the expected size before
issuing writes for performance reasons.
Jens Axboe [Thu, 7 Nov 2019 00:04:17 +0000 (17:04 -0700)]
Merge branch 'testing' of https://github.com/vincentkfu/fio
* 'testing' of https://github.com/vincentkfu/fio:
t/run-fio-tests: a script to automate running fio tests
t/jobs: fix t0011 syntax error
t/jobs: clean up t0009 and use only 4 CPUs
t/jobs: drop time_based in t0007
t/jobs: use current directory for test file for t0003 and t0004
t/jobs: fixup t0002 test jobs
t/stest: non-zero exit value on failure
t/sgunmap-test.py: drop six.moves dependency
t/steadystate_tests: better support automated testing
t/strided.py: change LFSR tests
t/readonly: replace shell script with python script
t/iee754: add return value
Vincent Fu [Fri, 18 Oct 2019 19:05:35 +0000 (15:05 -0400)]
t/run-fio-tests: a script to automate running fio tests
fio actually has a substantial test infrastructure but lacks a
convenient way to run existing tests in an automated fashion. This
script provides the means to run almost all of fio's existing tests with
a single command. A full run finishes in less than 10 minutes.
This script can be used to catch bugs before patches are submitted. It
can also be used to test new fio builds as patches are committed to the
repository.
Vincent Fu [Thu, 17 Oct 2019 15:53:04 +0000 (11:53 -0400)]
t/jobs: fix t0011 syntax error
Vincent Fu [Thu, 17 Oct 2019 18:38:24 +0000 (14:38 -0400)]
t/jobs: clean up t0009 and use only 4 CPUs
There are two runtime= lines in t0009. Comment out the one that is
ignored.
Also reduce the number of CPUs required from 6 to 4 so that this can be
run on more systems. The failure still occurs on systems with four CPUs.
Vincent Fu [Thu, 17 Oct 2019 15:52:27 +0000 (11:52 -0400)]
t/jobs: drop time_based in t0007
This is an extraneous option that produces a warning message which
complicates using this job file for automated testing.
Vincent Fu [Thu, 17 Oct 2019 15:54:03 +0000 (11:54 -0400)]
t/jobs: use current directory for test file for t0003 and t0004
This makes it easier to investigate problems when these job files are
used for automated testing.
Vincent Fu [Wed, 6 Nov 2019 16:22:10 +0000 (11:22 -0500)]
t/jobs: fixup t0002 test jobs
- add .fio extension to match other job files
- change line endings from DOS to UNIX
- use local file instead of device
Vincent Fu [Wed, 16 Oct 2019 19:03:32 +0000 (15:03 -0400)]
t/stest: non-zero exit value on failure
To facilitate automated testing, return a non-zero exit value when a
smalloc failure occurs.
Vincent Fu [Wed, 30 Oct 2019 23:55:21 +0000 (19:55 -0400)]
t/sgunmap-test.py: drop six.moves dependency
range was never actually used in this script.
Vincent Fu [Tue, 29 Oct 2019 10:09:54 +0000 (06:09 -0400)]
t/steadystate_tests: better support automated testing
To better support using this script for automated testing, do the
following:
- Use a more portable python interpreter reference
- Drop six.moves dependency
- Eliminate two potential divide-by-zero runtime errors
- Count and report the number of passed and failed tests
- Keep the output of the test jobs for inspection in case of failure
- Exit code 0 if all tests pass and non-zero otherwise
Vincent Fu [Mon, 28 Oct 2019 23:45:31 +0000 (19:45 -0400)]
t/strided.py: change LFSR tests
LFSR init fails on occasion with small zoneranges. This patch increases
the zonerange for LFSR tests to eliminate these failures.
Also fix a typo in the last test job.
Vincent Fu [Wed, 16 Oct 2019 18:52:07 +0000 (14:52 -0400)]
t/readonly: replace shell script with python script
The new script better supports automated testing than the previous one.
Also get rid of the t/jobs/readonly-?.fio job files. Using jobs files
with the --readonly parameter produces the same behavior as when
--readonly precedes --rw=XXX on the command line.
Vincent Fu [Wed, 16 Oct 2019 16:32:42 +0000 (12:32 -0400)]
t/iee754: add return value
To facilitate automated testing, return 0 when all tests succeed and a
non-zero value otherwise.
Also add one more test value.
Khem Raj [Tue, 5 Nov 2019 13:41:55 +0000 (06:41 -0700)]
arch-arm: Consider armv7ve arch as well
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 3 Nov 2019 13:35:44 +0000 (06:35 -0700)]
Merge branch 'master' of https://github.com/aphreet/fio
* 'master' of https://github.com/aphreet/fio:
Enable io_uring engine on powerpc arch
Mikhail Malygin [Sun, 3 Nov 2019 09:53:12 +0000 (09:53 +0000)]
Enable io_uring engine on powerpc arch
Signed-off-by: Mikhail Malygin <mikhail@malygin.me>
Jens Axboe [Sat, 2 Nov 2019 20:48:24 +0000 (14:48 -0600)]
engines/libaio.c: remove unused 'hipri' setting
Leftover from when we had aio-ring with polling, now dead.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 29 Oct 2019 19:39:14 +0000 (13:39 -0600)]
Merge branch 'patch-1' of https://github.com/hannesweisbach/fio
* 'patch-1' of https://github.com/hannesweisbach/fio:
Fix output redirection of exec_prerun/_postrun
Jens Axboe [Tue, 29 Oct 2019 16:05:34 +0000 (10:05 -0600)]
Merge branch '1029_tet' of https://github.com/optimistyzy/fio
* '1029_tet' of https://github.com/optimistyzy/fio:
backend: fix the memory leak if fio_memalign fails,
Ziye Yang [Tue, 29 Oct 2019 14:35:38 +0000 (22:35 +0800)]
backend: fix the memory leak if fio_memalign fails,
We should not use break, we should use return 1 and explictly fails this,
then the cleanup routine will work. If not, this function may still
return 0.
Signed-off-by: Gang Cao <cao.gang@intel.com>
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
hannesweisbach [Tue, 29 Oct 2019 07:35:24 +0000 (08:35 +0100)]
Fix output redirection of exec_prerun/_postrun
The previous version used the bashism "&>" to redirect stdout and stderr of the prerun/postrun commands to an output file. Since system() executes the command using "/bin/sh sh -c …". If /bin/sh is not bash, this wont work.
Use the "posixly-correct" way of redirecting stdout to the output file and then redirect stderr to stdout.
Signed-off-by: Hannes Weisbach <hannes.weisbach@gmail.com>
Jens Axboe [Thu, 24 Oct 2019 18:04:09 +0000 (12:04 -0600)]
Merge branch 'doc_fixes' of https://github.com/sitsofe/fio
* 'doc_fixes' of https://github.com/sitsofe/fio:
doc: delete repeated word "will"
man: don't use non-breaking minuses when they're not necessary
Sitsofe Wheeler [Mon, 24 Sep 2018 22:51:56 +0000 (23:51 +0100)]
doc: delete repeated word "will"
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 12 Aug 2018 20:00:57 +0000 (21:00 +0100)]
man: don't use non-breaking minuses when they're not necessary
Technically you only need to escape a dash in troff when you
1. Want it to be a typographical minus
2. Need to prevent the typesetter potentially doing a line break at it
(e.g. to prevent a -option being split apart)
so convert lots of "\-" to just "-" .
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Mon, 21 Oct 2019 14:46:43 +0000 (08:46 -0600)]
Merge branch 'android-log-fix' of https://github.com/kdrag0n/fio
* 'android-log-fix' of https://github.com/kdrag0n/fio:
Makefile: Link to the system logging library on Android
Danny Lin [Mon, 21 Oct 2019 01:43:08 +0000 (18:43 -0700)]
Makefile: Link to the system logging library on Android
Android replaces syslog calls with calls to the Android system logging
library, liblog. Link to liblog when compiling for Android to fix the
following undefined reference errors:
aarch64-linux-android-ld: log.o: in function `android_polyfill_vsyslog':
/usr/include/syslog.h:185: undefined reference to `__android_log_print'
aarch64-linux-android-ld: /usr/include/syslog.h:182: undefined reference to `__android_log_vprint'
Signed-off-by: Danny Lin <danny@kdrag0n.dev>
Jens Axboe [Tue, 15 Oct 2019 15:27:06 +0000 (09:27 -0600)]
Merge branch 'fix-fsync-on-close' of https://github.com/sitsofe/fio
* 'fix-fsync-on-close' of https://github.com/sitsofe/fio:
backend: fix final fsync behaviour
Vincent Fu [Tue, 15 Oct 2019 14:29:58 +0000 (10:29 -0400)]
testing: add test script for zonemode=strided
Python script to run jobs with randommap, with LFSR, and with
nonrandommap. Uses null ioengine or a real file.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 15 Oct 2019 14:29:57 +0000 (10:29 -0400)]
filesetup: use zonerange for map and LFSR with zonemode=strided
When a random map is enabled, cover only the current zone when run with
zonemode=strided. This ensures that when we reach the end of the zone we
wrap around back into the current zone instead of accessing blocks
outside of the zone.
In addition, when an LFSR is used, constrain the blocks generated to the
current zone. Previously, the LFSR random_generator would ignore the
strided zonemode setting.
Fixes: https://github.com/axboe/fio/issues/809
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 15 Oct 2019 14:29:56 +0000 (10:29 -0400)]
io_u: skip to the next zone when zoneskip is set to zero
If zoneskip is not set or set to 0 in zonemode=strided mode, all IO
occurs in a single zone. If zoneskip is non-zero, the next zone begins
zoneskip bytes beyond the end of the current zone. Thus, it's not
possible to access contiguous zones since zoneskip bytes will separate
the end of one zone and the beginning of the next zone.
This patch allows fio to move to the next zone when zoneskip is
explicitly set to 0, making it possible to issue IO to contiguous zones.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 14 Oct 2019 14:03:53 +0000 (08:03 -0600)]
parse: improve detection of bad input string
If we attempt to do number conversion, and strtoll() returns 0 AND
the end pointer is the same as the starting string, then fail the
conversion.
Fixes: https://github.com/axboe/fio/issues/841
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 13 Oct 2019 17:02:00 +0000 (11:02 -0600)]
Merge branch 'fix-corrupt-hist-log' of https://github.com/sitsofe/fio
* 'fix-corrupt-hist-log' of https://github.com/sitsofe/fio:
stat: fix corruption in histogram logs
Sitsofe Wheeler [Sun, 13 Oct 2019 08:55:51 +0000 (09:55 +0100)]
stat: fix corruption in histogram logs
Commit
6cc0e5aa9eddd2487dfa9ac80be1264151058409 ("Fix overflow of
counters incremented on each I/O operation") changed various structures
from [unsigned] int -> [u]int64_t but the size used when memcpy-ing
histogram data was still the older (smaller) size. This led to un-copied
values and usage of uninitialised data. An example job that demonstrates
the issue is:
$ valgrind ./fio --size=256k --thread=1 --rate_iops=5 --direct=1 \
--number_ios=15 --name=hist_test --filename=fio.test \
--log_hist_msec=1000 --write_hist_log=fio.test --log_hist_coarseness=6
==1743== Memcheck, a memory error detector
[...]
fio-3.16-8-gf951
Starting 1 thread
==1743== Thread 3: IOPS: [R(1)][23.1%][r=20KiB/s][r=5 IOPS][eta 00m:10s]
==1743== Use of uninitialised value of size 8
==1743== at 0x609D86B: _itoa_word (_itoa.c:179)
==1743== by 0x60A0F0D: vfprintf (vfprintf.c:1642)
==1743== by 0x60A9E53: fprintf (fprintf.c:32)
==1743== by 0x47FA6C: flush_hist_samples (iolog.c:869)
==1743== by 0x47F637: flush_log (iolog.c:1212)
==1743== by 0x4813C5: finish_log (iolog.c:1238)
==1743== by 0x4812CD: __write_log (iolog.c:1571)
==1743== by 0x481202: write_clat_hist_log (iolog.c:1625)
==1743== by 0x480045: td_writeout_logs (iolog.c:1721)
==1743== by 0x485C7A: thread_main (backend.c:1889)
==1743== by 0x5C296DA: start_thread (pthread_create.c:463)
==1743== by 0x616688E: clone (clone.S:95)
This commit fixes the issue by using the new size in the memcpy.
Fixes: https://github.com/axboe/fio/issues/827
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Wed, 9 Oct 2019 02:58:57 +0000 (20:58 -0600)]
Merge branch 'bumpflocks' of https://github.com/earlephilhower/fio
* 'bumpflocks' of https://github.com/earlephilhower/fio:
Increase MAX_FILELOCKS for highly parallel IO test
Earle F. Philhower, III [Tue, 8 Oct 2019 23:13:02 +0000 (16:13 -0700)]
Increase MAX_FILELOCKS for highly parallel IO test
When running hundreds of threads and collecting CLAT, SLAT, IOS on a
per-thread basis, FIO can need a lot of file locks to complete. When
it can't get them, it gives the cryptic error message:
> fio: filelock.c:182: __fio_lock_file: Assertion `!trylock' failed.
Bump the maximum from 128 to 1024 to give headroom for high core count,
high thread count testing.
Sitsofe Wheeler [Sun, 6 Oct 2019 07:58:28 +0000 (08:58 +0100)]
backend: fix final fsync behaviour
Previously, fsync_on_close's "final" fsync was done after fio's
accounting had finished thus causing the bandwidth reported to be too
high (due to the reported time being too low). An example job that
demonstrates the issue (on machines with a few gigabytes of RAM and
non-fast disks) is the following:
./fio --gtod_reduce=1 --filename=fio.tmp --size=5G --bs=2M --rw=write \
--stonewall --name=end_fsync --end_fsync=1 --name=fsync_on_close \
--fsync_on_close=1
end_fsync: (g=0): rw=write, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1
fsync_on_close: (g=1): rw=write, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1
[...]
Run status group 0 (all jobs):
WRITE: bw=381MiB/s (400MB/s), 381MiB/s-381MiB/s (400MB/s-400MB/s), io=5120MiB (5369MB), run=13424-13424msec
Run status group 1 (all jobs):
WRITE: bw=1726MiB/s (1810MB/s), 1726MiB/s-1726MiB/s (1810MB/s-1810MB/s), io=5120MiB (5369MB), run=2966-2966msec
This patch fixes the issue by doing an fsync for fsync_on_close at the
same point as it does for end_fsync. Technically fsync_on_close will go
and do another fsync after this point too but this should be harmless as
there is no new write data.
This patch also fixes the assert seen with the following job
./fio --size=8k --io_limit=32k --filename=fio.tmp --rw=write \
--end_fsync=1 --name=extended_end_fsync
[...]
fio: filesetup.c:1682: void get_file(struct fio_file *): Assertion `fio_file_open(f)' failed.
by preventing auto-close on last I/O when it is being submitted as a
final fsync (due to end_fsync, fsync_on_close).
Finally it also fixes the strange IO depths seen with when using end_fsync:
./fio --size=4k --filename=fio.tmp --rw=write --end_fsync=1 \
--name=end_fsync_test
end_fsync_test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
[...]
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
by no longer including such fsyncs in accounting.
Fixes: https://github.com/axboe/fio/issues/831
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Tue, 8 Oct 2019 03:23:26 +0000 (21:23 -0600)]
Merge branch 'replay-blktrace-fixes' of https://github.com/shimrot/fio
* 'replay-blktrace-fixes' of https://github.com/shimrot/fio:
Fix assert error on blktrace replay containing trims
Jens Axboe [Sun, 6 Oct 2019 15:15:51 +0000 (09:15 -0600)]
Merge branch 'error-on-implicit-decl' of https://github.com/sitsofe/fio
* 'error-on-implicit-decl' of https://github.com/sitsofe/fio:
configure: stop enabling fdatasync on OSX
Sitsofe Wheeler [Sat, 5 Oct 2019 11:04:33 +0000 (12:04 +0100)]
configure: stop enabling fdatasync on OSX
Change configure compile probes to fail on implicit declarations. This
(correctly) stops us from enabling fdatasync on OSX which was
problematic because we were actually calling a syscall stub with
arguments different to what was declared in its prototype (see
https://github.com/gbrault/picoc/issues/145#issuecomment-
89734655 and
https://gitlab.freedesktop.org/xdg/shared-mime-info/issues/7 ).
Fixes: https://github.com/axboe/fio/issues/834 ("Drop macOS support for
fdatasync")
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Rebecca Cran [Mon, 30 Sep 2019 20:57:08 +0000 (14:57 -0600)]
Windows: Update URLs to https, and remove mention of WiX version
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Rebecca Cran [Mon, 30 Sep 2019 20:57:07 +0000 (14:57 -0600)]
Update the email and web address for Windows binaries.
Signed-off-by: Jens Axboe <axboe@kernel.dk>