Jens Axboe [Mon, 17 Aug 2020 22:42:16 +0000 (15:42 -0700)]
engines/windowsaio: only set IOCP thread affinity if specified
We set it by default, which seems like a bad idea. If the issuing
thread is running on some other CPU, we don't want to bind this
to what is probably CPU0.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Alexey Dobriyan [Thu, 13 Aug 2020 16:33:07 +0000 (19:33 +0300)]
fio: add for_each_rw_ddir() macro
Make it slightly easier to add Zone Append as fully fledged I/O type.
Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 17 Aug 2020 03:59:34 +0000 (20:59 -0700)]
Fio 3.22
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 14 Aug 2020 23:02:37 +0000 (16:02 -0700)]
Merge branch 'issue-1065' of https://github.com/XeS0r/fio into master
* 'issue-1065' of https://github.com/XeS0r/fio:
thread_options: Use unsigned int type for exit_what and stonewall
Signed-off-by: Jens Axboe <axboe@kernel.dk>
André Wild [Fri, 14 Aug 2020 13:52:09 +0000 (15:52 +0200)]
thread_options: Use unsigned int type for exit_what and stonewall
Fixes:
64402a8 ("Expand choices for exitall")
Fixes: https://github.com/axboe/fio/issues/1065
Signed-off-by: André Wild <wild.andre.ae@gmail.com>
Jens Axboe [Fri, 14 Aug 2020 20:16:18 +0000 (13:16 -0700)]
Fixup examples/exitwhat.fio
Use proper read/write names.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Alexey Dobriyan [Thu, 13 Aug 2020 16:36:32 +0000 (19:36 +0300)]
zbd: use ->min_zone, ->max_zone in more places
Check that ->min_zone is less than ->max_zone.
This assert found one bug here.
Use it in more places, now that it was added.
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 12 Aug 2020 17:48:15 +0000 (11:48 -0600)]
io_u: calculate incremental residuals correctly
We need to use the transfer buffer length for the residual calculation,
otherwise multiple shorts on the same IO will not calculate the right
value.
Fixes:
5fff95436922 ("Add support for >= 4G block sizes")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Dmitry Fomichev [Tue, 4 Aug 2020 01:38:28 +0000 (10:38 +0900)]
t/zbd: check log file for failed assertions
Currently, a ZBD test can succeed even if an fio assertion is raised
during its run. Search every ZBD test log file for failed assertions
and fail the test if any were found.
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Dmitry Fomichev [Tue, 4 Aug 2020 01:38:27 +0000 (10:38 +0900)]
zbd: simplify zone reset code
zbd_reset_range() function is only called once from zbd_reset_zone()
and it is always called for an individual zone, not a range.
Make zone reset flow simpler by moving all the code needed
to reset a single zone from zbd_reset_range() to zbd_reset_zone().
Therefore, zbd_reset_range() is now dropped.
zbc_reset_zone() is always called with the zone already locked. Remove
recursive zone locking inside this function and add a note in the
description of this function saying that the caller is expected to have
the zone locked when calling it.
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Dmitry Fomichev [Tue, 4 Aug 2020 01:38:26 +0000 (10:38 +0900)]
configure: check if pkg-config is installed
A few libraries need to be newer than a specific version in order to be
supported by fio and pkg-config utility is used to verify the versions
of the installed libraries. Since this step may fail because pkg-config
is not installed, verify pkg-config presence and warn the user if it
could not be found.
To avoid code duplication, add a common helper function to perform
these checks.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Dmitry Fomichev [Tue, 4 Aug 2020 01:38:25 +0000 (10:38 +0900)]
configure: improve libzbc version check
Avoid parsing pkg-config output and just use --atleast-version to
check if libzbc is present and has an up to date version.
Currently, support for libzbc ioengine is always included if libzbc is
found at the build system. Add the option to disable libzbc ioengine
support even if libzbc is found. This can be useful for
cross-compilation.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 03:40:59 +0000 (21:40 -0600)]
io_u: set io_u->verify_offset in fill_io_u()
This is arguably saner than the previous fix, since it also catches
cases where io_u->offset is adjusted later on.
Hopefully that's it...
Fixes:
4fff54ccba73 ("verify: use origina offset for verification")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 03:31:01 +0000 (21:31 -0600)]
io_u: get_next_offset() should always set io_u->verify_offset
Don't assume that the original get_next_block() is the only place
that sets it. Set it unconditionally at the success exit path
of get_next_offset().
Fixes:
4fff54ccba73 ("verify: use origina offset for verification")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 02:55:27 +0000 (20:55 -0600)]
verify: use origina offset for verification
Requeued IO might have a different offset, since we increment it for
retry conditions. Store the original offset for verification purposes.
Just like we have io_u->xfer_buf/io_u->buf and
io_u->xfer_buflen/io_u->buflen, we really should be treating
io_u->offset the same. But that's a major change, so just add this
special original offset so we know that verify has it. Currently we
treat io_u->offset like we would have io_u->xfer_offset, so this just
makes io_u->verify_offset what io_u->offset should be.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 02:54:46 +0000 (20:54 -0600)]
io_uring: notice short IO on completion path
Async IO engines need to signal residual from the completion path,
but we completely ignore those. Ensure we properly requeue IO that
was short, after adjusting the state.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 02:53:55 +0000 (20:53 -0600)]
io_u: reset ->resid on starting a requeue IO
We only clear this state for a new IO, we should also do it for starting
a requeued IO.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 02:52:09 +0000 (20:52 -0600)]
engines/io_uring: make sure state is updated for requeues
Currently we always use the start of the buffer and buffer length,
but this isn't valid for a requeue. Make sure we updated both
based on the actual io_u.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 8 Aug 2020 00:21:52 +0000 (18:21 -0600)]
Merge branch 'master' of https://github.com/donny372/fio into master
* 'master' of https://github.com/donny372/fio:
Add support for reading iolog from stdin.
donny372 [Wed, 17 Jun 2020 19:58:22 +0000 (12:58 -0700)]
Add support for reading iolog from stdin.
Add support for reading iolog from stdin and update HOWTO and man page.
Fixes: https://github.com/axboe/fio/issues/997
Signed-off-by: Shundong Zhou <donny372@gmail.com>
Jens Axboe [Sat, 1 Aug 2020 21:07:38 +0000 (15:07 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio into master
* 'master' of https://github.com/bvanassche/fio:
Prevent that fio hangs when using io_submit_mode=offload
Bart Van Assche [Sat, 1 Aug 2020 18:01:06 +0000 (11:01 -0700)]
Prevent that fio hangs when using io_submit_mode=offload
This patch has been tested by running the following shell command:
for ((i=0;i<1000;i++)); do echo $i; python3 t/run-fio-tests.py -o 10; done
Fixes:
d28174f0189c ("workqueue: ensure we see deferred error for IOs")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Mon, 27 Jul 2020 22:00:20 +0000 (16:00 -0600)]
iolog: ensure that dynamic log entries are at least queue depth sized
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 27 Jul 2020 22:00:03 +0000 (16:00 -0600)]
Add roundup_pow2() as a generic helper
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 27 Jul 2020 04:18:53 +0000 (22:18 -0600)]
t/jobs/t001[1-2].fio: run for 10 seconds instead of 3
These jobs often fails in CI testing, and it might be because the
runtime is simply too short to make it reliable.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 27 Jul 2020 00:13:43 +0000 (18:13 -0600)]
Merge branch 'fix_travis_libiscsi' of https://github.com/sitsofe/fio
* 'fix_travis_libiscsi' of https://github.com/sitsofe/fio:
travis: fix x86 libiscsi detection
travis: remove unneeded dependency
Sitsofe Wheeler [Sun, 26 Jul 2020 19:48:59 +0000 (20:48 +0100)]
travis: fix x86 libiscsi detection
When cross compiling to 32 bit systems, pkg-config has to find 32 bit
libraries. This can be done by one of the following
- Setting PKG_CONFIG_PATH/PKG_CONFIG_LIBDIR to an appropriate directory
- Using a 32 bit pkg-config binary
Choose the later for now by installing a 32 bit pkg-config (which will
replace the 64 bit one). This should fix failing CI on the 32 bit x86
build.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 19:10:28 +0000 (20:10 +0100)]
travis: remove unneeded dependency
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sun, 26 Jul 2020 13:50:46 +0000 (07:50 -0600)]
Merge branch 'enable_or_exit' of https://github.com/sitsofe/fio
* 'enable_or_exit' of https://github.com/sitsofe/fio:
configure: fail when explicit enabling doesn't succeed
Sitsofe Wheeler [Sat, 25 Jul 2020 22:10:27 +0000 (23:10 +0100)]
configure: fail when explicit enabling doesn't succeed
- Bail out if the probe fails when the user is explicitly enabling the
cuda, libiscsi or libnbd ioengines or the libaio io_uring option
- Normalize some of the code that deals with enabling/disabling the
above options
Fixes: https://github.com/axboe/fio/issues/1052
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sun, 26 Jul 2020 13:43:46 +0000 (07:43 -0600)]
Merge branch 'travis_cleanup' of https://github.com/sitsofe/fio
* 'travis_cleanup' of https://github.com/sitsofe/fio:
Makefile: fix incorrectly set libiscsi cflags
memory: fix incorrect pointer comparison when freeing cuda memory
travis: make CI install script bail out on first error
travis: enable libiscsi and cuda ioengines
travis: add dependency for cuda ioengine
travis: use install section instead of before_install section
configure: check for Debian/Ubuntu tcmalloc_minimal
travis: improve installation dependencies
travis: simplify yml file
Sitsofe Wheeler [Sun, 26 Jul 2020 10:27:22 +0000 (11:27 +0100)]
Makefile: fix incorrectly set libiscsi cflags
This fixes the warning
error: -liscsi: 'linker' input unused [-Werror,-Wunused-command-line-argument]
produced by clang.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 06:14:15 +0000 (07:14 +0100)]
memory: fix incorrect pointer comparison when freeing cuda memory
This fixes the following warning spotted by gcc 7:
memory.c: In function ‘free_mem_cudamalloc’:
memory.c:277:22: error: comparison between pointer and integer [-Werror]
if (td->dev_mem_ptr != NULL)
^~
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 05:51:16 +0000 (06:51 +0100)]
travis: make CI install script bail out on first error
This avoids the issue where we carry on when we can't find a requested
package to install.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 04:21:34 +0000 (05:21 +0100)]
travis: enable libiscsi and cuda ioengines
...but only specific Linux setups. We have to avoid enabling cuda on
i386/arm64 as we don't have the dependency in those environments so
introduce a CI_TARGET_ARCH variable and do some refactoring.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 04:07:40 +0000 (05:07 +0100)]
travis: add dependency for cuda ioengine
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 02:31:57 +0000 (03:31 +0100)]
travis: use install section instead of before_install section
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 02:23:27 +0000 (03:23 +0100)]
configure: check for Debian/Ubuntu tcmalloc_minimal
Debian derived distros have a libtcmalloc-minimal4 package that only has
a versioned libtcmalloc_minimal.so.4 inside, so try explicitly searching
for it when looking for tcmalloc.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 02:13:03 +0000 (03:13 +0100)]
travis: improve installation dependencies
- On Ubuntu add bison/flex dependencies for lex arithmetic, add
libgoogle-perftools-dev so we can link against tcmalloc.
- Invoke pip3 just the once on macOS
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 01:43:16 +0000 (02:43 +0100)]
travis: simplify yml file
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sat, 25 Jul 2020 19:55:16 +0000 (13:55 -0600)]
Merge branch 'test-cleanup' of https://github.com/vincentkfu/fio
* 'test-cleanup' of https://github.com/vincentkfu/fio:
t/run-fio-tests: add description to each test result line
ci: set PYTHONUNBUFFERED=TRUE
t/run-fio-tests: fix issues identified by pylint3 and pyflakes3
t/run-fio-tests: catch modprobe exceptions
Jens Axboe [Sat, 25 Jul 2020 19:37:15 +0000 (13:37 -0600)]
Merge branch 'check-atomics' of https://github.com/sitsofe/fio
* 'check-atomics' of https://github.com/sitsofe/fio:
configure: check for C11 atomics support
Vincent Fu [Sat, 25 Jul 2020 17:55:06 +0000 (13:55 -0400)]
t/run-fio-tests: add description to each test result line
Since no one remembers what the test numbers actually refer to,
include either the job name or executable in each test result line.
Vincent Fu [Sat, 25 Jul 2020 17:50:06 +0000 (13:50 -0400)]
ci: set PYTHONUNBUFFERED=TRUE
Set this environment variable so that the build output is consistent
across the different environments.
Vincent Fu [Sat, 25 Jul 2020 17:20:33 +0000 (13:20 -0400)]
t/run-fio-tests: fix issues identified by pylint3 and pyflakes3
- fix whitespace issue in function parameter list
- eliminate extra 'requirements' key for t0012
- t0013 does not need a separate test class
- specify string format arguments as logging function parameters
Vincent Fu [Sat, 25 Jul 2020 17:02:21 +0000 (13:02 -0400)]
t/run-fio-tests: catch modprobe exceptions
Catch exceptions that occur when running modprobe in case the build
environment has it disabled (e.g., AWS CodeBuild).
Sitsofe Wheeler [Sat, 25 Jul 2020 07:46:15 +0000 (08:46 +0100)]
configure: check for C11 atomics support
Since
3932f8be718fc4ca3b863c51c0d567821d75c003 ("arch/arch.h: Introduce
atomic_{load_acquire,store_release}()") fio has needed C11 atomic
support and this is only provided from gcc 4.9 / clang 3.6 onwards
(http://stdatomic.gforge.inria.fr/#sec-2 ). Make this clearer by
doing the following:
- Add configure check for C11 atomics support and bail out with a
(friendly) message pointing at compiler version if it's not there
- Remove the minimum compiler version check from compiler.h as it is now
redundant
I've avoided doing an explicit compiler version check because Apple's
clang can have a wildly different version to upstream
(https://stackoverflow.com/a/
33614612 ).
Tested on:
- CentOS 7 with default gcc 4.8.5 (correctly errors with message)
- CentOS 7 docker container using llvm-toolset-7 to provide clang 5.0.1
(works)
- CentOS 7 docker container using devtool-toolset-7 to provide gcc 7.3.1
(works)
- Ubuntu 16.04 with clang-3.5 (correctly errors with message)
- Ubuntu 16.04 with clang-3.6 installed (works)
- macOS 10.14.6 with clang-1001.0.46.4 (works)
v2:
- Rebase
- Reword failure message and stop mentioning fio version number
Fixes: https://github.com/axboe/fio/issues/1047
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sat, 25 Jul 2020 14:01:03 +0000 (08:01 -0600)]
configure: error early on too old compier
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 24 Jul 2020 17:05:21 +0000 (11:05 -0600)]
Merge branch 'sribs-patch-1039' of https://github.com/sribs/fio
* 'sribs-patch-1039' of https://github.com/sribs/fio:
stat: stop triggerring division by zero on bandwidth lower than 1KBps
io_u: fix exit failure case when using rates and timeout
sribs [Fri, 24 Jul 2020 16:53:50 +0000 (22:23 +0530)]
stat: stop triggerring division by zero on bandwidth lower than 1KBps
Before the integer arithmetic is performed to obtain
percentage of aggregation this conversion of rs->agg[ddir] into KBytes
could potentially turn out to be 0.
Adjust stat.c to no longer trigger division by zero on bandwidths
greater than 0 but less than 1kilobyte a second. The fact that this
is performed on multiple occasions, a function is preferred
Signed-off-by: sribs <sribs@microsoft.com>
sribs [Fri, 24 Jul 2020 10:06:58 +0000 (15:36 +0530)]
io_u: fix exit failure case when using rates and timeout
Previously when using a rate smaller than the minimum block size,
fio could end up failing to exit at a designated timeout and
while doing this, do check if usec takes in negative values which could
potentially not exit fio indefinitely.
Fixes: https://github.com/axboe/fio/issues/1039
Signed-off-by: Sriharsha B S <sribs@microsoft.com>
Shin'ichiro Kawasaki [Wed, 22 Jul 2020 09:26:33 +0000 (18:26 +0900)]
t/zbd: Improve pass condition of test case #49
The test case #49 runs write workloads with verify option. It checks read
bytes for verify, but it does not check written bytes. To make test pass
condition more accurate, add check of the written bytes.
Fixes:
4844bb4716ab ("t/zbd: Add test case to check zonecapacity option")
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 21 Jul 2020 15:40:07 +0000 (09:40 -0600)]
zbd: fix %lu -> %llu dprint() formatting
The types aren't correct for 32-bit builds.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Hans Holmberg [Fri, 17 Jul 2020 09:36:37 +0000 (18:36 +0900)]
t/zbd: Add test case to check zonecapacity option
Add test case #49 which confirms the newly introduced zonecapacity
option is working as expected with regular block devices.
Signed-off-by: Hans Holmberg <hans.holmberg@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Fri, 17 Jul 2020 09:36:36 +0000 (18:36 +0900)]
t/zbd: Support testing zone capacity smaller than zone size with null_blk
To test zone capacity support by fio using null_blk, add -zone-cap option
to run-tests-against-zoned-nullb. With this option, the script creates a
null_blk device which has zone capacity smaller than zone size and run
test-zbd-support script for the device.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Hans Holmberg [Fri, 17 Jul 2020 09:36:35 +0000 (18:36 +0900)]
t/zbd: Support testing zone capacity smaller than zone size
Add support for testing zoned block devices where zone capacity is less
than zone size. If the test case runs sequential workloads or workloads
with verify, use zone capacity instead of zone size to complete.
To use zone capacity of the test target zones, introduce a helper
function total_zone_capacity() which sums up zone capacity numbers of the
test target zones.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Hans Holmberg <hans.holmberg@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Fri, 17 Jul 2020 09:36:34 +0000 (18:36 +0900)]
t/zbd: Mandate blkzone capacity report for devices with zone capacity
For NVMe ZNS and null_blk zoned drives, a zone capacity can be smaller
than its size. To test zone capacity handling by fio for these devices,
t/zbd/test-zbd-support must be able to discover zone capacities to avoid
test failures.
To avoid the failures, make zone capacity report by blkzone mandatory if
the test target device is a ZNS device or a null_blk device.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Hans Holmberg [Fri, 17 Jul 2020 09:36:33 +0000 (18:36 +0900)]
options: Add zonecapacity option for zonemode=zbd
To test zone capacity handling by fio using regular block devices, add
zonecapacity option for zonemode=zbd. This new option allows specifying a
zone capacity smaller than the zone size. Similarly to the zonesize
option, the zonecapacity defined value applies to all emulated zones of
the device.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Hans Holmberg <hans.holmberg@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Fri, 17 Jul 2020 09:36:32 +0000 (18:36 +0900)]
zbd: Support zone capacity smaller than zone size
NVMe ZNS specification defines zone capacity. The report zone interface
of Linux kernel supports it. This patch adds support for this new
interface.
Modify configure to check availability of BLK_ZONE_REP_CAPACITY flag as
well as blkzoned.h header only when target OS is Linux. If this flag is
defined and set in the report zone, all sectors from the zone capacity up
to the zone size are not accessible. Prevent read and write operations to
these sectors by modifying zbd_adjust_block(). Of note is that this skips
the region between zone capacity and zone size then reduces total I/O
bytes of sequential workloads.
Introduce helper functions zbd_zone_end() and zbd_zone_capacity_end() to
improve readability of zone limit check code.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com>
Signed-off-by: Hans Holmberg <hans.holmberg@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 20 Jul 2020 22:37:50 +0000 (16:37 -0600)]
Fio 3.21
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 18 Jul 2020 14:39:04 +0000 (08:39 -0600)]
Merge branch 'patch-1' of https://github.com/avlasov-mos-de/fio
* 'patch-1' of https://github.com/avlasov-mos-de/fio:
Fix scale on gnuplot graphs
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:22 +0000 (13:13 +0900)]
t/zbd: Enable regular block devices for test case #47
The test case #47 was not able to run for regular block devices since it
missed zonesize option required for regular block devices.
Enable the test case for regular block devices by calling the
run_fio_on_seq() helper function which adds the zonesize option.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:21 +0000 (13:13 +0900)]
t/zbd: Remove write before random read/write from test case #17
The test case #17 writes data to the last zone before random read/write.
This avoided read failure when read is chosen for the first operation of
the random read and write. Such failure case was prevented by the commit
fb0259fb276a ("zbd: Ensure first I/O is write for random read/write to
sequential zones"). Then the write is no longer required. Remove it.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:20 +0000 (13:13 +0900)]
t/zbd: Combine write and read fio commands for test case #16
The test case #16 checks result of read job which runs after write job of
test case #15. It does not work if test case #16 is executed alone with
the -t option and the target device has all zones empty.
Specify both the write job and the read job to a single fio command using
write_and_run_one_fio_job() helper function instead of run_one_fio_job().
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:19 +0000 (13:13 +0900)]
t/zbd: Combine write and read fio commands for test case #15
The test case #15 called fio twice. Once with write job to prepare data
to read. The other with read job for testing. But the first run has no
effect on the second run when the zone status is reinitialized.
Specify both the write job and the read job to a single fio command using
write_and_run_one_fio_job() helper function instead of run_one_fio_job().
Also apply same test pass condition for both regular block devices and
zoned block devices.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:18 +0000 (13:13 +0900)]
t/zbd: Combine write and read fio commands for test case #6
The test case #6 checks result of the read job which reads data written
by the write job of test case #5. It does not work if test case #6 is
executed alone with the -t option and the target device has all zones
empty.
Specify both the write job and the read job to a single fio command
using write_and_run_one_fio_job() helper function instead of
run_fio_on_seq() helper function.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:17 +0000 (13:13 +0900)]
t/zbd: Add write_and_run_one_fio_job() helper function
To test read workload behavior with zonemode=zbd and zoned block devices
with all zones empty, it is required to write data to the device before
the read workload. To avoid repeated codes for such test preparation,
introduce the helper function write_and_run_one_file_job(). It runs the
write job with specified offset and size to prepare the read data, then
run the specified job for read test.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:16 +0000 (13:13 +0900)]
t/zbd: Fix pass condition of test case #3
The test case #3 checks that the read workload for empty zones ends
without reading any data. When zonemode=zbd, it is expected for both
regular block devices and zoned block devices. However, the test case
allowed for regular block devices to read data from empty zones. Remove
this special pass condition for regular block devices.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:15 +0000 (13:13 +0900)]
zbd: Fix initial zone write pointer of regular block devices
With zonemode=zbd and regular block devices, fio initializes zone status
as empty. However, write pointers are set not at zone start but at zone
end. This inconsistency between write pointer position and zone status
caused different behavior from zoned block devices. Fix the inconsistency
by setting initial write pointer at zone start.
Of note is that every fio run with zonemode=zbd and regular block devices
starts with empty zones. Then read workloads just ends without any data
to read. To run read workload, specify read_beyond_wp option. Or specify
two jobs to a single fio command: write job to prepare data for read and
read job which waits for the write job completion.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 14 Jul 2020 16:10:23 +0000 (10:10 -0600)]
Merge branch 'fix_devdax' of https://github.com/harish-24/fio
* 'fix_devdax' of https://github.com/harish-24/fio:
Fix: dev-dax engine building with make
Harish [Tue, 14 Jul 2020 16:00:24 +0000 (21:30 +0530)]
Fix: dev-dax engine building with make
With recent changes to Makefile dev-dax skips building. This patch
fixes that.
Signed-off-by: Harish <harish@linux.ibm.com>
Jens Axboe [Mon, 13 Jul 2020 14:01:52 +0000 (08:01 -0600)]
Merge branch 'io_uring-opt' of https://github.com/antonblanchard/fio
* 'io_uring-opt' of https://github.com/antonblanchard/fio:
io_uring: Avoid needless update of completion queue head pointer
Anton Blanchard [Mon, 13 Jul 2020 01:34:48 +0000 (11:34 +1000)]
io_uring: Avoid needless update of completion queue head pointer
I'm seeing a slowdown in io_uring performance on a POWER9 box when
the userspace and kernel polling threads are on two cores that
share an L2 cache.
fio_ioring_cqring_reap() always stores to the completion queue head
pointer, even if nothing was reaped and the value hasn't changed.
Changing this to only update the head pointer when it changes results
in a 95% improvement in performance on this particular test.
Signed-off-by: Anton Blanchard <anton@ozlabs.org>
Jens Axboe [Wed, 8 Jul 2020 21:48:11 +0000 (15:48 -0600)]
t/io_uring: make bs and polled IO configurable at runtime
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 4 Jul 2020 23:29:32 +0000 (17:29 -0600)]
Merge branch 'windows' of https://github.com/bvanassche/fio
* 'windows' of https://github.com/bvanassche/fio:
os/windows/posix.c: Use INVALID_SOCKET instead of < 0
os/windows/posix.c: Strip trailing whitespace
Bart Van Assche [Sat, 4 Jul 2020 03:20:00 +0000 (20:20 -0700)]
os/windows/posix.c: Use INVALID_SOCKET instead of < 0
From
https://docs.microsoft.com/en-us/windows/win32/winsock/socket-data-type-2:
"Because the SOCKET type is unsigned, compiling existing source code from,
for example, a UNIX environment may lead to compiler warnings about
signed/unsigned data type mismatches.
This means, for example, that checking for errors when the socket and
accept functions return should not be done by comparing the return value
with -1, or seeing if the value is negative (both common and legal
approaches in UNIX). Instead, an application should use the manifest
constant INVALID_SOCKET as defined in the Winsock2.h header file."
This patch fixes the following build error for the latest version of
the MinGW64 cross-compiler on Cygwin:
os/windows/posix.c:886:17: error: comparison of unsigned expression < 0
is always false
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Sat, 4 Jul 2020 03:16:07 +0000 (20:16 -0700)]
os/windows/posix.c: Strip trailing whitespace
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Sat, 4 Jul 2020 19:46:46 +0000 (13:46 -0600)]
Merge branch 'workqueue' of https://github.com/bvanassche/fio
* 'workqueue' of https://github.com/bvanassche/fio:
workqueue: Fix race conditions in the workqueue mechanism
workqueue: Rework while loop locking
Bart Van Assche [Sat, 4 Jul 2020 16:44:56 +0000 (09:44 -0700)]
workqueue: Fix race conditions in the workqueue mechanism
From https://pubs.opengroup.org/onlinepubs/
9699919799/functions/pthread_cond_wait.html:
"The application shall ensure that these functions are called with mutex
locked by the calling thread; otherwise, an error (for
PTHREAD_MUTEX_ERRORCHECK and robust mutexes) or undefined behavior (for
other mutexes) results.
From https://pubs.opengroup.org/onlinepubs/
9699919799/functions/pthread_cond_signal.html:
"however, if predictable scheduling behavior is required, then that mutex
shall be locked by the thread calling pthread_cond_broadcast() or
pthread_cond_signal()."
Hence always hold the associated mutex around pthread_cond_wait() and
pthread_signal() calls.
This patch fixes the hangs reported by Travis and Appveyor for test case
t0013.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Sat, 4 Jul 2020 16:39:54 +0000 (09:39 -0700)]
workqueue: Rework while loop locking
Reduce the number of pthread_mutex_lock() and unlock calls by moving
these outside while loops. Other than protecting the all_sw_idle() call with
the wq->flush_lock, this patch does not change any functionality.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Sat, 4 Jul 2020 14:28:43 +0000 (08:28 -0600)]
Merge branch 'overlap' of https://github.com/bvanassche/fio
* 'overlap' of https://github.com/bvanassche/fio:
Add a test for serialize_overlap=1
Enable error checking for the mutex that serializes overlapping I/O
Fix a potentially infinite loop in check_overlap()
t/run-fio-tests.py: Increase IOPS tolerance further
.appveyor.yml: Select the latest software image
Bart Van Assche [Sun, 14 Jun 2020 19:09:45 +0000 (12:09 -0700)]
Add a test for serialize_overlap=1
This test causes fio to call check_overlap().
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Sat, 13 Jun 2020 18:33:37 +0000 (11:33 -0700)]
Enable error checking for the mutex that serializes overlapping I/O
Since it is nontrivial to verify that pthread_mutex_lock() and unlock
calls are paired for the overlap_check mutex, enable runtime verification.
The PTHREAD_ERRORCHECK_MUTEX_INITIALIZER_NP initializer causes
pthread_mutex_lock() to fail with EDEADLK if the mutex was already locked
and causes pthread_mutex_unlock() to fail with EPERM if the mutex is owned
by another thread.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Sat, 13 Jun 2020 18:23:39 +0000 (11:23 -0700)]
Fix a potentially infinite loop in check_overlap()
If the following happens:
* check_overlap() finds an overlap.
* All other threads finish after the overlap has been found and before
the next iteration of the do/while loop starts.
Then the do/while loop in check_overlap() will iterate forever. Fix this
by rewriting check_overlap() such that this cannot happen.
This patch fixes the following Coverity complaint:
CID 184174 (#2 of 2): Double lock (LOCK)
Fixes:
c06379a65d46 ("fio: enable overlap checking with offload submission")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Sat, 4 Jul 2020 04:19:23 +0000 (21:19 -0700)]
t/run-fio-tests.py: Increase IOPS tolerance further
From an Appveyor build:
DEBUG:root:Test 12: iops1: 984.676882
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Sat, 4 Jul 2020 04:13:31 +0000 (21:13 -0700)]
.appveyor.yml: Select the latest software image
This is necessary to make the overlap test (t0013) pass.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Fri, 3 Jul 2020 14:34:56 +0000 (08:34 -0600)]
configure: fix bad indentation
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Yigal Korman [Fri, 3 Jul 2020 12:38:43 +0000 (15:38 +0300)]
fio: better info when engine not found
Jens suggested we provide the user with the likely reasons why we failed
to load the given engine.
This should help better resolve these issues.
Signed-off-by: Yigal Korman <ykorman@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Yigal Korman [Fri, 3 Jul 2020 12:38:42 +0000 (15:38 +0300)]
configure: new --dynamic-libengines build option
When enabled, some of the more dependency-heavy internal engines are
converted to "plugin" engines, i.e. they are built into separate object
files and are loaded by fio on demand.
This helps downstream distros package these engines separately and not
force a long list of package dependencies from the base fio package.
Signed-off-by: Yigal Korman <ykorman@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Yigal Korman [Fri, 3 Jul 2020 12:38:41 +0000 (15:38 +0300)]
configure/Makefile: engine LIBS consistency
Some engines were adding their LIBS in the Makefile and some were
declaring them in the configure script.
This is inconsistent and confusing and prevents easy bulk changes to the
engines.
Signed-off-by: Yigal Korman <ykorman@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Yigal Korman [Fri, 3 Jul 2020 12:38:40 +0000 (15:38 +0300)]
fio: don't retry engine search on failure
ioengine_load will try to load a given engine and if it doesn't match
the current one (default is always psync), it will remove the current
one and try again.
This makes sense when the engine is loaded successfully, but if it's
not, we try to load it twice for no reason.
Fix by failing if we were unable to load requested engine and not
retrying.
Signed-off-by: Yigal Korman <ykorman@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 3 Jul 2020 00:08:14 +0000 (18:08 -0600)]
Merge branch 'num2str' of https://github.com/bvanassche/fio
* 'num2str' of https://github.com/bvanassche/fio:
num2str(): Add the E (exa) prefix
Add a num2str() unit test
num2str(): Fix overflow handling
num2str(): Remove the fmt[] array
num2str(): Use asprintf() instead of malloc()
Bart Van Assche [Thu, 2 Jul 2020 23:31:35 +0000 (16:31 -0700)]
num2str(): Add the E (exa) prefix
This change guarantees that all 64-bit integers that should be assigned a
multiplier are assigned a multiplier.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Thu, 2 Jul 2020 22:18:37 +0000 (15:18 -0700)]
Add a num2str() unit test
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Thu, 2 Jul 2020 21:53:09 +0000 (14:53 -0700)]
num2str(): Fix overflow handling
This patch changes the output for the arguments (UINT64_MAX, 4, 1, 0,
N2S_NONE, "18.4") from "18.4(null)" into "18.4".
This patch fixes the following Coverity complaint:
CID 169307 (#1 of 1): Out-of-bounds read (OVERRUN)
27. overrun-local: Overrunning array of 6 8-byte elements at element
index 6 (byte offset 55) by dereferencing pointer unitprefix + post_index.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Thu, 2 Jul 2020 21:56:16 +0000 (14:56 -0700)]
num2str(): Remove the fmt[] array
This patch does not change any functionality.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Thu, 2 Jul 2020 21:47:01 +0000 (14:47 -0700)]
num2str(): Use asprintf() instead of malloc()
This patch reduces the size of the buffer allocated by num2str().
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Sun, 28 Jun 2020 13:08:36 +0000 (07:08 -0600)]
Merge branch 'pmemblk' of https://github.com/bvanassche/fio
* 'pmemblk' of https://github.com/bvanassche/fio:
Unbreak the pmemblk engine
Bart Van Assche [Sat, 27 Jun 2020 14:26:24 +0000 (07:26 -0700)]
Unbreak the pmemblk engine
Reported-by: Yi Zhang <yi.zhang@redhat.com>
Tested-by: Yi Zhang <yi.zhang@redhat.com>
Fixes:
e9c7be0e32e6 ("pmemblk: Fix a memory leak")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Wed, 24 Jun 2020 15:02:06 +0000 (09:02 -0600)]
Merge branch 'compiler' of https://github.com/bvanassche/fio
* 'compiler' of https://github.com/bvanassche/fio:
Merge compiler/gcc4.h into compiler/compiler.h