Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:06 +0000 (17:20 +0900)]
t/zbd: Remove unnecessary option for zbc_reset_zone
When -all option is specified, zbc_reset_zone command does not take
offset option. Remove it.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:05 +0000 (17:20 +0900)]
t/zbd: Reset all zones before test when max open zones is specified
When the test target device has maximum open zones limit, the zones in
test target region may not be opened up to the limit, because the zones
out of the test target region may have open zones. To ensure that the
test target zones can be opened up to the limit, reset all zones of the
test target device before the test cases with write work load starts.
Introduce the helper function prep_write() to check if all zone reset is
required and do the reset.
Also remove unnecessary reset_zone calls for test case #29 and #48. These
are no longer required by virtue of the improvement in zbd_setup_files()
to set up zones to meet max_open_zones limit.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:04 +0000 (17:20 +0900)]
t/zbd: Add -o option to t/zbd/test-zoned-support
To specify maximum open zones of the test target device, add -o option
to t/zbd/test-zoned-support. When this option is specified, add
max_open_zones option to all fio commands in the test script. The option
ensures the number of zones opened by fio is within the limit. This is
useful when the test target device has maximum open zones limit.
When the fio command does not have multiple jobs and the test case does
not specify max_open_zones, add single max_open_zones option to the fio
command line.
When the fio command takes multiple jobs, add max_open_zones option to
each job. Introduce job_var_opts variable to keep options to be added to
each job. To distinguish it from global options for all jobs, rename
var_opts variable to global_var_opts. When a test case with multiple jobs
always specifies max_open_zones option, exclude the max_open_zones option
from job_var_opts, using job_var_opts_exclude() helper function.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:03 +0000 (17:20 +0900)]
t/zbd: Improve usage message of test-zbd-support script
The usage message of t/zbd/test-zbd-support does not explain meaning of
each option. Elaborate meaning of the options.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:02 +0000 (17:20 +0900)]
zbd: Initialize open zones list referring zone status at fio start
When fio starts write workloads to zones with open status, fio does not
reflect the zone status to open zone list. This results in inconsistent
open zone accounting.
To avoid this inconsistency, initialize the open zone list referring the
zone status before workload start. If a zone in target range have open
status, call zbd_open_zone() function to build up the open zone list.
Since io_u is not available at that timing, make the function independent
from io_u. If the number of open zones exceeds maximum open zones limits,
reset the exceeding open zone by calling zbd_reset_zone().
Of note is that fio counts open zones within the write target range.
Open zones out of the range are not counted or checked.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:01 +0000 (17:20 +0900)]
oslib/linux-blkzoned: Allow reset zone before file set up
To reset zones before file set up completion, modify the helper function
blkzoned_reset_wp() to work even when fio_file struct does not have valid
file descriptor yet. If the file descriptor is not available, open and
close the reset target file within the function. Otherwise use the
available descriptor to avoid file open/close overhead.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:00 +0000 (17:20 +0900)]
zbd: Decrement open zones count at write command completion
When max_open_zones or job_max_open_zones option is provided, fio counts
number of open zones. This open zone accounting is done during submission
of write commands. When write command is submitted to a zone for the
first time, the zone is counted as open. When a write command which will
fill the zone gets submitted, the zone is counted as closed. However,
this count at write command submission may open zones more than the
limit. When writes to zones more than the limit are submitted with high
queue depth, those writes in-flight open zones more than the limit.
To avoid such writes beyond max open zones limit, decrement number of
open zones count not at write command submission but at write command
completion. By doing this, the number of zones with in-flight write
commands are kept within the limit with accuracy. Introduce the helper
function zbd_end_zone_io() for this decrement and zbd_close_zone() call.
The zbd_close_zone() function requires thread_data argument. To refer
thread_data at write command completion, add the argument to zbd_put_io()
and zbd_queue_io() callbacks. Also add a loop to the zbd_close_zone()
function which converts zone_info array index to open_zones array index
to avoid loop code duplication.
Even when io_u points to an open zone, the zone may not be valid for
write since in-flight write commands may make the zone full. Check this
in zbd_open_zone() to handle such zones as in full status.
Because of the zone close timing change, there might be no open zone when
zbd_convert_to_open_zone() is called. Do not handle such case as an
error and open other zones.
When zbd_convert_to_open_zone() finds all open zones can not be used for
next write, it opens other zones. This zone open fails when the number of
open zones hits one of the limits: 1) number of zones in the fio write
target region, 2) max_open_zones option or 3) job_max_open_zones option.
To avoid the zone open failure, wait for writes in-flight completes and
open zones get closed before opening other zones.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 31 Aug 2020 15:07:12 +0000 (09:07 -0600)]
options: flow should parse as FIO_OPT_INT
It's an unsigned int, if we parse it as FIO_OPT_ULL then we clobber
the variable around it in the options struct. This is apparent when
running the examples/flow.fio job file, as flow_sleep=1000 is set
before flow, but gets clobbered when flow is set. This results in a
flow_sleep value of 0 instead of 1000.
Fixes:
d4e74fda98b6 ("flow: add ability for weight-based flow control on multiple jobs")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 31 Aug 2020 14:52:25 +0000 (08:52 -0600)]
Merge branch 'multi_job_flow' of https://github.com/bardavid/fio into master
* 'multi_job_flow' of https://github.com/bardavid/fio:
flow: add ability for weight-based flow control on multiple jobs
flow: reclaim flow when job is reaped
David, Bar [Sun, 30 Aug 2020 14:29:24 +0000 (17:29 +0300)]
flow: add ability for weight-based flow control on multiple jobs
Fixes: #741 - fio how to run mix workload all 4 patterns in one job file
Previously, 'flow' was used to regulate activity between 2 or more jobs with
certain weight. However, the implementation was ineffective when regulating
more than 2 jobs with different weights, since it relied on a single,
shared counter, and was missing logic to track individual job's activity
and regulate it according to it's requested weight in proportion to other jobs'
requested weight and activity.
This commit modifies the 'flow' functionality to track each job's
activity and the total activity belonging to a 'flow_id' group.
Using this new information, the 'flow' logic is then modified to
regulate a job's activity in proportion to other jobs in the
same group. Similar to previous behavior, the regulation is done by
stalling a job's activity.
New jobs joining an existing 'flow_id' group or jobs repead from said
group will cause re-calibration of the weights and alter the activity as a
result. For example, 3 jobs (j1,j2,j3) with weights (1:2:3)
respectively will maintain activity ratio of 1:2:3. Thus, j1 will have
activity of 1/6, j2 - 2/6 and j3 - 1/2.
If j3 exits prematurely, the proportion is re-calibrated
such that j1 will strive to have activity of 1/3 and j2 - 2/3.
Re-calibration phase may cause a hiccup in the activity of one or more
jobs since the new desired proportion might not be achieved quickly
enough.
Signed-off-by: David, Bar <bardavvid@gmail.com>
David, Bar [Thu, 13 Aug 2020 09:40:55 +0000 (12:40 +0300)]
flow: reclaim flow when job is reaped
Signed-off-by: David, Bar <bardavvid@gmail.com>
Rebecca Cran [Sat, 29 Aug 2020 23:31:23 +0000 (23:31 +0000)]
Update the year to 2020 in os/windows/eula.rtf
Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Rebecca Cran [Sat, 29 Aug 2020 23:25:50 +0000 (23:25 +0000)]
Update os/windows/dobuild.cmd to support signing binaries/installer
Since organizations are increasingly requiring software to be digitally
signed, or have heuristics that penalize unsigned software, add support
to do build.cmd to sign both the .exe binaries and the .msi installer.
Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 28 Aug 2020 15:14:38 +0000 (09:14 -0600)]
Use fallthrough attribute
Fio currently triggers a bunch of fall through errors on clang 10,
since it doesn't work with the /* fall through */ method of
indicating fallthrough.
Normalize this a bit and use the correct compiler attribute for this.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 28 Aug 2020 14:04:15 +0000 (08:04 -0600)]
Merge branch 'master' of https://github.com/donny372/fio into master
* 'master' of https://github.com/donny372/fio:
Avoid multiple instance read iolog from stdin.
Aravind Ramesh [Thu, 13 Aug 2020 15:01:48 +0000 (17:01 +0200)]
zbd: don't read past the WP on a read only workload with verify
When running a read only workload with verify option enabled,
fio continues to read beyond the write pointer, and thus
reads a deallocated block, which contains data of a predefined
pattern, thus causing the verify to fail.
Thus, we cannot simply jump to accept for a read only workload
with verify option enabled, we must let the regular flow of
execution to continue, so that the check if offset > wp is
performed, so that we will return the address to a new zone,
rather than an address that is past the wp.
Reading past the wp is not a recent bug, it is possible to
reproduce this bug even before zone capacity was introduced.
However, with the introduction of zone capacity, filling a
zone now only writes up to zone capacity of a zone, making
it much easier to reproduce this bug.
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com>
Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 22 Aug 2020 17:26:39 +0000 (11:26 -0600)]
t/io_uring: cleanup vectored vs non-vectored
No functional changes, just makes it a bit more logical.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 22 Aug 2020 16:58:14 +0000 (10:58 -0600)]
t/io_uring: use non-vectored reads if available
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 22 Aug 2020 16:48:15 +0000 (10:48 -0600)]
engines/io_uring: use non-vectored read/write if available
There's the nonvectored option to set this, by default to it if it
wasn't set explicitly *and* we have it available. We issue a probe
to detect this.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
donny372 [Tue, 18 Aug 2020 17:36:05 +0000 (10:36 -0700)]
Avoid multiple instance read iolog from stdin.
https://github.com/axboe/fio/pull/1019 added support for reading iolog from stdin, but we could have multiple instance of read iolog, only one can read from stdin.
Jens Axboe [Fri, 21 Aug 2020 22:22:43 +0000 (16:22 -0600)]
engines/io_uring: don't attempt to set RLIMITs
This was meant to be helpful, but it's actively harmful if we're not
running as root. Just get rid of it, it's really no different than
needing appropriate memlock limits for just the ring itself.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 21 Aug 2020 14:52:21 +0000 (08:52 -0600)]
Merge branch 'adjusting-libpmem' of https://github.com/lukaszstolarczuk/fio into master
* 'adjusting-libpmem' of https://github.com/lukaszstolarczuk/fio:
engines/libpmem: adjust for PMDK >=1.5 usage
Łukasz Stolarczuk [Fri, 17 Jul 2020 13:14:55 +0000 (15:14 +0200)]
engines/libpmem: adjust for PMDK >=1.5 usage
- libpmem engine adjusted to take full advantage of new PMDK versions,
- code redundancy cleaned (some code was copy-pasted from PMDK),
- libpmem example adjusted.
Behavior of direct and sync parameters changed:
- sync=1 means that pmem_drain() is executed for each write operation,
- depends on direct option, flags are set for pmem_memcpy() call:
- direct=1 - PMEM_F_MEM_NONTEMPORAL,
- direct=0 - PMEM_F_MEM_TEMPORAL.
Jens Axboe [Wed, 19 Aug 2020 19:02:42 +0000 (13:02 -0600)]
file: provider fio_file_free() helper
Since we have two spots, provide a helper to do the freeing.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 19 Aug 2020 18:10:26 +0000 (12:10 -0600)]
init: add_job() needs to use right file freeing functions
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 19 Aug 2020 18:04:38 +0000 (12:04 -0600)]
file: track allocation origin
We can't always rely on the engine options for the file, as files
can come from various places. So mark the file as having come from
smalloc/calloc and use that when freeing it.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 19 Aug 2020 17:23:42 +0000 (11:23 -0600)]
Merge branch 'force-windows-artifact' of https://github.com/sitsofe/fio into master
* 'force-windows-artifact' of https://github.com/sitsofe/fio:
ci: always upload Windows MSI if smoke test passes
Sitsofe Wheeler [Wed, 19 Aug 2020 15:54:15 +0000 (16:54 +0100)]
ci: always upload Windows MSI if smoke test passes
When the test_script stage fails the artifacts stage isn't run. This is
problematic because the long running tests (on all platforms) can be
flaky and Windows folk are accustomed to not having to do builds
themselves when testing the tip of a project. This could lead to a
situation where no there's no pre-built Windows binaries due to a
spurious failures...
Solve the above by moving the smoke test and artifact upload to be
earlier, remove a useless use of bash during smoke Appveyor smoke
testing and change the "Deployment name" from msi to fio.msi.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Wed, 19 Aug 2020 13:12:11 +0000 (07:12 -0600)]
engines/windowsaio: fix silly thinky on IO thread creation
I inadvertently made the setup fail unless the affinity option was
set, the intent was obviously to only run the affinitization if we
asked for it.
Fixes:
09c1aa8b3227 ("engines/windowsaio: only set IOCP thread affinity if specified")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Dmitry Fomichev [Mon, 17 Aug 2020 23:47:08 +0000 (08:47 +0900)]
configure: fix syntax error with NetBSD
The recent patch to detect for pkg-config presence has introduced some
bash-specific code to configure script. This lead to syntax errors
while running configure under some other shells, such as (d)ash.
Avoid ${!var} indirect substitution syntax and stop using "local"
keyword which is non-POSIX-standard. Address a few minor shellcheck
complaints about the code in same function.
Fixes:
162f8c2a96ae ("configure: check if pkg-config is installed")
Reviewed-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 18 Aug 2020 15:33:49 +0000 (08:33 -0700)]
Merge branch 'filelock_assert_fix' of https://github.com/bardavid/fio into master
* 'filelock_assert_fix' of https://github.com/bardavid/fio:
filelock: fix wrong file trylock assertion.
Jens Axboe [Tue, 18 Aug 2020 14:45:06 +0000 (07:45 -0700)]
Merge branch 'asprintf1' of https://github.com/kusumi/fio into master
* 'asprintf1' of https://github.com/kusumi/fio:
oslib: fix asprintf build failure
Tomohiro Kusumi [Tue, 18 Aug 2020 13:34:07 +0000 (22:34 +0900)]
oslib: fix asprintf build failure
oslib/asprintf.h needs to include <stdarg.h> for va_list.
This started to appear on NetBSD since
38b00241e3("num2str(): Use asprintf() instead of malloc()").
--
In file included from lib/num2str.c:7:0:
lib/../oslib/asprintf.h:5:45: error: unknown type name 'va_list'; did you mean '__va_list'?
int vasprintf(char **strp, const char *fmt, va_list ap);
^~~~~~~
__va_list
CC lib/pattern.o
gmake: *** [Makefile:447: lib/num2str.o] Error 1
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Jens Axboe [Mon, 17 Aug 2020 22:42:16 +0000 (15:42 -0700)]
engines/windowsaio: only set IOCP thread affinity if specified
We set it by default, which seems like a bad idea. If the issuing
thread is running on some other CPU, we don't want to bind this
to what is probably CPU0.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Alexey Dobriyan [Thu, 13 Aug 2020 16:33:07 +0000 (19:33 +0300)]
fio: add for_each_rw_ddir() macro
Make it slightly easier to add Zone Append as fully fledged I/O type.
Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 17 Aug 2020 03:59:34 +0000 (20:59 -0700)]
Fio 3.22
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 14 Aug 2020 23:02:37 +0000 (16:02 -0700)]
Merge branch 'issue-1065' of https://github.com/XeS0r/fio into master
* 'issue-1065' of https://github.com/XeS0r/fio:
thread_options: Use unsigned int type for exit_what and stonewall
Signed-off-by: Jens Axboe <axboe@kernel.dk>
André Wild [Fri, 14 Aug 2020 13:52:09 +0000 (15:52 +0200)]
thread_options: Use unsigned int type for exit_what and stonewall
Fixes:
64402a8 ("Expand choices for exitall")
Fixes: https://github.com/axboe/fio/issues/1065
Signed-off-by: André Wild <wild.andre.ae@gmail.com>
Jens Axboe [Fri, 14 Aug 2020 20:16:18 +0000 (13:16 -0700)]
Fixup examples/exitwhat.fio
Use proper read/write names.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Alexey Dobriyan [Thu, 13 Aug 2020 16:36:32 +0000 (19:36 +0300)]
zbd: use ->min_zone, ->max_zone in more places
Check that ->min_zone is less than ->max_zone.
This assert found one bug here.
Use it in more places, now that it was added.
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 12 Aug 2020 17:48:15 +0000 (11:48 -0600)]
io_u: calculate incremental residuals correctly
We need to use the transfer buffer length for the residual calculation,
otherwise multiple shorts on the same IO will not calculate the right
value.
Fixes:
5fff95436922 ("Add support for >= 4G block sizes")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Dmitry Fomichev [Tue, 4 Aug 2020 01:38:28 +0000 (10:38 +0900)]
t/zbd: check log file for failed assertions
Currently, a ZBD test can succeed even if an fio assertion is raised
during its run. Search every ZBD test log file for failed assertions
and fail the test if any were found.
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Dmitry Fomichev [Tue, 4 Aug 2020 01:38:27 +0000 (10:38 +0900)]
zbd: simplify zone reset code
zbd_reset_range() function is only called once from zbd_reset_zone()
and it is always called for an individual zone, not a range.
Make zone reset flow simpler by moving all the code needed
to reset a single zone from zbd_reset_range() to zbd_reset_zone().
Therefore, zbd_reset_range() is now dropped.
zbc_reset_zone() is always called with the zone already locked. Remove
recursive zone locking inside this function and add a note in the
description of this function saying that the caller is expected to have
the zone locked when calling it.
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Dmitry Fomichev [Tue, 4 Aug 2020 01:38:26 +0000 (10:38 +0900)]
configure: check if pkg-config is installed
A few libraries need to be newer than a specific version in order to be
supported by fio and pkg-config utility is used to verify the versions
of the installed libraries. Since this step may fail because pkg-config
is not installed, verify pkg-config presence and warn the user if it
could not be found.
To avoid code duplication, add a common helper function to perform
these checks.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Dmitry Fomichev [Tue, 4 Aug 2020 01:38:25 +0000 (10:38 +0900)]
configure: improve libzbc version check
Avoid parsing pkg-config output and just use --atleast-version to
check if libzbc is present and has an up to date version.
Currently, support for libzbc ioengine is always included if libzbc is
found at the build system. Add the option to disable libzbc ioengine
support even if libzbc is found. This can be useful for
cross-compilation.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 03:40:59 +0000 (21:40 -0600)]
io_u: set io_u->verify_offset in fill_io_u()
This is arguably saner than the previous fix, since it also catches
cases where io_u->offset is adjusted later on.
Hopefully that's it...
Fixes:
4fff54ccba73 ("verify: use origina offset for verification")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 03:31:01 +0000 (21:31 -0600)]
io_u: get_next_offset() should always set io_u->verify_offset
Don't assume that the original get_next_block() is the only place
that sets it. Set it unconditionally at the success exit path
of get_next_offset().
Fixes:
4fff54ccba73 ("verify: use origina offset for verification")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 02:55:27 +0000 (20:55 -0600)]
verify: use origina offset for verification
Requeued IO might have a different offset, since we increment it for
retry conditions. Store the original offset for verification purposes.
Just like we have io_u->xfer_buf/io_u->buf and
io_u->xfer_buflen/io_u->buflen, we really should be treating
io_u->offset the same. But that's a major change, so just add this
special original offset so we know that verify has it. Currently we
treat io_u->offset like we would have io_u->xfer_offset, so this just
makes io_u->verify_offset what io_u->offset should be.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 02:54:46 +0000 (20:54 -0600)]
io_uring: notice short IO on completion path
Async IO engines need to signal residual from the completion path,
but we completely ignore those. Ensure we properly requeue IO that
was short, after adjusting the state.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 02:53:55 +0000 (20:53 -0600)]
io_u: reset ->resid on starting a requeue IO
We only clear this state for a new IO, we should also do it for starting
a requeued IO.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Aug 2020 02:52:09 +0000 (20:52 -0600)]
engines/io_uring: make sure state is updated for requeues
Currently we always use the start of the buffer and buffer length,
but this isn't valid for a requeue. Make sure we updated both
based on the actual io_u.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 8 Aug 2020 00:21:52 +0000 (18:21 -0600)]
Merge branch 'master' of https://github.com/donny372/fio into master
* 'master' of https://github.com/donny372/fio:
Add support for reading iolog from stdin.
donny372 [Wed, 17 Jun 2020 19:58:22 +0000 (12:58 -0700)]
Add support for reading iolog from stdin.
Add support for reading iolog from stdin and update HOWTO and man page.
Fixes: https://github.com/axboe/fio/issues/997
Signed-off-by: Shundong Zhou <donny372@gmail.com>
David, Bar [Thu, 6 Aug 2020 23:08:43 +0000 (02:08 +0300)]
filelock: fix wrong file trylock assertion.
Signed-off-by: David, Bar <bardavvid@gmail.com>
Jens Axboe [Sat, 1 Aug 2020 21:07:38 +0000 (15:07 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio into master
* 'master' of https://github.com/bvanassche/fio:
Prevent that fio hangs when using io_submit_mode=offload
Bart Van Assche [Sat, 1 Aug 2020 18:01:06 +0000 (11:01 -0700)]
Prevent that fio hangs when using io_submit_mode=offload
This patch has been tested by running the following shell command:
for ((i=0;i<1000;i++)); do echo $i; python3 t/run-fio-tests.py -o 10; done
Fixes:
d28174f0189c ("workqueue: ensure we see deferred error for IOs")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Mon, 27 Jul 2020 22:00:20 +0000 (16:00 -0600)]
iolog: ensure that dynamic log entries are at least queue depth sized
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 27 Jul 2020 22:00:03 +0000 (16:00 -0600)]
Add roundup_pow2() as a generic helper
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 27 Jul 2020 04:18:53 +0000 (22:18 -0600)]
t/jobs/t001[1-2].fio: run for 10 seconds instead of 3
These jobs often fails in CI testing, and it might be because the
runtime is simply too short to make it reliable.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 27 Jul 2020 00:13:43 +0000 (18:13 -0600)]
Merge branch 'fix_travis_libiscsi' of https://github.com/sitsofe/fio
* 'fix_travis_libiscsi' of https://github.com/sitsofe/fio:
travis: fix x86 libiscsi detection
travis: remove unneeded dependency
Sitsofe Wheeler [Sun, 26 Jul 2020 19:48:59 +0000 (20:48 +0100)]
travis: fix x86 libiscsi detection
When cross compiling to 32 bit systems, pkg-config has to find 32 bit
libraries. This can be done by one of the following
- Setting PKG_CONFIG_PATH/PKG_CONFIG_LIBDIR to an appropriate directory
- Using a 32 bit pkg-config binary
Choose the later for now by installing a 32 bit pkg-config (which will
replace the 64 bit one). This should fix failing CI on the 32 bit x86
build.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 19:10:28 +0000 (20:10 +0100)]
travis: remove unneeded dependency
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sun, 26 Jul 2020 13:50:46 +0000 (07:50 -0600)]
Merge branch 'enable_or_exit' of https://github.com/sitsofe/fio
* 'enable_or_exit' of https://github.com/sitsofe/fio:
configure: fail when explicit enabling doesn't succeed
Sitsofe Wheeler [Sat, 25 Jul 2020 22:10:27 +0000 (23:10 +0100)]
configure: fail when explicit enabling doesn't succeed
- Bail out if the probe fails when the user is explicitly enabling the
cuda, libiscsi or libnbd ioengines or the libaio io_uring option
- Normalize some of the code that deals with enabling/disabling the
above options
Fixes: https://github.com/axboe/fio/issues/1052
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sun, 26 Jul 2020 13:43:46 +0000 (07:43 -0600)]
Merge branch 'travis_cleanup' of https://github.com/sitsofe/fio
* 'travis_cleanup' of https://github.com/sitsofe/fio:
Makefile: fix incorrectly set libiscsi cflags
memory: fix incorrect pointer comparison when freeing cuda memory
travis: make CI install script bail out on first error
travis: enable libiscsi and cuda ioengines
travis: add dependency for cuda ioengine
travis: use install section instead of before_install section
configure: check for Debian/Ubuntu tcmalloc_minimal
travis: improve installation dependencies
travis: simplify yml file
Sitsofe Wheeler [Sun, 26 Jul 2020 10:27:22 +0000 (11:27 +0100)]
Makefile: fix incorrectly set libiscsi cflags
This fixes the warning
error: -liscsi: 'linker' input unused [-Werror,-Wunused-command-line-argument]
produced by clang.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 06:14:15 +0000 (07:14 +0100)]
memory: fix incorrect pointer comparison when freeing cuda memory
This fixes the following warning spotted by gcc 7:
memory.c: In function ‘free_mem_cudamalloc’:
memory.c:277:22: error: comparison between pointer and integer [-Werror]
if (td->dev_mem_ptr != NULL)
^~
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 05:51:16 +0000 (06:51 +0100)]
travis: make CI install script bail out on first error
This avoids the issue where we carry on when we can't find a requested
package to install.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 04:21:34 +0000 (05:21 +0100)]
travis: enable libiscsi and cuda ioengines
...but only specific Linux setups. We have to avoid enabling cuda on
i386/arm64 as we don't have the dependency in those environments so
introduce a CI_TARGET_ARCH variable and do some refactoring.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 04:07:40 +0000 (05:07 +0100)]
travis: add dependency for cuda ioengine
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 02:31:57 +0000 (03:31 +0100)]
travis: use install section instead of before_install section
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 02:23:27 +0000 (03:23 +0100)]
configure: check for Debian/Ubuntu tcmalloc_minimal
Debian derived distros have a libtcmalloc-minimal4 package that only has
a versioned libtcmalloc_minimal.so.4 inside, so try explicitly searching
for it when looking for tcmalloc.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 02:13:03 +0000 (03:13 +0100)]
travis: improve installation dependencies
- On Ubuntu add bison/flex dependencies for lex arithmetic, add
libgoogle-perftools-dev so we can link against tcmalloc.
- Invoke pip3 just the once on macOS
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 26 Jul 2020 01:43:16 +0000 (02:43 +0100)]
travis: simplify yml file
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sat, 25 Jul 2020 19:55:16 +0000 (13:55 -0600)]
Merge branch 'test-cleanup' of https://github.com/vincentkfu/fio
* 'test-cleanup' of https://github.com/vincentkfu/fio:
t/run-fio-tests: add description to each test result line
ci: set PYTHONUNBUFFERED=TRUE
t/run-fio-tests: fix issues identified by pylint3 and pyflakes3
t/run-fio-tests: catch modprobe exceptions
Jens Axboe [Sat, 25 Jul 2020 19:37:15 +0000 (13:37 -0600)]
Merge branch 'check-atomics' of https://github.com/sitsofe/fio
* 'check-atomics' of https://github.com/sitsofe/fio:
configure: check for C11 atomics support
Vincent Fu [Sat, 25 Jul 2020 17:55:06 +0000 (13:55 -0400)]
t/run-fio-tests: add description to each test result line
Since no one remembers what the test numbers actually refer to,
include either the job name or executable in each test result line.
Vincent Fu [Sat, 25 Jul 2020 17:50:06 +0000 (13:50 -0400)]
ci: set PYTHONUNBUFFERED=TRUE
Set this environment variable so that the build output is consistent
across the different environments.
Vincent Fu [Sat, 25 Jul 2020 17:20:33 +0000 (13:20 -0400)]
t/run-fio-tests: fix issues identified by pylint3 and pyflakes3
- fix whitespace issue in function parameter list
- eliminate extra 'requirements' key for t0012
- t0013 does not need a separate test class
- specify string format arguments as logging function parameters
Vincent Fu [Sat, 25 Jul 2020 17:02:21 +0000 (13:02 -0400)]
t/run-fio-tests: catch modprobe exceptions
Catch exceptions that occur when running modprobe in case the build
environment has it disabled (e.g., AWS CodeBuild).
Sitsofe Wheeler [Sat, 25 Jul 2020 07:46:15 +0000 (08:46 +0100)]
configure: check for C11 atomics support
Since
3932f8be718fc4ca3b863c51c0d567821d75c003 ("arch/arch.h: Introduce
atomic_{load_acquire,store_release}()") fio has needed C11 atomic
support and this is only provided from gcc 4.9 / clang 3.6 onwards
(http://stdatomic.gforge.inria.fr/#sec-2 ). Make this clearer by
doing the following:
- Add configure check for C11 atomics support and bail out with a
(friendly) message pointing at compiler version if it's not there
- Remove the minimum compiler version check from compiler.h as it is now
redundant
I've avoided doing an explicit compiler version check because Apple's
clang can have a wildly different version to upstream
(https://stackoverflow.com/a/
33614612 ).
Tested on:
- CentOS 7 with default gcc 4.8.5 (correctly errors with message)
- CentOS 7 docker container using llvm-toolset-7 to provide clang 5.0.1
(works)
- CentOS 7 docker container using devtool-toolset-7 to provide gcc 7.3.1
(works)
- Ubuntu 16.04 with clang-3.5 (correctly errors with message)
- Ubuntu 16.04 with clang-3.6 installed (works)
- macOS 10.14.6 with clang-1001.0.46.4 (works)
v2:
- Rebase
- Reword failure message and stop mentioning fio version number
Fixes: https://github.com/axboe/fio/issues/1047
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sat, 25 Jul 2020 14:01:03 +0000 (08:01 -0600)]
configure: error early on too old compier
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 24 Jul 2020 17:05:21 +0000 (11:05 -0600)]
Merge branch 'sribs-patch-1039' of https://github.com/sribs/fio
* 'sribs-patch-1039' of https://github.com/sribs/fio:
stat: stop triggerring division by zero on bandwidth lower than 1KBps
io_u: fix exit failure case when using rates and timeout
sribs [Fri, 24 Jul 2020 16:53:50 +0000 (22:23 +0530)]
stat: stop triggerring division by zero on bandwidth lower than 1KBps
Before the integer arithmetic is performed to obtain
percentage of aggregation this conversion of rs->agg[ddir] into KBytes
could potentially turn out to be 0.
Adjust stat.c to no longer trigger division by zero on bandwidths
greater than 0 but less than 1kilobyte a second. The fact that this
is performed on multiple occasions, a function is preferred
Signed-off-by: sribs <sribs@microsoft.com>
sribs [Fri, 24 Jul 2020 10:06:58 +0000 (15:36 +0530)]
io_u: fix exit failure case when using rates and timeout
Previously when using a rate smaller than the minimum block size,
fio could end up failing to exit at a designated timeout and
while doing this, do check if usec takes in negative values which could
potentially not exit fio indefinitely.
Fixes: https://github.com/axboe/fio/issues/1039
Signed-off-by: Sriharsha B S <sribs@microsoft.com>
Shin'ichiro Kawasaki [Wed, 22 Jul 2020 09:26:33 +0000 (18:26 +0900)]
t/zbd: Improve pass condition of test case #49
The test case #49 runs write workloads with verify option. It checks read
bytes for verify, but it does not check written bytes. To make test pass
condition more accurate, add check of the written bytes.
Fixes:
4844bb4716ab ("t/zbd: Add test case to check zonecapacity option")
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 21 Jul 2020 15:40:07 +0000 (09:40 -0600)]
zbd: fix %lu -> %llu dprint() formatting
The types aren't correct for 32-bit builds.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Hans Holmberg [Fri, 17 Jul 2020 09:36:37 +0000 (18:36 +0900)]
t/zbd: Add test case to check zonecapacity option
Add test case #49 which confirms the newly introduced zonecapacity
option is working as expected with regular block devices.
Signed-off-by: Hans Holmberg <hans.holmberg@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Fri, 17 Jul 2020 09:36:36 +0000 (18:36 +0900)]
t/zbd: Support testing zone capacity smaller than zone size with null_blk
To test zone capacity support by fio using null_blk, add -zone-cap option
to run-tests-against-zoned-nullb. With this option, the script creates a
null_blk device which has zone capacity smaller than zone size and run
test-zbd-support script for the device.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Hans Holmberg [Fri, 17 Jul 2020 09:36:35 +0000 (18:36 +0900)]
t/zbd: Support testing zone capacity smaller than zone size
Add support for testing zoned block devices where zone capacity is less
than zone size. If the test case runs sequential workloads or workloads
with verify, use zone capacity instead of zone size to complete.
To use zone capacity of the test target zones, introduce a helper
function total_zone_capacity() which sums up zone capacity numbers of the
test target zones.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Hans Holmberg <hans.holmberg@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Fri, 17 Jul 2020 09:36:34 +0000 (18:36 +0900)]
t/zbd: Mandate blkzone capacity report for devices with zone capacity
For NVMe ZNS and null_blk zoned drives, a zone capacity can be smaller
than its size. To test zone capacity handling by fio for these devices,
t/zbd/test-zbd-support must be able to discover zone capacities to avoid
test failures.
To avoid the failures, make zone capacity report by blkzone mandatory if
the test target device is a ZNS device or a null_blk device.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Hans Holmberg [Fri, 17 Jul 2020 09:36:33 +0000 (18:36 +0900)]
options: Add zonecapacity option for zonemode=zbd
To test zone capacity handling by fio using regular block devices, add
zonecapacity option for zonemode=zbd. This new option allows specifying a
zone capacity smaller than the zone size. Similarly to the zonesize
option, the zonecapacity defined value applies to all emulated zones of
the device.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Hans Holmberg <hans.holmberg@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Fri, 17 Jul 2020 09:36:32 +0000 (18:36 +0900)]
zbd: Support zone capacity smaller than zone size
NVMe ZNS specification defines zone capacity. The report zone interface
of Linux kernel supports it. This patch adds support for this new
interface.
Modify configure to check availability of BLK_ZONE_REP_CAPACITY flag as
well as blkzoned.h header only when target OS is Linux. If this flag is
defined and set in the report zone, all sectors from the zone capacity up
to the zone size are not accessible. Prevent read and write operations to
these sectors by modifying zbd_adjust_block(). Of note is that this skips
the region between zone capacity and zone size then reduces total I/O
bytes of sequential workloads.
Introduce helper functions zbd_zone_end() and zbd_zone_capacity_end() to
improve readability of zone limit check code.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com>
Signed-off-by: Hans Holmberg <hans.holmberg@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 20 Jul 2020 22:37:50 +0000 (16:37 -0600)]
Fio 3.21
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 18 Jul 2020 14:39:04 +0000 (08:39 -0600)]
Merge branch 'patch-1' of https://github.com/avlasov-mos-de/fio
* 'patch-1' of https://github.com/avlasov-mos-de/fio:
Fix scale on gnuplot graphs
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:22 +0000 (13:13 +0900)]
t/zbd: Enable regular block devices for test case #47
The test case #47 was not able to run for regular block devices since it
missed zonesize option required for regular block devices.
Enable the test case for regular block devices by calling the
run_fio_on_seq() helper function which adds the zonesize option.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:21 +0000 (13:13 +0900)]
t/zbd: Remove write before random read/write from test case #17
The test case #17 writes data to the last zone before random read/write.
This avoided read failure when read is chosen for the first operation of
the random read and write. Such failure case was prevented by the commit
fb0259fb276a ("zbd: Ensure first I/O is write for random read/write to
sequential zones"). Then the write is no longer required. Remove it.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:20 +0000 (13:13 +0900)]
t/zbd: Combine write and read fio commands for test case #16
The test case #16 checks result of read job which runs after write job of
test case #15. It does not work if test case #16 is executed alone with
the -t option and the target device has all zones empty.
Specify both the write job and the read job to a single fio command using
write_and_run_one_fio_job() helper function instead of run_one_fio_job().
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:19 +0000 (13:13 +0900)]
t/zbd: Combine write and read fio commands for test case #15
The test case #15 called fio twice. Once with write job to prepare data
to read. The other with read job for testing. But the first run has no
effect on the second run when the zone status is reinitialized.
Specify both the write job and the read job to a single fio command using
write_and_run_one_fio_job() helper function instead of run_one_fio_job().
Also apply same test pass condition for both regular block devices and
zoned block devices.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Wed, 15 Jul 2020 04:13:18 +0000 (13:13 +0900)]
t/zbd: Combine write and read fio commands for test case #6
The test case #6 checks result of the read job which reads data written
by the write job of test case #5. It does not work if test case #6 is
executed alone with the -t option and the target device has all zones
empty.
Specify both the write job and the read job to a single fio command
using write_and_run_one_fio_job() helper function instead of
run_fio_on_seq() helper function.
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>