Bart Van Assche [Thu, 13 Jul 2023 19:44:54 +0000 (12:44 -0700)]
diskutil: Report how many sectors have been read and written
It is useful to know how much data has been read and/or written. Report
this information.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Thu, 13 Jul 2023 17:04:06 +0000 (10:04 -0700)]
diskutil: Fix a debug statement in get_io_ticks()
Report correctly whether or not reading statistics succeeded.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Thu, 13 Jul 2023 17:03:08 +0000 (10:03 -0700)]
diskutil: Simplify get_io_ticks()
Remove the sectors[] array. Remove the set-but-not-used in_flight variable.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Thu, 13 Jul 2023 16:54:30 +0000 (09:54 -0700)]
diskutil: Remove casts from get_io_ticks()
This patch does not change the behavior of the code.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Thu, 13 Jul 2023 16:54:22 +0000 (09:54 -0700)]
diskutil: Improve disk utilization data structure documentation
Document the meaning of the members of struct disk_util_stats.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Vincent Fu [Thu, 13 Jul 2023 16:07:19 +0000 (12:07 -0400)]
options: add code for FDP pli selection use in client/server mode
For every new option, we need to convert it from the host's storage
format to the on-wire protocol and back in order to be able to use it in
client/server mode.
Fixes:
d3e310c531059fb606f04819c362b4d46c518b84 ("fdp: support random placement id selection")
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Ankit Kumar [Wed, 12 Jul 2023 10:20:41 +0000 (15:50 +0530)]
engines/xnvme: add support for fdp
Add FDP support to xnvme I/O engine. This support can be used only with
nvme-ns generic character device (/dev/ngXnY). The available backends are
--xnvme_async=io_uring_cmd and --xnvme_sync=nvme.
Add a xnvme-fdp config example file.
Update the minimum required xnvme version to 0.7.0
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Ankit Kumar [Wed, 12 Jul 2023 10:20:40 +0000 (15:50 +0530)]
fdp: support random placement id selection
Allow user to either roundrobin or select random placement ID from
the available placement IDs.
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Ankit Kumar [Wed, 12 Jul 2023 10:20:39 +0000 (15:50 +0530)]
fdp: fix placement id check
Number of reclaim unit handle descriptors are 1 based, whereas the
input placement id index / indices are 0 based. Add the correct check
for that.
Fixes:
a7e8aae0 ("fio: add fdp support ..")
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Ankit Kumar [Wed, 12 Jul 2023 10:20:38 +0000 (15:50 +0530)]
fdp: use macros
use macros for directive type and max ruhs.
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Bart Van Assche [Wed, 5 Jul 2023 21:29:15 +0000 (14:29 -0700)]
zbd: Support finishing zones on Android
BLKFINISHZONE is missing from older versions of the Android NDK header
files. Hence, define BLKFINISHZONE if it has not been defined and detect
at runtime whether or not the kernel supports finishing zones.
Cc: Damien Le Moal <dlemoal@kernel.org>
Cc: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20230705212915.3373438-1-bvanassche@acm.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 27 Jun 2023 20:17:10 +0000 (20:17 +0000)]
t/nvmept: add trim test with ioengine options enabled
Add a test for a trim workload with ioengine options enabled like the
ones we have for read and write. fixedbufs cannot be enabled for a
trim-only workload because fio does not allocate data buffers for these
workloads.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Tue, 27 Jun 2023 20:16:13 +0000 (20:16 +0000)]
t/nvmept: add check for iodepth
Make sure that we achieve the iodepth specified in the job options.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Tue, 27 Jun 2023 19:19:11 +0000 (19:19 +0000)]
engines/io_uring: remove dead code related to trim
Now that we are no longer using an NVMe IOCTL to send io_uring_cmd trim
commands we can just use the existing pathway to carry out synchronous
trim commands for zoned devices. So we no longer need this code.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Tue, 27 Jun 2023 18:48:53 +0000 (18:48 +0000)]
engines/io_uring_cmd: make trims async
Instead of using a synchronous IOCTL to send a trim/deallocate request, just
use the io_uring pass-through interface to send the dataset management
command with the deallocate request just like we already do for read and
write commands.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Wed, 5 Jul 2023 14:55:30 +0000 (08:55 -0600)]
Merge branch 'makefile-hardening-cpp-flags' of https://github.com/proact-de/fio
* 'makefile-hardening-cpp-flags' of https://github.com/proact-de/fio:
Keep C pre processor hardening build flags.
Martin Steigerwald [Tue, 4 Jul 2023 12:29:14 +0000 (14:29 +0200)]
Keep C pre processor hardening build flags.
This allows for distributions like Debian to apply hardening flags
without patching the makefile.
Signed-off-by: Martin Steigerwald <martin.steigerwald@proact.de>
Jens Axboe [Mon, 3 Jul 2023 15:16:45 +0000 (09:16 -0600)]
fio/server: fix confusing sk_out check
The previous assert check we had here just checked for sk_out->sk being
-1, but if sk_out itself was set.
Fixes:
83276370ce4d ("fixed compiler warnings if NDEBUG enabled in core code")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 3 Jul 2023 14:20:01 +0000 (08:20 -0600)]
Merge branch 'improvement/fix-warnings-if-NDEBUG-enabled' of https://github.com/dpronin/fio
* 'improvement/fix-warnings-if-NDEBUG-enabled' of https://github.com/dpronin/fio:
fixed compiler warnings if NDEBUG enabled in test code
fixed compiler warnings if NDEBUG enabled in core code
Jens Axboe [Mon, 3 Jul 2023 14:19:07 +0000 (08:19 -0600)]
Merge branch 'libaio/actual_min_algo_update' of https://github.com/dpronin/fio
* 'libaio/actual_min_algo_update' of https://github.com/dpronin/fio:
use 'min' macro to find out next value of actual_min in libaio
Vincent Fu [Tue, 20 Jun 2023 18:11:36 +0000 (14:11 -0400)]
init: don't adjust time units again for subjobs
We adjust max_latency and latency_target values to be nsec internally.
Make sure we do this only once for the parent job and don't do it a
second time for a subjob.
Fixes: https://github.com/axboe/fio/issues/1582
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 16 Jun 2023 14:41:25 +0000 (10:41 -0400)]
docs: clarify opendir description
Make explicit how opendir deals with colons in the path.
Fixes: https://github.com/axboe/fio/issues/1573
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Shin'ichiro Kawasaki [Fri, 9 Jun 2023 23:59:14 +0000 (08:59 +0900)]
zbd: avoid Coverity defect report
Coverity reported a defect related to the local variable "in_flight":
Using an unreliable value of "in_flight" inside the second locked
section. If the data that "in_flight" depends on was changed by
another thread, this use might be incorrect.
The variable "in_flight" is thread local and other threads can not
change its value. Then the report should be false-positive. Just to
suppress the report, change reference timing of the valuable.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Link: https://lore.kernel.org/r/20230609235914.1376567-1-shinichiro.kawasaki@wdc.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Shin'ichiro Kawasaki [Thu, 8 Jun 2023 07:06:08 +0000 (16:06 +0900)]
t/zbd: test write zone accounting of trim workload
Recent commit fixed the bug of the write zone accounting of trim
workload. Add a test case which confirms the fix.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Shin'ichiro Kawasaki [Thu, 8 Jun 2023 07:06:07 +0000 (16:06 +0900)]
t/zbd: test write zone accounting of almost full zones
Recent commit fixed the bug of the write zone accounting for almost full
zones. Add a test case which confirms the fix.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Shin'ichiro Kawasaki [Thu, 8 Jun 2023 07:06:06 +0000 (16:06 +0900)]
t/zbd: reset zones before tests with max_open_zones option
After the recent fix, fio no longer resets zones when it finds more
zones in open condition than the max_open_zones option. This results in
failure of test cases 12, 13, 29, 32, 48 and 51. To avoid the failures,
reset zones at the beginning of the test cases.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Shin'ichiro Kawasaki [Thu, 8 Jun 2023 07:06:05 +0000 (16:06 +0900)]
zbd: fix write zone accounting of trim workload
The commit
e3be810bf0fd ("zbd: Support zone reset by trim") supported
trim for zonemode=zbd by introducing the function zbd_do_io_u_trim(),
which calls zbd_reset_zone(). However, it did not call
zbd_write_zone_put() to the trim target zone, then trim operation
resulted in wrong accounting of write zones.
To fix the issue, call zbd_write_zone_put() from zbd_reset_zone(). To
cover the case to reset zones without a zbd_write_zone_put() call,
prepare another function __zbd_reset_zone(). While at it, simplify
zbd_reset_zones() by calling the modified zbd_reset_zone().
Of note is that the constifier of the argument td of do_io_u_trim() is
removed since zbd_write_zone_put() requires changes in that argument.
Fixes:
e3be810bf0fd ("zbd: Support zone reset by trim")
Suggested-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Shin'ichiro Kawasaki [Thu, 8 Jun 2023 07:06:04 +0000 (16:06 +0900)]
zbd: fix write zone accounting of almost full zones
For zonemode=zbd, fio checks condition of each zone and account it as
write target zone if it has open condition. However, when such a zone in
open condition is almost full and its remainder area for write is
smaller than the block size, fio does not handle it as a write target
zone. This causes difference between open zones accounting on the device
and write target zones accounting by fio. It results in unexpected
max_open_zones limit failure.
Avoid the zone accounting difference by handling the almost full zones
as write target zones at fio start. Introduce the helper function
__zbd_write_zone_get() which does same operation as zbd_write_zone_get()
except the check for the almost full zones. At fio start, call
__zbd_write_zone_get() so that almost full zones are added to write
target zones. During fio workload run, call zbd_write_zone_get() so that
the almost full zones are not chosen for write target.
Suggested-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Shin'ichiro Kawasaki [Thu, 8 Jun 2023 07:06:03 +0000 (16:06 +0900)]
zbd: do not reset extra zones in open conditions
The commit
954217b90191 ("zbd: Initialize open zones list referring zone
status at fio start") introduced zone resets for zones in open condition
which exceeds the limit of max_open_zones. However, this zone reset may
break data in the zones even when fio does no write to them. Avoid the
zone reset and report it as an error.
Fixes:
954217b90191 ("zbd: Initialize open zones list referring zone status at fio start")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Shin'ichiro Kawasaki [Thu, 8 Jun 2023 07:06:02 +0000 (16:06 +0900)]
zbd: rename 'open zones' to 'write zones'
Current fio code for zonemode=zbd uses the word 'open zone' to mean the
zones that fio jobs write to. Before fio starts writing to a zone, it
calls zbd_open_zone(). When fio completes writing to a zone, it calls
zbd_close_zone(). This wording is good for zoned block devices with
max_open_zones limit, such as ZBC and ZAC devices. The devices use same
word 'open' to express the zone condition that the devices assign
resources for data write to zones. However, the word 'open' gets
confusing to support zoned block devices which has max_active_zones
limit, such as ZNS devices. These devices have both 'open' and 'active'
keywords to mean two different kinds of resources on the device. This
'active' status does not fit with the 'open zone' wording in the fio
code. Also, the word 'open' zone in fio code does not always match with
the 'open' condition of zones on the device (e.g. when
--ignore_zone_limits option is specified).
To avoid the confusion, stop using the word 'open zone' in the fio code.
Instead, use the word 'write zone' to mean that the zone is the write
target. When fio starts a write to a zone, it adds the zone to
write_zones array. When fio completes writing to a zone, it removes the
zone from the write_zones array. For this purpose, rename struct fields,
functions and a macro:
ZBD_MAX_OPEN_ZONES -> ZBD_MAX_WRITE_ZONES
struct fio_zone_info
open -> write
struct thread_data
num_open_zones -> num_write_zones
struct zoned_block_device_info:
max_open_zones -> max_write_zones
num_open_zones -> num_write_zones
open_zones[] -> write_zones[]
zbd_open_zone() -> zbd_write_zone_get()
zbd_close_zone() -> zbd_write_zone_put()
zbd_convert_to_open_zone() -> zbd_convert_to_write_zone()
To match up these changes, rename local variables and goto labels. Also
rephrase code comments.
Of note is that this rename is only for the fio code. The fio options
max_open_zones and job_max_open_zones are not renamed to not confuse
users.
Suggested-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 7 Jun 2023 22:45:49 +0000 (22:45 +0000)]
t/strided: increase minumum recommended size to 64MiB
Test 8 has offset=32M. So it fails with a 32MiB file. Increase the
minimum recommended test file/device size to 64MiB.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 7 Jun 2023 21:24:40 +0000 (21:24 +0000)]
t/strided: adapt to use fiotestlib
Use the FioJobCmdTest class and the run_fio_test runner to execute the
tests.
Also update run-fio-tests.py to accommodate the new way of specifying
the path to the fio executable.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 7 Jun 2023 21:21:50 +0000 (21:21 +0000)]
t/fiotestlib: add ability to ingest iops logs
Enhance the FioJobCmdTest class with the ability to read in an IOPS log
if one was generated during the course of the test run.
This reads in only the first IOPS log written as a result of
--write_iops_log.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 7 Jun 2023 16:49:18 +0000 (16:49 +0000)]
t/nvmept: adapt to use fiotestlib
Use the FioJobCmdTest class and the test runner from fiotestlib.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 7 Jun 2023 15:21:31 +0000 (15:21 +0000)]
t/readonly: adapt to use fiotestlib
Use the test runner and testclass provided in fiotestlib.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 7 Jun 2023 00:00:23 +0000 (20:00 -0400)]
t/random_seed: fixes from pylint
- Use % formatting for logging.debug
- Eliminate f-string without interpolated values
- Iterate with items() instead of just over keys
- Remove unused subprocess import
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 7 Jun 2023 13:49:18 +0000 (13:49 +0000)]
t/random_seed: use methods provided in fiotestlib to run tests
Adapt random_seed.py to use the new FioJobCmdTest class. Also use the
test runner from fiotestlib. This reduces code duplication.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Tue, 6 Jun 2023 22:13:15 +0000 (22:13 +0000)]
t/random_seed: use logging module for debug prints
Instead of having 'if self.debug:' lines all over the place, use the
logging module for debug messages.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 7 Jun 2023 13:47:53 +0000 (13:47 +0000)]
t/fiotestlib: add class for command-line fio job
Add a separate class for running a fio job with all of the options
provided on the command line (instead of in a job file).
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 7 Jun 2023 00:18:34 +0000 (20:18 -0400)]
t/fiotestlib: record test command in more useful format
Instead of recording the test command as a Python object, record it as a
string with spaces delimiting each of the paramters. This way we can use
it to directly issue the command in a shell.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Tue, 6 Jun 2023 20:38:21 +0000 (20:38 +0000)]
t/fiotestlib: rearrange constructor and setup steps
Make the constructor and setup steps more sensible. The constructor now
sets variables that should be fixed throughout the execution of the
test. Some of the setup steps may depend on values set by the
constructor. So keeps those separate.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Mon, 5 Jun 2023 18:06:17 +0000 (18:06 +0000)]
t/fiotestlib: use f-string for formatting
Take advantage of running on modern Python versions and use an f-string
for formatting.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Mon, 5 Jun 2023 17:59:55 +0000 (17:59 +0000)]
t/fiotestlib: use 'with' for opening files
Use with when opening files to relieve us of the need to explicitly
close the files.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Mon, 5 Jun 2023 17:46:19 +0000 (17:46 +0000)]
t/fiotestlib: use dictionaries for filenames and paths
Instead of separate variables for all the different filenames and paths,
use dictionaries for the filenames and paths.
This involves changing the function signature for the constructor and
setup() functions so that parameters are passed to setup() instead of
the constructor. This is because the base class defines the filenames
dictionary but FioJobFileTest needs to put an entry into the dictionary
before calling the base class constructor.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 2 Jun 2023 19:04:31 +0000 (19:04 +0000)]
t/run-fio-tests: move get_file outside of FioJobFileTest
There's no reason for this helper function to be part of a class. Just
make it an independent helper.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 2 Jun 2023 18:55:08 +0000 (18:55 +0000)]
t/run-fio-tests: rename FioJobTest to FioJobFileTest
We have two different types of tests based on fio jobs. The ones
referred to by FioJobFileTest consist of running a fio job file. It's
also possible to run a fio job test with options specified on the
command line. Such a class will be defined later.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 2 Jun 2023 16:25:50 +0000 (16:25 +0000)]
t/run-fio-tests: split source file
This is the first step in creating a test library that can be used by
other python test scripts to reduce code duplication.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Thu, 1 Jun 2023 14:20:17 +0000 (14:20 +0000)]
t/run-fio-tests: address issues identified by pylint
- use f strings
- use Python3 syntax for parent classes
- other small cleanups
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 31 May 2023 19:55:55 +0000 (19:55 +0000)]
t/run-fio-tests: integrate t/nvmept.py
Running t/nvmept.py requires that the test runner provide an NVMe
character device test target. t/nvmept.py will be skipped unless an NVMe
character device test target is provided.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 26 May 2023 23:05:12 +0000 (23:05 +0000)]
t/nvmept.py: test script for io_uring_cmd NVMe pass through
Test basic functionality of the io_uring_cmd ioengine with NVMe
pass-through commands.
This script starts with basic sequential and random read, write, trim,
readwrite, and trimwrite workloads. It also runs sequential and random
write workloads with verification. Finally it runs random read and write
workloads with almost all of the ioengine options enabled. The only
ioengine option not exercised is hipri because the NVMe driver must be
loaded with poll_queues set for this to work.
Since this is a destructive test the target device must be specified on
the command line.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Sat, 27 May 2023 00:09:53 +0000 (20:09 -0400)]
ci: disable tls for msys2 builds
The same tls issue that occurred with AppVeyor msys2 builds has appeared
in GitHub Actions msys2 builds. Disable thread local storage for GitHub
Actions msys2 builds as well.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Tue, 25 Apr 2023 18:22:26 +0000 (18:22 +0000)]
ci: upload tagged GitHub Actions Windows installers as releases
Since we are no long using AppVeyor for Windows tests, we can use the
installers built by GitHub Actions for Windows releases.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 24 May 2023 13:40:55 +0000 (09:40 -0400)]
ci: stop using AppVeyor for Windows builds
Stop running Windows tests using AppVeyor since it only runs tests
serially. GitHub Actions Windows tests have been running for a few weeks
and seem to be working ok.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Tue, 23 May 2023 18:33:03 +0000 (12:33 -0600)]
Fio 3.35
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 23 May 2023 12:42:56 +0000 (08:42 -0400)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
zbd: Report the zone capacity
zbd: Make an error message more detailed
Bart Van Assche [Wed, 22 Feb 2023 21:54:18 +0000 (13:54 -0800)]
zbd: Report the zone capacity
The zone capacity is important information. Hence report the zone
capacity if it is identical for all zones and if ZBD debugging is enabled.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Sat, 20 May 2023 01:30:38 +0000 (19:30 -0600)]
io_ur: make sure that sync errors are noticed upfront
This could probably be cleaner in the error handling, but jump to
the normal error handling case for ddir_sync() as well.
Fixes: https://github.com/axboe/fio/issues/1577
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 19 May 2023 16:38:56 +0000 (09:38 -0700)]
zbd: Make an error message more detailed
Knowing which part of a zone report is invalid is useful when debugging
firmware of zoned devices. Hence report the number of zones that have
been parsed successfully when reporting a report zones error.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Fri, 19 May 2023 14:21:04 +0000 (08:21 -0600)]
Merge branch 'master' of https://github.com/huajingyun01/fio
* 'master' of https://github.com/huajingyun01/fio:
Add LoongArch64 support
Jingyun Hua [Fri, 19 May 2023 09:14:32 +0000 (17:14 +0800)]
Add LoongArch64 support
Signed-off-by: Jingyun Hua <huajingyun@loongson.cn>
Vincent Fu [Thu, 18 May 2023 17:25:07 +0000 (13:25 -0400)]
ci: fix ups for 32-bit GitHub Actions Linux builds
Fix the dependency install problem for 32-bit builds by explicitly
installing libc6:i386 and libgcc-s1:i386 as mentioned in the link below.
https://github.com/actions/runner-images/issues/4589#issuecomment-
1552409942
Deleting microsoft-prod.list and adding --allow-downgrades were not
needed for our use case.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Thu, 18 May 2023 17:23:49 +0000 (13:23 -0400)]
Revert "ci: stop testing Linux 32-bit builds"
This reverts commit
83b2d4b78374055c3a2261136eedf03b5fbfc335.
There now seems to be a solution to the 32-bit GitHub Actions Ubuntu
breakage.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 17 May 2023 14:44:49 +0000 (14:44 +0000)]
docs: document no_completion_thread
Describe in the HOWTO and man page the windowsaio ioengine
no_completion_thread option.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 17 May 2023 14:25:02 +0000 (14:25 +0000)]
docs: move experimental_verify description
Move the description for experimental_verify next to the other
verify-related options.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 17 May 2023 14:21:59 +0000 (14:21 +0000)]
docs: move rate_cycle description
Move the description for rate_cycle from the I/O latency section to the
I/O rate section.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Ankit Kumar [Tue, 16 May 2023 12:17:17 +0000 (17:47 +0530)]
engines/io_uring: fix coverity issue
*** CID 455020: Integer handling issues (BAD_SHIFT)
/engines/io_uring.c: 1201 in fio_ioring_cmd_open_file()
In expression "1 << data->lba_shift", left shifting by more than 31 bits
has undefined behavior. The shift amount, "data->lba_shift", is
4294967295
Fixes:
345fa8f ("engines/io_uring_cmd: add extended LBA support")
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20230516121717.28508-2-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Mon, 15 May 2023 12:51:27 +0000 (08:51 -0400)]
ci: stop testing Linux 32-bit builds
The GitHub Actions image has had a longstanding problem with the
dependencies required for 32-bit builds. The current work around no
longer fixes the issue. Stop testing 32-bit builds to avoid false
positive failures with our CI testing.
We can re-enable 32-bit builds when this issue is fixed upstream.
Details: https://github.com/actions/runner-images/issues/4589
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Ankit Kumar [Mon, 15 May 2023 11:03:40 +0000 (16:33 +0530)]
engines/io_uring_cmd: add extended LBA support
The io_uring_cmd ioengine assumes that logical block size is always
power of 2. But with namespace formats where metadata is transferred at
the end of logical block i.e. an extended logical block this is not
true.
This patch calculates the correct extended logical block size and uses
division operation for start and number of logical block calculation.
The existing calculation for power of 2 logical block size will remain
same.
This also add checks to verify that block sizes are multiple of LBA
size.
This current implementation however doesn't support protection info and
metadata transferred as separate buffer. Return error for those.
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
[Vincent: edited commit message]
Ankit Kumar [Mon, 15 May 2023 11:03:39 +0000 (16:33 +0530)]
engines/nvme: support for 64 LBA formats
The NVM command set specification 1.0c supports 64 LBA formats.
The 0-based nlbaf field specifies the number of LBA formats.
The flbas field is used to calculate the current LBA format, in which
bit 0-3 indicates lsb and bit 5-6 indicates msb of the format index.
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Thu, 11 May 2023 00:10:56 +0000 (20:10 -0400)]
docs: expand description for interval-based bw and iops statistics
Update HOWTO.rst and fio.1 to provide more details about the descriptive
statistics interval-based bw and iops measurements.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 10 May 2023 21:29:02 +0000 (17:29 -0400)]
t/run-fio-test: fix comment
The test job that measures rates is actually t0011.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Wed, 10 May 2023 15:16:55 +0000 (09:16 -0600)]
engines/rdma: remove dead code
We can never reach this branch.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 10 May 2023 15:16:01 +0000 (09:16 -0600)]
t/read-to-pipe-async: remove dead code
'work' will be overwritten shortly, no point in clearing it to NULL.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 10 May 2023 14:04:40 +0000 (08:04 -0600)]
README: remove reference to the bsdio installer
These are no longer being updated, and we do have images on the
github page. Remove the reference.
Suggested-by: Rebecca Cran <rebecca@bsdio.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Denis Pronin [Tue, 9 May 2023 09:11:38 +0000 (12:11 +0300)]
use 'min' macro to find out next value of actual_min in libaio
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Jens Axboe [Thu, 27 Apr 2023 23:08:41 +0000 (17:08 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
ci: Also test the Android recovery environment
Detect ASharedMemory_create() support
Bart Van Assche [Thu, 27 Apr 2023 19:57:00 +0000 (12:57 -0700)]
ci: Also test the Android recovery environment
The android library is not available in the Android recovery
environment. Add a CI test for building and linking fio without the
android library.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Thu, 27 Apr 2023 19:57:00 +0000 (12:57 -0700)]
Detect ASharedMemory_create() support
Make linking with the android library optional.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Anuj Gupta [Thu, 27 Apr 2023 19:44:15 +0000 (01:14 +0530)]
t/io_uring: avoid null-ptr dereference in case setup_ring fails
s->sq_ring.ring_entries and s->cq_ring.ring_entries will be NULL,
incase setup_ring fails. This will cause a segmentation fault. Avoid
dereferencing them in such a scenario.
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://lore.kernel.org/r/20230427194415.1160701-1-anuj20.g@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Wed, 26 Apr 2023 22:04:34 +0000 (22:04 +0000)]
ci: work around for GitHub Actions Cygwin sed issue
gcc and other compilers create dependency files that are included in
the Makefile for builds subsequent to the first build after a clean
checkout.
Dependency files produced by gcc look like this:
parse.o: parse.c config-host.h compiler/compiler.h parse.h flist.h \
debug.h lib/types.h log.h lib/output_buffer.h options.h optgroup.h \
minmax.h lib/ieee754.h lib/pow2.h lib/types.h
The Makefile rule for .o files manipulates the gcc dependencies file to
look like this:
parse.o: parse.c config-host.h compiler/compiler.h parse.h flist.h \
debug.h lib/types.h log.h lib/output_buffer.h options.h optgroup.h \
minmax.h lib/ieee754.h lib/pow2.h lib/types.h
parse.c:
config-host.h:
compiler/compiler.h:
parse.h:
flist.h:
debug.h:
lib/types.h:
log.h:
lib/output_buffer.h:
options.h:
optgroup.h:
minmax.h:
lib/ieee754.h:
lib/pow2.h:
lib/types.h:
For some reason the GitHub Actions Windows Cygwin sed does not execute
-e 's/\\$$//' properly to remove the backslashes at the end of each
line. On this platform the dependencies file after processing looks
like:
parse.o: parse.c config-host.h compiler/compiler.h parse.h flist.h \
debug.h lib/types.h log.h lib/output_buffer.h options.h optgroup.h \
minmax.h lib/ieee754.h lib/pow2.h lib/types.h
parse.c:
config-host.h:
compiler/compiler.h:
parse.h:
flist.h:
\:
debug.h:
lib/types.h:
log.h:
lib/output_buffer.h:
options.h:
optgroup.h:
\:
minmax.h:
lib/ieee754.h:
lib/pow2.h:
lib/types.h:
Once these dependency files are created subsequent invocations of make
(i.e., 'make test') produce Makefile parsing errors like:
parse.d:9: *** missing separator. Stop.
All of this works fine on GitHub Actions msys2 and AppVeyor's Cygwin and
msys2.
Work around this problem by deleting the dependencies files before
running the tests.
For more details describing the processing of dependency files see
https://scottmcpeak.com/autodepend/autodepend.html
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 21 Apr 2023 18:56:59 +0000 (18:56 +0000)]
ci: add Windows Cygwin and msys2 builds to GitHub Actions
AppVeyor will only run builds one at a time whereas GitHub Actions will
run all builds in parallel.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Thu, 27 Apr 2023 13:20:53 +0000 (07:20 -0600)]
t/io_uring: make submitter_init() return < 0 on error
Rather than fudge around with setting some finish variables, have
it error it directly and have the callers actually check for that.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Niklas Cassel [Thu, 27 Apr 2023 09:24:23 +0000 (11:24 +0200)]
Revert "zbd: Report the zone capacity"
This reverts commit
067c18eb3700373d029a9f2795d662453fc09cf4.
In the Linux zoned block devices API (blkzoned.h) (and in the NVMe Zoned
Namespace Command Set Specification), the zone capacity can be different
for each zone (unlike zone size, which has to be the same for all zones).
In order to not mislead the user to think that the zone capacity has to
be the same for all zones, remove the zone capacity for zone0 from the
per device print.
The zone capacity can be printed in prints that are related to a specific
zone, e.g. when encountering an error when performing I/O to a zone.
Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Link: https://lore.kernel.org/r/20230427092423.605250-1-nks@flawful.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Thu, 20 Apr 2023 15:12:22 +0000 (15:12 +0000)]
filesetup: better handle non-uniform distributions
When we have a random workload with multiple files, randrepeat=0, and a
non-uniform random distribution, the offsets touched will follow the
same sequence for all files:
$ ./fio --name=test --nrfiles=2 --randrepeat=0 --filesize=16k \
--debug=io --rw=randread --norandommap \
--random_distribution=normal:50 | grep complete:
io 23042 complete: io_u 0x55bd982f6000: off=0x2000,len=0x1000,ddir=0,file=test.0.0
io 23042 complete: io_u 0x55bd982f6000: off=0x2000,len=0x1000,ddir=0,file=test.0.1
io 23042 complete: io_u 0x55bd982f6000: off=0x3000,len=0x1000,ddir=0,file=test.0.0
io 23042 complete: io_u 0x55bd982f6000: off=0x3000,len=0x1000,ddir=0,file=test.0.1
io 23042 complete: io_u 0x55bd982f6000: off=0x2000,len=0x1000,ddir=0,file=test.0.0
io 23042 complete: io_u 0x55bd982f6000: off=0x2000,len=0x1000,ddir=0,file=test.0.1
io 23042 complete: io_u 0x55bd982f6000: off=0x3000,len=0x1000,ddir=0,file=test.0.0
io 23042 complete: io_u 0x55bd982f6000: off=0x3000,len=0x1000,ddir=0,file=test.0.1
Notice that the blocks touched for test.0.0 and test.0.1 follow the same
sequence.
This patch allows the sequence of offsets touched to differ between
files by always involving the filename in the seed used for each file.
The randrepeat setting will still be respected as it is involved in
determining the value for td->rand_seeds[FIO_RAND_BLOCK_OFF].
With the patch applied the above invocation produces output like:
io 23022 complete: io_u 0x55ed2cd2c000: off=0x2000,len=0x1000,ddir=0,file=test.0.0
io 23022 complete: io_u 0x55ed2cd2c000: off=0x2000,len=0x1000,ddir=0,file=test.0.1
io 23022 complete: io_u 0x55ed2cd2c000: off=0x3000,len=0x1000,ddir=0,file=test.0.0
io 23022 complete: io_u 0x55ed2cd2c000: off=0x2000,len=0x1000,ddir=0,file=test.0.1
io 23022 complete: io_u 0x55ed2cd2c000: off=0x2000,len=0x1000,ddir=0,file=test.0.0
io 23022 complete: io_u 0x55ed2cd2c000: off=0x3000,len=0x1000,ddir=0,file=test.0.1
io 23022 complete: io_u 0x55ed2cd2c000: off=0x1000,len=0x1000,ddir=0,file=test.0.0
io 23022 complete: io_u 0x55ed2cd2c000: off=0x2000,len=0x1000,ddir=0,file=test.0.1
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Sat, 15 Apr 2023 00:52:47 +0000 (00:52 +0000)]
fio: replace malloc+memset with calloc
Clean up the code base by replacing malloc+memset with calloc. This
patch was generated from the Coccinelle script below.
The script below is inspired by similar scripts used elsewhere:
https://lore.kernel.org/linux-btrfs/cover.
1443546000.git.silvio.fricke@gmail.com/
https://github.com/coccinelle/coccinellery/blob/master/simple_kzalloc/simple_kzalloc1.cocci
@@
expression x,y;
statement s;
type T;
@@
-x = malloc(y * sizeof(T));
+x = calloc(y, sizeof(T));
(
if (!x) s
|
if (x == NULL) s
|
)
-memset(x, 0, y * sizeof(T));
@@
expression x,y,z;
statement s;
@@
-x = malloc(y * sizeof(z));
+x = calloc(y, sizeof(z));
(
if (!x) s
|
if (x == NULL) s
|
)
-memset(x, 0, y * sizeof(z));
@@
expression e,x;
statement s;
@@
-x = malloc(e);
+x = calloc(1, e);
(
if (!x) s
|
if (x == NULL) s
|
)
-memset(x, 0, e);
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Sat, 15 Apr 2023 00:51:44 +0000 (00:51 +0000)]
engines: separate declaration and assignment
Separate the declaraction and assignment for some variables.
This is a prep patch for a Coccinelle script that converts
mallloc+memset(,0,) to calloc.
Part of this patch was created using the Coccinelle script below.
@@
identifier x;
expression y;
statement s;
type T;
@@
-T x = malloc(y);
+T x;
+x = malloc(y);
(
if (!x) s
|
if (x == NULL) s
|
)
memset(x, 0, y);
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Sat, 15 Apr 2023 00:50:58 +0000 (00:50 +0000)]
engines: cleanup casts and move memset
Drop type casting of malloc's return value. Move a memset call closer to
its corresponding malloc call.
This is a prep patch for a Coccinelle script that converts
malloc+memset(,0,) to calloc.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 19 Apr 2023 14:40:07 +0000 (10:40 -0400)]
ci: disable __thread support for Windows msys2 build
Our Windows msys2 build is now broken after AppVeyor's recent image
update from clang/lld 15.0.7-3 to 16.0.0-1.
Here is the error:
CC unittests/oslib/strcasestr.o
CC unittests/oslib/strsep.o
LINK fio
ld.lld: error: undefined symbol: static_tv_valid
>>> referenced by gettime.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Rishabh Shukla investigated and found that clang + ld builds
successfully. So the problem seems to be with lld 16.0.0-1.
Until we find the root cause let's just disable __thread support on
AppVeyor msys2 builds.
Link: https://github.com/axboe/fio/issues/1559
Suggested-by: Rishabh Shukla <rishabh.sh@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 24 Mar 2023 18:46:07 +0000 (18:46 +0000)]
docs: update documentation for randrepeat and allrandrepeat
Note that allrandrepeat is an alias for randrepeat and that randrepeat
actually sets random seeds for all of fio's main random number
generators.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 24 Mar 2023 14:16:14 +0000 (14:16 +0000)]
test: improve evaluation of t0020.fio and t0021.fio
We previously used an arbirtrary test based on a count of consecutive
offsets to decide whether the offsets from t0020.fio and t0021.fio were
random or not. Replace this with a Wald-Wolfowitz runs test. This new
test calculates a statistic based on whether each offset is in the first
or second half of the LBA space. So it is not particularly powerful
although it has a more sound statistical justification than what we
originally had.
Recent changes to fio's default random number seeds produced false
positives with t0020.fio and prompted this change.
Consider calculating the test statistic directly instead of relying on
the statsmodel import in order to reduce the dependencies required.
References:
https://www.statology.org/runs-test-python/
https://www.geeksforgeeks.org/runs-test-of-randomness-in-python/
https://www.investopedia.com/terms/r/runs_test.asp
https://www.statsmodels.org/stable/generated/statsmodels.sandbox.stats.runs.runstest_1samp.html
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Thu, 23 Feb 2023 22:53:34 +0000 (22:53 +0000)]
t/random_seed: python script to test random seed options
This new script tests combinations of randrepeat, allrandrepeat, and
randseed.
Also add this script to t/run-fio-tests.py.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 24 Mar 2023 18:16:29 +0000 (18:16 +0000)]
init: clean up random seed options
- make allrandrepeat a synonym of randrepeat. allrandrepeat is
superfluous because the seeds set by randrepeat already encompass
random number generators beyond the one used for random offsets.
- allow randseed to override [all]randrepeat: this is what the
documentation implies but was not previously the case
This is a breaking change for users relying on the values of fio's
default random seeds.
Link: https://github.com/axboe/fio/pull/1546
Fixes: https://github.com/axboe/fio/issues/1502
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 24 Mar 2023 18:08:25 +0000 (18:08 +0000)]
init: get rid of td_fill_rand_seeds_internal
Do all the work of td_fill_rand_seeds_internal in td_fill_rand_seeds
since td_fill_rand_seeds was basically emptied by the previous patch.
No functional change.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 24 Mar 2023 17:47:14 +0000 (17:47 +0000)]
init: refactor random seed setting
td->rand_seed was modified in three different places. Put all this code
in setup_random_seeds() to make it easier to understand and more
maintanable. Also put setup_random_seeds() next to the other
random-seed-related functions in init.c.
init_rand_seed() was called in three different places for fio's main random
number generators. Also put these three sets of invocations in the same
place.
Always initialize all of fio's main set of random states instead of
skipping some for sequential workloads. This makes debugging easier.
No functional change.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Tue, 21 Feb 2023 22:53:00 +0000 (22:53 +0000)]
rand: print out random seeds for debugging
In order to debug how we handle random seeds let's print out the random
seeds we use in the FD_RANDOM debug output.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Xiaoguang Wang [Thu, 13 Apr 2023 10:47:14 +0000 (18:47 +0800)]
t/io_uring: fix max_blocks calculation in nvme passthrough mode
nvme_id_ns's nsze has already been counted in logical blocks, so
there is no need to divide by bs.
Signed-off-by: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230413104714.57703-1-xiaoguang.wang@linux.alibaba.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 10 Apr 2023 21:40:45 +0000 (15:40 -0600)]
Merge branch 'libaio-hang' of https://github.com/lrumancik/fio
* 'libaio-hang' of https://github.com/lrumancik/fio:
engines/io_uring: update getevents max to reflect previously seen events
Leah Rumancik [Mon, 10 Apr 2023 18:57:15 +0000 (11:57 -0700)]
engines/io_uring: update getevents max to reflect previously seen events
To ensure we don't return more than the requested number of events,
update the max events variable after reaping events before subsequent
calls to io_getevents system call.
Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>
Jens Axboe [Fri, 7 Apr 2023 22:42:07 +0000 (16:42 -0600)]
Merge branch 'libaio-hang' of https://github.com/lrumancik/fio
* 'libaio-hang' of https://github.com/lrumancik/fio:
engines/libaio: fix io_getevents min/max events arguments
Leah Rumancik [Fri, 7 Apr 2023 20:41:03 +0000 (13:41 -0700)]
engines/libaio: fix io_getevents min/max events arguments
If the io_getevents system call is interrupted, it can succeed while
returning less than the min_nr requested events. If this occurs during
the final reaping of events, libaio will call io_getevents again wtih
the same minimum events argument. Since some of the events have already
been reaped, this results in a hang. To fix this, update the arguments
for io_getevents based on the previously seen events.
Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>