fio.git
2 years agoci: Reporting installed msys2 packages
Erwan Velu [Wed, 2 Jun 2021 13:52:06 +0000 (15:52 +0200)]
ci: Reporting installed msys2 packages

When reproducing a build locally, it's important to be on the same
release as the CI.

So let's listi the installed packages so we can compare the two builds more easily.

Signed-off-by: Erwan Velu <e.velu@criteo.com>
2 years agoci: Installing missing toolchain
Erwan Velu [Wed, 2 Jun 2021 13:05:17 +0000 (15:05 +0200)]
ci: Installing missing toolchain

When trying to rebuild a failed build on a real windows system,
the toolchain is missing.

Let's add the toolchain here so we can reuse the script locally too.

Signed-off-by: Erwan Velu <e.velu@criteo.com>
2 years agozbd: add a new --ignore_zone_limits option
Niklas Cassel [Thu, 27 May 2021 11:12:32 +0000 (11:12 +0000)]
zbd: add a new --ignore_zone_limits option

In commit d2f442bc0bd5 ("ioengines: add get_max_open_zones zoned block
device operation") we added a check that verifies that the specified
--max_open_zones value is lower than the max value reported by the device.

For ZNS devices there is a max open zones and a max active zones limit.
For ZAC/ZBC devices there is only a max open zones limit.

On ZAC/ZBC, there is thus no limit on the amount of zones that can be
in zone state closed.
When doing a write to an empty or closed zone, a ZAC/ZBC drive will
close an arbitrary implicit open zone in order to handle the write.

The ZNS specification has no requirement on closing a zone in order to
handle a write to an empty or closed zone. The drive is free to return
an error.

Even on ZAC/ZBC, you do not want to exceed the max open zones limit,
since it will lead to additional implicit close zone and implicit open
zone operations, which may degrade performance.
However, it seems that this is sometimes done on purpose, in order to
measure the overhead of these additional operations. Therefore, add
an option that allows the user to ignore the reported device limits.

Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agozbd: add missing client/server support for option max_open_zones
Niklas Cassel [Thu, 27 May 2021 11:12:31 +0000 (11:12 +0000)]
zbd: add missing client/server support for option max_open_zones

Ensure that we convert the max_open_zones option for client/server.

Use __cpu_to_le32()/__le32_to_cpu() rather than
cpu_to_le32()/le32_to_cpu(), since max_open_zones is defined
as int rather than unsigned int in thread_options.h.

Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'fix-libpmem' of https://github.com/lukaszstolarczuk/fio
Jens Axboe [Thu, 27 May 2021 22:04:16 +0000 (16:04 -0600)]
Merge branch 'fix-libpmem' of https://github.com/lukaszstolarczuk/fio

* 'fix-libpmem' of https://github.com/lukaszstolarczuk/fio:
  engines/libpmem: do not call drain on close
  engines/libpmem: cleanup a little code, comments and example
  engines/libpmem: set file open/create mode always to RW

2 years agoFio 3.27 fio-3.27
Jens Axboe [Wed, 26 May 2021 16:10:32 +0000 (10:10 -0600)]
Fio 3.27

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'master' of https://github.com/DevriesL/fio
Jens Axboe [Tue, 25 May 2021 22:54:13 +0000 (16:54 -0600)]
Merge branch 'master' of https://github.com/DevriesL/fio

* 'master' of https://github.com/DevriesL/fio:
  android: add support for NDK sharedmem

2 years agoandroid: add support for NDK sharedmem
DevriesL [Tue, 25 May 2021 15:45:11 +0000 (23:45 +0800)]
android: add support for NDK sharedmem

Android add support for NDK sharedmem since API level 26 and prohibit
the directly use of ashmem since API level 29, so we can use sharedmem
if targeting API level is higher than 26.

Signed-off-by: DevriesL <therkduan@gmail.com>
2 years agoMerge branch 'taras/nfs-upstream' of https://github.com/tarasglek/fio-1
Jens Axboe [Tue, 18 May 2021 23:34:38 +0000 (17:34 -0600)]
Merge branch 'taras/nfs-upstream' of https://github.com/tarasglek/fio-1

* 'taras/nfs-upstream' of https://github.com/tarasglek/fio-1:
  clean up nfs example
  skip skeleton comments
  single line bodies
  C-style comments
  NFS configure fixes
  NFS engine

2 years agoMerge branch '2021-05-13/stat-fix-integer-overflow' of https://github.com/flx42/fio
Jens Axboe [Fri, 14 May 2021 15:36:59 +0000 (09:36 -0600)]
Merge branch '2021-05-13/stat-fix-integer-overflow' of https://github.com/flx42/fio

* '2021-05-13/stat-fix-integer-overflow' of https://github.com/flx42/fio:
  stat: fix integer overflow in convert_agg_kbytes_percent

2 years agoengines/libzbc: add support for the get_max_open_zones io op
Niklas Cassel [Fri, 14 May 2021 12:53:15 +0000 (12:53 +0000)]
engines/libzbc: add support for the get_max_open_zones io op

Add support for the new .get_max_open_zones io operation.

zbc.c will only ever call this callback for host-managed devices.

Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoioengines: add get_max_open_zones zoned block device operation
Niklas Cassel [Fri, 14 May 2021 12:53:14 +0000 (12:53 +0000)]
ioengines: add get_max_open_zones zoned block device operation

Define a new IO engine operation to get the maximum number of open zones.
Like the existing IO engine operations: .get_zoned_model, .report_zones,
and .reset_wp, this new IO engine operation is only valid for zoned block
devices.

Similarly to the other zbd IO engine operations, also provide a default
implementation inside oslib/linux-blkzoned.c that will be used if the
ioengine does not override it.

The default Linux oslib implementation is implemented similarly to
blkzoned_get_zoned_model(), i.e. it will return a successful error code
even when the sysfs attribute does not exist.
This is because the sysfs max_open_zones attribute was introduced first
in Linux v5.9.
All error handling is still there, so an ioengine that provides its own
implementation will still have its error code respected properly.

Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooslib/linux-blkzoned: move sysfs reading into its own function
Niklas Cassel [Fri, 14 May 2021 12:52:51 +0000 (12:52 +0000)]
oslib/linux-blkzoned: move sysfs reading into its own function

Move the sysfs reading into its own function so that it can be reused.
This new function will be reused in a following patch.

No functional change intended.

Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agozbd: only put an upper limit on max open zones once
Niklas Cassel [Fri, 14 May 2021 12:52:51 +0000 (12:52 +0000)]
zbd: only put an upper limit on max open zones once

There is an upper limit that is checked for each td, and for each file,
even though a file has a pointer to a zoned_block_device_info that has
already been created. Multiple files, from the same or from another td
can point to the same zoned_block_device_info.
All zoned_block_device_info:s have already been created earlier in the
call chain.

Simplify this by only checking the upper limit on max open zones when a
zoned_block_device_info is created.

This way, max_open_zones is handled from a single location, instead of
potentially being reassigned from a completely different location.

Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agostat: fix integer overflow in convert_agg_kbytes_percent
Felix Abecassis [Fri, 14 May 2021 00:02:40 +0000 (17:02 -0700)]
stat: fix integer overflow in convert_agg_kbytes_percent

Assuming that "int" is 32-bit, for high bandwidth values (> 21.5 GB/s)
the expression "mean * 100" will cause an integer overflow before the
conversion to "double" happens.

Signed-off-by: Felix Abecassis <fabecassis@nvidia.com>
2 years agofix fio2gnuplot to work with new logging format
Lars Kellogg-Stedman [Wed, 12 May 2021 22:41:43 +0000 (18:41 -0400)]
fix fio2gnuplot to work with new logging format

The logging format updates documented in 1a953d97 were never
propagated to fio2gnuplot, which since then has been failing with a
ValueError exception.

This commit explicits limits fio2gnuplot to only reading the first
4 columns in the log file.

Closes #928

2 years agoos: define EDQUOT to EIO if the OS doesn't provide it
Jens Axboe [Tue, 11 May 2021 13:58:03 +0000 (07:58 -0600)]
os: define EDQUOT to EIO if the OS doesn't provide it

Fixes: 418f53993b07 ("Make fill_device to stop writing on EDQUOT")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMake fill_device to stop writing on EDQUOT
Martin Bukatovic [Tue, 11 May 2021 07:38:55 +0000 (09:38 +0200)]
Make fill_device to stop writing on EDQUOT

Option fill_device stops writing when we run out of quota as well.

Signed-off-by: Martin Bukatovic <martin.bukatovic@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'z_unit_docs' of https://github.com/ahribeng/fio
Jens Axboe [Tue, 11 May 2021 03:16:58 +0000 (21:16 -0600)]
Merge branch 'z_unit_docs' of https://github.com/ahribeng/fio

* 'z_unit_docs' of https://github.com/ahribeng/fio:
  Add Documentation for z unit

2 years agooslib/linux-blkzoned: make sure that we always support zone capacity
Niklas Cassel [Thu, 6 May 2021 13:18:45 +0000 (13:18 +0000)]
oslib/linux-blkzoned: make sure that we always support zone capacity

A common problem is that users upgrade their kernel to support NVMe ZNS
devices, however, they still use the kernel uapi headers provided by their
distro.

This means that even if the kernel will populate the zone capacity fields
for each zone in the zone report returned by the ioctl, fio will not know
how to interpret that data.

This leads to fio writing past the zone capacity, which will lead to
I/O errors.

It is not trivial for a user to realize that the kernel uapi headers
provided by their distro is the reason for these I/O errors.

In order to make it easier for these users, provide a copy of the current
zoned block device kernel uapi structs.

If the kernel uapi headers installed on the system are too old to support
zone capacity, use the locally defined structs instead.
If the installed headers are new enough to support zone capacity, use the
installed headers.

This way, fio will always be able to handle zone capacity (if the kernel
supports it). At the same time, we will not redefine any structs from the
installed headers if they are newer than our locally defined structs.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agodon't access dlclose'd dynamic ioengine object after close
Eric Sandeen [Fri, 7 May 2021 21:13:05 +0000 (16:13 -0500)]
don't access dlclose'd dynamic ioengine object after close

Alexey reported this bug when using dynamically loaded IO engines;
a segfault on the line where we set the dlhandle to NULL after
the dlclose.

I think this is because ops points to the thing we obtained from dlsym:

ops = dlsym(dlhandle, engine_lib);

and after the final dlclose, the object no longer exists and efforts
to set the handle within it will fail for obvious reasons.
I'm not sure why I hadn't seen this before.

Fixes-RH-Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1956963
Reported-by: Alexey Dobriyan <adobriyan@gmail.com>
Fixes: f6931a1 ("fio: move dynamic library handle to io_ops structure")
Tested-by: Alexey Dobriyan <adobriyan@gmail.com>
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoAdd Documentation for z unit
Gonzalez [Thu, 6 May 2021 18:15:41 +0000 (11:15 -0700)]
Add Documentation for z unit

2 years agoconfigure: fix check_min_lib_version() eval
Stefan Hajnoczi [Thu, 6 May 2021 16:23:31 +0000 (17:23 +0100)]
configure: fix check_min_lib_version() eval

The following shell statement:

  if eval "echo \$$_feature" = "yes" ; then

executes:

  echo $... = "yes"

It does not actually compare the variable named by $_feature to the
string "yes".

Add the missing "test" call so the comparison happens as intended and
wrap the eval so it doesn't include the = "yes".

Fixes: 3e48f7c9de61 ("configure: fix syntax error with NetBSD")
Cc: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoclean up nfs example
Taras Glek [Wed, 5 May 2021 16:11:06 +0000 (09:11 -0700)]
clean up nfs example

2 years agoskip skeleton comments
Taras Glek [Wed, 5 May 2021 16:04:56 +0000 (09:04 -0700)]
skip skeleton comments

2 years agosingle line bodies
Taras Glek [Wed, 5 May 2021 16:02:20 +0000 (09:02 -0700)]
single line bodies

2 years agoC-style comments
Taras Glek [Wed, 5 May 2021 16:00:13 +0000 (09:00 -0700)]
C-style comments

2 years agoioengines: don't call zbd_put_io_u() for engines not implementing commit
Niklas Cassel [Tue, 27 Apr 2021 17:41:14 +0000 (17:41 +0000)]
ioengines: don't call zbd_put_io_u() for engines not implementing commit

Commit d9ed3e63e528 ("zbd: Fix zone locking for async I/O engines") added
a call to zbd_put_io_u() in the case where td->io_ops->commit callback
is not implemented on an ioengine.

The commit in question fails to mention why this zbd_put_io_u() call was
added for ioengines not implementing the commit callback.

The code in td_io_queue() looks like this:

ret = td->io_ops->queue(td, io_u);
zbd_queue_io_u(td, io_u, ret);

if (!td->io_ops->commit) {
io_u_mark_submit(td, 1);
io_u_mark_complete(td, 1);
zbd_put_io_u(td, io_u);
}

SYNC I/O engine case (e.g. psync):
The zone will be locked by zbd_adjust_block(), td->io_ops->queue(td, io_u),
which for a sync I/O engine will return FIO_Q_COMPLETED.

This return value will be send in to zbd_queue_io_u(), which at the end
of the function, unlocks the zone if the return value from ->queue()
differs from FIO_Q_QUEUED. For a sync I/O engine, the zone will be
unlocked here, and io_u->zbd_put_io function pointer will be set to NULL.

psync does not implement the ->commit() callback, so it will call
zbd_put_io_u(), which will do nothing, because the io_u->zbd_put_io
pointer is NULL.

ASYNC I/O engine case (e.g. io_uring):
The zone will be locked by zbd_adjust_block(), td->io_ops->queue(td, io_u),
which for an async I/O engine will return FIO_Q_QUEUED.

This return value will be send in to zbd_queue_io_u(), which at the end
of the function, unlocks the zone if the return value from ->queue()
differs from FIO_Q_QUEUED. For an async I/O engine, the zone will not be
unlocked here, so the io_u->zbd_put_io function pointer will still be set.

io_uring does implement the ->commit() callback, so it will not call
zbd_put_io_u() here at all.

Instead zbd_put_io_u() will be called by do_io() -> wait_for_completions()
-> io_u_queued_complete() -> ios_completed() -> put_io_u() -> zbd_put_io_u(),
which will unlock the zone and will set the io_u->zbd_put_io function pointer
to NULL.

In conclusion, the zbd_put_io_u() should never had been added in the case
where the ->commit() callback wasn't implemented in the first place,
and removing it shouldn't affect ioengines psync or io_uring.

Commit d9ed3e63e528 ("zbd: Fix zone locking for async I/O engines")
probably made the assumption that an async I/O engine == the ->commit()
callback is implemented, however, this is not true, there are async
I/O engines in tree (and out of tree), that does not implement the
->commit() callback. Instead, an async I/O engine is recognized by
the ->queue() callback returning FIO_Q_QUEUED.

Removing the invalid zbd_put_io_u() call will ensure that a zone is not
prematurely unlocked for async I/O engines that do not implement the
->commit() callback. Unlocking a zone prematurely leads to I/O errors.

Fixes: d9ed3e63e528 ("zbd: Fix zone locking for async I/O engines")
Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoThe GPL isn't a EULA: remove it and introduce WixUI_Minimal_NoEULA
Rebecca Cran [Thu, 22 Apr 2021 02:32:25 +0000 (20:32 -0600)]
The GPL isn't a EULA: remove it and introduce WixUI_Minimal_NoEULA

The GPL shouldn't be used as a EULA in an installer.
Remove it, and since the WixUI_Minimal dialog set requires a EULA
create a custom WixUI_Minimal_NoEULA set.

Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'pthread_getaffinity_1' of https://github.com/kusumi/fio
Jens Axboe [Sun, 25 Apr 2021 16:23:34 +0000 (10:23 -0600)]
Merge branch 'pthread_getaffinity_1' of https://github.com/kusumi/fio

* 'pthread_getaffinity_1' of https://github.com/kusumi/fio:
  gettime: Fix compilation on non-Linux with pthread_getaffinity_np()

3 years agogettime: Fix compilation on non-Linux with pthread_getaffinity_np()
Tomohiro Kusumi [Sun, 25 Apr 2021 15:10:40 +0000 (00:10 +0900)]
gettime: Fix compilation on non-Linux with pthread_getaffinity_np()

874d55e50c("os/os-linux: add pthread CPU affinity helper") and a few
commits after that broke compilation on non-Linux platforms which support
pthread_getaffinity_np().

Define fio_get_thread_affinity() on non-Linux platforms, and make gettime
test FIO_HAVE_GET_THREAD_AFFINITY which may or may not depend on pthread.
FIO_HAVE_GET_THREAD_AFFINITY is currently not defined on Windows.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
3 years agoMerge branch 'gpspm-add-optional-use-rpma_conn_completion_wait-function' of https...
Jens Axboe [Fri, 23 Apr 2021 14:39:21 +0000 (08:39 -0600)]
Merge branch 'gpspm-add-optional-use-rpma_conn_completion_wait-function' of https://github.com/ldorau/fio

* 'gpspm-add-optional-use-rpma_conn_completion_wait-function' of https://github.com/ldorau/fio:
  rpma: gpspm: introduce the busy_wait_polling toggle

3 years agorpma: gpspm: introduce the busy_wait_polling toggle
Oksana Salyk [Fri, 23 Apr 2021 06:09:44 +0000 (08:09 +0200)]
rpma: gpspm: introduce the busy_wait_polling toggle

The performance of the librpma_gpspm engine depends heavily
on how much CPU power it can use to its work.
One can want either to take all available CPU power
and see what the maximum possible performance is
or configure it less aggressively and collect the results
when the CPU is not solely dedicated to doing this one task.

The librpma_gpspm engine allows toggling between one and another
by either waiting for incoming requests in the kernel
using rpma_conn_completion_wait() (busy_wait_polling=0)
or trying to collect the completion as soon as it appears
by polling all the time using rpma_conn_completion_get()
(busy_wait_polling=1).

Signed-off-by: Oksana Salyk <oksana.salyk@intel.com>
3 years agoMerge branch 'zbd-no-parallel-init' of https://github.com/floatious/fio
Jens Axboe [Thu, 22 Apr 2021 17:18:23 +0000 (11:18 -0600)]
Merge branch 'zbd-no-parallel-init' of https://github.com/floatious/fio

* 'zbd-no-parallel-init' of https://github.com/floatious/fio:
  init: zonemode=zbd does not work with create_serialize=0

3 years agoinit: zonemode=zbd does not work with create_serialize=0
Niklas Cassel [Thu, 22 Apr 2021 09:17:58 +0000 (11:17 +0200)]
init: zonemode=zbd does not work with create_serialize=0

zbd_init_zone_info() has a comment that it only works correctly if it
called before the first fio fork() call.
However, right now, there is nothing that ensures this.

If the user specifies --create_serialize=0 and --numjobs=2, each thread
will get their own version of zbd_info.

zbd_info contains one mutex per zone, so if the threads get different
zbd_info, two threads can manage to lock the same zone at the same time,
which will lead to I/O errors.

Explicitly disallow --zonemode=zbd together with --create_serialize=0,
so that we know that all threads will use the same zbd_info, instead of
silently misbehaving.

Analysis:
setup_files() calls zbd_init_files() which calls zbd_init_zone_info().
zbd_init_zone_info() does a for_each_td(), where it checks if zbd_info
(for the same filename) has already been allocated by another thread.
This only works if create_serialize=1 (default).
If create_serialize=0, zbd_init_zone_info() will get called in parallel,
and in this case when the second thread checks if any other thread has
allocated zbd_info, the check will fail, since the first thread has not
yet been running long enough to allocate zbd_info.

Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
3 years agoNFS configure fixes
Taras Glek [Tue, 20 Apr 2021 18:02:18 +0000 (11:02 -0700)]
NFS configure fixes

3 years agobackend: fix switch_ioscheduler()
Damien Le Moal [Thu, 15 Apr 2021 02:16:54 +0000 (11:16 +0900)]
backend: fix switch_ioscheduler()

The backend.c function switch_ioscheduler() suffers from several
problems:
1) This function only considers the first file of a job. For jobs using
   multiple files, the ioscheduler switch will done only for that file.
2) If the job file is a character device, a pipe or a regular file for
   which the hosting block device file cannot be determined (e.g. a
   remote file), thring to switch the IO scheduler causes a crash as the
   file disk_util field is NULL.
Fix both problems by introducing the helper function set_ioscheduler()
and changing switch_ioscheduler() to repeatdly call this helper for all
files of the job, ignoring character device files, pipe files and files
without a hosting device information.

Also update the man page to better explain when the ioscheduler option
applies.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoengines: add engine for file delete
Friendy.Su@sony.com [Wed, 14 Apr 2021 03:57:09 +0000 (03:57 +0000)]
engines: add engine for file delete

This engine is to measure the performance of deleting files.

In practice, it is an important benchmark for a file system that
how quick it can release the disk space of deleted files.

Signed-off-by: friendy-su <friendy.su@sony.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'parse-signedness-warn' of https://github.com/floatious/fio
Jens Axboe [Tue, 13 Apr 2021 13:51:17 +0000 (07:51 -0600)]
Merge branch 'parse-signedness-warn' of https://github.com/floatious/fio

* 'parse-signedness-warn' of https://github.com/floatious/fio:
  parse: fix parse_is_percent() warning

3 years agoparse: fix parse_is_percent() warning
Niklas Cassel [Tue, 13 Apr 2021 10:03:31 +0000 (12:03 +0200)]
parse: fix parse_is_percent() warning

When compiling an out of tree ioengine, such as the SPDK fio plugin,
which has -Wextra in CFLAGS, the compiler gives the following warning:

parse.h: In function ‘parse_is_percent’:
parse.h:134:13: warning: comparison of integer expressions of different signedness: ‘long long unsigned int’ and ‘int’ [-Wsign-compare]

Since this warning was introduced recently by fio
commit b75c0fae6612 ("parse: simplify parse_is_percent()"),
and since this is the only warning seen when compiling the SPDK fio
plugin, readd the ULL prefix to parse_is_percent().

Fixes: b75c0fae6612 ("parse: simplify parse_is_percent()")
Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
3 years agoengines/libpmem: do not call drain on close
Łukasz Stolarczuk [Thu, 14 Jan 2021 17:19:43 +0000 (18:19 +0100)]
engines/libpmem: do not call drain on close

no matter if direct was 1 or 0, it's not necessary.
It's either covered by non-temporal stores or it's not desired
by user (if 0 was set).

Signed-off-by: Łukasz Stolarczuk <lukasz.stolarczuk@intel.com>
3 years agoengines/libpmem: cleanup a little code, comments and example
Łukasz Stolarczuk [Wed, 13 Jan 2021 16:43:03 +0000 (17:43 +0100)]
engines/libpmem: cleanup a little code, comments and example

Signed-off-by: Łukasz Stolarczuk <lukasz.stolarczuk@intel.com>
3 years agoengines/libpmem: set file open/create mode always to RW
Łukasz Stolarczuk [Mon, 11 Jan 2021 12:41:54 +0000 (13:41 +0100)]
engines/libpmem: set file open/create mode always to RW

previously, when created file with a 'write' job it couldn't be open
later on, when a 'read' job was ran.

Signed-off-by: Łukasz Stolarczuk <lukasz.stolarczuk@intel.com>
3 years agoNFS engine
Taras Glek [Wed, 26 Feb 2020 17:39:52 +0000 (09:39 -0800)]
NFS engine

3 years agot/zbd: test repeated async write with block size unaligned to zone size
Shin'ichiro Kawasaki [Mon, 12 Apr 2021 07:02:25 +0000 (16:02 +0900)]
t/zbd: test repeated async write with block size unaligned to zone size

A recently fixed bug was caused by zone reset during asynchronous IOs
in-flight. The bug symptom was unaligned command error which was
observed using random write workload with libaio engine and block size
not a divisor of zone size. To confirm the bug fix and to prevent future
regression, add a test case which runs the workload.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agozbd: avoid zone reset during asynchronous IOs in-flight
Shin'ichiro Kawasaki [Mon, 12 Apr 2021 07:02:24 +0000 (16:02 +0900)]
zbd: avoid zone reset during asynchronous IOs in-flight

When fio repeats same workload on zoned block devices, zbd_file_reset()
is called for each repetition. This function resets target zones when
one of two conditions are met: 1) the write pointer of the zone has
offset from the device start unaligned to block size, or 2) the workload
is verify and verification is not in process. When the workload runs
with block size not a divisor of the zone size, the offsets of write
pointers from device start (not from zone start) become unaligned to
block size, then zbd_file_reset() resets target zones. This zone reset
happens even when the asynchronous IOs are in-flight and causes
unexpected IO results. Especially if write requests are in-flight, they
fail with unaligned write command error. A single thread may do both the
zone reset and the write request submit, recursive zone locks can not
prevent the zone reset during the writes.

The write pointer check for block size alignment is not working as
intended. It should have checked offset not from device start but from
zone start. Having said that, regardless of this write pointer check
correctness, the zone reset is not required since the zones are reset in
zbd_adjust_block() anyway when remainder of the zone between write
pointer and zone end is smaller than block size.

To avoid the zone reset during asynchronous IOs, do not reset zones in
zbd_file_reset() when the write pointer offset from the device start is
unaligned to block size. Modify zbd_file_reset() to call the helper
function zbd_reset_zones() only when the workload is verify and
verification is not in process. The function zbd_reset_zones() had an
argument 'all_zones' to inform that the zones should be reset when its
write pointer is unaligned to block size. This argument is not required.
Remove it and simplify the function accordingly.

The zone reset for verify workloads is still required. It does not
conflict with asynchronous IOs, since fio waits for IO completion at
verification end, then IOs are not in-flight when zbd_file_reset() is
called for repetition after verification.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'wip-rados-dont-zerowrite' of https://github.com/aclamk/fio
Jens Axboe [Sat, 10 Apr 2021 17:46:30 +0000 (11:46 -0600)]
Merge branch 'wip-rados-dont-zerowrite' of https://github.com/aclamk/fio

* 'wip-rados-dont-zerowrite' of https://github.com/aclamk/fio:
  engine/rados: Add option to skip object creation

3 years agogettime: cleanup ifdef mess
Jens Axboe [Wed, 31 Mar 2021 02:13:16 +0000 (20:13 -0600)]
gettime: cleanup ifdef mess

Let's just set mask to all CPUs for the non-affinity case, and we
can use the same mask test case throughout.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agogettime: check affinity for thread, if we have it
Jens Axboe [Tue, 30 Mar 2021 23:38:53 +0000 (17:38 -0600)]
gettime: check affinity for thread, if we have it

If we have fio_get_thread_affinity(), we can support a smaller
(and sparse) CPU mask for the clock test.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoos/os-linux: add pthread CPU affinity helper
Jens Axboe [Tue, 30 Mar 2021 23:38:33 +0000 (17:38 -0600)]
os/os-linux: add pthread CPU affinity helper

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoconfigure: add test case for pthread_getaffinity_np()
Jens Axboe [Tue, 30 Mar 2021 23:38:02 +0000 (17:38 -0600)]
configure: add test case for pthread_getaffinity_np()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'unified-merge' of https://github.com/jeffreyalien/fio
Jens Axboe [Thu, 18 Mar 2021 16:19:57 +0000 (10:19 -0600)]
Merge branch 'unified-merge' of https://github.com/jeffreyalien/fio

* 'unified-merge' of https://github.com/jeffreyalien/fio:
  Add functionality to the unified_rw_reporting parameter to output separate and mixed stats when set to 'both' or 2.

3 years agoAdd functionality to the unified_rw_reporting parameter to output
Brandon Paupore [Wed, 9 Dec 2020 16:47:40 +0000 (10:47 -0600)]
Add functionality to the unified_rw_reporting parameter to output
separate and mixed stats when set to 'both' or 2.

To achieve this, the stats are gathered on a per/data direction
(read/write/etc) basis and then summed into the mixed stats when
generating output.

This functionality also allows setting unified_rw_reporting to
'none' or 'mixed' instead of 0 or 1 respectively.

Signed-off-by: Brandon Paupore <brandon.paupore@wdc.com>
Co-authored-by: Jeff Lien <jeff.lien@wdc.com>
3 years agoMerge branch 'add-librpma-engines' of https://github.com/janekmi/fio
Jens Axboe [Thu, 18 Mar 2021 14:47:05 +0000 (08:47 -0600)]
Merge branch 'add-librpma-engines' of https://github.com/janekmi/fio

* 'add-librpma-engines' of https://github.com/janekmi/fio:
  rpma: add librpma_apm_* and librpma_gpspm_* engines

3 years agoMerge branch 'free-dump-options' of https://github.com/floatious/fio
Jens Axboe [Wed, 17 Mar 2021 15:25:46 +0000 (09:25 -0600)]
Merge branch 'free-dump-options' of https://github.com/floatious/fio

* 'free-dump-options' of https://github.com/floatious/fio:
  options: free dump options list on exit

3 years agoMerge branch 'patch-1' of https://github.com/ihsinme/fio
Jens Axboe [Wed, 17 Mar 2021 15:25:05 +0000 (09:25 -0600)]
Merge branch 'patch-1' of https://github.com/ihsinme/fio

* 'patch-1' of https://github.com/ihsinme/fio:
  fix loop with unreachable exit condition

3 years agoMerge branch 'dfs_engine' of https://github.com/johannlombardi/fio
Jens Axboe [Wed, 17 Mar 2021 15:07:55 +0000 (09:07 -0600)]
Merge branch 'dfs_engine' of https://github.com/johannlombardi/fio

* 'dfs_engine' of https://github.com/johannlombardi/fio:
  engines/dfs: add DAOS File System (dfs) engine
  Disable pthread_condattr_setclock on cygwin

3 years agoengines/dfs: add DAOS File System (dfs) engine
Johann Lombardi [Sun, 14 Feb 2021 14:23:47 +0000 (15:23 +0100)]
engines/dfs: add DAOS File System (dfs) engine

DAOS is a new scale-out open-source object store.
See https://github.com/daos-stack/daos for more information.

This patch adds a new fio engine to support the filesystem layer built
on top of DAOS called DFS (DAOS File System). It supports asynchronous
read/write operations.

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
Co-authored-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
Co-authored-by: Vishwanath Venkatesan <vishwanath.venkatesan@intel.com>
Co-authored-by: Ioannis Galanis <ioannis.galanis@intel.com>
3 years agoDisable pthread_condattr_setclock on cygwin
Johann Lombardi [Wed, 17 Mar 2021 13:39:54 +0000 (14:39 +0100)]
Disable pthread_condattr_setclock on cygwin

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
3 years agorpma: add librpma_apm_* and librpma_gpspm_* engines
Jan Michalski [Thu, 11 Feb 2021 11:47:06 +0000 (12:47 +0100)]
rpma: add librpma_apm_* and librpma_gpspm_* engines

The Remote Persistent Memory Access (RPMA) Library is a C library
created to simplify accessing persistent memory on remote hosts over
Remote Direct Memory Access (RDMA).

The librpma_apm_client and librpma_apm_server is a pair of engines
which allows benchmarking persistent writes achieved via
the Appliance Persistency Method (APM; natively supported by
the librpma library) and regular reads (a part of the RDMA standard).

The librpma_gpspm_client and librpma_gpspm_server is a pair of
engines which allows benchmarking persistent writes achieved via
the General Purpose Persistency Method (GPSPM; build on top of
the librpma API).

The librpma library is available here: https://github.com/pmem/rpma
along with the set of scripts using the newly introduced engines
to construct miscellaneous benchmarking scenarios:
https://github.com/pmem/rpma/tree/master/tools/perf

The full history of the development of the librpma fio engines
is available at: https://github.com/pmem/fio/tree/rpma

Co-Authored-By: Lukasz Dorau <lukasz.dorau@intel.com>
Co-Authored-By: Tomasz Gromadzki <tomasz.gromadzki@intel.com>
Co-Authored-By: Jan Michalski <jan.m.michalski@intel.com>
Co-Authored-By: Oksana Salyk <oksana.salyk@intel.com>
3 years agoMerge branch 'dev_luye_github' of https://github.com/louisluSCU/fio
Jens Axboe [Thu, 11 Mar 2021 18:50:49 +0000 (11:50 -0700)]
Merge branch 'dev_luye_github' of https://github.com/louisluSCU/fio

* 'dev_luye_github' of https://github.com/louisluSCU/fio:
  Fix reading multiple blktrace replay files

3 years agoFix reading multiple blktrace replay files
luye [Thu, 11 Mar 2021 03:30:51 +0000 (11:30 +0800)]
Fix reading multiple blktrace replay files

During fio replay, the input file would be checked first by is_blktrace()
if it is a blktrace output file. If so, however, this would treat the
read_iolog option as one single file and ignore multiple files seperated by ":"

This commit contains the fixation that read input file name by index
before doing the check

Signed-off-by: luye <luye.yelu@bytedance.com>
3 years agoMerge branch 'fallock-blkdev' of https://github.com/dmonakhov/fio
Jens Axboe [Wed, 10 Mar 2021 20:30:23 +0000 (13:30 -0700)]
Merge branch 'fallock-blkdev' of https://github.com/dmonakhov/fio

* 'fallock-blkdev' of https://github.com/dmonakhov/fio:
  engines/falloc: add blockdevice as a target

3 years agoengines/falloc: add blockdevice as a target
Dmitry Monakhov [Wed, 10 Mar 2021 16:33:35 +0000 (19:33 +0300)]
engines/falloc: add blockdevice as a target

fallocate supoport for blockdevices was added five years ago.
It is time to stress-test it via fio.

3 years agoMerge branch 'master' of https://github.com/venkatrag1/fio
Jens Axboe [Wed, 10 Mar 2021 15:11:54 +0000 (08:11 -0700)]
Merge branch 'master' of https://github.com/venkatrag1/fio

* 'master' of https://github.com/venkatrag1/fio:
  options: allow separate values for max_latency

3 years agooptions: allow separate values for max_latency
Venkat Ramesh [Mon, 8 Feb 2021 04:57:40 +0000 (20:57 -0800)]
options: allow separate values for max_latency

Adding the ability to specify max_latency limits separately for read, write, trim.
Uses comma-separated values similar to blocksize, rate parameters etc.

Also modifies the behavior for the one option only case for max_latency and latency_target
by excluding syncs. This makes the latency limits similar to the rate checks, in that they
only apply to reads, writes and trims.

Extends the output printed on latency exceeded event, to display io_unit information using format
similar to that iolog write.

Signed-off-by: Venkat Ramesh <venkatraghavan@fb.com>
3 years agoMerge branch 'master' of https://github.com/DevriesL/fio
Jens Axboe [Tue, 9 Mar 2021 14:58:39 +0000 (07:58 -0700)]
Merge branch 'master' of https://github.com/DevriesL/fio

* 'master' of https://github.com/DevriesL/fio:
  engines/io_uring: fix compilation conflict with Android NDK

3 years agoengines/io_uring: fix compilation conflict with Android NDK
DevriesL [Tue, 9 Mar 2021 14:44:08 +0000 (22:44 +0800)]
engines/io_uring: fix compilation conflict with Android NDK

Android NDK already uses __errno in errno.h, rename variable __errno
to fix this conflict.

Signed-off-by: DevriesL <therkduan@gmail.com>
3 years agoFio 3.26 fio-3.26
Jens Axboe [Tue, 9 Mar 2021 00:44:38 +0000 (17:44 -0700)]
Fio 3.26

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/io_uring: SQPOLL fixes
Jens Axboe [Mon, 8 Mar 2021 22:13:06 +0000 (15:13 -0700)]
t/io_uring: SQPOLL fixes

Use the proper primitives to load/store the ring head/tail. And as
importantly, account any submit as a full submit, even if we don't
need to call io_uring_enter().

This fixes hangs with using SQPOLL.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'clock_monotonic_unused' of https://github.com/foxeng/fio
Jens Axboe [Sat, 6 Mar 2021 22:37:56 +0000 (15:37 -0700)]
Merge branch 'clock_monotonic_unused' of https://github.com/foxeng/fio

* 'clock_monotonic_unused' of https://github.com/foxeng/fio:
  configure: remove unused CLOCK_MONOTONIC_* symbols

3 years agoconfigure: remove unused CLOCK_MONOTONIC_* symbols
Fotis Xenakis [Sat, 6 Mar 2021 20:47:42 +0000 (22:47 +0200)]
configure: remove unused CLOCK_MONOTONIC_* symbols

The symbols are not used in the code and the respective probes were
removed in commits 187f39063e1b0a7baeda20a9f4f2406327ec0d41 and
69212fc41c0420f8caf272a0cc270194edbddfe7.

Signed-off-by: Fotis Xenakis <foxen@windowslive.com>
3 years agoengines/filecreate: remove improper message print
Friendy.Su@sony.com [Wed, 3 Mar 2021 06:27:27 +0000 (06:27 +0000)]
engines/filecreate: remove improper message print

'fallocate' should not be printed in filecreate engine

Signed-off-by: friendy-su <friendy.su@sony.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agofix loop with unreachable exit condition
ihsinme [Wed, 24 Feb 2021 21:37:14 +0000 (00:37 +0300)]
fix loop with unreachable exit condition

the essence of the error lies in a possible hit in the eternal cycle while, when exiting the higher cycle by means of break.

Signed-off-by: Sergey Valentey <ihsinme@gmail.com>
3 years agozbd: support 'z' suffix for zone granularity
Alexey Dobriyan [Mon, 22 Feb 2021 13:27:36 +0000 (16:27 +0300)]
zbd: support 'z' suffix for zone granularity

Allow users to pass some options with zone granularity which is natural
for ZBD workloads.

This is nifty for writing quick tests and when firmware guys change
zone sizes.

Converted options are

io_size=
offset=
offset_increment=
size=
zoneskip=

Example:

rw=write
numjobs=2
offset=1z
offset_increment=10z
size=5z
io_size=6z

Thread 1 will write zones 1, 2, 3, 4, 5, 1.
Thread 2 will write zones 11, 12, 13, 14, 15, 11.

Note:
zonemode=strided doesn't create ZBD zone structure but requires
value recalculation. This is why 2 functions are split.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agozbd: fix check against 32-bit zone size
Alexey Dobriyan [Mon, 22 Feb 2021 13:24:01 +0000 (16:24 +0300)]
zbd: fix check against 32-bit zone size

Zone size can be bigger than 4GB.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agozbd: simplify zoneskip= validness check
Alexey Dobriyan [Mon, 22 Feb 2021 13:21:22 +0000 (16:21 +0300)]
zbd: simplify zoneskip= validness check

Simply check the remainder:

(zoneskip % zone_size) > 0

It will do the right thing for all zoneskip= values, and
zone size being positive is checked earlier.

Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoparse: simplify parse_is_percent()
Alexey Dobriyan [Mon, 22 Feb 2021 13:20:03 +0000 (16:20 +0300)]
parse: simplify parse_is_percent()

Check "unsigned long long val <= -1ULL" is tautologically true
because of how value conversions work.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoAdd a new file to gitignore
Hongwei Qin [Sun, 21 Feb 2021 09:39:44 +0000 (17:39 +0800)]
Add a new file to gitignore

Add t/fuzz/fuzz_parseini to the ignore list.

Signed-off-by: Hongwei Qin <glqinhongwei@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agooptions: free dump options list on exit
Niklas Cassel [Thu, 4 Feb 2021 14:46:05 +0000 (14:46 +0000)]
options: free dump options list on exit

Fix the following LeakSanitizer warnings:

Indirect leak of 224 byte(s) in 7 object(s) allocated from:
    #0 0x7f7377b21bc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x563951e5e09d in add_to_dump_list /home/nks/src/fio/parse.c:1135
    #2 0x563951e5e09d in add_to_dump_list /home/nks/src/fio/parse.c:1127
    #3 0x563951e5e09d in parse_cmd_option /home/nks/src/fio/parse.c:1162

Indirect leak of 43 byte(s) in 7 object(s) allocated from:
    #0 0x7f7377aaa3dd in strdup (/lib/x86_64-linux-gnu/libasan.so.5+0x963dd)
    #1 0x563951e5e0a8 in add_to_dump_list /home/nks/src/fio/parse.c:1136
    #2 0x563951e5e0a8 in add_to_dump_list /home/nks/src/fio/parse.c:1127
    #3 0x563951e5e0a8 in parse_cmd_option /home/nks/src/fio/parse.c:1162

Indirect leak of 36 byte(s) in 7 object(s) allocated from:
    #0 0x7f7377aaa3dd in strdup (/lib/x86_64-linux-gnu/libasan.so.5+0x963dd)
    #1 0x563951e5e0b9 in add_to_dump_list /home/nks/src/fio/parse.c:1138
    #2 0x563951e5e0b9 in add_to_dump_list /home/nks/src/fio/parse.c:1127
    #3 0x563951e5e0b9 in parse_cmd_option /home/nks/src/fio/parse.c:1162

by moving fio_dump_options_free() to options.h,
so that we can call it during exit.

Reproducer:
LD_PRELOAD=libasan.so.5 fio --name=test --filename=/dev/nullb0 --runtime=2

Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
3 years agoengines/io_uring: add verbose error for ENOSYS
Jens Axboe [Tue, 16 Feb 2021 19:07:14 +0000 (12:07 -0700)]
engines/io_uring: add verbose error for ENOSYS

If we get ENOSYS for setting up the rings, then the kernel is too old
to support io_uring. Mention that explicitly.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoengine/rados: Add option to skip object creation
Adam Kupczyk [Mon, 8 Feb 2021 11:58:05 +0000 (06:58 -0500)]
engine/rados: Add option to skip object creation

When `touch_objects` is false, connection procedure no longer attempts to create objects.
In such case ceph caches are untouched and do not affect quality of performance measurement.
But if `touch_object` is false, tests that read objects without creating them previously
will now fail.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
3 years agoMerge branch 'per-engine-pre-write-function' of https://github.com/lukaszstolarczuk/fio
Jens Axboe [Sun, 14 Feb 2021 20:21:05 +0000 (13:21 -0700)]
Merge branch 'per-engine-pre-write-function' of https://github.com/lukaszstolarczuk/fio

* 'per-engine-pre-write-function' of https://github.com/lukaszstolarczuk/fio:
  filesetup: add engine's io_ops to prepopulate file with data

3 years agoMerge branch 'taras/clientuid' of https://github.com/tarasglek/fio-1
Jens Axboe [Wed, 10 Feb 2021 20:22:04 +0000 (13:22 -0700)]
Merge branch 'taras/clientuid' of https://github.com/tarasglek/fio-1

* 'taras/clientuid' of https://github.com/tarasglek/fio-1:
  $clientuid keyword to differentiate clients in client/server mode.

3 years ago$clientuid keyword to differentiate clients in client/server mode.
Taras Glek [Wed, 10 Feb 2021 19:08:52 +0000 (11:08 -0800)]
$clientuid keyword to differentiate clients in client/server mode.

Prior to this change getting fio to include IP as part of filename was a struggle. One had to use directory=/ to trigger IP-inclusion code and there was no way to customize that.

Signed-off-by: Taras Glek <taras@glek.net>
3 years agofilesetup: add engine's io_ops to prepopulate file with data
Łukasz Stolarczuk [Fri, 22 Jan 2021 12:12:43 +0000 (13:12 +0100)]
filesetup: add engine's io_ops to prepopulate file with data

In some cases (e.g. engine marked as diskless) files are not laid out.
If the first job is a read job, results are higher than expected
(because reading zero page). Each engine should deliver func to
prepopulate file with data to avoid this situation.

Signed-off-by: Łukasz Stolarczuk <lukasz.stolarczuk@intel.com>
3 years agozbd: relocate Coverity annotation
Dmitry Fomichev [Sat, 30 Jan 2021 03:35:24 +0000 (12:35 +0900)]
zbd: relocate Coverity annotation

The Coverity annotation added earlier to suppress a false positive
about missing unlock in zbd_adjust_block() didn't work because it
was placed not before the return statement, but earlier in the code.

Move the annotation to the right place to avoid the warning.

Reported-by: Bart Van Assche <bvanassche@acm.org>
Fixes: 8e4b2e55512f("zbd: don't unlock zone mutex after verify replay")
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agozbd: fix 32-bit compile warnings for logging
Jens Axboe [Fri, 29 Jan 2021 19:20:48 +0000 (12:20 -0700)]
zbd: fix 32-bit compile warnings for logging

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: avoid looping on invalid command line options
Dmitry Fomichev [Wed, 27 Jan 2021 04:19:45 +0000 (13:19 +0900)]
t/zbd: avoid looping on invalid command line options

t/zbd/test-zbd-support loops indefinitely if an unrecognized option
is specified in the command line. Add a switch case to display usage
and exit the script.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: increase timeout in test #48
Dmitry Fomichev [Wed, 27 Jan 2021 04:19:44 +0000 (13:19 +0900)]
t/zbd: increase timeout in test #48

Test #48 runs some i/o to the test device for 30 seconds and then waits
45 seconds for fio to finish. If this wait times out, the test assumes
that fio is hung because of a zone locking issue and fails. It is
observed that 45s may not be enough for some HDDs, especially the ones
running specialized firmware.

Increase the timeout to 180 seconds to avoid any false positives.
There is no change in test duration for the most common devices.
The test will wait for the full 180 seconds only if it fails, otherwise
it will finish very soon after the 30 second i/o period ends.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: show elapsed time in test-zbd-support
Dmitry Fomichev [Wed, 27 Jan 2021 04:19:43 +0000 (13:19 +0900)]
t/zbd: show elapsed time in test-zbd-support

This script may take quite a lot of time to run against large
zoned HDDs. At the end of every run, show exactly how much time
it took.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: add test #54 to exercise ZBD verification
Dmitry Fomichev [Wed, 27 Jan 2021 04:19:42 +0000 (13:19 +0900)]
t/zbd: add test #54 to exercise ZBD verification

Add a new test case to perform 75/25 read/write workload with varying
i/o size and verification on. It is very important to use a good random
generator for this test. Setting experimental_verify=1 is required for
this test to operate correctly.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: add checks for offline zone condition
Dmitry Fomichev [Wed, 27 Jan 2021 04:19:41 +0000 (13:19 +0900)]
t/zbd: add checks for offline zone condition

Some tests, e.g. #39 an #40, try to read the first zone of the drive.
It is assumed that the first zone is readable. However, if the first
zone is offline, the read fails along with the entire test.

This commit adds two functions to perform zone report and find the
first and the last zones that are not offline. Several test cases
now call these functions to avoid test failures described above.

Fixes for two more test failures are included in this commit -

Test #14 tries to write to conventional zones if they are found at
the beginning of the LBA range of the drive, but it assumes that
these zones are online. This may not always be the case. Add "offset"
to avoid the i/o to be attempted to run against any preceding offline
zones.

Similarly, in test #17, the script tries to find the last zone.
Check for the case when the last zone is offline. The test doesn't
set the i/o file size, but it works OK in most of the cases because
typically this test operates on the last physical zone. With the
online lookup in place, this may not always be the case and if there
are any offline zones that trail the last non-offline zone,
then the i/o will try to access that zone and fail. Add the "size"
to avoid the i/o to be attempted to run against any trailing offline
zones.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: prevent test #31 from looping
Dmitry Fomichev [Wed, 27 Jan 2021 04:19:40 +0000 (13:19 +0900)]
t/zbd: prevent test #31 from looping

The test 31 starts i/o to 128 zones in parallel.
There are two corner cases that are not properly handled in the
existing implementation -
1) If the total number of zones on the device is < 128, the test
will loop indefinitely because the loop increment is calculated as
zero by the script.
2) If the number of max_open_zones of the device is < 128, the
test will fail due to exceeding max_open_zones limit as the code
expects it to be >= 128.

Limit the number of open zones to the reported maximum
and skip the test if there is not enough zones on the device.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: add an option to bail on a failed test
Dmitry Fomichev [Wed, 27 Jan 2021 04:19:39 +0000 (13:19 +0900)]
t/zbd: add an option to bail on a failed test

Sometimes, it can be useful to inspect the state of the zones of the
test device, usually right after a test failure. Currently,
test-zbd-support script just keeps running and proper examination of
device zones can be difficult.

Add the -q option to test/zbd/support to quit immediately upon
encountering any test failure. Additionally, define the same option
in run-tests-against-nullb to propagate it to test/zbd/support.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: fix wrong units in test case #37
Shin'ichiro Kawasaki [Wed, 27 Jan 2021 04:19:38 +0000 (13:19 +0900)]
t/zbd: fix wrong units in test case #37

The second argument of the function total_zone_capacity is expected to
be in bytes. However, the call in test case #37 provides this argument
in sectors and this results in a wrong capacity calculation. Make sure
that the value that is passed to this function is converted to bytes.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: test random I/O direction in all-conventional case
Shin'ichiro Kawasaki [Wed, 27 Jan 2021 04:19:37 +0000 (13:19 +0900)]
t/zbd: test random I/O direction in all-conventional case

The number of 'sectors with data' is counted and used to determine the
direction of the first I/O of a random read/write ZBD workload. To
initialize the number, min_zone and max_zone fields in struct fio_file
are referred. There was a code bug that was recently fixed where
min_zone and max_zone fields were not initialized when all zones in I/O
region were conventional zones. This led to an uninitialized number of
sectors with data, and the write direction was always set for random
workloads.

Add a test case to perform random read/write workload on an I/O region
with both sequential and conventional zones. Check that both read and
write I/Os are executed.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: test that zone_reset_threshold calculation is correct
Shin'ichiro Kawasaki [Wed, 27 Jan 2021 04:19:36 +0000 (13:19 +0900)]
t/zbd: test that zone_reset_threshold calculation is correct

The option "zone_reset_threshold" specifies the ratio of logical blocks
with data to trigger zone resets. When the I/O range includes
conventional zones, only blocks in sequential zones must be used to
track this value. A recently fixed bug has uncovered that the number of
blocks in conventional zones were erroneously counted as the blocks
with data.

To prevent future regressions, add a test case to confirm that the
logical blocks accounting does not include conventional zones.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: test that conventional zones are not locked during random i/o
Shin'ichiro Kawasaki [Wed, 27 Jan 2021 04:19:35 +0000 (13:19 +0900)]
t/zbd: test that conventional zones are not locked during random i/o

A recently fixed bug was caused by an unexpected conventional zone lock
during random I/O adjustment. Only sequential zones are supposed to be
locked, but the conventional zone lock was observed with a random
workload against an I/O region with mixed conventional and sequential
zones.

Add two test cases with the same workload to ensure that no similar
regression happens in the future. One case tests reads and the other
is for writes. As a related change, add the helper function
require_conv_zones() to check that the test target device has enough
conventional zones available.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/zbd: skip tests that need too many sequential zones
Shin'ichiro Kawasaki [Wed, 27 Jan 2021 04:19:34 +0000 (13:19 +0900)]
t/zbd: skip tests that need too many sequential zones

Test cases #3, #4, #28, #29 and #48 require rather large numbers of
sequential zones to run properly and they fail if the test target
device has not enough of such zones in its zone configuration.

Check how many sequential zones are present on the test device and
skip any test cases for which this number is not enough.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>