fio.git
2 days agoEnable crc32c accelleration for arm64 on OSX master
Jens Axboe [Wed, 25 Jan 2023 15:01:30 +0000 (08:01 -0700)]
Enable crc32c accelleration for arm64 on OSX

Before:

jensaxboe@Jenss-MacBook-Pro fio % ./fio --crctest=crc32c
crc32c:   440.18 MiB/sec

After:

ensaxboe@Jenss-MacBook-Pro fio % ./fio --crctest=crc32c
crc32c: 23923.00 MiB/sec

We know we have it on osx on arm hardware, enabling it is pretty
straightforward with that assumption.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 days agoMakefile: add -Wno-stringop-truncation for y.tab.o
Jens Axboe [Wed, 25 Jan 2023 03:54:48 +0000 (20:54 -0700)]
Makefile: add -Wno-stringop-truncation for y.tab.o

This file is auto-generated, and it currently spews the following
warning for me:

In function ‘setup_to_parse_string’,
    inlined from ‘evaluate_arithmetic_expression’ at y.tab.c:1571:2:
y.tab.c:1559:9: warning: ‘strncpy’ specified bound depends on the length of the source argument [-Wstringop-truncation]
 1559 |         strncpy(lexer_input_buffer, string, len);
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
y.tab.c:1556:19: note: length computed here
 1556 |         if (len > strlen(string))
      |                   ^~~~~~~~~~~~~~

when compiled with:

gcc (Debian 12.2.0-14) 12.2.0

Just set -Wno-stringop-truncation unconditionally in the Makefile for
this file, don't think there's any point in checking if this warning
has been enabled manually.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 days agotools/fiograph: accommodate job files not ending in .fio
Vincent Fu [Thu, 19 Jan 2023 00:58:08 +0000 (19:58 -0500)]
tools/fiograph: accommodate job files not ending in .fio

For job files not ending in .fio, fiograph will overwrite the job file
with a graphviz script and then delete it if --keep is not specified.

Fix this by creating temporary files to contain the graphviz script and
image file. Then rename or delete the script and image file as directed
by the user specified options.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 days agoexamples: remove test.png
Vincent Fu [Thu, 19 Jan 2023 18:10:22 +0000 (13:10 -0500)]
examples: remove test.png

test.png doesn't correspond to any example job files. It seems to have
been inadvertantly added to the repository.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
9 days agotools/fiograph: improve default config file search
Vincent Fu [Wed, 18 Jan 2023 16:14:14 +0000 (11:14 -0500)]
tools/fiograph: improve default config file search

When a config file is not explicitly specified the current default is to
search for fiograph.conf only in the current directory. Change this to
try to use fiograph.conf in the directory where fiograph.py is located
when fiograph.conf is not found in the current directory.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
9 days agotools/fiograph: improve default output file name
Vincent Fu [Wed, 18 Jan 2023 15:52:31 +0000 (10:52 -0500)]
tools/fiograph: improve default output file name

Instead of removing all occurrences of '.fio' in the job filename, only
remove '.fio' when it is at the end of the job filename.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
9 days agotools/fiograph: add link to file formats
Vincent Fu [Wed, 18 Jan 2023 15:43:18 +0000 (10:43 -0500)]
tools/fiograph: add link to file formats

To the fiograph help text add a link to a list of the supported image
file output formats.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 weeks agoexamples: add missing fiograph diagram for sg_write_same_ndob.fio
Vincent Fu [Wed, 11 Jan 2023 20:22:40 +0000 (15:22 -0500)]
examples: add missing fiograph diagram for sg_write_same_ndob.fio

This fiograph diagram was missed in the earlier patch that added missing
fiograph diagrams for the example job files. Now each example job file
should have a fiograph diagram.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 weeks agodoc: clarify the usage of rw_sequencer
Ankit Kumar [Wed, 11 Jan 2023 06:48:05 +0000 (12:18 +0530)]
doc: clarify the usage of rw_sequencer

Update man page clarifying the usage of rw_sequencer=sequential
Added few examples explaining the offset generation for rw_sequencer.

Fixes: https://github.com/axboe/fio/issues/1223

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
5 weeks agoengines/xnvme: add support for picking mem backend
Ankit Kumar [Thu, 22 Dec 2022 04:39:51 +0000 (10:09 +0530)]
engines/xnvme: add support for picking mem backend

Add option to the xnvme fio engine for picking a
memory backend. Update the fio document.

Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
5 weeks agoengines/xnvme: add subnqn to fio-options
Ankit Kumar [Thu, 22 Dec 2022 04:39:50 +0000 (10:09 +0530)]
engines/xnvme: add subnqn to fio-options

For fio to utilize a fabrics target with multiple systems, it needs a
way for the user to specify which subsystem to use. This is done by
providing 'subnqn' as fio-option.

Signed-off-by: Simon A. F. Lund <simon.lund@samsung.com>
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
5 weeks agoengines/xnvme: user space vfio based backend
Ankit Kumar [Thu, 22 Dec 2022 04:39:49 +0000 (10:09 +0530)]
engines/xnvme: user space vfio based backend

Add an option to use user-space VFIO-based backend,
implemented using libvfn.
Update xnvme engine options for missing backends.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
5 weeks agoengines/xnvme: fixes for xnvme ioengine
Ankit Kumar [Thu, 22 Dec 2022 04:39:48 +0000 (10:09 +0530)]
engines/xnvme: fixes for xnvme ioengine

1. fix error-handling in xnvme_fioe_queue()
2. fix resource-leak in xnvme_fioe_init()

Signed-off-by: Simon A. F. Lund <simon.lund@samsung.com>
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
6 weeks agot/run-fio-tests: relax acceptance criteria for t0008
Vincent Fu [Tue, 13 Dec 2022 21:34:20 +0000 (21:34 +0000)]
t/run-fio-tests: relax acceptance criteria for t0008

t0008 runs a 50/50 seq read/write job on a 32M file. Since a random number
generator determines whether fio issues a read or a write command, the
read/write ratio will not be exactly 50/50.  This patch relaxes the acceptance
criteria to avoid false positives when we do not use the default random number
seeds.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
6 weeks agoexamples: add missing fiograph diagrams
Vincent Fu [Fri, 16 Dec 2022 18:20:08 +0000 (13:20 -0500)]
examples: add missing fiograph diagrams

We have added multiple example job files recently without including
fiograph diagrams for them. This patch adds fiograph diagrams for
example job files where they were missing.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
6 weeks agotools/fiograph: update config file
Vincent Fu [Fri, 16 Dec 2022 18:44:30 +0000 (13:44 -0500)]
tools/fiograph: update config file

Add 'specific_options' for some new ioengines. Also add a missing
ioengine option for the sg ioengine.

We should have some sort of script to fill in these ioengine options.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
6 weeks agoexample: add a zoned block device write example with GC by trim workload
Shin'ichiro Kawasaki [Thu, 15 Dec 2022 01:56:06 +0000 (10:56 +0900)]
example: add a zoned block device write example with GC by trim workload

Add an example job file which shows how to simulate garbage collection
of zoned block devices using a job with trim workload and flow options.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoexample: add a zoned block device write example with GC by zone resets
Shin'ichiro Kawasaki [Thu, 15 Dec 2022 01:56:05 +0000 (10:56 +0900)]
example: add a zoned block device write example with GC by zone resets

Add an example job file which shows how to simulate garbage collection
of zoned block devices using options zone_reset_threshold and
zone_reset_frequency.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoHOWTO/man: improve descriptions of max open zones options
Shin'ichiro Kawasaki [Thu, 15 Dec 2022 01:56:04 +0000 (10:56 +0900)]
HOWTO/man: improve descriptions of max open zones options

The options max_open_zones and job_max_open_zones control the number of
open zones for zonemode=zbd. However, descriptions in HOWTO and man
about these options are not clear enough since it is not well described
what an open zone is. Improve the description by adding explanation of
the open zone state. Also explain the default values for these options.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Link: https://lore.kernel.org/r/20221215015606.2767187-3-shinichiro.kawasaki@wdc.com
[axboe: apply wording suggestion from Damien]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoman: fix troff warning
Shin'ichiro Kawasaki [Thu, 15 Dec 2022 01:56:03 +0000 (10:56 +0900)]
man: fix troff warning

'make doc' reports a warning:

  troff: <standard input>:2554: warning: can't find font 'b'

To avoid it, add missing 'P' for troff built-in command '\f'.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agot/io_uring: adjust IORING_REGISTER_MAP_BUFFERS value
Jens Axboe [Mon, 12 Dec 2022 23:57:52 +0000 (16:57 -0700)]
t/io_uring: adjust IORING_REGISTER_MAP_BUFFERS value

This isn't upstream yet, but current test patches have it at 26.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agoengines/libblkio: Share a single blkio instance among threads in same process
Alberto Faria [Thu, 1 Dec 2022 22:08:03 +0000 (22:08 +0000)]
engines/libblkio: Share a single blkio instance among threads in same process

fio groups all subjobs that set option 'thread' into a single process.
Have them all share a single `struct blkio` instance, with one `struct
blkioq` per thread/subjob. This allows benchmarking multi-queue setups.

Note that `struct blkio` instances cannot be shared across different
processes.

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoengines/libblkio: Add options for some driver-specific properties
Alberto Faria [Thu, 1 Dec 2022 22:08:02 +0000 (22:08 +0000)]
engines/libblkio: Add options for some driver-specific properties

The properties are either common to several drivers or particularly
relevant for benchmarking, so this should help write cleaner workload
files.

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoengines/libblkio: Add option libblkio_force_enable_completion_eventfd
Alberto Faria [Thu, 1 Dec 2022 22:08:01 +0000 (22:08 +0000)]
engines/libblkio: Add option libblkio_force_enable_completion_eventfd

When set, the queue's completion fd is enabled even when it isn't used,
i.e., even if option libblkio_wait_mode is _not_ set to "eventfd".

Depending on the libblkio driver, this can have an impact on
performance. This option allows evaluating that overhead.

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoengines/libblkio: Add option libblkio_wait_mode
Alberto Faria [Thu, 1 Dec 2022 22:08:00 +0000 (22:08 +0000)]
engines/libblkio: Add option libblkio_wait_mode

It allows configuring how the engine waits for request completions,
instead of always using a blocking blkioq_do_io() call.

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoengines/libblkio: Add option libblkio_write_zeroes_on_trim
Alberto Faria [Thu, 1 Dec 2022 22:07:59 +0000 (22:07 +0000)]
engines/libblkio: Add option libblkio_write_zeroes_on_trim

When set, trim IOs will be submitted as blkioq_write_zeroes() requests
instead of blkioq_discard() requests.

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoengines/libblkio: Add option libblkio_vectored
Alberto Faria [Thu, 1 Dec 2022 22:07:58 +0000 (22:07 +0000)]
engines/libblkio: Add option libblkio_vectored

When enabled, read and write requests are submitted as vectored requests
using blkioq_{readv,writev}(), instead of using blkioq_{read,write}().

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoengines/libblkio: Add support for poll queues
Alberto Faria [Thu, 1 Dec 2022 22:07:57 +0000 (22:07 +0000)]
engines/libblkio: Add support for poll queues

Configure a poll queue instead of a "regular" queue when option hipri is
set.

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoengines/libblkio: Allow setting option mem/iomem
Alberto Faria [Thu, 1 Dec 2022 22:07:56 +0000 (22:07 +0000)]
engines/libblkio: Allow setting option mem/iomem

This allows users to customize data buffer memory using fio's existing
options. Users become responsible for ensuring that the allocated memory
satisfies all constraints imposed by the libblkio driver under use.

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoAdd engine flag FIO_SKIPPABLE_IOMEM_ALLOC
Alberto Faria [Thu, 1 Dec 2022 22:07:55 +0000 (22:07 +0000)]
Add engine flag FIO_SKIPPABLE_IOMEM_ALLOC

It makes it valid to set option mem/iomem even when the engine specifies
iomem_alloc and iomem_free callbacks, allowing users to optionally use
fio's customizable memory allocation logic instead of the engine's.

This is in preparation for giving libblkio engine users the choice
between controlling memory allocation or delegating it to the libblkio
library.

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoAdd a libblkio engine
Alberto Faria [Thu, 1 Dec 2022 22:07:54 +0000 (22:07 +0000)]
Add a libblkio engine

The libblkio library provides a unified API for efficiently accessing
block devices using modern high-performance block I/O interfaces like
io_uring and vhost-user-blk. Using libblkio reduces the amount of code
needed for interfacing with storage devices and allows developers to
focus on their applcations.

Add a libblkio engine that uses libblkio to perform I/O. This is useful
to benchmark the library itself, and also adds support for storage
interfaces and devices otherwise not supported by fio, such as
virtio-blk PCI, vhost-user, and vhost-vDPA devices.

See the libblkio documentation [2] or KVM Forum 2022 [3] presentation
for more information on the library itself.

[1] https://gitlab.com/libblkio/libblkio
[2] https://libblkio.gitlab.io/libblkio/index.html
[3] https://static.sched.com/hosted_files/kvmforum2022/8c/libblkio-kvm-forum-2022.pdf

Signed-off-by: Alberto Faria <afaria@redhat.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agodoc: update about size
Ankit Kumar [Thu, 1 Dec 2022 05:08:32 +0000 (10:38 +0530)]
doc: update about size

In few cases with fio option size the number of bytes of data transferred
is actually less than what we specified. This can happen if there are
gaps or holes while doing I/O's or if we are running a mix of sequential
and random workload.
Update the documentation for that.

Fixes: https://github.com/axboe/fio/issues/1486

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agobackend: respect return value of init_io_u_buffers
Shin'ichiro Kawasaki [Thu, 1 Dec 2022 02:44:25 +0000 (11:44 +0900)]
backend: respect return value of init_io_u_buffers

When workloads require large buffer for I/O, fio fails to allocate I/O
buffer but does not report meaningful error message. It just accesses
to null pointer and fail with signal 11. This symptom is observed with
the command line below:

$ fio --name=job --filename=/tmp/fio --rw=write --bs=1g --size=1g \
      --iodepth=128 --ioengine=libaio

The I/O buffer allocation is done in function init_io_u_buffers. The
allocation failure is not reported because return value of the function
is ignored. Check the return value and report to the higher layer.

Fixes: 71e6e5a2fd5c ("iolog replay: Realloc io_u buffers to adapt to operation size.")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Link: https://lore.kernel.org/r/20221201024425.2340442-1-shinichiro.kawasaki@wdc.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agodocs: description for experimental_verify
Vincent Fu [Tue, 29 Nov 2022 22:09:41 +0000 (17:09 -0500)]
docs: description for experimental_verify

Explain how experimental_verify differs from standard verify.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agodocs: synchronize fio.1 and HOWTO changes
Vincent Fu [Tue, 29 Nov 2022 18:05:51 +0000 (13:05 -0500)]
docs: synchronize fio.1 and HOWTO changes

A couple recent patches changed only one of fio.1 or HOWTO. This patch
synchronizes the changes between the two documents.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
8 weeks agoMerge branch 'lintian-manpage-fixes' of https://github.com/hoexter/fio
Jens Axboe [Mon, 28 Nov 2022 19:54:53 +0000 (12:54 -0700)]
Merge branch 'lintian-manpage-fixes' of https://github.com/hoexter/fio

* 'lintian-manpage-fixes' of https://github.com/hoexter/fio:
  Use correct backslash escape in man 1 fio
  Spelling: Fix allows to -> allows one to in man 1 fio

8 weeks agoUse correct backslash escape in man 1 fio
Sven Hoexter [Mon, 28 Nov 2022 19:06:36 +0000 (20:06 +0100)]
Use correct backslash escape in man 1 fio

The usage of '\\' to create a literal backslash enclosed in
apostrophe is sometimes wrongly rendered as an acute accent.

Reported by lintian, a Debian package linter.

Signed-off-by: Sven Hoexter <sven@stormbind.net>
8 weeks agoSpelling: Fix allows to -> allows one to in man 1 fio
Sven Hoexter [Mon, 28 Nov 2022 18:47:48 +0000 (19:47 +0100)]
Spelling: Fix allows to -> allows one to in man 1 fio

Reported by lintian, a Debian package linter.

Signed-off-by: Sven Hoexter <sven@stormbind.net>
2 months agodoc: update about sqthread_poll
Ankit Kumar [Wed, 23 Nov 2022 11:27:38 +0000 (16:57 +0530)]
doc: update about sqthread_poll

Update that when sqthread_poll is enabled fio will not report submission
latency.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agoengines:io_uring: fix clat calculation for sqthread poll
Ankit Kumar [Wed, 23 Nov 2022 11:27:37 +0000 (16:57 +0530)]
engines:io_uring: fix clat calculation for sqthread poll

When sqthread_poll is specified for io_uring and io_uring_cmd I/O
engines, fio reports garbage value for completion latencies. This is
because the issue time was not recorded. Added a change for that.
On the other hand submission latency for sqthread poll is really just
the time it takes to fill in the SQ ring entries and any syscall
required to wake up the idle kernel thread. So there is really no need
to report those.

This fixes the issue: https://github.com/axboe/fio/issues/1484

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agoMerge branch 'patch-1' of https://github.com/chienfuchen32/fio
Jens Axboe [Wed, 23 Nov 2022 17:41:00 +0000 (10:41 -0700)]
Merge branch 'patch-1' of https://github.com/chienfuchen32/fio

* 'patch-1' of https://github.com/chienfuchen32/fio:
  update documentation typo

2 months agoupdate documentation typo
chienfuchen32 [Wed, 23 Nov 2022 14:33:29 +0000 (22:33 +0800)]
update documentation typo

Signed-off-by: chienfuchen32 <chienfuchen32@gmail.com>
2 months agotest: add large pattern test
Logan Gunthorpe [Fri, 18 Nov 2022 23:16:01 +0000 (16:16 -0700)]
test: add large pattern test

Add a test which writes a file with a 16KB buffer pattern, then
verify the file with the same pattern.

The test writes a single 16KB block and thus should be the same as
the patttern file. Verify that this is the case after the test is
run.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agolib/pattern: Support binary pattern buffers on windows
Logan Gunthorpe [Fri, 18 Nov 2022 23:16:00 +0000 (16:16 -0700)]
lib/pattern: Support binary pattern buffers on windows

On windows, binary files used as pattern buffers may be mangled or
truncated seeing the files are openned in text mode.

Fix this by passing O_BINARY on windows when openning the file.

Suggested-by: Vincent Fu <vincentfu@gmail.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agooptions: Support arbitrarily long pattern buffers
Logan Gunthorpe [Fri, 18 Nov 2022 23:15:59 +0000 (16:15 -0700)]
options: Support arbitrarily long pattern buffers

Dynamically allocate the pattern buffer to remove the 512B length
restriction. To accomplish this, store a pointer instead of a fixed
block of memory for the buffers in the thread_options structure.
Then introduce and use the function parse_and_fill_pattern_alloc()
which will calculate the approprite size of the buffer and allocate
it before filling it.

The buffers will be freed, along with a number of string buffers
in free_thread_options_to_cpu(). They will also be reallocated (if
necessary) when receiving them over the wire with
convert_thread_options_to_cpu().

This allows for specifying real world compressible data (eg. The
Calgary Corpus) for the buffer_pattern option.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agolib/pattern: Support short repeated read calls when loading from file
Logan Gunthorpe [Fri, 18 Nov 2022 23:15:58 +0000 (16:15 -0700)]
lib/pattern: Support short repeated read calls when loading from file

Once a pattern file can be much larger, it will be possible that
kernel will return a short read while loading the file and thus may
randomly only load part of the file.

Fix this by putting the read in a loop so the entire file will be
read.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agolib/pattern: Support NULL output buffer in parse_and_fill_pattern()
Logan Gunthorpe [Fri, 18 Nov 2022 23:15:57 +0000 (16:15 -0700)]
lib/pattern: Support NULL output buffer in parse_and_fill_pattern()

Support passing a NULL output buffer to parse_and_fill_pattern().
Each formatting function simply needs to avoid accessing the buffer
when it is NULL. This allows calculating the required size of the
buffer before one might be allocated.

This will be useful in a subsequent patch for dynamically allocating
the pattern buffers.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agocconv: Support pattern buffers of arbitrary size
Logan Gunthorpe [Fri, 18 Nov 2022 23:15:56 +0000 (16:15 -0700)]
cconv: Support pattern buffers of arbitrary size

Change the thread_options_pack structure to support pattern buffers
of arbitrary size by using a flexible array at the end of the the
structure to store both the verify_pattern and the buffer_pattern
in that order.

In this way, only the actual bytes of each pattern will be sent over
the wire and patterns of an arbitrary size can be used with the packed
structure.

In order to determine the required size of the structure the function
thread_options_pack_size() is introduced which returns the total
number of bytes required for a given thread_options instance.

The two callsites of convert_thread_options_to_net() are then converted
to dynamically allocate a pdu of the appropriate size and the
two callsites of convert_thread_options_to_cpu() are modified to
take the size of the received data to prevent buffer overruns.

Also add specific testing of this feature in fio_test_cconv().

Seeing this changes the client/server protocol, the FIO_SERVER_VER
is bumped.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agot/zbd: add test case to check experimental_verify option
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:13:08 +0000 (11:13 +0900)]
t/zbd: add test case to check experimental_verify option

The option experimental_verify does not work with zonemode=zbd. Add a
test case to check that fio errors out when the both options are
specified.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Tested-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agot/zbd: remove experimental_verify option from test case #54
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:13:07 +0000 (11:13 +0900)]
t/zbd: remove experimental_verify option from test case #54

The option experimental_verify does not work with zonemode=zbd. Remove
it from the test case #54.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Tested-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agot/zbd: add test case to check zone_reset_threshold/frequency with verify
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:13:06 +0000 (11:13 +0900)]
t/zbd: add test case to check zone_reset_threshold/frequency with verify

Recent commit fixed assertion failure observed with zone_reset_threshold
and zone_reset_frequency options together with verify. Add a test case
to confirm the fix. Run four types of workloads and confirm no error.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agot/zbd: modify test case #34 for block size unaligned to zone size
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:13:05 +0000 (11:13 +0900)]
t/zbd: modify test case #34 for block size unaligned to zone size

The test case #34 confirmed that the block size unaligned to zone size
is handled as an error. After recent fix, now fio is able to handle such
block sizes, then the check for the error is no longer required.

Instead of removing this unnecessary test case, change it to cover
verify with complex workload. It runs random write workload with high
queue depth with verify. Use two types of block sizes unaligned to zone
size. This test workload is same as test case #57 except the verify
option and block sizes.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agot/zbd: fix test case #33 for block size unaligned to zone size
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:13:04 +0000 (11:13 +0900)]
t/zbd: fix test case #33 for block size unaligned to zone size

Recent fix of zonemode=zbd support for block size unaligned to zone size
unveiled that the test case #33 has two issues.

First issue is test preparation. Before the fix, write was done to only
to the first zone due to a bug in zone selection which happens when
block size is not a divisor of zone size. Then, status of second zone
did not affect. After the fix, write count to the zones may vary if the
second zone has almost full status since the zone is skipped from write
target. Fix this by resetting the write target zones.

Second issue is expected written data size. The test case checks that
the written data size is larger than the io_size option value. This
expectation was fine before the fix because data write was repeated in
do_io() and the limit was checked with io_issue_bytes_exceeded(),
which triggers loop break when written data is larger than io_size.
However, after the fix, the limit is checked with keep_running() in
thread_main(). According to code and block comment in keep_running(),
fio job terminates even when written data size is smaller than io_size
if the gap is smaller than maximum IO size. Then the expected written
data size is the largest multiple of block size smaller than or equal to
the io_size. This io_size check change resulted in the test case
failure. Avoid the failure by fixing the expected written data size
calculation.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Tested-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agozbd: prevent experimental verify with zonemode=zbd
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:13:03 +0000 (11:13 +0900)]
zbd: prevent experimental verify with zonemode=zbd

When the experimental_verify option is specified, fio does not record
normal I/O metadata to create verify read unit. Instead, fio resets file
status after normal I/O and before verify start, then replay I/Os to
regenerate write unit and adjust it to verify read. This I/O replay does
not work for zonemode=zbd since zone status needs to be restored at
verify start to the status before the normal I/O. However, such status
restore moves write pointers and erases data pattern for verify.

Check that the options experimental_verify and zonemode=zbd are not
specified together and error out in case they are both specified. Also
remove the helper function zbd_replay_write_order() which is called from
zbd_adjust_block(). This function adjusts verify read unit to meet zone
write restrictions for experimental verify, but such adjustment does not
work because zone status is not restored before verify start.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Tested-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agozbd: fix zone reset condition for verify
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:13:02 +0000 (11:13 +0900)]
zbd: fix zone reset condition for verify

When data verification is requested, zbd_file_reset() resets zones only
when td->runstate is not TD_VERIFYING so that the data to read back for
verify is not wiped out. However, even when verify data to read is left,
td->runstate is not always TD_VERIFYING. When verify_backlog option is
set, or when block size is not divisor of zone size, zbd_file_reset()
can be called while td->runstate is TD_RUNNING. This causes verify
failures.

To avoid the failures, improve the check condition to reset zones in
zbd_file_reset(). On top of td->runstate, refer td->io_hist_len,
td->verify_batch and td->o.verify_backlog values to avoid zone reset.
This is same check as check_get_verify().

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agozbd, verify: verify before zone reset for zone_reset_threshold/frequency
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:13:01 +0000 (11:13 +0900)]
zbd, verify: verify before zone reset for zone_reset_threshold/frequency

When zone_reset_threshold and zone_reset_frequency options are specified
with zonemode=zbd, it resets zones not full. When verify option is
specified on top of that, the zone reset of non-full zones erases data
for verify and causes verify failure. Current implementation avoids this
scenario by assert.

To allow zone_reset_threshold/frequency options together with verify,
do verify before the zone reset. When zone reset is required to an open
zone with verify data, call get_next_verify() to get io_u for verify
read and return it from zbd_adjust_block(). When io_u->file is set,
get_next_verify() assumes the io_u is requeued and does nothing. Unset
io_u->file to tell get_next_verify() is not requeued.

Also modify verify_io_u() to skip rand_seed check when the option
zone_reset_frequency is set. When the option is set, random seed is not
reset for verify reads in same manner as verify_backlog option, then
this check is not valid.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agozbd: allow block size not divisor of zone size
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:13:00 +0000 (11:13 +0900)]
zbd: allow block size not divisor of zone size

Current implementation checks that block size is divisor of zone size
when verify work load is specified. After the recent fix of block size
unaligned to zone, this check is no longer valid. Remove the check.

The check had been valid since such block size left unwritten area at
each zone end and keeps the zones in open/active status until verify
read is done. It easily hit max open/active zones limitation. After the
fix, the zones with unwritten area are finished then they do not hit the
limitation.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Tested-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agozbd: finish zones with remainder smaller than minimum write block size
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:12:59 +0000 (11:12 +0900)]
zbd: finish zones with remainder smaller than minimum write block size

When zonemode is zbd and block size is not divisor of zone size, write
target zone selection does not work as expected. When the write is
random write and the device has max open zone limit, the random write is
repeated to the zones selected up to the max open zone limit. All writes
are repeated only to the zones. When the write is sequential write,
write is done only for the first zone. The cause of such unexpected zone
selection is current write target zone selection logic. It selects write
target zones within open zones. When block size is not divisor of zone
size, the selected open zone has only remainder of writable blocks
smaller than the block size. Fio resets such zone after zone selection
and continues writing to it. This zone reset is required not to exceed
the limit of max_open_zones option or max_active_zone limit of the zoned
device, but it does not simulate the workload.

To avoid the zone reset and unexpected write to same zone, fix write
target zone handling of zones with remainder smaller than write block
size. Do not reset but finish such zone not to exceed the max_open_zones
option and max_active_zone limit. Then choose the zone next to the
finished zone as write target. To implement this, add the helper
function zbd_finish_zone().

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Tested-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agozbd: add zbd_zone_remainder() helper function
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:12:58 +0000 (11:12 +0900)]
zbd: add zbd_zone_remainder() helper function

Add the helper function zbd_zone_remainder(), which returns the number
of bytes that are still available for writing before the zone gets full.
Use this function to improve readability. It will also be used in the
following patch.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Tested-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agoengines/libzbc: add libzbc_finish_zone() helper function
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:12:57 +0000 (11:12 +0900)]
engines/libzbc: add libzbc_finish_zone() helper function

To support zone finish operation to ZBC drives through libzbc, add
finish_zone() callback to struct ioengine_ops, and implement in libzbc
IO engine. This feature is used to keep the same zone handling by
zonemode=zbd for libzbc engine as other engines.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Tested-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agooslib: blkzoned: add blkzoned_finish_zone() helper function
Shin'ichiro Kawasaki [Mon, 14 Nov 2022 02:12:56 +0000 (11:12 +0900)]
oslib: blkzoned: add blkzoned_finish_zone() helper function

Add the helper function blkzoned_finish_zone() to support zone finish
operation to zoned block devices through ioctl. This feature will be
used to change status of zones which is not yet full but does not have
enough size to write next block, so that such zones do not exceed max
active zones limit. This function does zone finish only when kernel
supports the ioctl BLKFINISHZONE. Otherwise, it does nothing. This
should be fine since the kernel without BLKFINISHZONE does not report
max active zone limit through sysfs to user space.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Tested-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Vincent Fu [Mon, 14 Nov 2022 13:47:00 +0000 (08:47 -0500)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  os/os.h: Improve cpus_configured()
  configure: Fix the struct nvme_uring_cmd detection
  configure: Fix clock_gettime() detection

2 months agoos/os.h: Improve cpus_configured()
Bart Van Assche [Sun, 13 Nov 2022 23:10:10 +0000 (15:10 -0800)]
os/os.h: Improve cpus_configured()

Fix the following Coverity complaint:

1. negative_return: Calling sysconf, which might return a negative value.
2. return_negative_fn: Returning the return value of sysconf, which might be negative.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoconfigure: Fix the struct nvme_uring_cmd detection
Bart Van Assche [Sun, 13 Nov 2022 23:11:54 +0000 (15:11 -0800)]
configure: Fix the struct nvme_uring_cmd detection

Prevent that struct nvme_uring_cmd detection fails as follows:

error: unused variable 'cmd' [-Werror=unused-variable]

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoconfigure: Fix clock_gettime() detection
Bart Van Assche [Sun, 13 Nov 2022 23:09:22 +0000 (15:09 -0800)]
configure: Fix clock_gettime() detection

Prevent that the clock_gettime() and CLOCK_MONOTONIC tests fail as follows:

error: argument 2 null where non-null expected [-Werror=nonnull]

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Mon, 7 Nov 2022 23:20:04 +0000 (16:20 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Android: Enable zoned block device support
  Windows: Fix the build

2 months agoAndroid: Enable zoned block device support
Bart Van Assche [Mon, 7 Nov 2022 19:14:34 +0000 (11:14 -0800)]
Android: Enable zoned block device support

Enable support for --zonemode=zbd on Android.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoWindows: Fix the build
Bart Van Assche [Mon, 7 Nov 2022 20:37:16 +0000 (12:37 -0800)]
Windows: Fix the build

Fix the build errors about passing arguments without a prototype.

Fixes: 93bcfd20e37c ("Move Windows port to MinGW")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoFio 3.33 fio-3.33
Jens Axboe [Sun, 6 Nov 2022 20:55:41 +0000 (13:55 -0700)]
Fio 3.33

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agotest: use homebrew to install sphinx instead of pip on macOS
Vincent Fu [Fri, 4 Nov 2022 17:15:10 +0000 (13:15 -0400)]
test: use homebrew to install sphinx instead of pip on macOS

With the current GitHub Actions macOS image, pip3 install sphinx does
not appear to place sphinx-doc in the path. This results in
documentation build failures.

Resolve this by using homebrew to install sphinx-doc and add it to the
search path.

https://www.sphinx-doc.org/en/master/usage/installation.html
https://github.com/vincentkfu/fio/actions/runs/3395703049/jobs/5645918799

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agotest: change GitHub Actions macOS platform to macOS 12
Vincent Fu [Fri, 4 Nov 2022 17:04:34 +0000 (13:04 -0400)]
test: change GitHub Actions macOS platform to macOS 12

I am seeing some documentation build failures on macOS 11. See if the
issue is fixed in macOS 12.

https://github.com/vincentkfu/fio/actions/runs/3395509649/jobs/5645674083

This change ultimately did not resolve the issue but we might as well
switch to macOS 12 since it has been out for a year already.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agoio_uring: update documentation and small fix for sqthread_poll
Ankit Kumar [Fri, 21 Oct 2022 11:32:36 +0000 (17:02 +0530)]
io_uring: update documentation and small fix for sqthread_poll

type for sqthread_poll should be FIO_OPT_STR_SET, as this only
requires it to be set. This also fixes the issue:
https://github.com/axboe/fio/issues/927

Update the HOWTO and man page to specify the missing types for all
the remaining io_uring options.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agoHOWTO: clean up exit_what description
Vincent Fu [Wed, 2 Nov 2022 14:26:36 +0000 (10:26 -0400)]
HOWTO: clean up exit_what description

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agoHOWTO: update description for flow option
Vincent Fu [Tue, 1 Nov 2022 21:24:34 +0000 (17:24 -0400)]
HOWTO: update description for flow option

d4e74fda98b60dc175b4114492fcc7c21c617ddd updated the description for
`flow` in fio.1 but did not update the HOWTO. Make the HOWTO description
match what is in fio.1

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotest: add test for experimental verify with loops and time_based options
Shin'ichiro Kawasaki [Thu, 20 Oct 2022 06:38:54 +0000 (15:38 +0900)]
test: add test for experimental verify with loops and time_based options

Add a test case to confirm fix of numberio accounting issue of
experimental verify using loops and time_based options.

Of note is that this case fails on Windows with error "bad header
rand_seed". Until it gets fixed, require to non-Windows OS for this test
case so that CI does not report the known failure.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotest: add test for verify read back of experimental verify
Shin'ichiro Kawasaki [Thu, 20 Oct 2022 06:38:53 +0000 (15:38 +0900)]
test: add test for verify read back of experimental verify

Add a test case to confirm fix of bytes_done accounting issue of
experimental verify. The test job runs a simple workload with
experimental verify and confirm verify read amount is the file size.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agoverify: fix numberio accounting of experimental verify
Shin'ichiro Kawasaki [Thu, 20 Oct 2022 06:38:52 +0000 (15:38 +0900)]
verify: fix numberio accounting of experimental verify

As for non-experimental verify, numberio is compared between the numbers
saved in metadata and written data header. As for experimental verify,
the metadata is not available. Instead of numberio in metadata, it
refers td->io_issues[] as the numberio value for the comparison.
However, td->io_issues[] is used not only for verify reads but also for
normal I/Os. It results in comparison with wrong numberio value and
verification failure.

Fix this issue by adding a new field td->verify_read_issues which counts
up number of verify reads. Substitute td->verify_read_issues to
io_u->numberio to refer it for the comparison in experimental verify
path. Also move td->io_issues[] substitution to io_u->numberio out of
populate_verify_io_u() to keep same behavior in non-experimental verify
path.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agoverify: fix bytes_done accounting of experimental verify
Shin'ichiro Kawasaki [Thu, 20 Oct 2022 06:38:51 +0000 (15:38 +0900)]
verify: fix bytes_done accounting of experimental verify

The commit 55312f9f5572 ("Add ->bytes_done[] to struct thread_data")
moved bytes_done[] on stack to struct thread_data. However, this unified
two bytes_done[] in do_io() and do_verify() stacks into single
td->bytes_done[]. This caused wrong condition check in do_verify() in
experimental verify path since td->bytes_done[] holds values for do_io()
not for do_verify(). This caused unexpected loop break in do_verify()
and verify read skip when experimental_verify=1 option is specified.

To fix this, add bytes_verified to struct thread_data for do_verify() in
same manner as bytes_done[] for do_io(). Introduce a helper function
io_u_update_bytes_done() to factor out same code for bytes_done[] and
bytes_verified[].

Fixes: 55312f9f5572 ("Add ->bytes_done[] to struct thread_data")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agoMerge branch 'fix-cpus_allowed' of https://github.com/roxma/fio
Vincent Fu [Mon, 24 Oct 2022 14:10:50 +0000 (10:10 -0400)]
Merge branch 'fix-cpus_allowed' of https://github.com/roxma/fio

* 'fix-cpus_allowed' of https://github.com/roxma/fio:
  cpus_allowed: use __NRPROCESSORS_CONF instead of __SC_NPROCESSORS_ONLN for non-sequential CPU ids

3 months agocpus_allowed: use __NRPROCESSORS_CONF instead of __SC_NPROCESSORS_ONLN for non-sequen...
mayuanpeng [Fri, 21 Oct 2022 12:18:13 +0000 (20:18 +0800)]
cpus_allowed: use __NRPROCESSORS_CONF instead of __SC_NPROCESSORS_ONLN for non-sequential CPU ids

When disabling SMT on some systems, the ID of some available CPU may
be larger than the value of sysconf(_SC_NPROCESSORS_ONLN). Without
this patch, fio complains that the expected CPU ID is invalid. Here's
an example from my server:

    $ ./fio --cpus_allowed=14 --ioengine=libaio --direct=1 --name=test --numjobs=1 --blocksize=128k --iodepth=1 --rw=read --filename=/dev/nvme0n1
    fio: CPU 14 too large (max=11)
    fio: failed parsing cpus_allowed=14

System information:

    $ lscpu
    Architecture:         x86_64
    CPU op-mode(s):       32-bit, 64-bit
    Byte Order:           Little Endian
    CPU(s):               20
    On-line CPU(s) list:  0,2,4,6,8,10,12,14,16-19
    Off-line CPU(s) list: 1,3,5,7,9,11,13,15
    ...
    Model name:           12th Gen Intel(R) Core(TM) i7-12700
    BIOS Model name:      12th Gen Intel(R) Core(TM) i7-12700
    ...

    $ uname -a
    Linux localhost.localdomain 4.18.0-348.el8.x86_64 #1 SMP Tue Oct 19 15:14:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

    $ cat /etc/redhat-release
    CentOS Linux release 8.5.2111

    $ cat /proc/cmdline
    BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-348.el8.x86_64 root=/dev/mapper/cl-root ro nosmt isolcpus=0,2,4,6,8,10,12,14 crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet

    $ cat test.c
    #include <stdio.h>
    #include <unistd.h>
    int main(int argc, char *argv[])
    {
        printf("_SC_NPROCESSORS_ONLN=%ld _SC_NPROCESSORS_CONF=%ld\n",
               sysconf(_SC_NPROCESSORS_ONLN), sysconf(_SC_NPROCESSORS_CONF));
    }

    $ gcc test.c

    $ ./a.out
    _SC_NPROCESSORS_ONLN=12 _SC_NPROCESSORS_CONF=20

Signed-off-by: mayuanpeng <mayuanpeng@dapustor.com>
3 months agostat: fix segfault with fio option --bandwidth-log
Ankit Kumar [Fri, 21 Oct 2022 06:49:00 +0000 (12:19 +0530)]
stat: fix segfault with fio option --bandwidth-log

The log_params for aggregate read, write and trim only specify
log type. As a result the io_log doesn't have the relevant
thread_data structure. With fio option --bandwidth-log this results
in segmentation fault.
Add a check and use DEF_LOG_ENTRIES for such case.

Fixes: 0a852a50 ("fio: Introduce the log_entries option")
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
[vkf: added Fixes tag, lightly edited commit message]
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agoMerge branch 'issue-1213' of https://github.com/SystemFabricWorks/fio
Vincent Fu [Wed, 19 Oct 2022 16:04:50 +0000 (12:04 -0400)]
Merge branch 'issue-1213' of https://github.com/SystemFabricWorks/fio

* 'issue-1213' of https://github.com/SystemFabricWorks/fio:
  libcufile: use generic_get_file_size
  fix configure probe for libcufile

3 months agolibcufile: use generic_get_file_size
Brian T. Smith [Tue, 18 Oct 2022 18:14:51 +0000 (13:14 -0500)]
libcufile: use generic_get_file_size

In libcufile ioengine_ops, set get_file_size to
generic_get_file_size.

Fixes #1213.

Signed-off-by: Brian T. Smith <bsmith@systemfabricworks.com>
3 months agofix configure probe for libcufile
Brian T. Smith [Mon, 26 Sep 2022 17:59:18 +0000 (12:59 -0500)]
fix configure probe for libcufile

The libcufile probe now requires -ldl passsed to the linker.

Signed-off-by: Brian T. Smith <bsmith@systemfabricworks.com>
3 months agotest: change GitHub actions checkout from v2 to v3
Vincent Fu [Tue, 18 Oct 2022 14:43:54 +0000 (10:43 -0400)]
test: change GitHub actions checkout from v2 to v3

GitHub actions is reporting a warning that actions/checkout@v2 includes
the deprecated Node.js 12. So switch to actions/checkout@v3 which uses
Node.js 16.

As far as I can tell this has no impact on our workflow.

For details see:
https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/
https://github.com/actions/checkout

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agot/zbd: add a CLI option to force io_uring
Dmitry Fomichev [Sun, 16 Oct 2022 01:43:09 +0000 (10:43 +0900)]
t/zbd: add a CLI option to force io_uring

Many modern Linux distros don't include libaio-dev or libaio-devel
package by default and this leads to libaio ioengine being
unavailable in fio instances built at such systems.

Approximately one third of the test cases in the fio test script for
zoned block devices, t/zbd/test-zbd-support, use libaio ioengine.

In order to allow users to run the entire set of ZBD tests without
libaio, introduce a new command line option, -u, to the test script.
When this option is added to the script command line, all tests that
normally use libaio will be modified to use io_uring ioengine.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Link: https://lore.kernel.org/r/20221016014309.53682-2-dmitry.fomichev@wdc.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agot/zbd: fix max_open_zones determination in tests
Dmitry Fomichev [Sun, 16 Oct 2022 01:43:08 +0000 (10:43 +0900)]
t/zbd: fix max_open_zones determination in tests

The test script erroneously falls back to using libzbc for finding out
the max_open_zones setting if the device being tested is not a SCSI
device. This causes false positives if libzbc is not installed in the
test system. To fix, unless the option to use libzbc is explicitly set,
set this value to 0 for non-SCSI devices to let fio find the actual
max_open_zones value using the system block interface.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Link: https://lore.kernel.org/r/20221016014309.53682-1-dmitry.fomichev@wdc.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agofio: warn about "ioengine=psync" and "iodepth >= 1"
Alexey Dobriyan [Sat, 15 Oct 2022 19:41:08 +0000 (22:41 +0300)]
fio: warn about "ioengine=psync" and "iodepth >= 1"

Some users are using both ioengine=psync and iodepth=2+
without blinking in front of the millions:

https://youtu.be/coShLkCriXc?t=840
https://youtu.be/coShLkCriXc?t=890

Help them with helpful message.

Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
Link: https://lore.kernel.org/r/Y0sM1PJKEzfv47ZB@localhost.localdomain
[axboe: style fixup]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoMerge branch 'readme-update' of https://github.com/nikoandpiko/fio
Jens Axboe [Sat, 15 Oct 2022 15:05:32 +0000 (09:05 -0600)]
Merge branch 'readme-update' of https://github.com/nikoandpiko/fio

* 'readme-update' of https://github.com/nikoandpiko/fio:
  Update to README

3 months agoUpdate to README
Nicholas Roma [Sat, 15 Oct 2022 11:41:52 +0000 (20:41 +0900)]
Update to README

Typo and grammatical fix for README

Signed-off-by: Nick Roma <nikoandpiko@gmail.com>
3 months agoengines/io_uring: set coop taskrun, single issuer and defer taskrun
Ankit Kumar [Tue, 11 Oct 2022 10:59:35 +0000 (16:29 +0530)]
engines/io_uring: set coop taskrun, single issuer and defer taskrun

Add missing changes to io_uring_cmd I/O engine for COOP_TASKRUN
This was introduced for io_uring in commit 4d22c103

Add missing changes to io_uring_cmd I/O engine to set single issuer
and defer taskrun. This was introduced for io_uring in commit e453f369

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20221011105935.10455-2-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoexamples: uring-cmd-zoned: expand the reasoning behind QD1
Pankaj Raghav [Fri, 7 Oct 2022 12:05:28 +0000 (14:05 +0200)]
examples: uring-cmd-zoned: expand the reasoning behind QD1

Expand the reasoning behind using QD1 for zoned devices with io_uring_cmd
engine.

Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotest: make t0014.fio time_based
Vincent Fu [Wed, 5 Oct 2022 22:34:41 +0000 (18:34 -0400)]
test: make t0014.fio time_based

Make the job time_based to ensure we run for long enough to obtain the
expected log lines. If the device is too fast fio will consume 100G
and write fewer than the expected log lines.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotest: improve run-fio-tests.py file open method
Vincent Fu [Wed, 5 Oct 2022 20:53:20 +0000 (13:53 -0700)]
test: improve run-fio-tests.py file open method

We have several instances where we try to open a file and fail the test
if accessing the file fails. Encapsulate all of this within a single
method to reduce code duplication.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotest: fix t/run-fio-tests.py style issues identified by pylint
Vincent Fu [Wed, 5 Oct 2022 20:46:40 +0000 (13:46 -0700)]
test: fix t/run-fio-tests.py style issues identified by pylint

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotest: add basic tests for trimwrite workloads
Vincent Fu [Wed, 5 Oct 2022 19:14:25 +0000 (12:14 -0700)]
test: add basic tests for trimwrite workloads

We have some tests for randtrimwrite; it's easy enough to reuse the code
to test trimwrite workloads.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotest: fix style issues in run-fio-tests.py
Vincent Fu [Wed, 5 Oct 2022 19:13:43 +0000 (12:13 -0700)]
test: fix style issues in run-fio-tests.py

Fix some style issues identified by pylint.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotest: check all offsets touched for randtrimwrite
Vincent Fu [Wed, 5 Oct 2022 00:20:09 +0000 (17:20 -0700)]
test: check all offsets touched for randtrimwrite

Make sure that all of the offsets are touched for the randtrimwrite test
when randommap is enabled.

This exposes a problem where mixed ddir workloads that use a random map
may not have all blocks touched when the different data directions use
different block sizes. Resolve this for now by setting the same block
size for all data directions for the job named bs.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotest: clean up randtrimwrite test
Vincent Fu [Tue, 4 Oct 2022 21:10:17 +0000 (14:10 -0700)]
test: clean up randtrimwrite test

Simplify the job file by putting write_bw_log in the [global] section.

Rename check_seq to the clearer check_trimwrite since it makes sure that
trims are followed by writes.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 4 Oct 2022 20:25:09 +0000 (14:25 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Android: Enable the 'sg' engine
  Android: Fix the build of the 'sg' engine