Vincent Fu [Fri, 30 Sep 2022 19:48:56 +0000 (12:48 -0700)]
test: test job for randtrimwrite
This test exposes a problem with randtrimwrite when norandommap is
enabled.
Currently we check to see if the last write and last trim ended at the
same position. If so, we decide that the next operation should be a
trim. This is ok in most cases but if norandommap is enabled, it could
happen by chance that we just finished a trim + write on an offset and
randomly chose that same offset again. This would fool fio into thinking
that it already finished a trim + write pair when it actually issued
only a trim operation.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Thu, 29 Sep 2022 23:58:17 +0000 (16:58 -0700)]
randtrimwrite: write at same offset as trim
We need to do a little bit more to make sure that the writes land on the
offsets that were trimmed.
We only have a single random seed for offsets. So we need to just use
the offset from trim commands when issuing writes.
When we have variable block sizes we need to make sure that the trim and
write commands are the same size.
When randommap is enabled we have to let it adjust the block size for
trim commands to make sure that the trim command does not touch any
blocks that have already been touched.
For sizes of write commands just use the size of the trim command.
Fixes:
c16dc793a3c45780f67ce65244b6e91323dee014 "Add randtrimwrite data
direction"
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Anuj Gupta [Mon, 3 Oct 2022 03:31:52 +0000 (09:01 +0530)]
engines/io_uring: add fixedbufs support for io_uring_cmd
This patch enables fixedbufs support for io_uring_cmd. This has already
been done in t/io_uring, just do it here as well.
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://lore.kernel.org/r/20221003033152.314763-1-anuj20.g@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 28 Sep 2022 15:11:41 +0000 (09:11 -0600)]
Add randtrimwrite data direction
This works like trimwrite, except we use random offsets for the trim
location rather than sequential ones.
Suggested-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 28 Sep 2022 13:28:58 +0000 (07:28 -0600)]
t/io_uring: get rid of useless read barriers
We don't need a read barrier when we have acquire semantics on reading
the CQ ring tail.
We also don't need acquire semantics on the SQ ring head, unless we're
using SQPOLL. A syscall transition will have happened for !SQPOLL.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 27 Sep 2022 14:43:49 +0000 (07:43 -0700)]
doc: build manpage from fio_doc.rst instead of fio_man.rst
Sphinx prints warnings when it encounters duplicate labels. In HOWTO.rst
are labels for int, irange, and bool. We include HOWTO.rst in both
fio_doc.rst and fio_man.rst. Since labels must be unique across all
files, Sphinx prints warnings for these labels.
For an unknown reason, Sphinx previously did not issue warnings for the
duplicate labels mentioned above until 5.2.0. But Sphinx 5.2.1 is now
installed for the macOS 11 image in GitHub Actions. So now we see Sphinx
warnings when building documentation in GitHub Actions.
Our CI treats Sphinx warnings as test failures. So our macOS builds are
marked as failures.
Resolve this problem by eliminating the separate fio_man.rst file and
just building the manpage from the largely equivalent fio_doc.rst.
Successful build with 5.1.1: https://github.com/axboe/fio/actions/runs/
3106980788/jobs/
5034529793
Failed build with 5.2.1: https://github.com/axboe/fio/actions/runs/
3129974184/jobs/
5079696775
Link: https://github.com/sphinx-doc/sphinx/pull/10781
Link: https://github.com/sphinx-doc/sphinx/issues/10870
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Thu, 22 Sep 2022 16:03:51 +0000 (10:03 -0600)]
arm64: ensure CPU clock retrieval issues isb()
This prevents counter value speculation, which will otherwise cause
issues with the CPU clock.
Link: https://github.com/axboe/fio/issues/1472
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Mon, 19 Sep 2022 13:34:13 +0000 (09:34 -0400)]
configure: change grep -P to grep -E
grep -P is not supported on FreeBSD. So use grep -E when we check
whether the value provided for --seed-buckets is a number or not.
Reported-by: Rebecca Cran <rebecca@bsdio.com>
Link: https://lore.kernel.org/fio/26076f36-54d5-ca6e-0b96-a7371b1c5e49@bsdio.com/T/#t
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 14 Sep 2022 17:50:52 +0000 (10:50 -0700)]
gettime: cleanups
Fix a comment and get rid of the extra new lines in the debug print.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Thu, 15 Sep 2022 15:02:49 +0000 (11:02 -0400)]
Merge branch 'fix-example-disk-zone-profile' of github.com:cvubrugier/fio
* 'fix-example-disk-zone-profile' of github.com:cvubrugier/fio:
examples: fix bandwidth logs generation in disk-zone-profile.fio
examples: set zonemode to strided in disk-zone-profile.fio
Vincent Fu [Thu, 15 Sep 2022 14:53:24 +0000 (10:53 -0400)]
Merge branch 'master' of github.com:uniontech-lilinjie/fio
* 'master' of github.com:uniontech-lilinjie/fio:
fix spelling error
Ankit Kumar [Tue, 13 Sep 2022 10:45:27 +0000 (16:15 +0530)]
backend: number of ios not as expected for trimwrite
number_ios should be twice for trimwrite, just like size or
io_size. Update the documentation for "rw=trimwrite" to explain the
changes.
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Reviewed-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220913104527.18734-2-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
lilinjie [Tue, 13 Sep 2022 10:35:20 +0000 (18:35 +0800)]
fix spelling error
Signed-off-by: lilinjie <lilinjie@uniontech.com>
Vincent Fu [Mon, 12 Sep 2022 14:43:17 +0000 (10:43 -0400)]
stat: fix comment about memory consumption
We changed from unsigned int to uint64_t for the latency frequency
counters. Reflect this in the comment describing how much memory the
latency frequency counters require.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Christophe Vu-Brugier [Mon, 12 Sep 2022 12:17:49 +0000 (14:17 +0200)]
examples: fix bandwidth logs generation in disk-zone-profile.fio
Because the job name is set to "/dev/sdb", the bandwidth logs are
written to /dev/sdb_bw.1.log which is unexpected.
This commit changes the job name to "disk-zone-profile" so that the
bandwidth logs are written to "disk-zone-profile_bw.1.log".
Moreover, the `log_offset` option is enabled to log the I/O
offset. This is required to plot the HDD zones.
Signed-off-by: Christophe Vu-Brugier <christophe.vu-brugier@seagate.com>
Christophe Vu-Brugier [Mon, 12 Sep 2022 12:13:04 +0000 (14:13 +0200)]
examples: set zonemode to strided in disk-zone-profile.fio
This fixes the following error:
# fio disk-zone-profile.fio
fio: --zonemode=none and --zonesize are not compatible.
Signed-off-by: Christophe Vu-Brugier <christophe.vu-brugier@seagate.com>
Jens Axboe [Tue, 6 Sep 2022 12:58:48 +0000 (06:58 -0600)]
Merge branch 'rpma-add-support-for-libpmem2-to-the-librpma-engine' of https://github.com/ldorau/fio
* 'rpma-add-support-for-libpmem2-to-the-librpma-engine' of https://github.com/ldorau/fio:
rpma: add support for libpmem2 to librpma engine in GPSPM mode
rpma: add support for libpmem2 to librpma engine in APM mode
rpma: simplify server_cmpl_process()
Jens Axboe [Tue, 6 Sep 2022 12:57:16 +0000 (06:57 -0600)]
Merge branch 'ci-build-the-librpma-fio-engine' of https://github.com/ldorau/fio
* 'ci-build-the-librpma-fio-engine' of https://github.com/ldorau/fio:
ci: remove the unused travis-install-pmdk.sh file
ci: build the librpma fio engine
Lukasz Dorau [Tue, 6 Sep 2022 12:17:40 +0000 (14:17 +0200)]
ci: remove the unused travis-install-pmdk.sh file
Remove the unused travis-install-pmdk.sh file.
The libpmem and libpmem2 libraries are installed from packages.
Signed-off-by: Lukasz Dorau <lukasz.dorau@intel.com>
Lukasz Dorau [Tue, 6 Sep 2022 12:11:43 +0000 (14:11 +0200)]
ci: build the librpma fio engine
Install the librpma library and two required packges
(libpmem2-dev and libprotobuf-c-dev) to have
the librpma fio engine built.
Signed-off-by: Lukasz Dorau <lukasz.dorau@intel.com>
Kacper Stefanski [Thu, 24 Feb 2022 14:59:03 +0000 (15:59 +0100)]
rpma: add support for libpmem2 to librpma engine in GPSPM mode
Add support for libpmem2 to librpma fio engine in the GPSPM mode.
Signed-off-by: Kacper Stefanski <kacper.stefanski@intel.com>
Kacper Stefanski [Sun, 13 Feb 2022 23:02:23 +0000 (00:02 +0100)]
rpma: add support for libpmem2 to librpma engine in APM mode
Add support for libpmem2 to librpma fio engine in the APM mode.
Signed-off-by: Kacper Stefanski <kacper.stefanski@intel.com>
Lukasz Dorau [Mon, 28 Feb 2022 14:25:53 +0000 (15:25 +0100)]
rpma: simplify server_cmpl_process()
Simplify the code of the server_cmpl_process() function.
Signed-off-by: Lukasz Dorau <lukasz.dorau@intel.com>
Jens Axboe [Sat, 3 Sep 2022 17:02:10 +0000 (11:02 -0600)]
t/io_uring: enable support for registered buffers for passthrough
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 3 Sep 2022 17:01:19 +0000 (11:01 -0600)]
t/io_uring: properly detect numa nodes for passthrough mode
We need to use a different path for the char nodes. With this, my
setup correctly detects the nodes for all devices:
submitter=0, tid=4746, file=/dev/ng0n1, node=6
submitter=1, tid=4747, file=/dev/ng1n1, node=6
submitter=2, tid=4748, file=/dev/ng2n1, node=6
submitter=3, tid=4749, file=/dev/ng3n1, node=6
submitter=4, tid=4750, file=/dev/ng4n1, node=4
submitter=5, tid=4751, file=/dev/ng5n1, node=4
submitter=6, tid=4752, file=/dev/ng6n1, node=4
submitter=7, tid=4753, file=/dev/ng7n1, node=4
submitter=8, tid=4754, file=/dev/ng8n1, node=4
submitter=9, tid=4755, file=/dev/ng9n1, node=4
submitter=10, tid=4756, file=/dev/ng10n1, node=4
submitter=12, tid=4758, file=/dev/ng12n1, node=10
submitter=11, tid=4757, file=/dev/ng11n1, node=4
submitter=13, tid=4759, file=/dev/ng13n1, node=10
[...]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 3 Sep 2022 14:49:37 +0000 (08:49 -0600)]
Merge branch 'fix/howto-spelling' of https://github.com/scop/fio
* 'fix/howto-spelling' of https://github.com/scop/fio:
HOWTO: spelling fixes
Jens Axboe [Sat, 3 Sep 2022 14:49:16 +0000 (08:49 -0600)]
Merge branch 'doc/showcmd-usage' of https://github.com/scop/fio
* 'doc/showcmd-usage' of https://github.com/scop/fio:
doc: fix --showcmd usage
Jens Axboe [Sat, 3 Sep 2022 14:48:45 +0000 (08:48 -0600)]
Merge branch 'fix/help-terse-version-5' of https://github.com/scop/fio
* 'fix/help-terse-version-5' of https://github.com/scop/fio:
init: include 5 in --terse-version help
Ville Skyttä [Sat, 3 Sep 2022 06:12:55 +0000 (09:12 +0300)]
doc: fix --showcmd usage
It does not take an argument, but operates on the job files and other
arguments given on the command line.
Thanks-to: Koichi Murase <myoga.murase@gmail.com>
Signed-off-by: Ville Skyttä <ville.skytta@upcloud.com>
Ville Skyttä [Sat, 3 Sep 2022 05:54:03 +0000 (08:54 +0300)]
HOWTO: spelling fixes
Signed-off-by: Ville Skyttä <ville.skytta@upcloud.com>
Ville Skyttä [Sat, 3 Sep 2022 05:49:16 +0000 (08:49 +0300)]
init: include 5 in --terse-version help
Thanks-to: Koichi Murase <myoga.murase@gmail.com>
Signed-off-by: Ville Skyttä <ville.skytta@upcloud.com>
Jens Axboe [Fri, 2 Sep 2022 23:29:45 +0000 (17:29 -0600)]
Merge branch 'fiopr_windows_log_compression_storage_fixes' of https://github.com/PCPartPicker/fio
* 'fiopr_windows_log_compression_storage_fixes' of https://github.com/PCPartPicker/fio:
Fix log compression storage on windows
aggieNick02 [Fri, 2 Sep 2022 23:19:43 +0000 (18:19 -0500)]
Fix log compression storage on windows
Set the file open mode to be binary instead of text when dealing with
compressed log files. This fixes log compression storage not working on
windows, and lets the test added in PR
https://github.com/axboe/fio/pull/1458
pass on windows.
Signed-off-by: Nick Neumann <nick@pcpartpicker.com>
Jens Axboe [Thu, 1 Sep 2022 18:03:23 +0000 (12:03 -0600)]
Merge branch 'fiopr_compressfixes' of https://github.com/PCPartPicker/fio
* 'fiopr_compressfixes' of https://github.com/PCPartPicker/fio:
Fix fio silently dropping log entries when using log_compression
aggieNick02 [Thu, 1 Sep 2022 17:26:45 +0000 (12:26 -0500)]
Fix fio silently dropping log entries when using log_compression
This fixes issue 1457: https://github.com/axboe/fio/issues/1457
The fix is to just add a workqueue_flush to iolog_flush so that any log
chunks being compressed in the workqueue get flushed before we compress
and write the entries in the current active logs. Without this, log
entries in the workqueue can be omitted or be written out of order.
We also add a new test that ensures the log file has the right number of
entries in the correct order when using log_compression, both with and
without the store_log_compressed option.
There is still some cleanup to do with respect to the current
documentation and comments for the log_compression option, as it does
not affect the size of chunks, but instead is effectively a boolean that
controls whether chunks are compressed and removed. I'll leave that for
a separate PR.
Signed-off-by: Nick Neumann <nick@pcpartpicker.com>
Jens Axboe [Thu, 1 Sep 2022 14:34:32 +0000 (08:34 -0600)]
t/io_uring: minor optimizations to IO init fast path
1) Only read SQ ring head at the start of prep, we don't need to read
it at every iteration.
2) Initialize SQ ring index at init time, rather than in the fast path.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 1 Sep 2022 00:44:52 +0000 (18:44 -0600)]
t/io_uring: take advantage of new io_uring setup flags
Set COOP_TASKRUN, SINGLE_ISSUER, and DEFER_TASKRUN if they are available
on the host.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 1 Sep 2022 00:40:14 +0000 (18:40 -0600)]
t/io_uring: unify getting of the offset
Move it into a helper, and use it from all three methods. This also fixes
an issue with the aio method not honoring random IO or not.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 31 Aug 2022 20:14:29 +0000 (14:14 -0600)]
engines/io_uring: set single issuer and defer taskrun
If available, set these two flags as well. SINGLE_ISSUER tells the kernel
that it can expect that it's just a single task issuing requests, and
DEFER_TASKRUN tells the kernel that we're fine with deferring task_work
runs until we reap events.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 31 Aug 2022 19:45:32 +0000 (13:45 -0600)]
engines/io_uring: set COOP_TASKRUN for ring setup
If available, this will generally improve performance quite a bit.
Fio can easily use it, we just need to set the flag.
IORING_SETUP_COOP_TASKRUN tells the kernel that the application doesn't
need to get rescheduled to handle task_work, it's fine to defer this
to when we transition anyway.
For a QD=8 random read workload, this increases performance from
680K to 870K IOPS.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 31 Aug 2022 00:19:30 +0000 (18:19 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Merge os-android.h into os-linux.h
Split os-android.h
Linux: Use the byte order functions from <asm/byteorder.h>
Remove two casts from os-linux.h
Bart Van Assche [Wed, 24 Aug 2022 22:05:42 +0000 (15:05 -0700)]
Merge os-android.h into os-linux.h
Reduce code duplication by merging the os-android.h and os-linux.h
header files. The only functional change in this patch is that
FIO_HAVE_SGIO is now defined in the Android build and hence that the sg
I/O engine is enabled for Android.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Wed, 24 Aug 2022 21:55:51 +0000 (14:55 -0700)]
Split os-android.h
Move the shared memory emulation into a new header file in preparation
of merging the Linux and Android operating system header files.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Wed, 24 Aug 2022 22:05:32 +0000 (15:05 -0700)]
Linux: Use the byte order functions from <asm/byteorder.h>
Prepare for the unification of the Linux and Android header files by
switching to the <asm/byteorder.h> functions for Linux.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Wed, 24 Aug 2022 21:54:39 +0000 (14:54 -0700)]
Remove two casts from os-linux.h
This patch prepares for merging the os-linux.h and os-android.h header files.
While CPU_CLR() and CPU_SET() are defined as statement expressions in
the Linux header files, these macros have been defined as statements in
the Android header files. Remove the '(void)' casts to prevent the
introduction of a build error when merging the Linux and Android header
files.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Tue, 30 Aug 2022 16:51:13 +0000 (10:51 -0600)]
Fio 3.32
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 30 Aug 2022 16:48:18 +0000 (10:48 -0600)]
backend: revert bad memory leak fix
This essentially reverts the commit mentioned in the fixes line, as it
causes crashes with using a trigger timeout + command.
Fixes:
807473c36e10 ("fixed memory leak detected by ASAN")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 30 Aug 2022 13:59:55 +0000 (09:59 -0400)]
test: add tests for lfsr and norandommap
t0021 checks whether the lfsr random generator actually touches every
offset.
t0022 checks whether fio touches offsets more than once when
norandommap=1.
We should have automated tests for basic functionality to detect
problems early.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Mon, 29 Aug 2022 19:15:56 +0000 (15:15 -0400)]
test: get 32-bit Ubuntu 22.04 build working
Ubuntu 22.04 no longer has i386 builds for the packages libibverbs and
librdmacm. So stop trying to install those packages for the 32-bit
build.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Mon, 29 Aug 2022 18:24:16 +0000 (14:24 -0400)]
test: use Ubuntu 22.04 for 64-bit tests
On 22.04 there was a conflict among libunwind-14-dev, libunwind-dev, and
libunwind8 that was resolved by removing libunwind-14-dev.
The 32-bit Ubuntu setup steps require more attention to get them to work
on 22.04. Stay on 20.04 for now and figure it out later.
Starting pkgProblemResolver with broken count: 1
Starting 2 pkgProblemResolver with broken count: 1
Investigating (0) libunwind-14-dev:amd64 < 1:14.0.0-1ubuntu1 @ii K Ib >
Broken libunwind-14-dev:amd64 Breaks on libunwind-dev:amd64 < none -> 1.3.2-2build2 @un puN >
Considering libunwind-dev:amd64 -1 as a solution to libunwind-14-dev:amd64 2
Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
libunwind-14-dev : Breaks: libunwind-dev but 1.3.2-2build2 is to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Mon, 29 Aug 2022 15:30:30 +0000 (11:30 -0400)]
test: add some tests for seq and rand offsets
t/jobs/t0019.fio is a seq read test
t/jobs/t0020.fio is a rand read test
We don't have any automated tests which make sure that sequential access
patterns are actually sequential and that random access patterns are not
sequential. Add these two tests to help detect the possibility that
these features could break.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Fri, 26 Aug 2022 20:14:44 +0000 (14:14 -0600)]
t/io_uring: remove duplicate definition of gettid()
With a recent change, we now include os.h through nvme.h, and this
can cause a duplicate gettid() definition:
t/io_uring.c:499:12: error: redefinition of ‘gettid’
static int gettid(void)
^~~~~~
In file included from t/../engines/../os/os.h:39,
from t/../engines/../thread_options.h:5,
from t/../engines/../fio.h:18,
from t/../engines/nvme.h:10,
from t/io_uring.c:38:
t/../engines/../os/os-linux.h:147:19: note: previous definition of
‘gettid’ was here
static inline int gettid(void)
^~~~~~
Include os.h directly to make it clear that we use it, and remove the
gettid() definition from io_uring.c.
Reported-by: Yi Zhang <yi.zhang@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Thu, 25 Aug 2022 19:08:33 +0000 (12:08 -0700)]
test: add basic test for io_uring ioengine
We should have a quick smoke test for the io_uring ioengine to
automatically detect breakage.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Fri, 26 Aug 2022 13:52:54 +0000 (07:52 -0600)]
t/io_uring: fix 64-bit cast on 32-bit archs
gcc complains that:
t/io_uring.c: In function ‘init_io_pt’:
t/io_uring.c:618:52: error: left shift count >= width of type [-Werror=shift-count-overflow]
618 | sqe->user_data |= ((unsigned long)s->clock_index << 32);
| ^~
we're shifting more than the size of the type. Cast to a 64-bit value
so that it'll work on 32-bit as well.
Fixes:
7d04588a7663 ("t/io_uring: add support for async-passthru")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Anuj Gupta [Fri, 26 Aug 2022 11:33:06 +0000 (17:03 +0530)]
t/io_uring: add support for async-passthru
This patch adds support for async-passthru in t/io_uring. User needs to
specify -u1 option in the command
Example commandline:
t/io_uring -b512 -d128 -c32 -s32 -p0 -F1 -B0 -O0 -n1 -u1 /dev/ng0n1
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://lore.kernel.org/r/20220826113306.4139-3-anuj20.g@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Anuj Gupta [Fri, 26 Aug 2022 11:33:05 +0000 (17:03 +0530)]
t/io_uring: prep for including engines/nvme.h in t/io_uring
Change page_size and cal_clat_percentiles name to something different
as these are indirectly picked from engines/nvme.h (fio.h and stat.h)
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://lore.kernel.org/r/20220826113306.4139-2-anuj20.g@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 25 Aug 2022 17:19:34 +0000 (11:19 -0600)]
engines/io_uring: delete debug code
This was inadvertently introduced by a previous commit, get rid
of it.
Fixes:
1816895b788e ("engines/io_uring: pass back correct error value when interrupted")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 25 Aug 2022 02:09:29 +0000 (20:09 -0600)]
Merge branch 'master' of https://github.com/kraj/fio
* 'master' of https://github.com/kraj/fio:
io_uring: Replace pthread_self with s->tid
Khem Raj [Thu, 25 Aug 2022 01:08:53 +0000 (18:08 -0700)]
io_uring: Replace pthread_self with s->tid
__init_rand64 takes 64bit value and srand48 takes unsigned 32bit value,
pthread_t is opaque type and some libcs ( e.g. musl ) do not define them
in plain old data types and ends up with errors
| t/io_uring.c:809:32: error: incompatible pointer to integer conversion passing 'pthread_t' (aka 'struct __pthread *') to parameter of type 'uint64_t' (aka 'unsigned long') [-Wint-conver
sion]
| __init_rand64(&s->rand_state, pthread_self());
| ^~~~~~~~~~~~~~
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Jens Axboe [Wed, 24 Aug 2022 21:52:34 +0000 (15:52 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Enable CPU affinity support on Android
Bart Van Assche [Wed, 24 Aug 2022 20:42:29 +0000 (13:42 -0700)]
Enable CPU affinity support on Android
This patch enables the --cpumask=, --cpus_allowed= and
--cpus_allowed_policy= fio options.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Wed, 24 Aug 2022 18:01:39 +0000 (12:01 -0600)]
engines/io_uring: pass back correct error value when interrupted
Running with an io_uring engine and using a USR1 signal to show
current status will end up terminating the job with:
fio: pid=91726, err=-4/file:ioengines.c:320, func=get_events, error=Unknown error -4
sfx: (groupid=0, jobs=1): err=-4 (file:ioengines.c:320, func=get_events, error=Unknown error -4): pid=91726: Wed Aug 24 11:59:51 2022
Ensure that the return value is set correctly based on the errno.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 23 Aug 2022 21:09:25 +0000 (17:09 -0400)]
Merge branch 'sigbreak-wait' of github.com:bjpaupor/fio
* 'sigbreak-wait' of github.com:bjpaupor/fio:
Add wait for handling SIGBREAK
Vincent Fu [Tue, 23 Aug 2022 16:38:51 +0000 (12:38 -0400)]
Revert "Fix multithread issues when operating on a single shared file"
This reverts commit
acbda87c34c743ff2d9e125d9539bcfbbf49eb75.
This commit introduced a lot of unintended consequences for
create_serialize=0. The aim of the commit can be accomplished with a
combination of filesize and io_size.
Fixes: https://github.com/axboe/fio/issues/1442
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Tue, 23 Aug 2022 16:38:20 +0000 (12:38 -0400)]
Revert "Minor style fixups"
This reverts commit
48f8268e88629d408ffd09b1601ad13366bd4ce1.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Brandon Paupore [Fri, 5 Aug 2022 17:57:27 +0000 (12:57 -0500)]
Add wait for handling SIGBREAK
When closing a command prompt window or terminating it using something
like the taskkill command, each child process (such as a running FIO
workload) is sent a SIGBREAK signal. Once those child processes have
responded to that signal, Windows terminates them if they're still
executing.
This change has the main thread to wait for others to exit when handling
a SIGBREAK signal, such that each job will still have time to wrap-up
and give stats before the entire program terminates.
Signed-off-by: Brandon Paupore <brandon.paupore@wdc.com>
Jens Axboe [Wed, 17 Aug 2022 01:54:17 +0000 (19:54 -0600)]
Merge branch 'clarify-io-errors' of https://github.com/Hi-Angel/fio
* 'clarify-io-errors' of https://github.com/Hi-Angel/fio:
doc: clarify that I/O errors may go unnoticed without direct=1
doc: get rid of trailing whitespace
Konstantin Kharlamov [Tue, 16 Aug 2022 16:10:38 +0000 (19:10 +0300)]
doc: clarify that I/O errors may go unnoticed without direct=1
Fixes: https://github.com/axboe/fio/issues/1443
Reported-by: Konstantin Kharlamov <Hi-Angel@yandex.ru>
Signed-off-by: Konstantin Kharlamov <Hi-Angel@yandex.ru>
Vincent Fu [Mon, 15 Aug 2022 15:40:58 +0000 (11:40 -0400)]
test: fix hash for t0016
I used the wrong hash for t0016 in the original commit. Fix it to refer
to the hash that fixed the issue in this tree.
Fixes:
de31fe9a ("testing: add test for slat + clat = tlat")
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Mon, 15 Aug 2022 15:34:43 +0000 (11:34 -0400)]
test: add latency test using posixaio ioengine
Make sure that mean(slat) + mean(clat) = mean(total lat).
Tests 15 and 16 use the libaio and null ioengines, respectively. Both of
those ioengines have commit hooks. Add this new test using the posixaio
ioengine which does not have a commit hook so that we can better cover
the possible ways that latency is calcualted.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Konstantin Kharlamov [Tue, 16 Aug 2022 16:14:13 +0000 (19:14 +0300)]
doc: get rid of trailing whitespace
Signed-off-by: Konstantin Kharlamov <Hi-Angel@yandex.ru>
Ankit Kumar [Tue, 16 Aug 2022 05:38:21 +0000 (11:08 +0530)]
doc: update fio doc for xnvme engine
- Elaborate about the various sync, async and admin
interfaces.
- add missing io_uring_cmd async backend entry.
- xnvme ioengine doesn't support file stat.
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220816053821.440-3-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Ankit Kumar [Tue, 16 Aug 2022 05:38:20 +0000 (11:08 +0530)]
engines/xnvme: fix segfault issue with xnvme ioengine
fix segfault when xnvme ioengine is called without thread=1.
The segfault happens because td->io_ops_data is accessed at
two locations xnvme_fioe_cleanup and xnvme_fioe_iomem_free,
during the error handling call.
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220816053821.440-2-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Mon, 15 Aug 2022 14:37:57 +0000 (10:37 -0400)]
README: link to GitHub releases for Windows
Note that Windows installers are now available as releases on GitHub.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Thu, 11 Aug 2022 19:39:02 +0000 (15:39 -0400)]
Merge branch 's3_crypto' of github.com:hualongfeng/fio
Friendy.Su@sony.com [Mon, 8 Aug 2022 08:35:50 +0000 (08:35 +0000)]
ioengines: merge filecreate, filestat, filedelete engines to fileoperations.c
file operation engines have similar structure, implement them
in one file.
Signed-off-by: friendy-su <friendy.su@sony.com>
Feng, Hualong [Thu, 28 Jul 2022 01:47:48 +0000 (01:47 +0000)]
doc: Add usage and example about s3 storage class and crypto
There add option usage about s3 storage class
`http_s3_storage_class` and
s3 SSE server side encryption
`http_s3_sse_customer_key` and
`http_s3_sse_customer_algorithm`
And add example file in example folder.
Signed-off-by: Feng, Hualong <hualong.feng@intel.com>
Feng, Hualong [Wed, 20 Jul 2022 01:41:35 +0000 (09:41 +0800)]
engines/http: Add s3 crypto options for s3
Server-side encryption is about protecting data at rest.
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html)
When we want to test server-side encryption, we need to specify
server-side encryption with customer-provided encryption keys (SSE-C).
The two option **http_s3_sse_customer_key** and
**http_s3_sse_customer_algorithm** is for server-side encryption.
Signed-off-by: Feng, Hualong <hualong.feng@intel.com>
Feng, Hualong [Wed, 20 Jul 2022 04:01:35 +0000 (12:01 +0800)]
engines/http: Add storage class option for s3
Amazon S3 offers a range of storage classes that you can choose from
based on the data access, resiliency, and cost requirements of your
workloads. (https://aws.amazon.com/s3/storage-classes/)
For example, we have **STANDARD** storage class to test normal
workload, and have **COLD** storage class to test the workload
with gzip compression. It is convenient to select which
storage class to access for different kinds data.
Signed-off-by: Feng, Hualong <hualong.feng@intel.com>
Jens Axboe [Wed, 10 Aug 2022 15:51:49 +0000 (09:51 -0600)]
lib/rand: get rid of unused MAX_SEED_BUCKETS
It's only used to size the array, we don't need it.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 10 Aug 2022 15:50:52 +0000 (09:50 -0600)]
Merge branch 'multi_seed_refill' of https://github.com/sungup/fio
* 'multi_seed_refill' of https://github.com/sungup/fio:
lib/rand: Enhance __fill_random_buf using the multi random seed
Sungup Moon [Mon, 8 Aug 2022 08:21:46 +0000 (17:21 +0900)]
lib/rand: Enhance __fill_random_buf using the multi random seed
The __fill_random_buf fills a buffer using the random 8byte integer to
write. But, this mechanism is depend on the CPU performance and could
not reach the max performance on the PCIe Gen5 devices. I have tested
128KB single worker sequential write on PCIe Gen5 NVMe, but it cannot
reach write throughput 6.0GB/s.
So, I have reviewed the __fill_random_buf and focused the multiplier
dependency to generate the random number. So, I have changed
__fill_random_buf using the multiple-random-seed to reduce the
dependencies in the small data filling loop.
I'll attach detail analysis result in the PR of this branch.
Signed-off-by: Sungup Moon <sungup.moon@samsung.com>
Jens Axboe [Tue, 9 Aug 2022 20:41:25 +0000 (14:41 -0600)]
Fio 3.31
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 9 Aug 2022 14:45:53 +0000 (10:45 -0400)]
ci: drop master branch requirement for AppVeyor releases
Even if we tag a commit on the master branch AppVeyor won't upload the
installers as GitHub release artifacts.
https://ci.appveyor.com/project/axboe/fio/builds/
44416420/job/t1elejqo2yk3r72b
So just drop the requirement for AppVeyor to only upload release
artifacts from the master branch.
This seems to get things working on my fork:
https://ci.appveyor.com/project/vincentkfu/fio/builds/
44417490/job/e6xtrwc6xlccrn0j
It's good to have the flexibility to store installers from other
branches in any case.
Signed-off-by: Vincent Fu <vincentfu@gmail.com>
Jens Axboe [Tue, 9 Aug 2022 13:07:39 +0000 (07:07 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio
* 'master' of ssh://git.kernel.dk/data/git/fio:
testing: add test for slat + clat = tlat
engines/null: add FIO_ASYNCIO_SETS_ISSUE_TIME flag
testing: add test for slat + clat = tlat
Vincent Fu [Mon, 8 Aug 2022 17:51:51 +0000 (17:51 +0000)]
ci: upload tagged AppVeyor installers as GitHub releases
Deploy Windows installers built by AppVeyor as releases on GitHub. This
process is triggered only when tags are uploaded to the repository.
Patch was created using this guide:
https://www.appveyor.com/docs/deployment/github/
- Generate a GitHub authentication token with public_repo scope:
https://github.com/settings/tokens
- Login to AppVeyor and encrypt GitHub token: Account -> Encrypt YAML
https://ci.appveyor.com/tools/encrypt
- Insert encrypted token in appveyor.yml on the secure: line under
auth_token:
With APPVEYOR_REPO_TAG set to true only tags pushed to GitHub will have
Cygwin MSI installers uploaded as releases to GitHub.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220808175133.37920-2-vincent.fu@samsung.com
[axboe: replace token with right one]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Fri, 5 Aug 2022 20:35:30 +0000 (13:35 -0700)]
testing: add test for slat + clat = tlat
Add this test for the null ioengine which can run on more platforms.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 5 Aug 2022 20:31:11 +0000 (13:31 -0700)]
engines/null: add FIO_ASYNCIO_SETS_ISSUE_TIME flag
Add a flag to prevent td_io_queue from setting issue_time since this
ioengine already does it in its commit function when iodepth > 1.
We also need to make sure the changes to the ioengine flags are
propogated to td->flags.
Fixes:
56b6bc25 ("engines/null: fill issue_time during commit")
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 5 Aug 2022 20:07:21 +0000 (13:07 -0700)]
testing: add test for slat + clat = tlat
The sum of submission and completion latency should equal total latency.
Add a test case for this for the libaio ioengine.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 3 Aug 2022 19:21:11 +0000 (12:21 -0700)]
filesetup: use correct random seed for non-uniform distributions
The index in the random seed array for generating offsets is
FIO_RAND_BLOCK_OFF. So this is the index that should be used to find the
random seed when fio generates offsets following the Zipf, Pareto, and
Gaussian distributions. The previous index 4 actually corresponds to
FIO_RAND_MIX_OFF.
This change means that the default sequences of non-uniform random
offsets generated before and after this patch will differ. So users
relying on the repeatability of I/O patterns will have new repeatable
patterns after this change.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 3 Aug 2022 19:20:56 +0000 (12:20 -0700)]
engines/null: fill issue_time during commit
For iodepth > 1, the null ioengine is an asynchronous ioengine, yet it
does not report submission latency. This patch adds code for setting
issue_time in order to calculate submission latency.
The extra work does reduce performance for iodepth > 1 but this
reduction can be mitigated with options like gtod_reduce.
- there are no differeces at qd 1 (5837K vs 5864K)
- default options at qd 256 shows an 8% IOPS reduction (3583K vs 3303K).
- gtod_reduce=1 at qd 256 shows a 3% IOPS reduction (5119K vs 4966K).
$ ../fio-canonical/fio --name=test --ioengine=null --size=1T --norandommap --ramp_time=5s --runtime=15s --cpus_allowed=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.30-73-gd622
<snip>
read: IOPS=5837k, BW=22.3GiB/s (23.9GB/s)(334GiB/15001msec)
<snip>
$ ./fio --name=test --ioengine=null --size=1T --norandommap --ramp_time=5s --runtime=15s --cpus_allowed=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.30-74-gcfbc2
<snip>
read: IOPS=5864k, BW=22.4GiB/s (24.0GB/s)(336GiB/15001msec)
<snip>
$ ../fio-canonical/fio --name=test --ioengine=null --size=1T --norandommap --iodepth=256 --ramp_time=5s --runtime=15s --cpus_allowed=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=256
fio-3.30-73-gd622
<snip>
read: IOPS=3583k, BW=13.7GiB/s (14.7GB/s)(205GiB/15001msec)
<snip>
$ ./fio --name=test --ioengine=null --size=1T --norandommap --iodepth=256 --ramp_time=5s --runtime=15s --cpus_allowed=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=256
fio-3.30-74-gcfbc2
<snip>
read: IOPS=3303k, BW=12.6GiB/s (13.5GB/s)(189GiB/15001msec)
<snip>
$ ../fio-canonical/fio --name=test --ioengine=null --size=1T --norandommap --iodepth=256 --ramp_time=5s --runtime=15s --cpus_allowed=1 --gtod_reduce=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=256
fio-3.30-73-gd622
<snip>
read: IOPS=5119k, BW=19.5GiB/s (21.0GB/s)(293GiB/15001msec)
<snip>
$ ./fio --name=test --ioengine=null --size=1T --norandommap --iodepth=256 --ramp_time=5s --runtime=15s --cpus_allowed=1 --gtod_reduce=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=256
fio-3.30-74-gcfbc2
<snip>
read: IOPS=4966k, BW=18.9GiB/s (20.3GB/s)(284GiB/15001msec)
<snip>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Wed, 3 Aug 2022 17:07:36 +0000 (13:07 -0400)]
examples: fix ioengine in zbd-rand-write.fio
Make the comment and actual ioengine setting consistent to reduce
possible confusion.
Fixes: https://github.com/axboe/fio/issues/1433
Reported-by: thunderex@gmail.com
Signed-off-by: Vincent Fu <vincentfu@gmail.com>
Jens Axboe [Tue, 2 Aug 2022 16:20:31 +0000 (10:20 -0600)]
Merge branch 'io_uring-numa'
* io_uring-numa:
t/io_uring: support NUMA placement
Jens Axboe [Tue, 2 Aug 2022 15:47:12 +0000 (09:47 -0600)]
t/io_uring: support NUMA placement
A few other changes in here as well that I didn't bother splitting
out:
- Use M for millions of IOPS
- Cleanup the init prints a bit
- Don't track file depths, becomes prohibitive on large number of files
or devices.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 31 Jul 2022 18:06:12 +0000 (12:06 -0600)]
t/io_uring: switch to GiB/sec if numbers get large
Easier to read if we're above 2GiB/sec in bandwidth.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Wed, 27 Jul 2022 15:43:53 +0000 (15:43 +0000)]
.github: add pull request template
We frequently remind contributors about expectations for commit
messages. Add a template so that guidelines show up when contributors
open a pull request on GitHub.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Thu, 28 Jul 2022 03:04:31 +0000 (21:04 -0600)]
README: add maintainer section
Note that Vincent is also maintaining fio - and while in there, also
make a note not to email any of us directly. The public resources
should be used instead.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 28 Jul 2022 02:56:32 +0000 (20:56 -0600)]
Minor style fixups
Missing space for a comparison, and braces that aren't really needed.
Fixes:
acbda87c34c7 ("Fix multithread issues when operating on a single shared file")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 28 Jul 2022 02:54:30 +0000 (20:54 -0600)]
Merge branch 'proposed_fix' of https://github.com/weberc-ntap/fio
* 'proposed_fix' of https://github.com/weberc-ntap/fio:
Fix multithread issues when operating on a single shared file
Chris Weber [Fri, 22 Jul 2022 03:34:02 +0000 (03:34 +0000)]
Fix multithread issues when operating on a single shared file
When nrfiles=1, numjobs>1 and create_serialize=0, multiple threads
try to create the single shared file in parallel. If the file was
pre-existing, but an incorrect size, then multiple threads are
deleting and creating at the same time. When all of this happens
in parallel, there is a chance that the file can end up the
incorrect size (the chance increases as numjobs increases).
These changes handle the corner case described above by having
a single thread create/extend the file prior to running all of
the threads in parallel. By doing this step early, when
setup_files() is called later, it should no longer need to
create or extend the file, avoiding the race condition. The user
still needs to set a fallocate option other than 'none' or the
file will end up 0 bytes in size and the race condition will
still occur. It would be simple to add a ftruncate() to the code
to force this, but that would override the user's choice of
fallocate options.
Signed-off-by: Chris Weber <weberc@netapp.com>