fio.git
3 months agolib/rand: get rid of unused MAX_SEED_BUCKETS
Jens Axboe [Wed, 10 Aug 2022 15:51:49 +0000 (09:51 -0600)]
lib/rand: get rid of unused MAX_SEED_BUCKETS

It's only used to size the array, we don't need it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoMerge branch 'multi_seed_refill' of https://github.com/sungup/fio
Jens Axboe [Wed, 10 Aug 2022 15:50:52 +0000 (09:50 -0600)]
Merge branch 'multi_seed_refill' of https://github.com/sungup/fio

* 'multi_seed_refill' of https://github.com/sungup/fio:
  lib/rand: Enhance __fill_random_buf using the multi random seed

3 months agolib/rand: Enhance __fill_random_buf using the multi random seed
Sungup Moon [Mon, 8 Aug 2022 08:21:46 +0000 (17:21 +0900)]
lib/rand: Enhance __fill_random_buf using the multi random seed

The __fill_random_buf fills a buffer using the random 8byte integer to
write. But, this mechanism is depend on the CPU performance and could
not reach the max performance on the PCIe Gen5 devices. I have tested
128KB single worker sequential write on PCIe Gen5 NVMe, but it cannot
reach write throughput 6.0GB/s.

So, I have reviewed the __fill_random_buf and focused the multiplier
dependency to generate the random number. So, I have changed
__fill_random_buf using the multiple-random-seed to reduce the
dependencies in the small data filling loop.

I'll attach detail analysis result in the PR of this branch.

Signed-off-by: Sungup Moon <sungup.moon@samsung.com>
3 months agoFio 3.31 fio-3.31
Jens Axboe [Tue, 9 Aug 2022 20:41:25 +0000 (14:41 -0600)]
Fio 3.31

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoci: drop master branch requirement for AppVeyor releases test-tag-2022-08-09-2
Vincent Fu [Tue, 9 Aug 2022 14:45:53 +0000 (10:45 -0400)]
ci: drop master branch requirement for AppVeyor releases

Even if we tag a commit on the master branch AppVeyor won't upload the
installers as GitHub release artifacts.

https://ci.appveyor.com/project/axboe/fio/builds/44416420/job/t1elejqo2yk3r72b

So just drop the requirement for AppVeyor to only upload release
artifacts from the master branch.

This seems to get things working on my fork:

https://ci.appveyor.com/project/vincentkfu/fio/builds/44417490/job/e6xtrwc6xlccrn0j

It's good to have the flexibility to store installers from other
branches in any case.

Signed-off-by: Vincent Fu <vincentfu@gmail.com>
3 months agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio test-tag-2022-08-09
Jens Axboe [Tue, 9 Aug 2022 13:07:39 +0000 (07:07 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

* 'master' of ssh://git.kernel.dk/data/git/fio:
  testing: add test for slat + clat = tlat
  engines/null: add FIO_ASYNCIO_SETS_ISSUE_TIME flag
  testing: add test for slat + clat = tlat

3 months agoci: upload tagged AppVeyor installers as GitHub releases
Vincent Fu [Mon, 8 Aug 2022 17:51:51 +0000 (17:51 +0000)]
ci: upload tagged AppVeyor installers as GitHub releases

Deploy Windows installers built by AppVeyor as releases on GitHub. This
process is triggered only when tags are uploaded to the repository.

Patch was created using this guide:
https://www.appveyor.com/docs/deployment/github/

- Generate a GitHub authentication token with public_repo scope:
  https://github.com/settings/tokens
- Login to AppVeyor and encrypt GitHub token: Account -> Encrypt YAML
  https://ci.appveyor.com/tools/encrypt
- Insert encrypted token in appveyor.yml on the secure: line under
  auth_token:

With APPVEYOR_REPO_TAG set to true only tags pushed to GitHub will have
Cygwin MSI installers uploaded as releases to GitHub.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220808175133.37920-2-vincent.fu@samsung.com
[axboe: replace token with right one]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agotesting: add test for slat + clat = tlat
Vincent Fu [Fri, 5 Aug 2022 20:35:30 +0000 (13:35 -0700)]
testing: add test for slat + clat = tlat

Add this test for the null ioengine which can run on more platforms.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agoengines/null: add FIO_ASYNCIO_SETS_ISSUE_TIME flag
Vincent Fu [Fri, 5 Aug 2022 20:31:11 +0000 (13:31 -0700)]
engines/null: add FIO_ASYNCIO_SETS_ISSUE_TIME flag

Add a flag to prevent td_io_queue from setting issue_time since this
ioengine already does it in its commit function when iodepth > 1.

We also need to make sure the changes to the ioengine flags are
propogated to td->flags.

Fixes: 56b6bc25 ("engines/null: fill issue_time during commit")
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agotesting: add test for slat + clat = tlat
Vincent Fu [Fri, 5 Aug 2022 20:07:21 +0000 (13:07 -0700)]
testing: add test for slat + clat = tlat

The sum of submission and completion latency should equal total latency.
Add a test case for this for the libaio ioengine.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agofilesetup: use correct random seed for non-uniform distributions
Vincent Fu [Wed, 3 Aug 2022 19:21:11 +0000 (12:21 -0700)]
filesetup: use correct random seed for non-uniform distributions

The index in the random seed array for generating offsets is
FIO_RAND_BLOCK_OFF. So this is the index that should be used to find the
random seed when fio generates offsets following the Zipf, Pareto, and
Gaussian distributions. The previous index 4 actually corresponds to
FIO_RAND_MIX_OFF.

This change means that the default sequences of non-uniform random
offsets generated before and after this patch will differ. So users
relying on the repeatability of I/O patterns will have new repeatable
patterns after this change.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agoengines/null: fill issue_time during commit
Vincent Fu [Wed, 3 Aug 2022 19:20:56 +0000 (12:20 -0700)]
engines/null: fill issue_time during commit

For iodepth > 1, the null ioengine is an asynchronous ioengine, yet it
does not report submission latency. This patch adds code for setting
issue_time in order to calculate submission latency.

The extra work does reduce performance for iodepth > 1 but this
reduction can be mitigated with options like gtod_reduce.

- there are no differeces at qd 1 (5837K vs 5864K)
- default options at qd 256 shows an 8% IOPS reduction (3583K vs 3303K).
- gtod_reduce=1 at qd 256 shows a 3% IOPS reduction (5119K vs 4966K).

$ ../fio-canonical/fio --name=test --ioengine=null --size=1T --norandommap --ramp_time=5s --runtime=15s --cpus_allowed=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.30-73-gd622
<snip>
  read: IOPS=5837k, BW=22.3GiB/s (23.9GB/s)(334GiB/15001msec)
<snip>

$ ./fio --name=test --ioengine=null --size=1T --norandommap --ramp_time=5s --runtime=15s --cpus_allowed=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.30-74-gcfbc2
<snip>
  read: IOPS=5864k, BW=22.4GiB/s (24.0GB/s)(336GiB/15001msec)
<snip>

$ ../fio-canonical/fio --name=test --ioengine=null --size=1T --norandommap --iodepth=256 --ramp_time=5s --runtime=15s --cpus_allowed=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=256
fio-3.30-73-gd622
<snip>
  read: IOPS=3583k, BW=13.7GiB/s (14.7GB/s)(205GiB/15001msec)
<snip>

$ ./fio --name=test --ioengine=null --size=1T --norandommap --iodepth=256 --ramp_time=5s --runtime=15s --cpus_allowed=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=256
fio-3.30-74-gcfbc2
<snip>
  read: IOPS=3303k, BW=12.6GiB/s (13.5GB/s)(189GiB/15001msec)
<snip>

$ ../fio-canonical/fio --name=test --ioengine=null --size=1T --norandommap --iodepth=256 --ramp_time=5s --runtime=15s --cpus_allowed=1 --gtod_reduce=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=256
fio-3.30-73-gd622
<snip>
  read: IOPS=5119k, BW=19.5GiB/s (21.0GB/s)(293GiB/15001msec)
<snip>

$ ./fio --name=test --ioengine=null --size=1T --norandommap --iodepth=256 --ramp_time=5s --runtime=15s --cpus_allowed=1 --gtod_reduce=1
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=256
fio-3.30-74-gcfbc2
<snip>
  read: IOPS=4966k, BW=18.9GiB/s (20.3GB/s)(284GiB/15001msec)
<snip>

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
4 months agoexamples: fix ioengine in zbd-rand-write.fio
Vincent Fu [Wed, 3 Aug 2022 17:07:36 +0000 (13:07 -0400)]
examples: fix ioengine in zbd-rand-write.fio

Make the comment and actual ioengine setting consistent to reduce
possible confusion.

Fixes: https://github.com/axboe/fio/issues/1433
Reported-by: thunderex@gmail.com
Signed-off-by: Vincent Fu <vincentfu@gmail.com>
4 months agoMerge branch 'io_uring-numa'
Jens Axboe [Tue, 2 Aug 2022 16:20:31 +0000 (10:20 -0600)]
Merge branch 'io_uring-numa'

* io_uring-numa:
  t/io_uring: support NUMA placement

4 months agot/io_uring: support NUMA placement
Jens Axboe [Tue, 2 Aug 2022 15:47:12 +0000 (09:47 -0600)]
t/io_uring: support NUMA placement

A few other changes in here as well that I didn't bother splitting
out:

- Use M for millions of IOPS
- Cleanup the init prints a bit
- Don't track file depths, becomes prohibitive on large number of files
  or devices.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agot/io_uring: switch to GiB/sec if numbers get large
Jens Axboe [Sun, 31 Jul 2022 18:06:12 +0000 (12:06 -0600)]
t/io_uring: switch to GiB/sec if numbers get large

Easier to read if we're above 2GiB/sec in bandwidth.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months ago.github: add pull request template
Vincent Fu [Wed, 27 Jul 2022 15:43:53 +0000 (15:43 +0000)]
.github: add pull request template

We frequently remind contributors about expectations for commit
messages. Add a template so that guidelines show up when contributors
open a pull request on GitHub.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
4 months agoREADME: add maintainer section
Jens Axboe [Thu, 28 Jul 2022 03:04:31 +0000 (21:04 -0600)]
README: add maintainer section

Note that Vincent is also maintaining fio - and while in there, also
make a note not to email any of us directly. The public resources
should be used instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoMinor style fixups
Jens Axboe [Thu, 28 Jul 2022 02:56:32 +0000 (20:56 -0600)]
Minor style fixups

Missing space for a comparison, and braces that aren't really needed.

Fixes: acbda87c34c7 ("Fix multithread issues when operating on a single shared file")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoMerge branch 'proposed_fix' of https://github.com/weberc-ntap/fio
Jens Axboe [Thu, 28 Jul 2022 02:54:30 +0000 (20:54 -0600)]
Merge branch 'proposed_fix' of https://github.com/weberc-ntap/fio

* 'proposed_fix' of https://github.com/weberc-ntap/fio:
  Fix multithread issues when operating on a single shared file

4 months agoFix multithread issues when operating on a single shared file
Chris Weber [Fri, 22 Jul 2022 03:34:02 +0000 (03:34 +0000)]
Fix multithread issues when operating on a single shared file

When nrfiles=1, numjobs>1 and create_serialize=0, multiple threads
try to create the single shared file in parallel. If the file was
pre-existing, but an incorrect size, then multiple threads are
deleting and creating at the same time. When all of this happens
in parallel, there is a chance that the file can end up the
incorrect size (the chance increases as numjobs increases).
These changes handle the corner case described above by having
a single thread create/extend the file prior to running all of
the threads in parallel. By doing this step early, when
setup_files() is called later, it should no longer need to
create or extend the file, avoiding the race condition. The user
still needs to set a fallocate option other than 'none' or the
file will end up 0 bytes in size and the race condition will
still occur. It would be simple to add a ftruncate() to the code
to force this, but that would override the user's choice of
fallocate options.

Signed-off-by: Chris Weber <weberc@netapp.com>
4 months agoci: install libnfs for linux and macos builds
Vincent Fu [Tue, 12 Jul 2022 15:30:33 +0000 (15:30 +0000)]
ci: install libnfs for linux and macos builds

We should try to build as many ioengines as possible in our automated
builds in order to detect breakage sooner.

Signed-off-by: Vincent Fu <vincentfu@gmail.com>
4 months agoengines/nfs: remove commit hook
Vincent Fu [Tue, 12 Jul 2022 15:09:37 +0000 (15:09 +0000)]
engines/nfs: remove commit hook

The nfs ioengine does not need a commit hook because requests are
already committed when the queue hook is called. The queue hook calls
nfs_{pread/pwrite}_async and there is no separate commit step necessary
for the IO to be fully submitted.

This change also allows submission latencies to be reported because the
commit hook was not setting the io_u's issue_time.

Signed-off-by: Vincent Fu <vincentfu@gmail.com>
4 months agoconfigure: cleanups for nfs ioengine
Vincent Fu [Mon, 11 Jul 2022 15:51:13 +0000 (15:51 +0000)]
configure: cleanups for nfs ioengine

By default, build the nfs ioengine if configure can find libnfs.

Add a --disable-libnfs to avoid building the nfs ionegine even if
support could be found on the system.

Change the --enable-libnfs option to abort the build if libnfs cannot be
found.

Also use pkg-config to find the libnfs path instead of hard coding the
path.

Finally, drop CONFIG_NFS and use CONFIG_LIBNFS everywhere. There's no
reason to have two separate symbols.

Signed-off-by: Vincent Fu <vincentfu@gmail.com>
4 months agodocs: clarify write_iolog description
Vincent Fu [Fri, 15 Jul 2022 20:20:55 +0000 (20:20 +0000)]
docs: clarify write_iolog description

Note that the log file is opened in append mode to avoid confusion when
the same log file is used repeatedly.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
4 months agoengines/http: silence openssl 3.0 deprecation warnings
Jens Axboe [Tue, 19 Jul 2022 19:21:19 +0000 (13:21 -0600)]
engines/http: silence openssl 3.0 deprecation warnings

Maybe someone will fix these one day, for now shut up the annoying
warning about certain functions being deprecated. Example:

engines/http.c: In function ‘_gen_hex_md5’:
engines/http.c:261:9: warning: ‘MD5’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
  261 |         MD5((unsigned char*)p, len, hash);
      |         ^~~
In file included from engines/http.c:28:
/usr/include/openssl/md5.h:52:38: note: declared here
   52 | OSSL_DEPRECATEDIN_3_0 unsigned char *MD5(const unsigned char *d, size_t n,
      |                                      ^~~
engines/http.c: In function ‘_hmac’:
engines/http.c:273:9: warning: ‘HMAC_CTX_new’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
  273 |         ctx = HMAC_CTX_new();
      |         ^~~

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoMerge branch 'giubacc-misplaced-goto'
Vincent Fu [Tue, 19 Jul 2022 14:59:38 +0000 (10:59 -0400)]
Merge branch 'giubacc-misplaced-goto'

4 months agoFixed misplaced goto in http.c
Giuseppe Baccini [Tue, 19 Jul 2022 07:39:46 +0000 (09:39 +0200)]
Fixed misplaced goto in http.c

- In http.c:fio_http_queue function, when the user specifies rw=write
  and if curl_easy_perform fails then the control reaches line 565 and
  incorrectly prints: "WARNING: Only DDIR_READ/DDIR_WRITE/DDIR_TRIM are
supported!".
  Fix to this consists in moving statement: "goto err" at the end of the
  block as already has been done for trim/read blocks.

Signed-off-by: Giuseppe Baccini <giuseppe.baccini@suse.com>
4 months agoUpdate README.rst to specify secure protocols where possible
Rebecca Cran [Mon, 11 Jul 2022 02:55:47 +0000 (20:55 -0600)]
Update README.rst to specify secure protocols where possible

Change git clone commands to specify https instead of git or http.
GitHub has removed support for the git protocol
(https://github.blog/changelog/2022-03-15-removed-unencrypted-git-protocol-and-certain-ssh-keys/)
and http accesses will normally redirect to https.

Update links to use https where possible: all the sites with updated links
automatically redirect http accesses to https.

Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
Link: https://lore.kernel.org/r/20220711025547.11917-1-rebecca@bsdio.com
[axboe: fix git.kernel.dk location]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoMerge branch 'client-hist-le64' of https://github.com/tuan-hoang1/fio
Jens Axboe [Thu, 7 Jul 2022 12:33:25 +0000 (06:33 -0600)]
Merge branch 'client-hist-le64' of https://github.com/tuan-hoang1/fio

* 'client-hist-le64' of https://github.com/tuan-hoang1/fio:
  client: only do le64_to_cpu() on io_sample_data member if iolog is histogram

4 months agoclient: only do le64_to_cpu() on io_sample_data member if iolog is histogram
Tuan Hoang [Thu, 7 Jul 2022 07:53:52 +0000 (09:53 +0200)]
client: only do le64_to_cpu() on io_sample_data member if iolog is histogram

In the case of histogram iolog, the union io_sample_data member is a
pointer of struct io_u_plat_entry, while in the case of normal iolog, it
is an uint64_t. Thus only need to do the byteswap in case it is an
uint64_t.

This has been done similarly in server code.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
Signed-off-by: Tuan Hoang <tuan.hoang1@ibm.com>
4 months agoMerge branch 'server-hist-le64' of https://github.com/tuan-hoang1/fio
Jens Axboe [Wed, 6 Jul 2022 22:38:07 +0000 (16:38 -0600)]
Merge branch 'server-hist-le64' of https://github.com/tuan-hoang1/fio

* 'server-hist-le64' of https://github.com/tuan-hoang1/fio:
  server: only do cpu_to_le64() on io_sample_data member if iolog is histogram

4 months agoserver: only do cpu_to_le64() on io_sample_data member if iolog is histogram
Tuan Hoang [Wed, 6 Jul 2022 22:35:09 +0000 (00:35 +0200)]
server: only do cpu_to_le64() on io_sample_data member if iolog is histogram

In the case of histogram iolog, the union io_sample_data member is a
pointer of struct io_u_plat_entry, while in the case of normal iolog, it
is an uint64_t. Thus only need to do the byteswap in case it is an
uint64_t.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
Signed-off-by: Tuan Hoang <tuan.hoang1@ibm.com>
4 months agoconfigure: revert NFS configure change
Jens Axboe [Tue, 5 Jul 2022 13:18:54 +0000 (07:18 -0600)]
configure: revert NFS configure change

It's broken and nobody seems to be stepping up to fix it, revert the
known problematic change.

Fixes: 165b8a70f919 ("NFS configure fixes")
Link: https://github.com/axboe/fio/pull/1216
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agohash: cleanups
Jens Axboe [Fri, 1 Jul 2022 21:03:39 +0000 (15:03 -0600)]
hash: cleanups

- Use __hash_u64() for __fill_random_buffer()
- Convert rdma to use GOLDEN_RATIO_64

That's the last user of GOLDEN_RATIO_PRIME, which due to bit sparseness
isn't really useful for our purposes.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agolib/rand: improve __fill_random_buf()
Jens Axboe [Fri, 1 Jul 2022 21:02:06 +0000 (15:02 -0600)]
lib/rand: improve __fill_random_buf()

This won't be equivalent to what we have, but I _think_ the randomness
is good enough for this purpose.

This improves performance by about 30% for me, tested on both aarch64
and x86-64.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agoMerge branch 'fill-random-smaller' of https://github.com/gsauthof/fio
Jens Axboe [Fri, 1 Jul 2022 21:01:25 +0000 (15:01 -0600)]
Merge branch 'fill-random-smaller' of https://github.com/gsauthof/fio

* 'fill-random-smaller' of https://github.com/gsauthof/fio:
  Simplify and optimize __fill_random_buf

5 months agoSimplify and optimize __fill_random_buf
Georg Sauthoff [Thu, 30 Jun 2022 21:30:46 +0000 (23:30 +0200)]
Simplify and optimize __fill_random_buf

This reduces the number of source lines and the code size.

For example, when compiling with GCC 12.1 (-O3 -march=skylake), the
resulting assembly shrinks from 33 to 27 instructions and the number of
jump instructions is reduced from 4 to 3.

NB: GCC is able to eliminate the memcpy() call.

NB: Even if a compiler doesn't eliminate the memcpy() call, it's very
unlikely to ever get called since the buffer sizes are expected to be
powers of two (>= 8), usually.

Signed-off-by: Georg Sauthoff <mail@gms.tf>
5 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Thu, 23 Jun 2022 14:20:22 +0000 (08:20 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  ci: Verify the Android build
  ci/travis-*: Fix shellcheck warnings

5 months agoci: Verify the Android build
Bart Van Assche [Wed, 22 Jun 2022 22:37:29 +0000 (15:37 -0700)]
ci: Verify the Android build

Let the continuous integration infrastructure verify the fio Android build
in order to detect regressions in the Android build quickly.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
5 months agoci/travis-*: Fix shellcheck warnings
Bart Van Assche [Thu, 23 Jun 2022 13:26:34 +0000 (06:26 -0700)]
ci/travis-*: Fix shellcheck warnings

Although these scripts are no longer used, fix the shellcheck warnings in
these scripts. This patch does not change any functionality.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
5 months agoconfigure: add option to disable xnvme build
Ankit Kumar [Wed, 22 Jun 2022 11:25:46 +0000 (16:55 +0530)]
configure: add option to disable xnvme build

Add option to disable xnvme build even if found.
Remove enable xnvme build option, as xnvme support
is already being probed.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220622112546.13503-2-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agogettime: fix whitespace damage
Jens Axboe [Sun, 19 Jun 2022 18:04:19 +0000 (12:04 -0600)]
gettime: fix whitespace damage

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agoMerge branch 'master' of https://github.com/useche/fio
Jens Axboe [Thu, 16 Jun 2022 00:38:41 +0000 (18:38 -0600)]
Merge branch 'master' of https://github.com/useche/fio

* 'master' of https://github.com/useche/fio:
  Init file_cache to invalid (maj, min)

5 months agoInit file_cache to invalid (maj, min)
Luis Useche [Wed, 15 Jun 2022 22:44:54 +0000 (15:44 -0700)]
Init file_cache to invalid (maj, min)

In the very unlikely case that the trace was taken from a device with
major and minor zeroes, the cache will wrongly hit the first time. We
are hitting this problem when trying to replay generated traces with
major and minor zeroes. Initializing with ~0U fixes this problem.

Signed-off-by: Luis Useche <useche@gmail.com>
5 months agoioengines: clean up latency accounting for 3 ioengines
Vincent Fu [Tue, 14 Jun 2022 15:58:29 +0000 (15:58 +0000)]
ioengines: clean up latency accounting for 3 ioengines

The librpma_apm_client, librpma_gpspm_client, and rdma ioengines have
commit functions that record submission latency. In order to avoid
setting issue_time twice add the FIO_ASYNCIO_SETS_ISSUE_TIME flag. Also
add code to update the iolog issue time when needed.

I don't have the means to test this patch.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220614155822.307771-6-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agoioengines: update last_issue if we set issue_time
Vincent Fu [Tue, 14 Jun 2022 15:58:29 +0000 (15:58 +0000)]
ioengines: update last_issue if we set issue_time

If we're not updating issue_time it doesn't make sense to update
last_issue. We should also be updating last_issue in libaio and io_uring
when we record issue_time.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220614155822.307771-5-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agoHOWTO: improve description of latency measures
Vincent Fu [Tue, 14 Jun 2022 15:58:29 +0000 (15:58 +0000)]
HOWTO: improve description of latency measures

- clarify how submission latency is calculated for async ioengines
- it is slat (not clat) that is near zero for sync ioengines
- Note that submission latency + completion latency = total latency

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220614155822.307771-4-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agoioengines: don't record issue_time if ioengines already do it
Vincent Fu [Tue, 14 Jun 2022 15:58:29 +0000 (15:58 +0000)]
ioengines: don't record issue_time if ioengines already do it

io_uring, io_uring_cmd, and libaio record issue_time inside the ioengine
code when their commit functions are called. So we don't need to record
issue_time again for these ioengines in td_io_queue.

If we do fill issue_time twice, then mean(slat) + mean(clat) !=
mean(lat):

user@ubuntu:~/fio-dev$ fio-canonical/fio --name=test --ioengine=io_uring --number_ios=1 --rw=randread --size=1M
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.30-48-g26fa
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=172145: Mon Jun  6 16:12:42 2022
  read: IOPS=1000, BW=4000KiB/s (4096kB/s)(4096B/1msec)
    slat (nsec): min=61424, max=61424, avg=61424.00, stdev= 0.00
    clat (nsec): min=242709, max=242709, avg=242709.00, stdev= 0.00
     lat (nsec): min=308346, max=308346, avg=308346.00, stdev= 0.00

61424 + 242709 = 304133 != 308346

If we fill issue_time only once, then the equality will hold (as it
should):

user@ubuntu:~/fio-dev$ fio-latency/fio --name=test --ioengine=io_uring --number_ios=1 --rw=randread --size=1M
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.30-48-g26fa-dirty
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=172220: Mon Jun  6 16:12:47 2022
  read: IOPS=1000, BW=4000KiB/s (4096kB/s)(4096B/1msec)
    slat (nsec): min=53701, max=53701, avg=53701.00, stdev= 0.00
    clat (nsec): min=259566, max=259566, avg=259566.00, stdev= 0.00
     lat (nsec): min=313267, max=313267, avg=313267.00, stdev= 0.00

53701 + 259566 = 313267

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220614155822.307771-3-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agoioengines: add helper for trims with async ioengines
Vincent Fu [Tue, 14 Jun 2022 15:58:29 +0000 (15:58 +0000)]
ioengines: add helper for trims with async ioengines

Async ioengines support trim commands but trims are synchronous
operations. We need to provide special handling when measuring latency
for these commands. Create a helper function to help us identify when an
async ioengine is issuing a sync trim command. This makes the code more
readable.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220614155822.307771-2-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 14 Jun 2022 00:14:26 +0000 (18:14 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  configure: Fix libzbc detection on SUSE Linux
  configure: Support gcc 12

5 months agoconfigure: Fix libzbc detection on SUSE Linux
Bart Van Assche [Mon, 13 Jun 2022 23:27:08 +0000 (16:27 -0700)]
configure: Fix libzbc detection on SUSE Linux

The path of the libzbc header file is /usr/include/libzbc/libzbc/zbc.h
instead of /usr/include/libzbc/zbc.h on SUSE Linux systems. Add support
for the SUSE libzbc include path.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
5 months agoconfigure: Support gcc 12
Bart Van Assche [Mon, 13 Jun 2022 23:18:09 +0000 (16:18 -0700)]
configure: Support gcc 12

Fix the following classes of errors reported by gcc 12:
* ${variable} may be used uninitialized.
* ${variable} is set but not used.
* argument 2 is null but the corresponding size argument 3 value is 1 [-Werror=nonnull]

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
6 months agot/zbd: skip test case #13 when max_open_zones is too small
Shin'ichiro Kawasaki [Thu, 2 Jun 2022 09:13:10 +0000 (18:13 +0900)]
t/zbd: skip test case #13 when max_open_zones is too small

Test case #13 of t/zbd/test-zbd-support fails when the test target
device has max_open_zones smaller than 4. To avoid the failure, add a
helper function require_max_open_zones and use it to skip the test case
when the device does not have required minimum max_open_zones.

Reported-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/fio/20220602015316.6ismlexb22fwd5ko@shindev/
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Link: https://lore.kernel.org/r/20220602091310.97189-1-shinichiro.kawasaki@wdc.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoengines/nvme: ioctl return value is an int
Jens Axboe [Thu, 2 Jun 2022 09:57:22 +0000 (03:57 -0600)]
engines/nvme: ioctl return value is an int

Fix comparing an unsigned int to less than zero, it can never be true.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoengines/nvme: fix 'fd' leak in error handling
Jens Axboe [Thu, 2 Jun 2022 09:56:51 +0000 (03:56 -0600)]
engines/nvme: fix 'fd' leak in error handling

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoengines/io_uring: cleanup supported case
Jens Axboe [Thu, 2 Jun 2022 08:22:36 +0000 (02:22 -0600)]
engines/io_uring: cleanup supported case

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoexamples: add 2 example job file for io_uring_cmd engine
Anuj Gupta [Tue, 31 May 2022 13:31:55 +0000 (19:01 +0530)]
examples: add 2 example job file for io_uring_cmd engine

examples/uring-cmd-ng.fio has usage for conventional nvme-ns char device
examples/uring-cmd-zoned.fio has usage for ZNS nvme-ns char device

Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Co-authored-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220531133155.17493-10-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoengines/io_uring: Enable zone device support for io_uring_cmd I/O engine
Ankit Kumar [Tue, 31 May 2022 13:31:54 +0000 (19:01 +0530)]
engines/io_uring: Enable zone device support for io_uring_cmd I/O engine

Add zone device specific ioengine_ops for io_uring_cmd.
* get_zoned_model
* report_zones
* reset_wp
* get_max_open_zones

Add the necessary NVMe ZNS specfication opcodes and structures. Add
helper functions to submit admin and I/O passthrough commands for these
new NVMe ZNS specific commands.

For write workload iodepth must be set to 1 as there is no IO scheduler

Tested-by: Vincent Fu <vincent.fu@samsung.com>
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220531133155.17493-9-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agozbd: Check for direct flag only if its block device
Ankit Kumar [Tue, 31 May 2022 13:31:53 +0000 (19:01 +0530)]
zbd: Check for direct flag only if its block device

nvme-ns generic character devices currently do not support O_DIRECT flag.
Check for fio option for direct flag only if filetype is a block device.

t/zbd skip test case #1 for character devices as they don't require
direct I/O.

Tested-by: Vincent Fu <vincent.fu@samsung.com>
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Link: https://lore.kernel.org/r/20220531133155.17493-8-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agodocs: document options for io_uring_cmd I/O engine
Ankit Kumar [Tue, 31 May 2022 13:31:52 +0000 (19:01 +0530)]
docs: document options for io_uring_cmd I/O engine

Update documentation with io_uring_cmd I/O engine specific options.
Add missing io_uring I/O engine entry from fio man page.
Update docs with missing io_uring engine specific options.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220531133155.17493-7-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoengines/io_uring: add new I/O engine for uring passthrough support
Anuj Gupta [Tue, 31 May 2022 13:31:51 +0000 (19:01 +0530)]
engines/io_uring: add new I/O engine for uring passthrough support

Add a new I/O engine (io_uring_cmd) for sending uring passthrough
commands. It will also use most of the existing helpers from the
I/O engine io_uring. The I/O preparation, completion, file open,
file close and post init paths are going to differ and hence
io_uring_cmd will have its own helper for them.

Add a new io_uring_cmd engine specific option to support nvme
passthrough commands. Filename name for this specific option
must specify nvme-ns generic character device (dev/ngXnY).
This provides io_uring_cmd I/O engine a bandwidth to support
various passthrough commands in future.

The engine_pos and engine_data fields in struct fio_file are
separated now. This will help I/O engine io_uring_cmd to store
specific data as well as keep track of register files.

Added a new option cmd_type. This specifies the type of uring
command to submit. Currently it only supports nvme uring command

Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Co-authored-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220531133155.17493-6-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agonvme: add nvme opcodes, structures and helper functions
Ankit Kumar [Tue, 31 May 2022 13:31:50 +0000 (19:01 +0530)]
nvme: add nvme opcodes, structures and helper functions

Add NVMe specification opcodes, data structures and helper
functions to get identify namespace and form nvme uring command.

This will help the follow up patches to get nvme-ns generic
character device size, lba size.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Co-authored-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://lore.kernel.org/r/20220531133155.17493-5-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoinit: return error incase an invalid value is passed as option
Anuj Gupta [Tue, 31 May 2022 13:31:49 +0000 (19:01 +0530)]
init: return error incase an invalid value is passed as option

Currently, fio exits incase an invalid value is passed as fio
option, but continues even if an invalid value is passed for
I/O engine specific options. Exit in that scenario.

Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://lore.kernel.org/r/20220531133155.17493-4-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoconfigure: check nvme uring command support
Ankit Kumar [Tue, 31 May 2022 13:31:48 +0000 (19:01 +0530)]
configure: check nvme uring command support

Modify configure to check availability of nvme_uring_cmd, but only
when the target OS is Linux. This way in the follow up patch we
can define the missing structure to prevent compilation errors.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220531133155.17493-3-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoio_uring.h: add IORING_SETUP_SQE128 and IORING_SETUP_CQE32
Anuj Gupta [Tue, 31 May 2022 13:31:47 +0000 (19:01 +0530)]
io_uring.h: add IORING_SETUP_SQE128 and IORING_SETUP_CQE32

This asks the kernel to setup a ring with 128-byte SQE and
32-byte CQE entries. It may fail with -EINVAL if the kernel
doesn't support this feature. If the kernel does support this
feature, then the ring will support big-sqe/big-cqe entries
which some commands may require.

Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://lore.kernel.org/r/20220531133155.17493-2-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agodocs: update language setting for Sphinx build
Vincent Fu [Tue, 31 May 2022 21:49:11 +0000 (21:49 +0000)]
docs: update language setting for Sphinx build

Sphinx 5.0.0 no longer accepts None for language. Remove it and Sphinx
will use English as the default and no longer emit a warning.

For details see https://github.com/sphinx-doc/sphinx/issues/10474

Sphinx warnings cause our documentation build on GitHub Actions to fail.
Failures were observed for recent macOS builds which upgraded to Sphinx
5.0.0.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220531214857.169864-1-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoMerge branch 'wip-lmy-rados' of https://github.com/liangmingyuanneo/fio
Jens Axboe [Sun, 29 May 2022 15:32:18 +0000 (09:32 -0600)]
Merge branch 'wip-lmy-rados' of https://github.com/liangmingyuanneo/fio

* 'wip-lmy-rados' of https://github.com/liangmingyuanneo/fio:
  engines/ceph: add option for setting config file path

6 months agoengines/ceph: add option for setting config file path
liangmingyuan [Thu, 26 May 2022 18:49:16 +0000 (02:49 +0800)]
engines/ceph: add option for setting config file path

Specifies the configuration path of ceph cluster on rados test, so
conf file does not have to be /etc/ceph/ceph.conf.
To set the option, adding the next line to global section of fio:
conf=path_to_ceph_config_file

Signed-off-by: Mingyuan Liang <liangmingyuan@baidu.com>
6 months agodocs: update discussion of huge page sizes
Vincent Fu [Tue, 24 May 2022 14:23:24 +0000 (14:23 +0000)]
docs: update discussion of huge page sizes

Note that the default huge page size is either 2 or 4 MiB, depending on
the platform. Also mention /sys/kernel/mm/hugepages/ as another place to
see the supported huge page sizes.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220524142229.135808-6-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agot/run-fio-tests: improve json data decoding
Vincent Fu [Tue, 24 May 2022 14:23:24 +0000 (14:23 +0000)]
t/run-fio-tests: improve json data decoding

Instead of skipping up to five lines, just skip everything until the
opening curly brace when trying to decode JSON data.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220524142229.135808-5-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoHOWTO: add blank line for prettier formatting
Vincent Fu [Tue, 24 May 2022 14:23:24 +0000 (14:23 +0000)]
HOWTO: add blank line for prettier formatting

There needs to be a blank line between the name of an option and its
description in order for the formatting to look nice at
https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-ignore-zone-limits,

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220524142229.135808-4-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoconfigure: refer to zlib1g-dev package for zlib support
Vincent Fu [Tue, 24 May 2022 14:23:24 +0000 (14:23 +0000)]
configure: refer to zlib1g-dev package for zlib support

Recent Debian-based distributions provide zlib support in the zlib1g-dev
package.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220524142229.135808-3-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agosteadystate: delete incorrect comment
Vincent Fu [Tue, 24 May 2022 14:23:24 +0000 (14:23 +0000)]
steadystate: delete incorrect comment

Fio actually does not begin collecting steady state data until the
steady state ramp time has expired. Remove the comment that said steady
state data is collected from the start of the job.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220524142229.135808-2-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoMerge branch 'master' of https://github.com/guoanwu/fio
Jens Axboe [Wed, 25 May 2022 12:30:06 +0000 (06:30 -0600)]
Merge branch 'master' of https://github.com/guoanwu/fio

* 'master' of https://github.com/guoanwu/fio:
  pmemblk.c: fix one logic bug - read always with write

6 months agopmemblk.c: fix one logic bug - read always with write
dennis.wu [Sat, 21 May 2022 15:27:35 +0000 (23:27 +0800)]
pmemblk.c: fix one logic bug - read always with write

logic issue,if read success and return 0, then pmemblk_write called as well:
if (io_u->ddir == DDIR_READ &&
  0 != pmemblk_read(pmb->pmb_pool, buf, off)) {
io_u->error = errno;
break;
} else if (0 != pmemblk_write(pmb->pmb_pool, buf, off)) {
io_u->error = errno;
break;
}

Signed-off-by: dennis.wu <dennis.wu@intel.com>
6 months agoMakefile: Suppress `-Wimplicit-fallthrough` when compiling `lex.yy`
Ammar Faizi [Thu, 12 May 2022 16:43:33 +0000 (23:43 +0700)]
Makefile: Suppress `-Wimplicit-fallthrough` when compiling `lex.yy`

lex.yy.c is an auto generated C file. When compiling with clang-15, we
got the following warning:

```
      CC lex.yy.o
  lex.yy.c:1444:5: warning: unannotated fall-through between switch labels [-Wimplicit-fallthrough]
                                  case EOB_ACT_END_OF_FILE:
                                  ^
  lex.yy.c:1444:5: note: insert '__attribute__((fallthrough));' to silence this warning
                                  case EOB_ACT_END_OF_FILE:
                                  ^
                                  __attribute__((fallthrough));
  lex.yy.c:1444:5: note: insert 'break;' to avoid fall-through
                                  case EOB_ACT_END_OF_FILE:
                                  ^
                                  break;
  1 warning generated.
```

There is nothing we can do to fix lex.yy.c since it's an auto generated
file. Fix this by appending `-Wno-implicit-fallthrough` when compiling
this file if we have `-Wimplicit-fallthrough` flag enabled.

Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Link: https://lore.kernel.org/r/20220512164333.46516-4-ammarfaizi2@gnuweeb.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agobackend: Fix indentation
Ammar Faizi [Thu, 12 May 2022 16:43:31 +0000 (23:43 +0700)]
backend: Fix indentation

Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Link: https://lore.kernel.org/r/20220512164333.46516-2-ammarfaizi2@gnuweeb.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoexamples: add example job file for xnvme engine usage
Ankit Kumar [Wed, 11 May 2022 16:30:19 +0000 (22:00 +0530)]
examples: add example job file for xnvme engine usage

Co-Authored-By: Ankit Kumar <ankit.kumar@samsung.com>
Co-Authored-By: Simon A. F. Lund <simon.lund@samsung.com>
Link: https://lore.kernel.org/r/20220511163019.5608-4-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agodocs: documentation for xnvme ioengine
Ankit Kumar [Wed, 11 May 2022 16:30:18 +0000 (22:00 +0530)]
docs: documentation for xnvme ioengine

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220511163019.5608-3-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 months agoengines/xnvme: add xnvme engine
Ankit Kumar [Wed, 11 May 2022 16:30:17 +0000 (22:00 +0530)]
engines/xnvme: add xnvme engine

This patch introduces a new fio engine to work with xNVMe >= 0.2.0.
xNVMe provides a user space library (libxnvme) to work with NVMe
devices. The NVMe driver being used by libxnvme is re-targetable and
can be any one of the GNU/Linux Kernel NVMe driver via libaio,
IOCTLs, io_uring, the SPDK NVMe driver, or your own custom NVMe driver.

For more info visit https://xnvme.io
https://github.com/OpenMPDK/xNVMe

Co-Authored-By: Ankit Kumar <ankit.kumar@samsung.com>
Co-Authored-By: Simon A. F. Lund <simon.lund@samsung.com>
Co-Authored-By: Mads Ynddal <m.ynddal@samsung.com>
Co-Authored-By: Michael Bang <mi.bang@samsung.com>
Co-Authored-By: Karl Bonde Torp <k.torp@samsung.com>
Co-Authored-By: Gurmeet Singh <gur.singh@samsung.com>
Co-Authored-By: Pierre Labat <plabat@micron.com>
Link: https://lore.kernel.org/r/20220511163019.5608-2-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 months agoMerge branch 'patch-1' of https://github.com/ferdnyc/fio
Jens Axboe [Sun, 1 May 2022 13:29:05 +0000 (07:29 -0600)]
Merge branch 'patch-1' of https://github.com/ferdnyc/fio

* 'patch-1' of https://github.com/ferdnyc/fio:
  README: Update Fedora pkg URL

7 months agoREADME: Update Fedora pkg URL
Frank Dana [Sun, 1 May 2022 08:07:34 +0000 (04:07 -0400)]
README: Update Fedora pkg URL

7 months agoMerge branch 'global_dedup' of https://github.com/bardavid/fio
Jens Axboe [Fri, 29 Apr 2022 22:30:50 +0000 (16:30 -0600)]
Merge branch 'global_dedup' of https://github.com/bardavid/fio

* 'global_dedup' of https://github.com/bardavid/fio:
  adding an example for dedupe_global usage and DRR testing
  Introducing support for generation of dedup buffers across jobs. The dedup buffers are spread evenly between the jobs that enabled the dedupe_global option

7 months agoadding an example for dedupe_global usage and DRR testing
Bar David [Thu, 28 Apr 2022 13:30:47 +0000 (16:30 +0300)]
adding an example for dedupe_global usage and DRR testing

Signed-off-by: Bar David <bardavvid@gmail.com>
7 months agoIntroducing support for generation of dedup buffers
Bar David [Sun, 24 Apr 2022 09:25:57 +0000 (12:25 +0300)]
Introducing support for generation of dedup buffers
across jobs. The dedup buffers are spread evenly
between the jobs that enabled the dedupe_global option

Note only dedupe_mode=working_set is supported.
Note compression is supported with the global dedup enabled

Signed-off-by: Bar David <bardavvid@gmail.com>
7 months agoMerge branch 'fix/json/strdup_memory_leak' of https://github.com/dpronin/fio
Jens Axboe [Sun, 17 Apr 2022 22:47:22 +0000 (16:47 -0600)]
Merge branch 'fix/json/strdup_memory_leak' of https://github.com/dpronin/fio

* 'fix/json/strdup_memory_leak' of https://github.com/dpronin/fio:
  updated logging of iops1, iops2, ratio in FioJobTest_iops_rate
  fixed bunch of memory leaks in json constructor

7 months agoupdated logging of iops1, iops2, ratio in FioJobTest_iops_rate
Denis Pronin [Sun, 17 Apr 2022 20:08:15 +0000 (23:08 +0300)]
updated logging of iops1, iops2, ratio in FioJobTest_iops_rate

log iops1 and iops2 upon reading json_data from results
log ratio as iops2 / iops1

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
7 months agofixed bunch of memory leaks in json constructor
Denis Pronin [Sun, 10 Apr 2022 12:41:47 +0000 (15:41 +0300)]
fixed bunch of memory leaks in json constructor

fixed memory leak produced by not freed string given by 'strdup' in
'json_object_add_value_string' function

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
7 months agoMerge branch 'fix/jobs_eta_memory_leak' of https://github.com/dpronin/fio
Jens Axboe [Sun, 17 Apr 2022 14:57:44 +0000 (08:57 -0600)]
Merge branch 'fix/jobs_eta_memory_leak' of https://github.com/dpronin/fio

* 'fix/jobs_eta_memory_leak' of https://github.com/dpronin/fio:
  use flist_first_entry instead of flist_entry applied to 'next' list item
  fixed memory leak of not freed jobs_eta in several cases

7 months agouse flist_first_entry instead of flist_entry applied to 'next' list item
Denis Pronin [Sun, 10 Apr 2022 12:22:46 +0000 (15:22 +0300)]
use flist_first_entry instead of flist_entry applied to 'next' list item

use flist_first_entry in 'handle_xmits' function

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
7 months agofixed memory leak of not freed jobs_eta in several cases
Denis Pronin [Sun, 10 Apr 2022 12:21:06 +0000 (15:21 +0300)]
fixed memory leak of not freed jobs_eta in several cases

used 'free' function in 'print_thread_status' and in
'show_thread_status_json' functions to free jobs_eta previously
allocated

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
7 months agoMerge branch 'fix/memory-leak' of https://github.com/dpronin/fio
Jens Axboe [Sun, 17 Apr 2022 13:23:59 +0000 (07:23 -0600)]
Merge branch 'fix/memory-leak' of https://github.com/dpronin/fio

* 'fix/memory-leak' of https://github.com/dpronin/fio:
  fixed possible and actual memory leaks

7 months agoMerge branch 'fix/remove-sudo-in-test-script' of https://github.com/dpronin/fio
Jens Axboe [Sun, 10 Apr 2022 21:18:42 +0000 (15:18 -0600)]
Merge branch 'fix/remove-sudo-in-test-script' of https://github.com/dpronin/fio

* 'fix/remove-sudo-in-test-script' of https://github.com/dpronin/fio:
  actions-full-test.sh, removed sudo from the script

7 months agoactions-full-test.sh, removed sudo from the script
Denis Pronin [Sat, 2 Apr 2022 08:46:47 +0000 (11:46 +0300)]
actions-full-test.sh, removed sudo from the script

the script might be called with 'sudo' from the call site, if required

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
7 months agofixed possible and actual memory leaks
Denis Pronin [Fri, 11 Mar 2022 12:13:46 +0000 (15:13 +0300)]
fixed possible and actual memory leaks

backend.c: release memory occupied by the parent process for options
ioengines.c: free_ioengine entry point is protected by 'assert' call
upon td and td->io_ops

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
7 months agoiolog: Use %llu for 64-bit
Jens Axboe [Fri, 8 Apr 2022 18:46:44 +0000 (12:46 -0600)]
iolog: Use %llu for 64-bit

The previous fix was a bit dump, we need %llu on 32-bit. Fi that and
the cast.

Fixes: 11cb686eeda8 ("iolog: fix warning for 32-bit compilation")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 months agoiolog: fix warning for 32-bit compilation
Jens Axboe [Fri, 8 Apr 2022 18:32:12 +0000 (12:32 -0600)]
iolog: fix warning for 32-bit compilation

Cast the 64-bit value, we can't print it directly as %lu.

Fixes: e8cf24e570e0 ("iolog: add iolog_write for version 3")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 months agoiolog: update man page for version 3
Mohamad Gebai [Thu, 7 Apr 2022 17:40:31 +0000 (10:40 -0700)]
iolog: update man page for version 3

Add documentation for iolog version 3, mainly the differences between
versions 2 and 3.

Signed-off-by: Mohamad Gebai <mogeb@fb.com>
Link: https://lore.kernel.org/r/20220407174031.599117-4-mogeb@fb.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 months agoiolog: add iolog_write for version 3
Mohamad Gebai [Thu, 7 Apr 2022 17:40:30 +0000 (10:40 -0700)]
iolog: add iolog_write for version 3

Add timestamps to all actions for iolog version 3. Fio now generates iolog
files using version 3 by default, and only supports writing using that
version. Reading iolog v2 still works as expected.

Signed-off-by: Mohamad Gebai <mogeb@fb.com>
Link: https://lore.kernel.org/r/20220407174031.599117-3-mogeb@fb.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>