fio.git
8 days agoMakefile: Suppress `-Wimplicit-fallthrough` when compiling `lex.yy` master
Ammar Faizi [Thu, 12 May 2022 16:43:33 +0000 (23:43 +0700)]
Makefile: Suppress `-Wimplicit-fallthrough` when compiling `lex.yy`

lex.yy.c is an auto generated C file. When compiling with clang-15, we
got the following warning:

```
      CC lex.yy.o
  lex.yy.c:1444:5: warning: unannotated fall-through between switch labels [-Wimplicit-fallthrough]
                                  case EOB_ACT_END_OF_FILE:
                                  ^
  lex.yy.c:1444:5: note: insert '__attribute__((fallthrough));' to silence this warning
                                  case EOB_ACT_END_OF_FILE:
                                  ^
                                  __attribute__((fallthrough));
  lex.yy.c:1444:5: note: insert 'break;' to avoid fall-through
                                  case EOB_ACT_END_OF_FILE:
                                  ^
                                  break;
  1 warning generated.
```

There is nothing we can do to fix lex.yy.c since it's an auto generated
file. Fix this by appending `-Wno-implicit-fallthrough` when compiling
this file if we have `-Wimplicit-fallthrough` flag enabled.

Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Link: https://lore.kernel.org/r/20220512164333.46516-4-ammarfaizi2@gnuweeb.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 days agobackend: Fix indentation
Ammar Faizi [Thu, 12 May 2022 16:43:31 +0000 (23:43 +0700)]
backend: Fix indentation

Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Link: https://lore.kernel.org/r/20220512164333.46516-2-ammarfaizi2@gnuweeb.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 days agoexamples: add example job file for xnvme engine usage
Ankit Kumar [Wed, 11 May 2022 16:30:19 +0000 (22:00 +0530)]
examples: add example job file for xnvme engine usage

Co-Authored-By: Ankit Kumar <ankit.kumar@samsung.com>
Co-Authored-By: Simon A. F. Lund <simon.lund@samsung.com>
Link: https://lore.kernel.org/r/20220511163019.5608-4-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 days agodocs: documentation for xnvme ioengine
Ankit Kumar [Wed, 11 May 2022 16:30:18 +0000 (22:00 +0530)]
docs: documentation for xnvme ioengine

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20220511163019.5608-3-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 days agoengines/xnvme: add xnvme engine
Ankit Kumar [Wed, 11 May 2022 16:30:17 +0000 (22:00 +0530)]
engines/xnvme: add xnvme engine

This patch introduces a new fio engine to work with xNVMe >= 0.2.0.
xNVMe provides a user space library (libxnvme) to work with NVMe
devices. The NVMe driver being used by libxnvme is re-targetable and
can be any one of the GNU/Linux Kernel NVMe driver via libaio,
IOCTLs, io_uring, the SPDK NVMe driver, or your own custom NVMe driver.

For more info visit https://xnvme.io
https://github.com/OpenMPDK/xNVMe

Co-Authored-By: Ankit Kumar <ankit.kumar@samsung.com>
Co-Authored-By: Simon A. F. Lund <simon.lund@samsung.com>
Co-Authored-By: Mads Ynddal <m.ynddal@samsung.com>
Co-Authored-By: Michael Bang <mi.bang@samsung.com>
Co-Authored-By: Karl Bonde Torp <k.torp@samsung.com>
Co-Authored-By: Gurmeet Singh <gur.singh@samsung.com>
Co-Authored-By: Pierre Labat <plabat@micron.com>
Link: https://lore.kernel.org/r/20220511163019.5608-2-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoMerge branch 'patch-1' of https://github.com/ferdnyc/fio
Jens Axboe [Sun, 1 May 2022 13:29:05 +0000 (07:29 -0600)]
Merge branch 'patch-1' of https://github.com/ferdnyc/fio

* 'patch-1' of https://github.com/ferdnyc/fio:
  README: Update Fedora pkg URL

2 weeks agoREADME: Update Fedora pkg URL
Frank Dana [Sun, 1 May 2022 08:07:34 +0000 (04:07 -0400)]
README: Update Fedora pkg URL

2 weeks agoMerge branch 'global_dedup' of https://github.com/bardavid/fio
Jens Axboe [Fri, 29 Apr 2022 22:30:50 +0000 (16:30 -0600)]
Merge branch 'global_dedup' of https://github.com/bardavid/fio

* 'global_dedup' of https://github.com/bardavid/fio:
  adding an example for dedupe_global usage and DRR testing
  Introducing support for generation of dedup buffers across jobs. The dedup buffers are spread evenly between the jobs that enabled the dedupe_global option

3 weeks agoadding an example for dedupe_global usage and DRR testing
Bar David [Thu, 28 Apr 2022 13:30:47 +0000 (16:30 +0300)]
adding an example for dedupe_global usage and DRR testing

Signed-off-by: Bar David <bardavvid@gmail.com>
3 weeks agoIntroducing support for generation of dedup buffers
Bar David [Sun, 24 Apr 2022 09:25:57 +0000 (12:25 +0300)]
Introducing support for generation of dedup buffers
across jobs. The dedup buffers are spread evenly
between the jobs that enabled the dedupe_global option

Note only dedupe_mode=working_set is supported.
Note compression is supported with the global dedup enabled

Signed-off-by: Bar David <bardavvid@gmail.com>
4 weeks agoMerge branch 'fix/json/strdup_memory_leak' of https://github.com/dpronin/fio
Jens Axboe [Sun, 17 Apr 2022 22:47:22 +0000 (16:47 -0600)]
Merge branch 'fix/json/strdup_memory_leak' of https://github.com/dpronin/fio

* 'fix/json/strdup_memory_leak' of https://github.com/dpronin/fio:
  updated logging of iops1, iops2, ratio in FioJobTest_iops_rate
  fixed bunch of memory leaks in json constructor

4 weeks agoupdated logging of iops1, iops2, ratio in FioJobTest_iops_rate
Denis Pronin [Sun, 17 Apr 2022 20:08:15 +0000 (23:08 +0300)]
updated logging of iops1, iops2, ratio in FioJobTest_iops_rate

log iops1 and iops2 upon reading json_data from results
log ratio as iops2 / iops1

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
4 weeks agofixed bunch of memory leaks in json constructor
Denis Pronin [Sun, 10 Apr 2022 12:41:47 +0000 (15:41 +0300)]
fixed bunch of memory leaks in json constructor

fixed memory leak produced by not freed string given by 'strdup' in
'json_object_add_value_string' function

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
4 weeks agoMerge branch 'fix/jobs_eta_memory_leak' of https://github.com/dpronin/fio
Jens Axboe [Sun, 17 Apr 2022 14:57:44 +0000 (08:57 -0600)]
Merge branch 'fix/jobs_eta_memory_leak' of https://github.com/dpronin/fio

* 'fix/jobs_eta_memory_leak' of https://github.com/dpronin/fio:
  use flist_first_entry instead of flist_entry applied to 'next' list item
  fixed memory leak of not freed jobs_eta in several cases

4 weeks agouse flist_first_entry instead of flist_entry applied to 'next' list item
Denis Pronin [Sun, 10 Apr 2022 12:22:46 +0000 (15:22 +0300)]
use flist_first_entry instead of flist_entry applied to 'next' list item

use flist_first_entry in 'handle_xmits' function

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
4 weeks agofixed memory leak of not freed jobs_eta in several cases
Denis Pronin [Sun, 10 Apr 2022 12:21:06 +0000 (15:21 +0300)]
fixed memory leak of not freed jobs_eta in several cases

used 'free' function in 'print_thread_status' and in
'show_thread_status_json' functions to free jobs_eta previously
allocated

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
4 weeks agoMerge branch 'fix/memory-leak' of https://github.com/dpronin/fio
Jens Axboe [Sun, 17 Apr 2022 13:23:59 +0000 (07:23 -0600)]
Merge branch 'fix/memory-leak' of https://github.com/dpronin/fio

* 'fix/memory-leak' of https://github.com/dpronin/fio:
  fixed possible and actual memory leaks

5 weeks agoMerge branch 'fix/remove-sudo-in-test-script' of https://github.com/dpronin/fio
Jens Axboe [Sun, 10 Apr 2022 21:18:42 +0000 (15:18 -0600)]
Merge branch 'fix/remove-sudo-in-test-script' of https://github.com/dpronin/fio

* 'fix/remove-sudo-in-test-script' of https://github.com/dpronin/fio:
  actions-full-test.sh, removed sudo from the script

5 weeks agoactions-full-test.sh, removed sudo from the script
Denis Pronin [Sat, 2 Apr 2022 08:46:47 +0000 (11:46 +0300)]
actions-full-test.sh, removed sudo from the script

the script might be called with 'sudo' from the call site, if required

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
5 weeks agofixed possible and actual memory leaks
Denis Pronin [Fri, 11 Mar 2022 12:13:46 +0000 (15:13 +0300)]
fixed possible and actual memory leaks

backend.c: release memory occupied by the parent process for options
ioengines.c: free_ioengine entry point is protected by 'assert' call
upon td and td->io_ops

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
5 weeks agoiolog: Use %llu for 64-bit
Jens Axboe [Fri, 8 Apr 2022 18:46:44 +0000 (12:46 -0600)]
iolog: Use %llu for 64-bit

The previous fix was a bit dump, we need %llu on 32-bit. Fi that and
the cast.

Fixes: 11cb686eeda8 ("iolog: fix warning for 32-bit compilation")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoiolog: fix warning for 32-bit compilation
Jens Axboe [Fri, 8 Apr 2022 18:32:12 +0000 (12:32 -0600)]
iolog: fix warning for 32-bit compilation

Cast the 64-bit value, we can't print it directly as %lu.

Fixes: e8cf24e570e0 ("iolog: add iolog_write for version 3")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoiolog: update man page for version 3
Mohamad Gebai [Thu, 7 Apr 2022 17:40:31 +0000 (10:40 -0700)]
iolog: update man page for version 3

Add documentation for iolog version 3, mainly the differences between
versions 2 and 3.

Signed-off-by: Mohamad Gebai <mogeb@fb.com>
Link: https://lore.kernel.org/r/20220407174031.599117-4-mogeb@fb.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoiolog: add iolog_write for version 3
Mohamad Gebai [Thu, 7 Apr 2022 17:40:30 +0000 (10:40 -0700)]
iolog: add iolog_write for version 3

Add timestamps to all actions for iolog version 3. Fio now generates iolog
files using version 3 by default, and only supports writing using that
version. Reading iolog v2 still works as expected.

Signed-off-by: Mohamad Gebai <mogeb@fb.com>
Link: https://lore.kernel.org/r/20220407174031.599117-3-mogeb@fb.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoiolog: add version 3 to support timestamp-based replay
Mohamad Gebai [Thu, 7 Apr 2022 17:40:29 +0000 (10:40 -0700)]
iolog: add version 3 to support timestamp-based replay

Version 3 format looks as follows:

    timestamp filename action offset length

All file and IO actions must have timestamps, including 'add'. The 'wait'
action is not allowed with version 3 so that we can leave all timing
functionality to timestamps.

Signed-off-by: Mohamad Gebai <mogeb@fb.com>
Link: https://lore.kernel.org/r/20220407174031.599117-2-mogeb@fb.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoFio 3.30 fio-3.30
Jens Axboe [Wed, 6 Apr 2022 23:10:00 +0000 (17:10 -0600)]
Fio 3.30

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agosmalloc: fix ptr address in redzone error message
Vincent Fu [Tue, 5 Apr 2022 17:32:49 +0000 (17:32 +0000)]
smalloc: fix ptr address in redzone error message

sfree_check_redzone is passed a pointer to the address of the *header*
of an allocated block. This does not match the address of any of the
buffers returned by smalloc. Adjust the value printed out to refer to
the address returned by smalloc associated with the header in question.
This makes debugging easier because it allows us to more easily identify
the buffer where over-/under-run occurred.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agoRename 'fallthrough' attribute to 'fio_fallthrough'
Jens Axboe [Wed, 30 Mar 2022 23:31:36 +0000 (17:31 -0600)]
Rename 'fallthrough' attribute to 'fio_fallthrough'

fallthrough is reserved in C++, so this causes issues with C++
programs pulling in the fio.h -> compiler.h header.

Rename it to something fio specific instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agoMerge branch 'status-interval-finished-jobs' of https://github.com/mmkayPL/fio
Jens Axboe [Tue, 29 Mar 2022 12:30:44 +0000 (06:30 -0600)]
Merge branch 'status-interval-finished-jobs' of https://github.com/mmkayPL/fio

* 'status-interval-finished-jobs' of https://github.com/mmkayPL/fio:
  Handle finished jobs when using status-interval

7 weeks agoHandle finished jobs when using status-interval
Kozlowski Mateusz [Tue, 29 Mar 2022 09:27:03 +0000 (11:27 +0200)]
Handle finished jobs when using status-interval

stat: When printing job stats with status-interval, don't keep adding values to
the total runtime if the jobs are already finished. This should fix the printing
of the intermediate runtime/average BW etc.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
7 weeks agoMerge branch 'master' of https://github.com/cccheng/fio
Jens Axboe [Mon, 28 Mar 2022 12:43:56 +0000 (06:43 -0600)]
Merge branch 'master' of https://github.com/cccheng/fio

* 'master' of https://github.com/cccheng/fio:
  Fix compile error of GCC 4

7 weeks agoFix compile error of GCC 4
Chung-Chiang Cheng [Mon, 28 Mar 2022 06:47:25 +0000 (14:47 +0800)]
Fix compile error of GCC 4

gcc-4.9.3 doesn't recognize __has_attribute(__fallthrough__) and reports
the following error.

  CC crc/crc32c-arm64.o
  In file included from crc/../os/../file.h:5:0,
                   from crc/../os/os-linux.h:32,
                   from crc/../os/os.h:39,
                   from crc/crc32c-arm64.c:2:
  crc/../os/../compiler/compiler.h:74:20: error: missing binary operator before token "("
  Makefile:501: recipe for target 'crc/crc32c-arm64.o' failed
  make: *** [crc/crc32c-arm64.o] Error 1

Signed-off-by: Chung-Chiang Cheng <cccheng@synology.com>
8 weeks agoMerge branch 'github-1372' of https://github.com/vincentkfu/fio
Jens Axboe [Thu, 24 Mar 2022 16:11:34 +0000 (10:11 -0600)]
Merge branch 'github-1372' of https://github.com/vincentkfu/fio

* 'github-1372' of https://github.com/vincentkfu/fio:
  io_u: produce bad offsets for some time_based jobs

8 weeks agoio_u: produce bad offsets for some time_based jobs
Vincent Fu [Wed, 23 Mar 2022 22:22:37 +0000 (18:22 -0400)]
io_u: produce bad offsets for some time_based jobs

Allow get_next_seq_offset to produce bad offsets for time_based jobs
when fio is accessing more than one file. Otherwise fio will not skip to
the next file once the current one is done.

Fixes: https://github.com/axboe/fio/issues/1372

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agoengines/null: use correct -include
Jens Axboe [Sun, 20 Mar 2022 15:31:20 +0000 (09:31 -0600)]
engines/null: use correct -include

Fixes: cef0a8357b3f ("engines/null: update external engine compilation")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'master' of https://github.com/jnoc/fio
Jens Axboe [Sun, 20 Mar 2022 14:07:31 +0000 (08:07 -0600)]
Merge branch 'master' of https://github.com/jnoc/fio

* 'master' of https://github.com/jnoc/fio:
  Added citation.cff for easy APA/BibTeX citation directly from the Github repository

2 months agoAdded citation.cff for easy APA/BibTeX citation directly from the Github repository
Jonathon Carter [Sun, 20 Mar 2022 02:56:50 +0000 (02:56 +0000)]
Added citation.cff for easy APA/BibTeX citation directly from the Github repository

Signed-off-by: Jonathon Carter <hello@jcarter.uk>
2 months agoengines/null: update external engine compilation
Jens Axboe [Sun, 20 Mar 2022 13:22:33 +0000 (07:22 -0600)]
engines/null: update external engine compilation

Everything needs to include config-host.h, and make sure that the C++
side uses the right type for the queue op.

Fixes: https://github.com/axboe/fio/issues/1371
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'flags-fix' of https://github.com/albertofaria/fio
Jens Axboe [Tue, 15 Mar 2022 23:21:41 +0000 (17:21 -0600)]
Merge branch 'flags-fix' of https://github.com/albertofaria/fio

* 'flags-fix' of https://github.com/albertofaria/fio:
  Properly encode engine flags in thread_data::flags

2 months agoProperly encode engine flags in thread_data::flags
Alberto Faria [Tue, 15 Mar 2022 21:06:39 +0000 (21:06 +0000)]
Properly encode engine flags in thread_data::flags

We have 16 engine flags and an 18-bit shift, so cast engine flags to
unsigned long long before shifting to avoid dropping
FIO_ASYNCIO_SYNC_TRIM and FIO_NO_OFFLOAD.

Also make thread_data::flags unsigned long long to ensure it fits all
flags even when longs are 32 bit, and fix TD_ENG_FLAG_MASK.

Signed-off-by: Alberto Faria <afaria@redhat.com>
2 months agot/dedupe: ensure that 'ret' is initialized
Jens Axboe [Fri, 11 Mar 2022 13:15:53 +0000 (06:15 -0700)]
t/dedupe: ensure that 'ret' is initialized

Fixes: 16b1e2456234 ("t/dedupe: handle errors more gracefully")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/dedupe: handle errors more gracefully
Jens Axboe [Fri, 11 Mar 2022 12:09:20 +0000 (05:09 -0700)]
t/dedupe: handle errors more gracefully

Don't assert for a deflate error, properly check for it and pass it
back up the stack so we can abort the thread.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'fuzz-cleanup' of https://github.com/vincentkfu/fio
Jens Axboe [Fri, 11 Mar 2022 02:11:24 +0000 (19:11 -0700)]
Merge branch 'fuzz-cleanup' of https://github.com/vincentkfu/fio

* 'fuzz-cleanup' of https://github.com/vincentkfu/fio:
  fuzz: avoid building t/fuzz/parse_ini by default

2 months agot/io_uring: only enable sync if we have preadv2
Jens Axboe [Fri, 11 Mar 2022 02:09:56 +0000 (19:09 -0700)]
t/io_uring: only enable sync if we have preadv2

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agofuzz: avoid building t/fuzz/parse_ini by default
Vincent Fu [Fri, 11 Mar 2022 00:45:46 +0000 (19:45 -0500)]
fuzz: avoid building t/fuzz/parse_ini by default

With a vanilla build t/fuzz/parse_ini will segfault because the symbol
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is not defined. If the symbol
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is defined, fio won't tear down
shared memory on program termination. Not tearing down shared memory is
necessary for t/fuzz/parse_ini to work correctly.

Don't build t/fuzz/parse_ini unless CFLAGS contains
-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
2 months agoMerge branch 'improvement/enable-asan' of https://github.com/dpronin/fio
Jens Axboe [Fri, 11 Mar 2022 00:50:30 +0000 (17:50 -0700)]
Merge branch 'improvement/enable-asan' of https://github.com/dpronin/fio

* 'improvement/enable-asan' of https://github.com/dpronin/fio:
  ASAN enabling when configuring

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'improvement/prevent-sigsegv-when-dup-files' of https://github.com/dpron...
Jens Axboe [Fri, 11 Mar 2022 00:48:01 +0000 (17:48 -0700)]
Merge branch 'improvement/prevent-sigsegv-when-dup-files' of https://github.com/dpronin/fio

* 'improvement/prevent-sigsegv-when-dup-files' of https://github.com/dpronin/fio:
  improvements in dup_files function

2 months agoMerge branch 'refactoring/configure' of https://github.com/dpronin/fio
Jens Axboe [Fri, 11 Mar 2022 00:47:44 +0000 (17:47 -0700)]
Merge branch 'refactoring/configure' of https://github.com/dpronin/fio

* 'refactoring/configure' of https://github.com/dpronin/fio:
  configure script refactoring

2 months agoMerge branch 'master' of https://github.com/dpronin/fio
Jens Axboe [Fri, 11 Mar 2022 00:46:51 +0000 (17:46 -0700)]
Merge branch 'master' of https://github.com/dpronin/fio

* 'master' of https://github.com/dpronin/fio:
  fixed memory leak detected by ASAN

2 months agoASAN enabling when configuring
Denis Pronin [Thu, 10 Mar 2022 11:15:40 +0000 (14:15 +0300)]
ASAN enabling when configuring

introduced opportunity for a user to enable ASAN for the compiler
when calling 'configure' script using '--enable-asan' option

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
2 months agofixed memory leak detected by ASAN
Denis Pronin [Wed, 9 Mar 2022 16:41:45 +0000 (19:41 +0300)]
fixed memory leak detected by ASAN

release memory occupied for td->files for each thread_data
in the parent process

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
2 months agoimprovements in dup_files function
Denis Pronin [Thu, 10 Mar 2022 10:34:53 +0000 (13:34 +0300)]
improvements in dup_files function

cleared allocation of td->files when duplicating files,
call 'assert', before iterating over original thread_data,
that prevents possible segmentation fault when duplicating files

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
2 months agoconfigure script refactoring
Denis Pronin [Thu, 10 Mar 2022 10:16:41 +0000 (13:16 +0300)]
configure script refactoring

tabs are replaced by whitespaces in help page

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
2 months agoMerge branch 'master' of https://github.com/dpronin/fio
Jens Axboe [Wed, 9 Mar 2022 13:20:31 +0000 (06:20 -0700)]
Merge branch 'master' of https://github.com/dpronin/fio

* 'master' of https://github.com/dpronin/fio:
  - freeing job_sections array of strings upon freeing each its item in init.c

2 months agoMerge branch 'fix/asan-memleak-forkdata' of https://github.com/dpronin/fio
Jens Axboe [Wed, 9 Mar 2022 13:19:44 +0000 (06:19 -0700)]
Merge branch 'fix/asan-memleak-forkdata' of https://github.com/dpronin/fio

* 'fix/asan-memleak-forkdata' of https://github.com/dpronin/fio:
  - fixed memory leak in parent process detected by ASAN when forking and not freeing memory in the parent process allocated for fork_data

2 months agoMerge branch 'fix/asan-memleak' of https://github.com/dpronin/fio
Jens Axboe [Wed, 9 Mar 2022 13:18:42 +0000 (06:18 -0700)]
Merge branch 'fix/asan-memleak' of https://github.com/dpronin/fio

* 'fix/asan-memleak' of https://github.com/dpronin/fio:
  - fixed memory leak, which is happening when parsing options, claimed by ASAN

2 months ago- fixed memory leak in parent process detected by ASAN when forking and not freeing...
Denis Pronin [Wed, 9 Mar 2022 12:37:01 +0000 (15:37 +0300)]
- fixed memory leak in parent process detected by ASAN when forking and not freeing memory in the parent process allocated for fork_data

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
2 months ago- fixed memory leak, which is happening when parsing options, claimed by ASAN
Denis Pronin [Wed, 9 Mar 2022 11:53:43 +0000 (14:53 +0300)]
- fixed memory leak, which is happening when parsing options, claimed by ASAN

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
2 months ago- freeing job_sections array of strings upon freeing each its item in init.c
Denis Pronin [Wed, 9 Mar 2022 10:03:36 +0000 (13:03 +0300)]
- freeing job_sections array of strings upon freeing each its item in init.c

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
2 months agoMerge branch 'master' of https://github.com/dpronin/fio
Jens Axboe [Tue, 8 Mar 2022 23:42:37 +0000 (16:42 -0700)]
Merge branch 'master' of https://github.com/dpronin/fio

* 'master' of https://github.com/dpronin/fio:
  - fixed typo in configure script

2 months ago- fixed typo in configure script
Denis Pronin [Tue, 8 Mar 2022 17:34:25 +0000 (20:34 +0300)]
- fixed typo in configure script

Signed-off-by: Denis Pronin <dannftk@yandex.ru>
2 months agoMakefile: get rid of fortify source
Jens Axboe [Mon, 7 Mar 2022 16:16:39 +0000 (09:16 -0700)]
Makefile: get rid of fortify source

Haven't seen anything useful come out of it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/io_uring: add missing CR
Jens Axboe [Mon, 7 Mar 2022 16:11:40 +0000 (09:11 -0700)]
t/io_uring: add missing CR

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/io_uring: support using preadv2
Jens Axboe [Mon, 7 Mar 2022 16:10:36 +0000 (09:10 -0700)]
t/io_uring: support using preadv2

Just for comparison for sync workloads, similarly to how we have support
for aio.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/io_uring: add support for registering the ring fd
Jens Axboe [Mon, 7 Mar 2022 16:07:06 +0000 (09:07 -0700)]
t/io_uring: add support for registering the ring fd

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoio_uring.h: sync with 5.18 kernel bits
Jens Axboe [Mon, 7 Mar 2022 16:01:49 +0000 (09:01 -0700)]
io_uring.h: sync with 5.18 kernel bits

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/io_uring: change fatal map buffers condition with multiple files
Jens Axboe [Mon, 7 Mar 2022 16:00:14 +0000 (09:00 -0700)]
t/io_uring: change fatal map buffers condition with multiple files

It _may_ not work with multiple files/devices, but for most common
cases it will. Just allow it for now with a warning.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/io_uring: change map buffers registration opcode
Jens Axboe [Mon, 7 Mar 2022 15:59:22 +0000 (08:59 -0700)]
t/io_uring: change map buffers registration opcode

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agowindowsaio: open file for write if we have syncs
Jens Axboe [Sat, 26 Feb 2022 17:43:20 +0000 (10:43 -0700)]
windowsaio: open file for write if we have syncs

Windows wants the file opened for write if we do a file sync, so
ensure we do that if we have syncs.

Fixes: https://github.com/axboe/fio/issues/1352
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoAdd TD_F_SYNCS thread flag
Jens Axboe [Sat, 26 Feb 2022 17:42:01 +0000 (10:42 -0700)]
Add TD_F_SYNCS thread flag

It's not enough to just track writes, some operating systems require
a file to be opened for write to issue a file sync. Which does kind
of make sense...

Add such a flag and set it for iolog/blktrace replay, if we see a sync
in there.

This does mean we need to bump the IO engine version, as the engine
flags need to get shifted.

Link: https://github.com/axboe/fio/issues/1352
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Thu, 24 Feb 2022 19:40:19 +0000 (12:40 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Fix three compiler warnings

2 months agoFix three compiler warnings
Bart Van Assche [Thu, 24 Feb 2022 19:05:41 +0000 (11:05 -0800)]
Fix three compiler warnings

Fix three occurrences of the following clang compiler warning:

warning: suggest braces around initialization of subobject [-Wmissing-braces]

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoio_uring: use syscall helpers for the hot path
Jens Axboe [Mon, 21 Feb 2022 16:43:48 +0000 (09:43 -0700)]
io_uring: use syscall helpers for the hot path

The only real hot system call here is the io_uring_enter(2) call,
as that'll happen during the IO submission/completion parts. The rest
are just setup function calls, we don't really care about those.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agox86-64: add system call definitions
Jens Axboe [Mon, 21 Feb 2022 16:43:15 +0000 (09:43 -0700)]
x86-64: add system call definitions

Avoid a libc function call, just define our own syscall wrappers for
this architecture. Lifted from liburing.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoaarch64: add system call definitions
Jens Axboe [Mon, 21 Feb 2022 16:41:53 +0000 (09:41 -0700)]
aarch64: add system call definitions

Avoid a libc function call, just define our own syscall wrappers for
this architecture. Lifted from liburing.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'genfio-tempfile' of https://github.com/scop/fio
Jens Axboe [Sun, 20 Feb 2022 19:39:11 +0000 (12:39 -0700)]
Merge branch 'genfio-tempfile' of https://github.com/scop/fio

* 'genfio-tempfile' of https://github.com/scop/fio:
  genfio: fix temporary file handling

2 months agoMerge branch 'spelling' of https://github.com/scop/fio
Jens Axboe [Sun, 20 Feb 2022 19:28:51 +0000 (12:28 -0700)]
Merge branch 'spelling' of https://github.com/scop/fio

* 'spelling' of https://github.com/scop/fio:
  Spelling and grammar fixes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'which-command-v-type-P' of https://github.com/scop/fio
Jens Axboe [Sun, 20 Feb 2022 19:26:52 +0000 (12:26 -0700)]
Merge branch 'which-command-v-type-P' of https://github.com/scop/fio

* 'which-command-v-type-P' of https://github.com/scop/fio:
  ci, t, tools: use `command` and `type` instead of `which`

2 months agoSpelling and grammar fixes
Ville Skyttä [Thu, 4 Nov 2021 07:39:32 +0000 (09:39 +0200)]
Spelling and grammar fixes

Signed-off-by: Ville Skyttä <ville.skytta@upcloud.com>
2 months agoci, t, tools: use `command` and `type` instead of `which`
Ville Skyttä [Thu, 4 Nov 2021 07:30:28 +0000 (09:30 +0200)]
ci, t, tools: use `command` and `type` instead of `which`

`which` is not POSIX, and cannot be assumed to installed everywhere.

`command -v` is available in POSIX and its predecessors at least since
1994: https://pubs.opengroup.org/onlinepubs/7908799/
It can be used as a replacement for `which` in a number of occurrences
in fio.

For bash scripts, `type -P` is available as a builtin replacement for
`which` and its $PATH search semantics.

Signed-off-by: Ville Skyttä <ville.skytta@upcloud.com>
2 months agogenfio: fix temporary file handling
Ville Skyttä [Tue, 2 Nov 2021 21:41:00 +0000 (23:41 +0200)]
genfio: fix temporary file handling

As a side effect, the template temp file is no longer left behind, as
a unique filename is used for it on each run.

Use the same method of figuring out the temp dir as in
check_status_file().

Signed-off-by: Ville Skyttä <ville.skytta@upcloud.com>
2 months agoMerge branch 'rpma-update-RPMA-engines-with-new-librpma-completions-API' of https...
Jens Axboe [Fri, 18 Feb 2022 16:02:03 +0000 (09:02 -0700)]
Merge branch 'rpma-update-RPMA-engines-with-new-librpma-completions-API' of https://github.com/ldorau/fio

* 'rpma-update-RPMA-engines-with-new-librpma-completions-API' of https://github.com/ldorau/fio:
  rpma: update RPMA engines with new librpma completions API
  rpma: RPMA engines require librpma>=v0.11.0 with rpma_cq_get_wc()

2 months agorpma: update RPMA engines with new librpma completions API
Oksana Salyk [Fri, 4 Feb 2022 19:00:36 +0000 (14:00 -0500)]
rpma: update RPMA engines with new librpma completions API

The API of librpma has been changed between v0.10.0 and v0.12.0
and fio has to be updated.

Signed-off-by: Oksana Salyk <oksana.salyk@intel.com>
2 months agorpma: RPMA engines require librpma>=v0.11.0 with rpma_cq_get_wc()
Lukasz Dorau [Fri, 18 Feb 2022 13:57:18 +0000 (14:57 +0100)]
rpma: RPMA engines require librpma>=v0.11.0 with rpma_cq_get_wc()

Signed-off-by: Lukasz Dorau <lukasz.dorau@intel.com>
3 months agoCorrect F_FULLSYNC -> F_FULLFSYNC
Jens Axboe [Thu, 17 Feb 2022 19:53:59 +0000 (12:53 -0700)]
Correct F_FULLSYNC -> F_FULLFSYNC

Apparently used a mix of the two, inconsistently.

Fixes: a04e0665cb5d ("Use fcntl(..., F_FULLSYNC) if available")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoUse fcntl(..., F_FULLSYNC) if available
Jens Axboe [Thu, 17 Feb 2022 19:08:41 +0000 (12:08 -0700)]
Use fcntl(..., F_FULLSYNC) if available

Some operating systems don't perform a data integrity flush when
fsync() is done, but provide fcntl(fd, F_FULLSYNC) to provide that kind
of guarantee.

To ensure that comparisons between operating systems is fair, use
fcntl() to do a proper sync if available.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agot/io_uring: align buffers correctly on non-4k page sizes
Jens Axboe [Thu, 17 Feb 2022 17:18:49 +0000 (10:18 -0700)]
t/io_uring: align buffers correctly on non-4k page sizes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agot/io_uring: allow non-power-of-2 queue depths
Jens Axboe [Thu, 17 Feb 2022 17:16:19 +0000 (10:16 -0700)]
t/io_uring: allow non-power-of-2 queue depths

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agodiskutil: include limits.h for PATH_MAX
Jens Axboe [Wed, 16 Feb 2022 00:11:06 +0000 (17:11 -0700)]
diskutil: include limits.h for PATH_MAX

On OmniOS, compilation fails because of a missing PATH_MAX definition:

$ gmake
    CC cconv.o
In file included from stat.h:6:0,
                 from thread_options.h:7,
                 from cconv.c:4:
diskutil.h:52:12: error: 'PATH_MAX' undeclared here (not in a function); did you mean 'INT8_MAX'?
  char path[PATH_MAX];
            ^~~~~~~~
            INT8_MAX
gmake: *** [Makefile:505: cconv.o] Error 1

Add limits.h to fix that.

Link: https://github.com/axboe/fio/issues/1344
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoMerge branch 'check_min_rate_cleanup' of https://github.com/PCPartPicker/fio
Jens Axboe [Tue, 15 Feb 2022 21:15:58 +0000 (14:15 -0700)]
Merge branch 'check_min_rate_cleanup' of https://github.com/PCPartPicker/fio

* 'check_min_rate_cleanup' of https://github.com/PCPartPicker/fio:
  Cleanup __check_min_rate

3 months agoMerge branch 'rand_nr_bugfix' of https://github.com/PCPartPicker/fio
Jens Axboe [Tue, 15 Feb 2022 20:54:21 +0000 (13:54 -0700)]
Merge branch 'rand_nr_bugfix' of https://github.com/PCPartPicker/fio

* 'rand_nr_bugfix' of https://github.com/PCPartPicker/fio:
  Fix :<nr> suffix with random read/write causing 0 initial offset

3 months agoFix :<nr> suffix with random read/write causing 0 initial offset
aggieNick02 [Tue, 15 Feb 2022 17:59:34 +0000 (11:59 -0600)]
Fix :<nr> suffix with random read/write causing 0 initial offset

When using the :<nr> suffix with random reads or writes, the initial
offset would be set to 0 for the first nr-1 operations. This happened
because td->ddir_seq_nr was initialized to the specified option value,
when it needs to always be initialized to 1, so that the first call to
get_next_offset leads to choosing a new random offset for the first nr
operations.

Signed-off-by: Nick Neumann nick@pcpartpicker.com
3 months agoMerge branch 'fix_bytesrate_eta' of https://github.com/PCPartPicker/fio
Jens Axboe [Tue, 15 Feb 2022 19:22:31 +0000 (12:22 -0700)]
Merge branch 'fix_bytesrate_eta' of https://github.com/PCPartPicker/fio

* 'fix_bytesrate_eta' of https://github.com/PCPartPicker/fio:
  Fix ETA display when rate and/or rate_min are specified

3 months agoFix ETA display when rate and/or rate_min are specified
aggieNick02 [Mon, 14 Feb 2022 21:13:50 +0000 (15:13 -0600)]
Fix ETA display when rate and/or rate_min are specified

The base passed to num2str in the ETA display code passed the wrong
base (0 instead of 1). Additionally, je->sig_figs was never set and
defaulted to 0. Both of these caused the desired range in the ETA code
to always display 0-0 when rate or rate_min was specified.

Signed-off-by: Nick Neumann nick@pcpartpicker.com
3 months agoci: detect Windows installer build failures
Vincent Fu [Tue, 15 Feb 2022 13:30:30 +0000 (13:30 +0000)]
ci: detect Windows installer build failures

When the Windows installer build fails, the build script actually
continues running and does not detect the failure. Use ls to determine
if the MSI file exists in order to detect whether or not the installer
build succeeded.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220215133027.931-1-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoCleanup __check_min_rate
aggieNick02 [Mon, 14 Feb 2022 03:42:27 +0000 (21:42 -0600)]
Cleanup __check_min_rate

This is a cleanup of __check_min_rate. In looking at stuff for previous fixes,
it seems like there are a lot of boolean checks of things that are always true
or always false. I'll explain my reasoning for each change; it is possible I'm
missing something somehow but I've run through it a few times.

Here's my logic:

1) td->rate_bytes and td->rate_blocks are 0 on first call to __check_min_rate,
and then are the previous iteration's value of td->this_io_bytes and
td->this_io_blocks on subsequent calls

2) bytes and iops are the current iteration's values of td->this_io_bytes and
td->this_io_blocks

3) The values of td->this_io_bytes and td->this_io_blocks are monotonic with
respect to each call of __check_min_rate

Therefore, bytes and iops are always greater than or equal to td->rate_bytes
and td->rate_blocks. This means the "if (bytes < td->rate_bytes[ddir]) {" on
line 176 can never happen.

Now, I want to say the same thing about line 197, but that line is weird/wrong
in another way. rate_iops is td->o.rate_iops, the specified desired iops rate
from the job. So I believe that is a bug - the specified desired iops rate
should not even be examined in this function, just like the same is true for
the desired bytes rate. I'm pretty sure what is meant is to compare iops to
td->rate_blocks just like bytes is compared to td->rate_bytes in line 176,
which would similarly always be false.

Now we can focus on the else caluses (lines 180-192 and lines 202-213). If
spent is 0, we should just be returning false early like in 169-170, so let's
move that case up with it. The "if (rate < ratemin || bytes <
td->rate_bytes[ddir]) {" and "if (rate < rate_iops_min || iops <
td->rate_blocks[ddir]) {" both have impossibilities as the second part of the
or clause. All we really want is to compare computed bytes rate to ratemin, and
computed iops rate to rate_iops_min.

With all of that, this function becomes a lot simpler. The rest of the cleanup
is renaming of variables to make what they are clearer, and some other simple
things (like initializing the variables directly instead of initializing to
zero and then doing +=). The renames are as follows:

- td->lastrate to td->last_rate_check_time, the last time a min rate check was
performed

- bytes to current_rate_check_bytes, the number of bytes transferred so far at
the time this call to __check_min_rate was made

- iops to current_rate_check_blocks, the number of blocks transferred so far at
the time this call to __check_min_rate was made

- rate to current_rate_bytes or current_rate_iops, depending on if it is used
as the current cycle's byte rate or block rate

- ratemin to option_rate_bytes_min, the user supplied desired minimum bytes
rate

- rate_iops eliminated - should not be used in this function

- rate_iops_min to option_rate_iops_min, the user supplied desired minimum
block rate

- td->rate_bytes to td->last_rate_check_bytes - the number of bytes transferred
the *last* time a minimum rate check was called *and* passed (not
shortcircuited because not enough time had elapsed for the cycle or settling)

- td->rate_blocks to td->last_rate_check_blocks - the number of blocks
transferred the *last* time a minimum rate check was called *and* passed (not
shortcircuited because not enough time had elapsed for the cycle or settling)

Signed-off-by: Nick Neumann nick@pcpartpicker.com
3 months agoMerge branch 'fio-docs-ci' of https://github.com/vincentkfu/fio
Jens Axboe [Fri, 11 Feb 2022 23:29:44 +0000 (16:29 -0700)]
Merge branch 'fio-docs-ci' of https://github.com/vincentkfu/fio

* 'fio-docs-ci' of https://github.com/vincentkfu/fio:
  windows: update the installer build for renamed files
  ci: install sphinx packages and add doc building to GitHub Actions
  HOWTO: combine two chunk_size listings into a single one
  HOWTO: combine separate hipri listings into a single one
  HOWTO: combine multiple pool option listings
  docs: rename HOWTO to HOWTO.rst
  docs: update Makefile in order to detect build failures
  docs: document cpumode option for the cpuio ioengine

3 months agowindows: update the installer build for renamed files
Vincent Fu [Fri, 11 Feb 2022 21:55:41 +0000 (16:55 -0500)]
windows: update the installer build for renamed files

Update the MSI build instructions to point to the new README.rst and
HOWTO.rst

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
3 months agoMerge branch 'fio_offload_fixes' of https://github.com/PCPartPicker/fio
Jens Axboe [Fri, 11 Feb 2022 21:25:33 +0000 (14:25 -0700)]
Merge branch 'fio_offload_fixes' of https://github.com/PCPartPicker/fio

* 'fio_offload_fixes' of https://github.com/PCPartPicker/fio:
  Fix issues (assert or uninit var, hang) with check_min_rate and offloading

3 months agoci: install sphinx packages and add doc building to GitHub Actions
Vincent Fu [Fri, 4 Feb 2022 21:19:04 +0000 (16:19 -0500)]
ci: install sphinx packages and add doc building to GitHub Actions

To better detect breakage in our documentation builds let's add them to
our GitHub Actions CI.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>