Denis Pronin [Fri, 11 Mar 2022 10:59:07 +0000 (13:59 +0300)]
fixed compiler warnings if NDEBUG enabled in core code
if NDEBUG is defined we check several return codes fairly
rather than ignore them in turning into nothing assert call
in RELEASE mode
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Jens Axboe [Sun, 17 Apr 2022 22:47:22 +0000 (16:47 -0600)]
Merge branch 'fix/json/strdup_memory_leak' of https://github.com/dpronin/fio
* 'fix/json/strdup_memory_leak' of https://github.com/dpronin/fio:
updated logging of iops1, iops2, ratio in FioJobTest_iops_rate
fixed bunch of memory leaks in json constructor
Denis Pronin [Sun, 17 Apr 2022 20:08:15 +0000 (23:08 +0300)]
updated logging of iops1, iops2, ratio in FioJobTest_iops_rate
log iops1 and iops2 upon reading json_data from results
log ratio as iops2 / iops1
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Denis Pronin [Sun, 10 Apr 2022 12:41:47 +0000 (15:41 +0300)]
fixed bunch of memory leaks in json constructor
fixed memory leak produced by not freed string given by 'strdup' in
'json_object_add_value_string' function
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Jens Axboe [Sun, 17 Apr 2022 14:57:44 +0000 (08:57 -0600)]
Merge branch 'fix/jobs_eta_memory_leak' of https://github.com/dpronin/fio
* 'fix/jobs_eta_memory_leak' of https://github.com/dpronin/fio:
use flist_first_entry instead of flist_entry applied to 'next' list item
fixed memory leak of not freed jobs_eta in several cases
Denis Pronin [Sun, 10 Apr 2022 12:22:46 +0000 (15:22 +0300)]
use flist_first_entry instead of flist_entry applied to 'next' list item
use flist_first_entry in 'handle_xmits' function
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Denis Pronin [Sun, 10 Apr 2022 12:21:06 +0000 (15:21 +0300)]
fixed memory leak of not freed jobs_eta in several cases
used 'free' function in 'print_thread_status' and in
'show_thread_status_json' functions to free jobs_eta previously
allocated
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Jens Axboe [Sun, 17 Apr 2022 13:23:59 +0000 (07:23 -0600)]
Merge branch 'fix/memory-leak' of https://github.com/dpronin/fio
* 'fix/memory-leak' of https://github.com/dpronin/fio:
fixed possible and actual memory leaks
Jens Axboe [Sun, 10 Apr 2022 21:18:42 +0000 (15:18 -0600)]
Merge branch 'fix/remove-sudo-in-test-script' of https://github.com/dpronin/fio
* 'fix/remove-sudo-in-test-script' of https://github.com/dpronin/fio:
actions-full-test.sh, removed sudo from the script
Denis Pronin [Sat, 2 Apr 2022 08:46:47 +0000 (11:46 +0300)]
actions-full-test.sh, removed sudo from the script
the script might be called with 'sudo' from the call site, if required
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Denis Pronin [Fri, 11 Mar 2022 12:13:46 +0000 (15:13 +0300)]
fixed possible and actual memory leaks
backend.c: release memory occupied by the parent process for options
ioengines.c: free_ioengine entry point is protected by 'assert' call
upon td and td->io_ops
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Jens Axboe [Fri, 8 Apr 2022 18:46:44 +0000 (12:46 -0600)]
iolog: Use %llu for 64-bit
The previous fix was a bit dump, we need %llu on 32-bit. Fi that and
the cast.
Fixes:
11cb686eeda8 ("iolog: fix warning for 32-bit compilation")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 8 Apr 2022 18:32:12 +0000 (12:32 -0600)]
iolog: fix warning for 32-bit compilation
Cast the 64-bit value, we can't print it directly as %lu.
Fixes:
e8cf24e570e0 ("iolog: add iolog_write for version 3")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Mohamad Gebai [Thu, 7 Apr 2022 17:40:31 +0000 (10:40 -0700)]
iolog: update man page for version 3
Add documentation for iolog version 3, mainly the differences between
versions 2 and 3.
Signed-off-by: Mohamad Gebai <mogeb@fb.com>
Link: https://lore.kernel.org/r/20220407174031.599117-4-mogeb@fb.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Mohamad Gebai [Thu, 7 Apr 2022 17:40:30 +0000 (10:40 -0700)]
iolog: add iolog_write for version 3
Add timestamps to all actions for iolog version 3. Fio now generates iolog
files using version 3 by default, and only supports writing using that
version. Reading iolog v2 still works as expected.
Signed-off-by: Mohamad Gebai <mogeb@fb.com>
Link: https://lore.kernel.org/r/20220407174031.599117-3-mogeb@fb.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Mohamad Gebai [Thu, 7 Apr 2022 17:40:29 +0000 (10:40 -0700)]
iolog: add version 3 to support timestamp-based replay
Version 3 format looks as follows:
timestamp filename action offset length
All file and IO actions must have timestamps, including 'add'. The 'wait'
action is not allowed with version 3 so that we can leave all timing
functionality to timestamps.
Signed-off-by: Mohamad Gebai <mogeb@fb.com>
Link: https://lore.kernel.org/r/20220407174031.599117-2-mogeb@fb.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 6 Apr 2022 23:10:00 +0000 (17:10 -0600)]
Fio 3.30
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Tue, 5 Apr 2022 17:32:49 +0000 (17:32 +0000)]
smalloc: fix ptr address in redzone error message
sfree_check_redzone is passed a pointer to the address of the *header*
of an allocated block. This does not match the address of any of the
buffers returned by smalloc. Adjust the value printed out to refer to
the address returned by smalloc associated with the header in question.
This makes debugging easier because it allows us to more easily identify
the buffer where over-/under-run occurred.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 30 Mar 2022 23:31:36 +0000 (17:31 -0600)]
Rename 'fallthrough' attribute to 'fio_fallthrough'
fallthrough is reserved in C++, so this causes issues with C++
programs pulling in the fio.h -> compiler.h header.
Rename it to something fio specific instead.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 29 Mar 2022 12:30:44 +0000 (06:30 -0600)]
Merge branch 'status-interval-finished-jobs' of https://github.com/mmkayPL/fio
* 'status-interval-finished-jobs' of https://github.com/mmkayPL/fio:
Handle finished jobs when using status-interval
Kozlowski Mateusz [Tue, 29 Mar 2022 09:27:03 +0000 (11:27 +0200)]
Handle finished jobs when using status-interval
stat: When printing job stats with status-interval, don't keep adding values to
the total runtime if the jobs are already finished. This should fix the printing
of the intermediate runtime/average BW etc.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Jens Axboe [Mon, 28 Mar 2022 12:43:56 +0000 (06:43 -0600)]
Merge branch 'master' of https://github.com/cccheng/fio
* 'master' of https://github.com/cccheng/fio:
Fix compile error of GCC 4
Chung-Chiang Cheng [Mon, 28 Mar 2022 06:47:25 +0000 (14:47 +0800)]
Fix compile error of GCC 4
gcc-4.9.3 doesn't recognize __has_attribute(__fallthrough__) and reports
the following error.
CC crc/crc32c-arm64.o
In file included from crc/../os/../file.h:5:0,
from crc/../os/os-linux.h:32,
from crc/../os/os.h:39,
from crc/crc32c-arm64.c:2:
crc/../os/../compiler/compiler.h:74:20: error: missing binary operator before token "("
Makefile:501: recipe for target 'crc/crc32c-arm64.o' failed
make: *** [crc/crc32c-arm64.o] Error 1
Signed-off-by: Chung-Chiang Cheng <cccheng@synology.com>
Jens Axboe [Thu, 24 Mar 2022 16:11:34 +0000 (10:11 -0600)]
Merge branch 'github-1372' of https://github.com/vincentkfu/fio
* 'github-1372' of https://github.com/vincentkfu/fio:
io_u: produce bad offsets for some time_based jobs
Vincent Fu [Wed, 23 Mar 2022 22:22:37 +0000 (18:22 -0400)]
io_u: produce bad offsets for some time_based jobs
Allow get_next_seq_offset to produce bad offsets for time_based jobs
when fio is accessing more than one file. Otherwise fio will not skip to
the next file once the current one is done.
Fixes: https://github.com/axboe/fio/issues/1372
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Sun, 20 Mar 2022 15:31:20 +0000 (09:31 -0600)]
engines/null: use correct -include
Fixes:
cef0a8357b3f ("engines/null: update external engine compilation")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 20 Mar 2022 14:07:31 +0000 (08:07 -0600)]
Merge branch 'master' of https://github.com/jnoc/fio
* 'master' of https://github.com/jnoc/fio:
Added citation.cff for easy APA/BibTeX citation directly from the Github repository
Jonathon Carter [Sun, 20 Mar 2022 02:56:50 +0000 (02:56 +0000)]
Added citation.cff for easy APA/BibTeX citation directly from the Github repository
Signed-off-by: Jonathon Carter <hello@jcarter.uk>
Jens Axboe [Sun, 20 Mar 2022 13:22:33 +0000 (07:22 -0600)]
engines/null: update external engine compilation
Everything needs to include config-host.h, and make sure that the C++
side uses the right type for the queue op.
Fixes: https://github.com/axboe/fio/issues/1371
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 15 Mar 2022 23:21:41 +0000 (17:21 -0600)]
Merge branch 'flags-fix' of https://github.com/albertofaria/fio
* 'flags-fix' of https://github.com/albertofaria/fio:
Properly encode engine flags in thread_data::flags
Alberto Faria [Tue, 15 Mar 2022 21:06:39 +0000 (21:06 +0000)]
Properly encode engine flags in thread_data::flags
We have 16 engine flags and an 18-bit shift, so cast engine flags to
unsigned long long before shifting to avoid dropping
FIO_ASYNCIO_SYNC_TRIM and FIO_NO_OFFLOAD.
Also make thread_data::flags unsigned long long to ensure it fits all
flags even when longs are 32 bit, and fix TD_ENG_FLAG_MASK.
Signed-off-by: Alberto Faria <afaria@redhat.com>
Jens Axboe [Fri, 11 Mar 2022 13:15:53 +0000 (06:15 -0700)]
t/dedupe: ensure that 'ret' is initialized
Fixes:
16b1e2456234 ("t/dedupe: handle errors more gracefully")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 11 Mar 2022 12:09:20 +0000 (05:09 -0700)]
t/dedupe: handle errors more gracefully
Don't assert for a deflate error, properly check for it and pass it
back up the stack so we can abort the thread.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 11 Mar 2022 02:11:24 +0000 (19:11 -0700)]
Merge branch 'fuzz-cleanup' of https://github.com/vincentkfu/fio
* 'fuzz-cleanup' of https://github.com/vincentkfu/fio:
fuzz: avoid building t/fuzz/parse_ini by default
Jens Axboe [Fri, 11 Mar 2022 02:09:56 +0000 (19:09 -0700)]
t/io_uring: only enable sync if we have preadv2
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Fri, 11 Mar 2022 00:45:46 +0000 (19:45 -0500)]
fuzz: avoid building t/fuzz/parse_ini by default
With a vanilla build t/fuzz/parse_ini will segfault because the symbol
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is not defined. If the symbol
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is defined, fio won't tear down
shared memory on program termination. Not tearing down shared memory is
necessary for t/fuzz/parse_ini to work correctly.
Don't build t/fuzz/parse_ini unless CFLAGS contains
-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Fri, 11 Mar 2022 00:50:30 +0000 (17:50 -0700)]
Merge branch 'improvement/enable-asan' of https://github.com/dpronin/fio
* 'improvement/enable-asan' of https://github.com/dpronin/fio:
ASAN enabling when configuring
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 11 Mar 2022 00:48:01 +0000 (17:48 -0700)]
Merge branch 'improvement/prevent-sigsegv-when-dup-files' of https://github.com/dpronin/fio
* 'improvement/prevent-sigsegv-when-dup-files' of https://github.com/dpronin/fio:
improvements in dup_files function
Jens Axboe [Fri, 11 Mar 2022 00:47:44 +0000 (17:47 -0700)]
Merge branch 'refactoring/configure' of https://github.com/dpronin/fio
* 'refactoring/configure' of https://github.com/dpronin/fio:
configure script refactoring
Jens Axboe [Fri, 11 Mar 2022 00:46:51 +0000 (17:46 -0700)]
Merge branch 'master' of https://github.com/dpronin/fio
* 'master' of https://github.com/dpronin/fio:
fixed memory leak detected by ASAN
Denis Pronin [Thu, 10 Mar 2022 11:15:40 +0000 (14:15 +0300)]
ASAN enabling when configuring
introduced opportunity for a user to enable ASAN for the compiler
when calling 'configure' script using '--enable-asan' option
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Denis Pronin [Wed, 9 Mar 2022 16:41:45 +0000 (19:41 +0300)]
fixed memory leak detected by ASAN
release memory occupied for td->files for each thread_data
in the parent process
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Denis Pronin [Thu, 10 Mar 2022 10:34:53 +0000 (13:34 +0300)]
improvements in dup_files function
cleared allocation of td->files when duplicating files,
call 'assert', before iterating over original thread_data,
that prevents possible segmentation fault when duplicating files
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Denis Pronin [Thu, 10 Mar 2022 10:16:41 +0000 (13:16 +0300)]
configure script refactoring
tabs are replaced by whitespaces in help page
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Jens Axboe [Wed, 9 Mar 2022 13:20:31 +0000 (06:20 -0700)]
Merge branch 'master' of https://github.com/dpronin/fio
* 'master' of https://github.com/dpronin/fio:
- freeing job_sections array of strings upon freeing each its item in init.c
Jens Axboe [Wed, 9 Mar 2022 13:19:44 +0000 (06:19 -0700)]
Merge branch 'fix/asan-memleak-forkdata' of https://github.com/dpronin/fio
* 'fix/asan-memleak-forkdata' of https://github.com/dpronin/fio:
- fixed memory leak in parent process detected by ASAN when forking and not freeing memory in the parent process allocated for fork_data
Jens Axboe [Wed, 9 Mar 2022 13:18:42 +0000 (06:18 -0700)]
Merge branch 'fix/asan-memleak' of https://github.com/dpronin/fio
* 'fix/asan-memleak' of https://github.com/dpronin/fio:
- fixed memory leak, which is happening when parsing options, claimed by ASAN
Denis Pronin [Wed, 9 Mar 2022 12:37:01 +0000 (15:37 +0300)]
- fixed memory leak in parent process detected by ASAN when forking and not freeing memory in the parent process allocated for fork_data
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Denis Pronin [Wed, 9 Mar 2022 11:53:43 +0000 (14:53 +0300)]
- fixed memory leak, which is happening when parsing options, claimed by ASAN
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Denis Pronin [Wed, 9 Mar 2022 10:03:36 +0000 (13:03 +0300)]
- freeing job_sections array of strings upon freeing each its item in init.c
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Jens Axboe [Tue, 8 Mar 2022 23:42:37 +0000 (16:42 -0700)]
Merge branch 'master' of https://github.com/dpronin/fio
* 'master' of https://github.com/dpronin/fio:
- fixed typo in configure script
Denis Pronin [Tue, 8 Mar 2022 17:34:25 +0000 (20:34 +0300)]
- fixed typo in configure script
Signed-off-by: Denis Pronin <dannftk@yandex.ru>
Jens Axboe [Mon, 7 Mar 2022 16:16:39 +0000 (09:16 -0700)]
Makefile: get rid of fortify source
Haven't seen anything useful come out of it.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 7 Mar 2022 16:11:40 +0000 (09:11 -0700)]
t/io_uring: add missing CR
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 7 Mar 2022 16:10:36 +0000 (09:10 -0700)]
t/io_uring: support using preadv2
Just for comparison for sync workloads, similarly to how we have support
for aio.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 7 Mar 2022 16:07:06 +0000 (09:07 -0700)]
t/io_uring: add support for registering the ring fd
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 7 Mar 2022 16:01:49 +0000 (09:01 -0700)]
io_uring.h: sync with 5.18 kernel bits
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 7 Mar 2022 16:00:14 +0000 (09:00 -0700)]
t/io_uring: change fatal map buffers condition with multiple files
It _may_ not work with multiple files/devices, but for most common
cases it will. Just allow it for now with a warning.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 7 Mar 2022 15:59:22 +0000 (08:59 -0700)]
t/io_uring: change map buffers registration opcode
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 26 Feb 2022 17:43:20 +0000 (10:43 -0700)]
windowsaio: open file for write if we have syncs
Windows wants the file opened for write if we do a file sync, so
ensure we do that if we have syncs.
Fixes: https://github.com/axboe/fio/issues/1352
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 26 Feb 2022 17:42:01 +0000 (10:42 -0700)]
Add TD_F_SYNCS thread flag
It's not enough to just track writes, some operating systems require
a file to be opened for write to issue a file sync. Which does kind
of make sense...
Add such a flag and set it for iolog/blktrace replay, if we see a sync
in there.
This does mean we need to bump the IO engine version, as the engine
flags need to get shifted.
Link: https://github.com/axboe/fio/issues/1352
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 24 Feb 2022 19:40:19 +0000 (12:40 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Fix three compiler warnings
Bart Van Assche [Thu, 24 Feb 2022 19:05:41 +0000 (11:05 -0800)]
Fix three compiler warnings
Fix three occurrences of the following clang compiler warning:
warning: suggest braces around initialization of subobject [-Wmissing-braces]
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Mon, 21 Feb 2022 16:43:48 +0000 (09:43 -0700)]
io_uring: use syscall helpers for the hot path
The only real hot system call here is the io_uring_enter(2) call,
as that'll happen during the IO submission/completion parts. The rest
are just setup function calls, we don't really care about those.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 21 Feb 2022 16:43:15 +0000 (09:43 -0700)]
x86-64: add system call definitions
Avoid a libc function call, just define our own syscall wrappers for
this architecture. Lifted from liburing.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 21 Feb 2022 16:41:53 +0000 (09:41 -0700)]
aarch64: add system call definitions
Avoid a libc function call, just define our own syscall wrappers for
this architecture. Lifted from liburing.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 20 Feb 2022 19:39:11 +0000 (12:39 -0700)]
Merge branch 'genfio-tempfile' of https://github.com/scop/fio
* 'genfio-tempfile' of https://github.com/scop/fio:
genfio: fix temporary file handling
Jens Axboe [Sun, 20 Feb 2022 19:28:51 +0000 (12:28 -0700)]
Merge branch 'spelling' of https://github.com/scop/fio
* 'spelling' of https://github.com/scop/fio:
Spelling and grammar fixes
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 20 Feb 2022 19:26:52 +0000 (12:26 -0700)]
Merge branch 'which-command-v-type-P' of https://github.com/scop/fio
* 'which-command-v-type-P' of https://github.com/scop/fio:
ci, t, tools: use `command` and `type` instead of `which`
Ville Skyttä [Thu, 4 Nov 2021 07:39:32 +0000 (09:39 +0200)]
Spelling and grammar fixes
Signed-off-by: Ville Skyttä <ville.skytta@upcloud.com>
Ville Skyttä [Thu, 4 Nov 2021 07:30:28 +0000 (09:30 +0200)]
ci, t, tools: use `command` and `type` instead of `which`
`which` is not POSIX, and cannot be assumed to installed everywhere.
`command -v` is available in POSIX and its predecessors at least since
1994: https://pubs.opengroup.org/onlinepubs/
7908799/
It can be used as a replacement for `which` in a number of occurrences
in fio.
For bash scripts, `type -P` is available as a builtin replacement for
`which` and its $PATH search semantics.
Signed-off-by: Ville Skyttä <ville.skytta@upcloud.com>
Ville Skyttä [Tue, 2 Nov 2021 21:41:00 +0000 (23:41 +0200)]
genfio: fix temporary file handling
As a side effect, the template temp file is no longer left behind, as
a unique filename is used for it on each run.
Use the same method of figuring out the temp dir as in
check_status_file().
Signed-off-by: Ville Skyttä <ville.skytta@upcloud.com>
Jens Axboe [Fri, 18 Feb 2022 16:02:03 +0000 (09:02 -0700)]
Merge branch 'rpma-update-RPMA-engines-with-new-librpma-completions-API' of https://github.com/ldorau/fio
* 'rpma-update-RPMA-engines-with-new-librpma-completions-API' of https://github.com/ldorau/fio:
rpma: update RPMA engines with new librpma completions API
rpma: RPMA engines require librpma>=v0.11.0 with rpma_cq_get_wc()
Oksana Salyk [Fri, 4 Feb 2022 19:00:36 +0000 (14:00 -0500)]
rpma: update RPMA engines with new librpma completions API
The API of librpma has been changed between v0.10.0 and v0.12.0
and fio has to be updated.
Signed-off-by: Oksana Salyk <oksana.salyk@intel.com>
Lukasz Dorau [Fri, 18 Feb 2022 13:57:18 +0000 (14:57 +0100)]
rpma: RPMA engines require librpma>=v0.11.0 with rpma_cq_get_wc()
Signed-off-by: Lukasz Dorau <lukasz.dorau@intel.com>
Jens Axboe [Thu, 17 Feb 2022 19:53:59 +0000 (12:53 -0700)]
Correct F_FULLSYNC -> F_FULLFSYNC
Apparently used a mix of the two, inconsistently.
Fixes:
a04e0665cb5d ("Use fcntl(..., F_FULLSYNC) if available")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 17 Feb 2022 19:08:41 +0000 (12:08 -0700)]
Use fcntl(..., F_FULLSYNC) if available
Some operating systems don't perform a data integrity flush when
fsync() is done, but provide fcntl(fd, F_FULLSYNC) to provide that kind
of guarantee.
To ensure that comparisons between operating systems is fair, use
fcntl() to do a proper sync if available.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 17 Feb 2022 17:18:49 +0000 (10:18 -0700)]
t/io_uring: align buffers correctly on non-4k page sizes
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 17 Feb 2022 17:16:19 +0000 (10:16 -0700)]
t/io_uring: allow non-power-of-2 queue depths
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 16 Feb 2022 00:11:06 +0000 (17:11 -0700)]
diskutil: include limits.h for PATH_MAX
On OmniOS, compilation fails because of a missing PATH_MAX definition:
$ gmake
CC cconv.o
In file included from stat.h:6:0,
from thread_options.h:7,
from cconv.c:4:
diskutil.h:52:12: error: 'PATH_MAX' undeclared here (not in a function); did you mean 'INT8_MAX'?
char path[PATH_MAX];
^~~~~~~~
INT8_MAX
gmake: *** [Makefile:505: cconv.o] Error 1
Add limits.h to fix that.
Link: https://github.com/axboe/fio/issues/1344
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 15 Feb 2022 21:15:58 +0000 (14:15 -0700)]
Merge branch 'check_min_rate_cleanup' of https://github.com/PCPartPicker/fio
* 'check_min_rate_cleanup' of https://github.com/PCPartPicker/fio:
Cleanup __check_min_rate
Jens Axboe [Tue, 15 Feb 2022 20:54:21 +0000 (13:54 -0700)]
Merge branch 'rand_nr_bugfix' of https://github.com/PCPartPicker/fio
* 'rand_nr_bugfix' of https://github.com/PCPartPicker/fio:
Fix :<nr> suffix with random read/write causing 0 initial offset
aggieNick02 [Tue, 15 Feb 2022 17:59:34 +0000 (11:59 -0600)]
Fix :<nr> suffix with random read/write causing 0 initial offset
When using the :<nr> suffix with random reads or writes, the initial
offset would be set to 0 for the first nr-1 operations. This happened
because td->ddir_seq_nr was initialized to the specified option value,
when it needs to always be initialized to 1, so that the first call to
get_next_offset leads to choosing a new random offset for the first nr
operations.
Signed-off-by: Nick Neumann nick@pcpartpicker.com
Jens Axboe [Tue, 15 Feb 2022 19:22:31 +0000 (12:22 -0700)]
Merge branch 'fix_bytesrate_eta' of https://github.com/PCPartPicker/fio
* 'fix_bytesrate_eta' of https://github.com/PCPartPicker/fio:
Fix ETA display when rate and/or rate_min are specified
aggieNick02 [Mon, 14 Feb 2022 21:13:50 +0000 (15:13 -0600)]
Fix ETA display when rate and/or rate_min are specified
The base passed to num2str in the ETA display code passed the wrong
base (0 instead of 1). Additionally, je->sig_figs was never set and
defaulted to 0. Both of these caused the desired range in the ETA code
to always display 0-0 when rate or rate_min was specified.
Signed-off-by: Nick Neumann nick@pcpartpicker.com
Vincent Fu [Tue, 15 Feb 2022 13:30:30 +0000 (13:30 +0000)]
ci: detect Windows installer build failures
When the Windows installer build fails, the build script actually
continues running and does not detect the failure. Use ls to determine
if the MSI file exists in order to detect whether or not the installer
build succeeded.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Link: https://lore.kernel.org/r/20220215133027.931-1-vincent.fu@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
aggieNick02 [Mon, 14 Feb 2022 03:42:27 +0000 (21:42 -0600)]
Cleanup __check_min_rate
This is a cleanup of __check_min_rate. In looking at stuff for previous fixes,
it seems like there are a lot of boolean checks of things that are always true
or always false. I'll explain my reasoning for each change; it is possible I'm
missing something somehow but I've run through it a few times.
Here's my logic:
1) td->rate_bytes and td->rate_blocks are 0 on first call to __check_min_rate,
and then are the previous iteration's value of td->this_io_bytes and
td->this_io_blocks on subsequent calls
2) bytes and iops are the current iteration's values of td->this_io_bytes and
td->this_io_blocks
3) The values of td->this_io_bytes and td->this_io_blocks are monotonic with
respect to each call of __check_min_rate
Therefore, bytes and iops are always greater than or equal to td->rate_bytes
and td->rate_blocks. This means the "if (bytes < td->rate_bytes[ddir]) {" on
line 176 can never happen.
Now, I want to say the same thing about line 197, but that line is weird/wrong
in another way. rate_iops is td->o.rate_iops, the specified desired iops rate
from the job. So I believe that is a bug - the specified desired iops rate
should not even be examined in this function, just like the same is true for
the desired bytes rate. I'm pretty sure what is meant is to compare iops to
td->rate_blocks just like bytes is compared to td->rate_bytes in line 176,
which would similarly always be false.
Now we can focus on the else caluses (lines 180-192 and lines 202-213). If
spent is 0, we should just be returning false early like in 169-170, so let's
move that case up with it. The "if (rate < ratemin || bytes <
td->rate_bytes[ddir]) {" and "if (rate < rate_iops_min || iops <
td->rate_blocks[ddir]) {" both have impossibilities as the second part of the
or clause. All we really want is to compare computed bytes rate to ratemin, and
computed iops rate to rate_iops_min.
With all of that, this function becomes a lot simpler. The rest of the cleanup
is renaming of variables to make what they are clearer, and some other simple
things (like initializing the variables directly instead of initializing to
zero and then doing +=). The renames are as follows:
- td->lastrate to td->last_rate_check_time, the last time a min rate check was
performed
- bytes to current_rate_check_bytes, the number of bytes transferred so far at
the time this call to __check_min_rate was made
- iops to current_rate_check_blocks, the number of blocks transferred so far at
the time this call to __check_min_rate was made
- rate to current_rate_bytes or current_rate_iops, depending on if it is used
as the current cycle's byte rate or block rate
- ratemin to option_rate_bytes_min, the user supplied desired minimum bytes
rate
- rate_iops eliminated - should not be used in this function
- rate_iops_min to option_rate_iops_min, the user supplied desired minimum
block rate
- td->rate_bytes to td->last_rate_check_bytes - the number of bytes transferred
the *last* time a minimum rate check was called *and* passed (not
shortcircuited because not enough time had elapsed for the cycle or settling)
- td->rate_blocks to td->last_rate_check_blocks - the number of blocks
transferred the *last* time a minimum rate check was called *and* passed (not
shortcircuited because not enough time had elapsed for the cycle or settling)
Signed-off-by: Nick Neumann nick@pcpartpicker.com
Jens Axboe [Fri, 11 Feb 2022 23:29:44 +0000 (16:29 -0700)]
Merge branch 'fio-docs-ci' of https://github.com/vincentkfu/fio
* 'fio-docs-ci' of https://github.com/vincentkfu/fio:
windows: update the installer build for renamed files
ci: install sphinx packages and add doc building to GitHub Actions
HOWTO: combine two chunk_size listings into a single one
HOWTO: combine separate hipri listings into a single one
HOWTO: combine multiple pool option listings
docs: rename HOWTO to HOWTO.rst
docs: update Makefile in order to detect build failures
docs: document cpumode option for the cpuio ioengine
Vincent Fu [Fri, 11 Feb 2022 21:55:41 +0000 (16:55 -0500)]
windows: update the installer build for renamed files
Update the MSI build instructions to point to the new README.rst and
HOWTO.rst
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Jens Axboe [Fri, 11 Feb 2022 21:25:33 +0000 (14:25 -0700)]
Merge branch 'fio_offload_fixes' of https://github.com/PCPartPicker/fio
* 'fio_offload_fixes' of https://github.com/PCPartPicker/fio:
Fix issues (assert or uninit var, hang) with check_min_rate and offloading
Vincent Fu [Fri, 4 Feb 2022 21:19:04 +0000 (16:19 -0500)]
ci: install sphinx packages and add doc building to GitHub Actions
To better detect breakage in our documentation builds let's add them to
our GitHub Actions CI.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
aggieNick02 [Fri, 11 Feb 2022 20:46:12 +0000 (14:46 -0600)]
Fix issues (assert or uninit var, hang) with check_min_rate and offloading
Using rate_min/rate_iops_min when io_submit_mode=offload option is set
leads to intermittent asserts and doesn't work. The variable comp_time
is never set in do_io in backend.c in the offload case, and comp_time is
then used in the calls to check_min_rate. The time computations in
check_min_rate either assert and terminate fio, or return meaningless
values, so any rate checking is not correct.
This first issue is fixed by adding a call to fio_gettime in the
offloading case. Once that is done though, there is still another
problem remaining. When the min rate is not achieved (with the
offloading option), fio detects it and tries to exit but fails. It ends
up in a state where ctrl-C will not cause an exit either. This happens
because cleanup_pending_aio(td) in the error case hangs in its second
call to io_u_queued_complete. Calling workqueue_flush in the error case
(when offloading) fixes the problem by making sure nothing is left in
the work queues and cleanup can proceed as it does in the non-offload
case.
Signed-off-by: Nick Neumann <nick@pcpartpicker.com>
Jens Axboe [Fri, 11 Feb 2022 13:58:12 +0000 (06:58 -0700)]
Add aarch64 cpu clock support
We can use cntvct_el0 to read the CPU clock.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 11 Feb 2022 13:42:13 +0000 (06:42 -0700)]
t/io_uring: avoid unused `nr_batch` warning
If we have libaio support, but not an appropriate CPU clock, then the
build throws a warning on nr_batch being assigned but never used.
Mirror what was done on the io_uring init path and only defined and
set `nr_batch` if we have CPU clock support.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Eric Sandeen [Tue, 8 Feb 2022 16:00:39 +0000 (10:00 -0600)]
fio: really use LDFLAGS when linking dynamic engines
Fix stupid braino on my part.
Fixes:
2b3d4a6a924e ("fio: use LDFLAGS when linking dynamic engines")
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Link: https://lore.kernel.org/r/1644336039-12774-1-git-send-email-sandeen@redhat.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Vincent Fu [Fri, 4 Feb 2022 21:05:37 +0000 (16:05 -0500)]
HOWTO: combine two chunk_size listings into a single one
This resolves the documentation build warning about multiple listings
for the chunk_size option.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 4 Feb 2022 20:59:37 +0000 (15:59 -0500)]
HOWTO: combine separate hipri listings into a single one
Resolve doc build warnings about multiple appearances of the hipri
option.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 4 Feb 2022 20:34:10 +0000 (15:34 -0500)]
HOWTO: combine multiple pool option listings
Listing the pool option in multiple places makes it impossible to link
to it in the documentation. Combine the two pool option listings into
one to resolve the doc build warning.
Also clean up a few small formatting issues.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 4 Feb 2022 20:08:24 +0000 (15:08 -0500)]
docs: rename HOWTO to HOWTO.rst
Since the HOWTO uses the rst format, we should identify it that way. It
will display nicely on github.com.
Also update the documentation to refer to the new HOWTO.rst file.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Vincent Fu [Fri, 28 Jan 2022 18:50:11 +0000 (18:50 +0000)]
docs: update Makefile in order to detect build failures
With the -W option sphinx-docs will yield a non-zero return code when it
encounters warnings.
With the --keep-going option sphinx-docs will continue running to the
end of the build even if it encounters errors.
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>