fio.git
2 months agoMerge branch 'filestat1' of https://github.com/kusumi/fio
Jens Axboe [Thu, 23 Jan 2020 18:35:23 +0000 (11:35 -0700)]
Merge branch 'filestat1' of https://github.com/kusumi/fio

* 'filestat1' of https://github.com/kusumi/fio:
  engines/filestat: add "lstat" option to use lstat(2)

2 months agoengines/filestat: add "lstat" option to use lstat(2)
Tomohiro Kusumi [Thu, 23 Jan 2020 16:36:32 +0000 (01:36 +0900)]
engines/filestat: add "lstat" option to use lstat(2)

Use stat(2) by default, but lstat(2) if "lstat" bool option specified.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
2 months agostat: ensure we align correctly
Jens Axboe [Thu, 23 Jan 2020 02:53:14 +0000 (19:53 -0700)]
stat: ensure we align correctly

Fixes: b2a432bfbb6d ("Per-command priority: Priority logging and libaio/io_uring cmdprio_percentage")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'priorityQueuing' of https://github.com/Seagate/fio
Jens Axboe [Thu, 23 Jan 2020 02:50:41 +0000 (19:50 -0700)]
Merge branch 'priorityQueuing' of https://github.com/Seagate/fio

* 'priorityQueuing' of https://github.com/Seagate/fio:
  Per-command priority: Priority logging and libaio/io_uring cmdprio_percentage
  Whitespace standardization

2 months agoPer-command priority: Priority logging and libaio/io_uring cmdprio_percentage
Phillip Chen [Wed, 22 Jan 2020 23:28:38 +0000 (16:28 -0700)]
Per-command priority: Priority logging and libaio/io_uring cmdprio_percentage

Add cmdprio_percentage option to libaio and io_uring engines to set
ioprio on a per-command basis. Add tracking of high priority
commands to be displayed separately in human readable and JSON
outputs.

2 months agoWhitespace standardization
Phillip Chen [Tue, 21 Jan 2020 18:00:17 +0000 (11:00 -0700)]
Whitespace standardization

Remove trailing whitespace.

2 months agoengines: add engine for file stat
Su, Friendy [Wed, 22 Jan 2020 15:31:25 +0000 (08:31 -0700)]
engines: add engine for file stat

This engine is to measure performance of accessing file's meta data.
This is for the actual access pattern which does not do real IO, but
just look up the file and get file's attribute.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Sat, 18 Jan 2020 16:35:25 +0000 (09:35 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  client: Fix memory leaks in handle_job_opt()
  pmemblk: Fix a memory leak
  lib/memcpy: Suppress a Coverity leak report for setup_tests()
  stat: Fix another memory leak in add_ddir_status_json()
  stat: Fix a memory leak in add_ddir_status_json()
  stat: Remove several superfluous if-tests
  stat: Remove more superfluous casts
  stat: Remove two superfluous casts

2 months agoclient: Fix memory leaks in handle_job_opt()
Bart Van Assche [Sat, 18 Jan 2020 05:10:41 +0000 (21:10 -0800)]
client: Fix memory leaks in handle_job_opt()

This patch fixes the following Coverity complaint:

CID 169311 (#1 of 1): Resource leak (RESOURCE_LEAK)
9. leaked_storage: Variable p going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agopmemblk: Fix a memory leak
Bart Van Assche [Sat, 18 Jan 2020 05:08:10 +0000 (21:08 -0800)]
pmemblk: Fix a memory leak

This patch fixes the following Coverity complaint:

CID 169312 (#1 of 1): Resource leak (RESOURCE_LEAK)
5. leaked_storage: Variable path going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agolib/memcpy: Suppress a Coverity leak report for setup_tests()
Bart Van Assche [Sat, 18 Jan 2020 05:02:00 +0000 (21:02 -0800)]
lib/memcpy: Suppress a Coverity leak report for setup_tests()

Suppress the following Coverity memory leak complaint:

CID 169324 (#1 of 1): Resource leak (RESOURCE_LEAK)
7. leaked_storage: Variable src going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Fix another memory leak in add_ddir_status_json()
Bart Van Assche [Sat, 18 Jan 2020 05:00:01 +0000 (21:00 -0800)]
stat: Fix another memory leak in add_ddir_status_json()

Only compute the percentile_object and clat_bins_object data structures if
these will be stored inside another JSON object.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Fix a memory leak in add_ddir_status_json()
Bart Van Assche [Sat, 18 Jan 2020 04:42:27 +0000 (20:42 -0800)]
stat: Fix a memory leak in add_ddir_status_json()

This patch fixes the following Coverity complaint:

CID 173000 (#1 of 1): Resource leak (RESOURCE_LEAK)
18. leaked_storage: Variable ovals going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Remove several superfluous if-tests
Bart Van Assche [Sat, 18 Jan 2020 04:41:43 +0000 (20:41 -0800)]
stat: Remove several superfluous if-tests

Passing NULL to free() is allowed. From https://pubs.opengroup.org/onlinepubs/9699919799/functions/free.html#:

If ptr is a null pointer, no action shall occur.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Remove more superfluous casts
Bart Van Assche [Sat, 18 Jan 2020 04:50:19 +0000 (20:50 -0800)]
stat: Remove more superfluous casts

No cast is necessary to pass a char * argument to a function that accepts
a const char * argument.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Remove two superfluous casts
Bart Van Assche [Sat, 18 Jan 2020 04:40:41 +0000 (20:40 -0800)]
stat: Remove two superfluous casts

The C language supports implicit casting from a non-void to void pointer.
Remove two explicit casts from a non-void to a void pointer.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoMerge branch 'fix_verify_push' of https://github.com/gwendalcr/fio
Jens Axboe [Thu, 16 Jan 2020 22:44:44 +0000 (15:44 -0700)]
Merge branch 'fix_verify_push' of https://github.com/gwendalcr/fio

* 'fix_verify_push' of https://github.com/gwendalcr/fio:
  verify: Fix test to not check for numberio when verify_only is true

2 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Thu, 16 Jan 2020 22:44:16 +0000 (15:44 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  t/read-to-pipe-async: Complain if pthread_detach() fails
  t/read-to-pipe-async: Do not divide by zero
  t/memlock: Verify 'threads' argument
  server: Make it explicit that the setsockopt() return value is ignored
  blktrace: Fix memory leaks in error paths
  blktrace: Check value of 'merge_buf' pointer before using it
  blktrace: Pass a positive error code to td_verror()

2 months agoMerge branch 'issue-825' of https://github.com/LeaflessMelospiza/fio
Jens Axboe [Thu, 16 Jan 2020 22:43:10 +0000 (15:43 -0700)]
Merge branch 'issue-825' of https://github.com/LeaflessMelospiza/fio

* 'issue-825' of https://github.com/LeaflessMelospiza/fio:
  Moved diskutil reporting functions to stat.c

2 months agoverify: Fix test to not check for numberio when verify_only is true
Gwendal Grignou [Tue, 14 Jan 2020 00:35:10 +0000 (16:35 -0800)]
verify: Fix test to not check for numberio when verify_only is true

io->numberio can not be populated when verify_only is true, because
do_dry_run() build and complete IOs immediately, so it can not replicate
the numberio that was produced when the data was layered on the media.

Without this fix, using write_random
[write_stress]
filename=${FILENAME}
size=${FILESIZE}
verify_only=${VERIFY_ONLY}
readwrite=randwrite
bs=4k
ioengine=libaio
iodepth=32
direct=1
do_verify=1
verify=crc32c

'VERIFY_ONLY=1 FILENAME=/dev/sda1 FILESIZE=1M fio write_random' passes,
but
'VERIFY_ONLY=0 FILENAME=/dev/sda1 FILESIZE=1M fio write_random' fails:
"""verify_only option fails with verify: bad header numberio 1, wanted
0""".
The fix addresses the problem by not checking numberio.

Fixes #732

Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
2 months agoUse aux_path, if set, when loading verify state
Andrey Kuzmin [Thu, 16 Jan 2020 15:15:40 +0000 (08:15 -0700)]
Use aux_path, if set, when loading verify state

Fixes: https://github.com/axboe/fio/issues/908
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMoved diskutil reporting functions to stat.c
LeaflessMelospiza [Wed, 15 Jan 2020 10:13:15 +0000 (15:43 +0530)]
Moved diskutil reporting functions to stat.c

2 months agoengines/io_uring: use fixed opcodes for pre-mapped buffers
Keith Busch [Tue, 14 Jan 2020 21:24:24 +0000 (13:24 -0800)]
engines/io_uring: use fixed opcodes for pre-mapped buffers

Use the correct opcode when for reads and writes using the fixedbuf
option, otherwise EINVAL errors will be returned to these requests.

Fixes: b10b1e70a ("io_uring: add option for non-vectored read/write commands")
Signed-off-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agotime: limit usec_sleep() to maximum intervals of 1 second
Jens Axboe [Mon, 13 Jan 2020 21:51:35 +0000 (14:51 -0700)]
time: limit usec_sleep() to maximum intervals of 1 second

The total sleep may be much longer, but wakeup every ~1 second at least
to check if we got signaled to exit.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/read-to-pipe-async: Complain if pthread_detach() fails
Bart Van Assche [Sat, 11 Jan 2020 23:46:57 +0000 (15:46 -0800)]
t/read-to-pipe-async: Complain if pthread_detach() fails

This patch fixes the following Coverity complaint:

CID 280680 (#1 of 1): Unchecked return value (CHECKED_RETURN)
5. check_return: Calling pthread_detach without checking return value (as is done elsewhere 4 out of 5 times).

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agot/read-to-pipe-async: Do not divide by zero
Bart Van Assche [Sat, 11 Jan 2020 23:42:49 +0000 (15:42 -0800)]
t/read-to-pipe-async: Do not divide by zero

This patch fixes the following Coverity complaint:

23. zero_return: Function call utime_since(&s, &re) returns 0.
CID 280732 (#2 of 2): Division or modulo by zero (DIVIDE_BY_ZERO)
24. divide_by_zero: In expression bytes * 1000UL * 1000UL / utime_since(&s, &re), division by expression utime_since(&s, &re) which may be zero has undefined behavior.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agot/memlock: Verify 'threads' argument
Bart Van Assche [Sat, 11 Jan 2020 23:45:32 +0000 (15:45 -0800)]
t/memlock: Verify 'threads' argument

This patch fixes the following Coverity complaint:

CID 280681 (#1 of 1): Untrusted value as argument (TAINTED_SCALAR)
4. tainted_data: Passing tainted variable threads to a tainted sink.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoserver: Make it explicit that the setsockopt() return value is ignored
Bart Van Assche [Sun, 12 Jan 2020 00:31:14 +0000 (16:31 -0800)]
server: Make it explicit that the setsockopt() return value is ignored

This patch fixes the following Coverity complaint:

CID 169316 (#1 of 1): Unchecked return value from library (CHECKED_RETURN)
5. check_return: Calling setsockopt(sk, 1, 15, &opt, 4U) without checking return value. This library function may fail and return an error code.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoblktrace: Fix memory leaks in error paths
Bart Van Assche [Sun, 12 Jan 2020 00:07:39 +0000 (16:07 -0800)]
blktrace: Fix memory leaks in error paths

This patch fixes the following Coverity complaint:

CID 183977 (#1 of 1): Resource leak (RESOURCE_LEAK)
17. leaked_storage: Variable str going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoblktrace: Check value of 'merge_buf' pointer before using it
Bart Van Assche [Sun, 12 Jan 2020 00:10:42 +0000 (16:10 -0800)]
blktrace: Check value of 'merge_buf' pointer before using it

This patch fixes the following Coverity complaint:

CID 183976 (#1 of 1): Dereference null return value (NULL_RETURNS)
5. dereference: Dereferencing a pointer that might be null 'merge_fp' when calling 'setvbuf'.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoblktrace: Pass a positive error code to td_verror()
Bart Van Assche [Sat, 11 Jan 2020 23:58:26 +0000 (15:58 -0800)]
blktrace: Pass a positive error code to td_verror()

This patch fixes the following Coverity complaint:

CID 183980 (#1-2 of 2): Argument cannot be negative (NEGATIVE_RETURNS)
13. negative_returns: ret is passed to a parameter that cannot be negative.

Fixes: e28875637094 ("blktrace support: speedup reading of data")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoio_uring: set sqe iopriority, if prio/prioclass is set
Jens Axboe [Thu, 9 Jan 2020 21:58:51 +0000 (14:58 -0700)]
io_uring: set sqe iopriority, if prio/prioclass is set

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 7 Jan 2020 01:38:02 +0000 (18:38 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Windows >= 7: Make fio_getaffinity() error reporting more detailed
  Windows: Uninline CPU affinity functions
  Windows: Use snprintf() instead of StringCch*()
  Include "oslib/asprintf.h" where necessary
  Windows: Remove more unused OS dependent code
  Change off64_t into uint64_t
  Fix the build in case FIO_HAVE_DISK_UTIL is not defined
  Windows: Fix multiple configure tests
  configure: Improve the getopt_long_only() test
  configure: Improve the ibverbs test
  Makefile: Use 'tr' if 'fmt' is not available
  Makefile: Avoid duplicating code

3 months agoWindows >= 7: Make fio_getaffinity() error reporting more detailed
Bart Van Assche [Mon, 6 Jan 2020 04:25:36 +0000 (20:25 -0800)]
Windows >= 7: Make fio_getaffinity() error reporting more detailed

Report different error messages for GetProcessGroupAffinity() failures
and also if a process is associated with multiple process groups.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoWindows: Uninline CPU affinity functions
Bart Van Assche [Mon, 6 Jan 2020 04:17:41 +0000 (20:17 -0800)]
Windows: Uninline CPU affinity functions

None of the CPU affinity functions is in the hot path. Reduce the build
time by uninlining these functions.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoWindows: Use snprintf() instead of StringCch*()
Bart Van Assche [Mon, 6 Jan 2020 01:51:19 +0000 (17:51 -0800)]
Windows: Use snprintf() instead of StringCch*()

Use ANSI C functions instead of Windows-specific functions.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoInclude "oslib/asprintf.h" where necessary
Bart Van Assche [Sun, 5 Jan 2020 21:37:48 +0000 (13:37 -0800)]
Include "oslib/asprintf.h" where necessary

The header file "oslib/asprintf.h" must be included by every source file
that calls asprintf().

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoWindows: Remove more unused OS dependent code
Bart Van Assche [Sun, 5 Jan 2020 19:40:36 +0000 (11:40 -0800)]
Windows: Remove more unused OS dependent code

Since commit 46bfd4e5170e ("Remove old OS dependent (unused) random code")
rand_r() is no longer used. Hence remove its definition.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoChange off64_t into uint64_t
Bart Van Assche [Sun, 5 Jan 2020 19:10:30 +0000 (11:10 -0800)]
Change off64_t into uint64_t

off64_t is Linux specific while uint64_t is defined by a C standard. This
change makes fio easier to port.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoFix the build in case FIO_HAVE_DISK_UTIL is not defined
Bart Van Assche [Mon, 6 Jan 2020 16:47:17 +0000 (08:47 -0800)]
Fix the build in case FIO_HAVE_DISK_UTIL is not defined

This patch fixes the following build error:

stat.c: In function 'show_thread_status_terse_all':
stat.c:1243:31: error: suggest braces around empty body in an 'if' statement [-Werror=empty-body]
   show_disk_util(1, NULL, out);
                               ^

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoWindows: Fix multiple configure tests
Bart Van Assche [Mon, 6 Jan 2020 15:21:14 +0000 (07:21 -0800)]
Windows: Fix multiple configure tests

Auto-detect whether socklen_t, gettimeofday(), TCP_NODELAY and IPv6 are
supported instead of hard-coding that these are supported.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoconfigure: Improve the getopt_long_only() test
Bart Van Assche [Mon, 6 Jan 2020 19:00:24 +0000 (11:00 -0800)]
configure: Improve the getopt_long_only() test

Fix the following build error if configure is invoked with
--extra-cflags=-Werror:

/tmp/fio-conf-20209-133775-26866.c: In function 'main':
/tmp/fio-conf-20209-133775-26866.c:6:11: error: null argument where non-null required (argument 3) [-Werror=nonnull]
   int c = getopt_long_only(argc, argv, NULL, NULL, NULL);
           ^~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoconfigure: Improve the ibverbs test
Bart Van Assche [Mon, 6 Jan 2020 18:55:48 +0000 (10:55 -0800)]
configure: Improve the ibverbs test

This patch fixes the following configure test build error if configure is
invoked with --extra-cflags=-Werror:

/tmp/fio-conf-14710-130227-4466.c: In function 'main':
/tmp/fio-conf-14710-130227-4466.c:4:18: error: unused variable 'pd' [-Werror=unused-variable]
   struct ibv_pd *pd = ibv_alloc_pd(NULL);
                  ^~
cc1: all warnings being treated as errors

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoMakefile: Use 'tr' if 'fmt' is not available
Bart Van Assche [Mon, 6 Jan 2020 02:16:54 +0000 (18:16 -0800)]
Makefile: Use 'tr' if 'fmt' is not available

NetBSD is not the only OS that does not provide 'fmt'. A native MinGW shell
does not provide 'fmt' either. Hence detect at runtime whether or not 'fmt'
is available.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoMakefile: Avoid duplicating code
Bart Van Assche [Mon, 6 Jan 2020 02:16:20 +0000 (18:16 -0800)]
Makefile: Avoid duplicating code

Use the default compilation rule for building init.o instead of duplicating
it.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoMerge branch 'master' of https://github.com/vincentkfu/fio
Jens Axboe [Mon, 6 Jan 2020 17:46:26 +0000 (10:46 -0700)]
Merge branch 'master' of https://github.com/vincentkfu/fio

* 'master' of https://github.com/vincentkfu/fio:
  t/steadystate_tests: relax acceptance criterion
  t/run-fio-tests: automatically skip t/jobs/t0005 on Windows

3 months agot/steadystate_tests: relax acceptance criterion
Vincent Fu [Fri, 20 Dec 2019 19:52:12 +0000 (14:52 -0500)]
t/steadystate_tests: relax acceptance criterion

Relax the acceptance criterion for tests that do not attain steady
state. Originally, only a 10ms difference between expected and actual
runtime was allowed. Change this to 50ms. I observed a spurious failure
where a test ran for 10017ms instead of the expected 10000ms.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
3 months agot/run-fio-tests: automatically skip t/jobs/t0005 on Windows
Vincent Fu [Thu, 12 Dec 2019 22:31:14 +0000 (15:31 -0700)]
t/run-fio-tests: automatically skip t/jobs/t0005 on Windows

t0005 uses direct IO with a sync ioengine which is not supported under
Windows. Automatically skip this test when run-fio-tests is run under
Windows.

With this change we no longer need to skip test 5 in the appveyor setup.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
3 months agoMerge branch 'json1' of https://github.com/kusumi/fio
Jens Axboe [Mon, 6 Jan 2020 15:22:38 +0000 (08:22 -0700)]
Merge branch 'json1' of https://github.com/kusumi/fio

* 'json1' of https://github.com/kusumi/fio:
  json: remove two redundant json_print_array() prototypes

3 months agojson: remove two redundant json_print_array() prototypes
Tomohiro Kusumi [Mon, 6 Jan 2020 15:04:47 +0000 (00:04 +0900)]
json: remove two redundant json_print_array() prototypes

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
3 months agoMerge branch 'memalign1' of https://github.com/kusumi/fio
Jens Axboe [Sun, 5 Jan 2020 17:31:40 +0000 (10:31 -0700)]
Merge branch 'memalign1' of https://github.com/kusumi/fio

* 'memalign1' of https://github.com/kusumi/fio:
  lib/memalign: remove smalloc()/sfree() dependency

3 months agolib/memalign: remove smalloc()/sfree() dependency
Tomohiro Kusumi [Sun, 5 Jan 2020 15:01:10 +0000 (00:01 +0900)]
lib/memalign: remove smalloc()/sfree() dependency

fio_memalign()/fio_memfree() implementation shouldn't depend on
smalloc()/sfree() which has dependency on fio code itself.
e.g. This forces unittest code to prepare stab functions for
smalloc()/sfree().

This smalloc()/sfree() dependency was added by 3114b675fd
("fio: enable cross-thread overlap checking with processes").

Rename fio_memalign()/fio_memfree() to __fio_memalign()/__fio_memfree()
and take a function pointer instead of a boolean flag.
Add fio_memalign()/fio_memfree() as an inlined wrapper for
__fio_memalign()/__fio_memfree() without API change.
The only real change here is lib/memalign functions got renamed.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
3 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Sun, 5 Jan 2020 03:22:16 +0000 (20:22 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  t/read-to-pipe-async: Complain if option -f is specified multiple times
  t/memlock: Free allocated memory
  helper_thread: Complain if select() fails
  Block signals for the helper thread

3 months agot/read-to-pipe-async: Complain if option -f is specified multiple times
Bart Van Assche [Sat, 4 Jan 2020 22:10:47 +0000 (14:10 -0800)]
t/read-to-pipe-async: Complain if option -f is specified multiple times

This patch fixes the following Coverity complaint:

CID 280676:  Resource leaks  (RESOURCE_LEAK)
Overwriting "file" in "file = strdup(optarg)" leaks the storage that "file"
points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agot/memlock: Free allocated memory
Bart Van Assche [Sat, 4 Jan 2020 22:07:54 +0000 (14:07 -0800)]
t/memlock: Free allocated memory

This patch fixes the following Coverity complaint:

CID 280679:  Resource leaks  (RESOURCE_LEAK)
Variable "buf" going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agohelper_thread: Complain if select() fails
Bart Van Assche [Sat, 4 Jan 2020 21:59:02 +0000 (13:59 -0800)]
helper_thread: Complain if select() fails

This patch suppresses the following Coverity complaint:

CID 280687:  Error handling issues  (CHECKED_RETURN)
Calling "select(1, &rfds, NULL, &efds, &timeout)" without checking return
value. This library function may fail and return an error code.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoBlock signals for the helper thread
Bart Van Assche [Sat, 4 Jan 2020 21:58:52 +0000 (13:58 -0800)]
Block signals for the helper thread

Let another thread than the helper thread handle signals. This simplifies
error handling. As an example, the next patch will check the return value
of select() and won't have to check for EINTR due to this patch.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Sat, 4 Jan 2020 17:44:20 +0000 (10:44 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Fix a potential deadlock in helper_do_stat()
  Handle helper_thread_create() failures properly

3 months agoFix a potential deadlock in helper_do_stat()
Bart Van Assche [Fri, 3 Jan 2020 19:35:22 +0000 (11:35 -0800)]
Fix a potential deadlock in helper_do_stat()

pthread_mutex_lock(), pthread_cond_signal() and pthread_mutex_unlock() are
not async-signal-safe and hence must not be used inside a singal handler
implementation. Rework the code for communication with the helper thread
such that it becomes async-signal-safe.

Compared to commit 31eca641ad91, tests for the pipe() and pipe2() system
calls have been added and a pipe simulation for Windows has been added
(pipe_over_localhost()).

Fixes: a47591e4923f ("Improve logging accuracy")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoHandle helper_thread_create() failures properly
Bart Van Assche [Fri, 3 Jan 2020 21:21:53 +0000 (13:21 -0800)]
Handle helper_thread_create() failures properly

Report helper_thread_create() failures. Make sure that no crash occurs
if helper_thread_create() fails.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Sat, 4 Jan 2020 04:00:17 +0000 (21:00 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Improve the pthread_condattr_setclock() test
  Only build t/read-to-pipe-async if pread() is available
  .appveyor.yml: Convert to ASCII

3 months agoImprove the pthread_condattr_setclock() test
Bart Van Assche [Fri, 3 Jan 2020 20:52:47 +0000 (12:52 -0800)]
Improve the pthread_condattr_setclock() test

Link with the pthread library if pthread_condattr_setclock() is not an
inline function.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoOnly build t/read-to-pipe-async if pread() is available
Bart Van Assche [Fri, 3 Jan 2020 20:41:33 +0000 (12:41 -0800)]
Only build t/read-to-pipe-async if pread() is available

Reported-by: Vincent Fu <vincent.fu@wdc.com>
Fixes: 5ad80fa7cf5b ("Makefile: Build more test code")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months ago.appveyor.yml: Convert to ASCII
Bart Van Assche [Fri, 3 Jan 2020 21:09:34 +0000 (13:09 -0800)]
.appveyor.yml: Convert to ASCII

Change a single character in a comment from UTF-8 into ASCII.

Fixes: a2b3cb65a9d3 ("appveyor: minor refactoring, clarifications")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoMerge branch 'master' of https://github.com/kenbarr1/fio
Jens Axboe [Sat, 4 Jan 2020 00:15:31 +0000 (17:15 -0700)]
Merge branch 'master' of https://github.com/kenbarr1/fio

* 'master' of https://github.com/kenbarr1/fio:
  io_u: fix rate limiting to handle file wrap-around

3 months agoio_u: fix rate limiting to handle file wrap-around
Ken Barr [Fri, 3 Jan 2020 20:11:44 +0000 (15:11 -0500)]
io_u: fix rate limiting to handle file wrap-around

When using "rate" with "time_based", the amount of time to sleep should be based
on the invocation time (epoch) rather than the start time of a given loop
(start) because no other statistics are reset when a file is re-opened.
Prior to this change, a wrap-around would cause the rate_ddir code to compute
that ~0 time had elapsed since the last I/O (i.e., a near-infinite rate), and
trigger a very long sleep in an attempt to adjust to the given rate.

Testing done:
  Using configuration below...
    Before: starts at 100 MB/sec and drops to 0 after ~10 sec (upon wrap) before
    recovering at t ~ 20.
    After: maintains 100 MB/sec for the duration of the test

  [repro]
  # 1 GiB + 1 Byte, to force a fill_io_u failure, the file to close and
  # reopen, and the start timestamp to reset.
  size=1073741825
  bs=2097152
  direct=1
  ioengine=psync
  readwrite=write

  runtime=30s
  time_based
  rate=100m

  filename=data_file

3 months agoRevert "Fix a potential deadlock in helper_do_stat()"
Jens Axboe [Fri, 3 Jan 2020 19:32:07 +0000 (12:32 -0700)]
Revert "Fix a potential deadlock in helper_do_stat()"

This reverts commit 31eca641ad91634e5ffcf369cd756b0506a700c1.

Killing this for now as it breaks the Windows build.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Fri, 3 Jan 2020 19:20:41 +0000 (12:20 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Fix a potential deadlock in helper_do_stat()
  t/read-to-pipe-async: Use the monotonic clock when measuring time intervals
  Use CLOCK_MONOTONIC for condition variables used by pthread_cond_timedwait()
  Makefile: Build more test code
  Suppress a Coverity taint warning in check_status_file()
  Micro-optimize __load_ioengine()

3 months agoMerge branch 'unit1' of https://github.com/kusumi/fio
Jens Axboe [Fri, 3 Jan 2020 19:19:10 +0000 (12:19 -0700)]
Merge branch 'unit1' of https://github.com/kusumi/fio

* 'unit1' of https://github.com/kusumi/fio:
  unittests: add unittest suite for oslib/strsep.c
  unittests: add unittest suite for oslib/strcasestr.c

3 months agoFix a potential deadlock in helper_do_stat()
Bart Van Assche [Tue, 31 Dec 2019 17:37:55 +0000 (09:37 -0800)]
Fix a potential deadlock in helper_do_stat()

pthread_mutex_lock(), pthread_cond_signal() and pthread_mutex_unlock() are
not async-signal-safe and hence must not be used inside a singal handler
implementation. Rework the code for communication with the helper thread
such that it becomes async-signal-safe.

Fixes: a47591e4923f ("Improve logging accuracy")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agot/read-to-pipe-async: Use the monotonic clock when measuring time intervals
Bart Van Assche [Wed, 1 Jan 2020 23:01:53 +0000 (15:01 -0800)]
t/read-to-pipe-async: Use the monotonic clock when measuring time intervals

Make the output independent of wall clock changes applied by e.g. ntpd.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoUse CLOCK_MONOTONIC for condition variables used by pthread_cond_timedwait()
Bart Van Assche [Wed, 1 Jan 2020 00:23:32 +0000 (16:23 -0800)]
Use CLOCK_MONOTONIC for condition variables used by pthread_cond_timedwait()

This patch ensures that the time during which pthread_cond_timedwait()
waits is not affected by wall clock adjustments by e.g. ntpd.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoMakefile: Build more test code
Bart Van Assche [Wed, 1 Jan 2020 22:23:53 +0000 (14:23 -0800)]
Makefile: Build more test code

Build the t/io_uring, t/memlock and t/read-to-pipe-async executables if
'make all' is executed. Do not include t/time-test because it uses a
non-portable 128-bit integer type.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoSuppress a Coverity taint warning in check_status_file()
Bart Van Assche [Tue, 31 Dec 2019 17:35:10 +0000 (09:35 -0800)]
Suppress a Coverity taint warning in check_status_file()

Suppress the following Coverity complaint:

CID 24143 (#1 of 1): Use of untrusted string value (TAINTED_STRING)
tainted_string: Passing tainted string fio_status_file_path to unlink, which
cannot accept tainted data.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoMicro-optimize __load_ioengine()
Bart Van Assche [Wed, 1 Jan 2020 00:31:26 +0000 (16:31 -0800)]
Micro-optimize __load_ioengine()

Instead of copying the I/O engine name, use a pointer to the I/O engine
name. This patch suppresses the following false positive Valgrind complaint:

Conditional jump or move depends on uninitialised value(s)
   at 0x41D451: __load_ioengine (ioengines.c:133)
   by 0x41D451: load_ioengine (ioengines.c:161)
   by 0x41FA59: ioengine_load (init.c:1126)
   by 0x423B64: parse_cmd_line (init.c:2670)
   by 0x4241A3: parse_options (init.c:2965)
   by 0x40F587: main (fio.c:42)

The code for copying I/O engine names was introduced by commit 2866c82d598e
("[PATCH] Separate io engines into separate loadable objects").

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 months agoio_uring: Enable io_uring ioengine on aarch64 arch
Zhenyu Ye [Wed, 25 Dec 2019 15:02:57 +0000 (08:02 -0700)]
io_uring: Enable io_uring ioengine on aarch64 arch

This patch enables io_uring ioengine on aarch64 arch.

Signed-off-by: Zhenyu Ye <yezhenyu2@huawei.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoio_uring: add option for non-vectored read/write commands
Jens Axboe [Mon, 23 Dec 2019 15:54:25 +0000 (08:54 -0700)]
io_uring: add option for non-vectored read/write commands

Also syncs io_uring.h with the kernel 5.6 version.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agounittests: add unittest suite for oslib/strsep.c
Tomohiro Kusumi [Sun, 22 Dec 2019 05:26:25 +0000 (14:26 +0900)]
unittests: add unittest suite for oslib/strsep.c

See b8b0e1eea7780a02ff67f0caeba446cc403f1b37
("unittests: add CUnit based unittest framework") for usage.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
3 months agounittests: add unittest suite for oslib/strcasestr.c
Tomohiro Kusumi [Sat, 21 Dec 2019 16:46:28 +0000 (01:46 +0900)]
unittests: add unittest suite for oslib/strcasestr.c

See b8b0e1eea7780a02ff67f0caeba446cc403f1b37
("unittests: add CUnit based unittest framework") for usage.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
3 months agoMerge branch 'doc_fixes' of https://github.com/sitsofe/fio
Jens Axboe [Sat, 21 Dec 2019 14:04:58 +0000 (07:04 -0700)]
Merge branch 'doc_fixes' of https://github.com/sitsofe/fio

* 'doc_fixes' of https://github.com/sitsofe/fio:
  doc: fix up sphinx warnings
  HOWTO: fix up broken formatting in registerfiles section
  doc: stop saying backslashes need escaping

3 months agoMerge branch 'travis' of https://github.com/sitsofe/fio
Jens Axboe [Sat, 21 Dec 2019 14:04:34 +0000 (07:04 -0700)]
Merge branch 'travis' of https://github.com/sitsofe/fio

* 'travis' of https://github.com/sitsofe/fio:
  travis: switch to ubuntu 18.04 and install more libraries
  travis: remove duplicate xcode image and add comments

3 months agodoc: fix up sphinx warnings
Sitsofe Wheeler [Sat, 21 Dec 2019 09:53:19 +0000 (09:53 +0000)]
doc: fix up sphinx warnings

- Reference to merge-blktrace-only needs the leading dashes to work
  (seemingly because it takes no parameters?)
- Nothing seems to be html_static_path, we don't have an _static dir and
  it generates a warning "html_static_path entry '_static' does not
  exist" so disable it for now

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 months agoHOWTO: fix up broken formatting in registerfiles section
Sitsofe Wheeler [Sat, 21 Dec 2019 09:17:50 +0000 (09:17 +0000)]
HOWTO: fix up broken formatting in registerfiles section

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 months agodoc: stop saying backslashes need escaping
Sitsofe Wheeler [Wed, 30 Oct 2019 06:14:12 +0000 (06:14 +0000)]
doc: stop saying backslashes need escaping

In 02dd2689885c6d64d9d01a92e67d40f73c0afc72 ("HOWTO: update directory
and filename option descriptions") I added a change that stated that
backslashes needed escaping too but this is not (and never has been) the
case. Fix this thinko by removing all mentions of backslash escaping
being needed from the documentation.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 months agotravis: switch to ubuntu 18.04 and install more libraries
Sitsofe Wheeler [Sat, 5 Oct 2019 14:18:38 +0000 (15:18 +0100)]
travis: switch to ubuntu 18.04 and install more libraries

Also sort the package dependencies into alphabetical order to make it
easier to see what has changed in the future.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 months agotravis: remove duplicate xcode image and add comments
Sitsofe Wheeler [Sat, 5 Oct 2019 13:36:48 +0000 (14:36 +0100)]
travis: remove duplicate xcode image and add comments

We already build on whatever the default Travis xcode image is
which is currently 9.4 so remove the duplicate build targeting 9.4 and
add some comments explaining the purpose of the different builds.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 months agoMerge branch 'mine/patch-2' of https://github.com/hannesweisbach/fio
Jens Axboe [Thu, 19 Dec 2019 12:38:30 +0000 (05:38 -0700)]
Merge branch 'mine/patch-2' of https://github.com/hannesweisbach/fio

* 'mine/patch-2' of https://github.com/hannesweisbach/fio:
  Add example job file for exit_what
  Expand choices for exitall

3 months agoAdd example job file for exit_what
Hannes Weisbach [Thu, 19 Dec 2019 09:33:24 +0000 (10:33 +0100)]
Add example job file for exit_what

3 months agoExpand choices for exitall
Hannes Weisbach [Wed, 27 Nov 2019 09:57:49 +0000 (10:57 +0100)]
Expand choices for exitall

Add exit_what thread_option with the following choices:

- "group" (default) exit all jobs of the same group (equal to "exitall=1")
- "stonewall", quit all jobs until the next stonewall
- "all", quit *all* jobs

Signed-off-by: Hannes Weisbach <hannes.weisbach@gmail.com>
3 months agoMerge branch 'windows_mkdir' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 19 Dec 2019 03:13:58 +0000 (20:13 -0700)]
Merge branch 'windows_mkdir' of https://github.com/sitsofe/fio

* 'windows_mkdir' of https://github.com/sitsofe/fio:
  filesetup: fix directory creation issues

3 months agofilesetup: fix directory creation issues
Sitsofe Wheeler [Sun, 8 Dec 2019 18:03:57 +0000 (18:03 +0000)]
filesetup: fix directory creation issues

- Fix double directory prefix concatenation that was happening after
  95af8dd57a7a9a564f99492bb1e04bb6d13b95b1 ("filesetup: don't call
  create_work_dirs() for ioengine with FIO_DISKLESSIO") refactored some
  code
- Stop always using '/' and switch to using FIO_OS_PATH_SEPARATOR when
  concatenating the directory value to the filename in set_name_idx()
- Change the permissions directories are created with to have the
  execute bit so they can be entered(!)
- Fix bug where filenames that were absolute unix paths would prevent
  the creation of intermediate directories
- Remove the configure mkdir test and introduce fio_mkdir to hide mkdir
  platform differences
- Make Windows mkdir emulation more robust against multiple slashes
  after the drive letter and when dealing dealing with with paths
  containing drive letters in general

This commit makes the following work on Windows:
./fio --thread --name=doubleslashtest \
 --filename='C\:\\Windows\Temp\fio.tmp' --size=4k --bs=4k --rw=read

The next example now works when D:\1\ exists but D:\1\2\ doesn't:

./fio --thread --name=mkdirtest --directory='D\:\1' \
 --filename='2\fio.tmp' --size=4k --bs=4k --rw=read

The following now works on macOS/Linux when /tmp/1/ exists but /tmp/1/2/
doesn't:

./fio --name=mkdirtest --directory='/tmp/1' \
 --filename='2/fio.tmp' --size=4k --bs=4k --rw=read

Fixes https://github.com/axboe/fio/issues/784 ,
https://github.com/axboe/fio/issues/864 and
https://github.com/axboe/fio/issues/870

Tested-by: Astolfo Rueda <astolfor@microsoft.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 months agoMerge branch 'issue-878' of https://github.com/vincentkfu/fio
Jens Axboe [Wed, 18 Dec 2019 16:53:48 +0000 (09:53 -0700)]
Merge branch 'issue-878' of https://github.com/vincentkfu/fio

* 'issue-878' of https://github.com/vincentkfu/fio:
  client/server: add missing fsync data structures

3 months agoclient/server: add missing fsync data structures
Vincent Fu [Mon, 16 Dec 2019 15:15:11 +0000 (10:15 -0500)]
client/server: add missing fsync data structures

In client/server mode, fsync latencies were missing because a couple
data structures were missed in the client/server send/receive thread
stats code. This patch adds those data structures and now we are able to
see the fsync latencies in client/server mode.

Fixes: https://github.com/axboe/fio/issues/878
Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
3 months agoMerge branch 'cygwin-build-error' of https://github.com/vincentkfu/fio
Jens Axboe [Mon, 16 Dec 2019 23:14:40 +0000 (16:14 -0700)]
Merge branch 'cygwin-build-error' of https://github.com/vincentkfu/fio

* 'cygwin-build-error' of https://github.com/vincentkfu/fio:
  Makefile: add libssp for Windows

3 months agoMerge branch 'travis-xcode11.2-python' of https://github.com/vincentkfu/fio
Jens Axboe [Mon, 16 Dec 2019 23:08:35 +0000 (16:08 -0700)]
Merge branch 'travis-xcode11.2-python' of https://github.com/vincentkfu/fio

* 'travis-xcode11.2-python' of https://github.com/vincentkfu/fio:
  .travis.yml: xcode11.2 scipy issue

3 months agoFio 3.17 fio-3.17
Jens Axboe [Mon, 16 Dec 2019 22:48:43 +0000 (15:48 -0700)]
Fio 3.17

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoMakefile: add libssp for Windows
Vincent Fu [Thu, 12 Dec 2019 22:22:06 +0000 (15:22 -0700)]
Makefile: add libssp for Windows

Recent Cygwin mingw releases require linking against libssp to avoid
undefined symbols.

See https://github.com/msys2/MINGW-packages/issues/5868 for discussion.
https://github.com/msys2/MINGW-packages/issues/5803 also contains
discussion of possibly disabling the FORTIFY_SOURCE linker flag.

Fixes: https://github.com/axboe/fio/issues/881
Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
3 months ago.travis.yml: xcode11.2 scipy issue
Vincent Fu [Fri, 13 Dec 2019 19:58:23 +0000 (12:58 -0700)]
.travis.yml: xcode11.2 scipy issue

The previous commit that added support for xcode11.2 resolved the
steadystate_tests scipy dependency with 'pip3 install scipy'. However,
the travis xcode11.2 image changed and now we need 'pip install scipy'
to provide the scipy dependency. To forestall future issues, install
scipy for both python2 and python3.

Also enable debug output in run-fio-tests.py to provide more information
in case of failure.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
3 months agot/io_uring: check for CONFIG_HAVE_GETTID
Jens Axboe [Thu, 12 Dec 2019 19:34:50 +0000 (12:34 -0700)]
t/io_uring: check for CONFIG_HAVE_GETTID

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agot/run-fio-tests: relax acceptance criterion for t0011
Vincent Fu [Tue, 10 Dec 2019 17:54:42 +0000 (12:54 -0500)]
t/run-fio-tests: relax acceptance criterion for t0011

I saw a handful of benign failures for t0011 when run on
travis/appveyor. This patch allows the test to still pass with a
measured IOPS of 998 to 1002 and adds a debug print for the value
actually observed.

Signed-off-by: Jens Axboe <axboe@kernel.dk>