fio.git
23 hours agofio: fix interaction between offset/size limited threads and "max_open_zones" master
Alexey Dobriyan [Thu, 2 Apr 2020 19:21:02 +0000 (22:21 +0300)]
fio: fix interaction between offset/size limited threads and "max_open_zones"

If thread bumps into "max_open_zones" limit, it tries to close/reopen some
other zone before issuing IO. This scan is done over full list of block device's
opened zones. It means that a zone which doesn't belong to thread's working
area can be altered or IO can be retargeted at such zone.

If IO is retargeted then it will be dropped by "is_valid_offset()" check.

What happens with null block device testing is that one thread monopolises
IO and others threads do basically nothing.

This config will reliably succeed now:

[global]
zonemode=zbd
zonesize=1M
rw=randwrite
...
thread
numjobs=2
offset_increment=128M

[j]
max_open_zones=2
size=2M

Starting 2 threads
zbd      7991  /dev/nullb0: zbd model string: host-managed
zbd      7991  Device /dev/nullb0 has 1024 zones of size 1024 KB
zbd      8009  /dev/nullb0: examining zones 0 .. 2
zbd      8010  /dev/nullb0: examining zones 128 .. 130
zbd      8009  /dev/nullb0: opening zone 0
zbd      8010  /dev/nullb0: opening zone 128
zbd      8009  /dev/nullb0: queued I/O (0, 4096) for zone 0
zbd      8009  zbd_convert_to_open_zone(/dev/nullb0): starting from zone 128 (offset 1552384, buflen 4096)

retargeted for other thread's zone (zone 0 => zone 128)

zbd      8010  /dev/nullb0: queued I/O (134217728, 4096) for zone 128
zbd      8009  zbd_convert_to_open_zone(/dev/nullb0): returning zone 128
zbd      8009  Dropped request with offset 134221824

and dropped

Note: quasi-randomness is kind of necessary to spread I/O. Imagine index 0
is picked all the time, zone living there will be reopened constantly and
get relatively little I/O.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Tested-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 days agoMerge branch 'github-issue-947' of https://github.com/vincentkfu/fio
Jens Axboe [Tue, 31 Mar 2020 16:21:48 +0000 (10:21 -0600)]
Merge branch 'github-issue-947' of https://github.com/vincentkfu/fio

* 'github-issue-947' of https://github.com/vincentkfu/fio:
  stat: eliminate extra log samples

3 days agostat: eliminate extra log samples
Vincent Fu [Tue, 31 Mar 2020 11:26:16 +0000 (07:26 -0400)]
stat: eliminate extra log samples

b2a432bfbb6d inadvertently added extra log samples.

$ ./fio-canonical/fio --name=test --time_based --runtime=10s --write_lat_log=fio-07-b2a432 --log_avg_msec=1000 --size=1G --rw=rw
test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.17-93-gb2a4
Starting 1 process
...
$ cat fio-07-b2a432_clat.1.log
1000, 5851, 0, 0, 0
1000, 2551, 1, 0, 0
1000, 5028, 1, 0, 0
2000, 4175, 0, 0, 0
2000, 3214, 1, 0, 0
2000, 60619, 0, 0, 0
...

There should only be two lines at each timestamp (one for reads, one for
writes), but the first two timestamps have three lines each.

The cause is an inadvertent change in stat.c:add_log_sample() of
__add_stat_to_log to _add_stat_to_log. Reverting to the two-underscore
version resolves this issue.

Fixes: https://github.com/axboe/fio/issues/947
Fixes: b2a432bfbb6d ("Per-command priority: Priority logging and libaio/io_uring cmdprio_percentage")
Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 days agoMerge branch 'jsonplus2csv' of https://github.com/vincentkfu/fio
Jens Axboe [Thu, 26 Mar 2020 15:03:40 +0000 (09:03 -0600)]
Merge branch 'jsonplus2csv' of https://github.com/vincentkfu/fio

* 'jsonplus2csv' of https://github.com/vincentkfu/fio:
  .travis.yml: remove pip line from xcode11.2 config
  t/jsonplus2csv_test.py: test script for tools/fio_jsonplus_clat2csv
  tools/fio_jsonplus2csv: accommodate multiple lat measurements

8 days ago.travis.yml: remove pip line from xcode11.2 config
Vincent Fu [Wed, 25 Mar 2020 17:48:39 +0000 (13:48 -0400)]
.travis.yml: remove pip line from xcode11.2 config

travis-ci changed the xcode11.2 image and 'pip' is no longer available.
So only run 'pip install scipy' for the default xcode image.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 days agot/jsonplus2csv_test.py: test script for tools/fio_jsonplus_clat2csv
Vincent Fu [Wed, 25 Mar 2020 16:53:54 +0000 (12:53 -0400)]
t/jsonplus2csv_test.py: test script for tools/fio_jsonplus_clat2csv

Add a script to run a basic jsonplus to CSV conversion and then validate
the conversion.

Also integrate this test script with t/run-fio-tests.py and install the
python package 'six' to support fio_jsonplus_clat2csv in the AppVeyor
build/testing environment.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 days agotools/fio_jsonplus2csv: accommodate multiple lat measurements
Vincent Fu [Mon, 23 Mar 2020 22:14:40 +0000 (18:14 -0400)]
tools/fio_jsonplus2csv: accommodate multiple lat measurements

Add some intelligence to this script so that it works for any of
submission, completion, and total latency whenever they are present. The
CSV data format is changed to accommodate this.

While we're here also do the following:

add a way to generate optional debug output
add validate option that compares generated CSV data with the original
json+ data
fix style issues identified by pylint3
update documentation

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2 weeks agozbd: add test for stressing zone locking
Naohiro Aota [Fri, 28 Feb 2020 07:12:48 +0000 (16:12 +0900)]
zbd: add test for stressing zone locking

Add a test to stress zone locking mechanism by having a large number of
threads with a small number of max_open_zones. Run 30 seconds time-based
fio under the timeout command. After 45 seconds, "timeout" kill -KILL the
fio process. If a zone lock deadlocks, fio is killed by the timeout
command, and this test fails. If not, fio runs to the end and this test
success.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Tested-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoio_u: ensure io_u_quiesce() to process all the IOs
Naohiro Aota [Fri, 28 Feb 2020 07:12:47 +0000 (16:12 +0900)]
io_u: ensure io_u_quiesce() to process all the IOs

Currently, when IO have an error io_u_quiesce() stops processing
in-flight IOs there and leaves other IOs non-completed. This is not a
desired behavior for io_u_quiesce(). Fix it by continuing even on
error.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Tested-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agobackend: always clean up pending aios
Naohiro Aota [Fri, 28 Feb 2020 07:12:46 +0000 (16:12 +0900)]
backend: always clean up pending aios

cleanup_pending_aios() is called when a thread exits with error, so all the
call site of this function is under "if (td->error)". However, commit
d28174f0189c ("workqueue: ensure we see deferred error for IOs"), for some
reason, added "if (td->error) return" at the head of this function, making
this function practically void. Revert this part to ensure cleaning up
pending aios.

Besides, cleanup_pending_aios() should not return even when
io_u_queued_complete() failed. Because, it keeps in-flight aios left.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Tested-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agozbd: use zone_lock to lock a zone
Naohiro Aota [Fri, 28 Feb 2020 07:12:45 +0000 (16:12 +0900)]
zbd: use zone_lock to lock a zone

commit 6f0c608564c3 ("zbd: Avoid async I/O multi-job workload deadlock")
introduced io_u_quiesce() when it failed to lock a zone to avoid deadlock.
This situation can happen on the other locking place like
zbd_convert_to_open_zone(). Thus, introduce common helper "zone_lock" to
lock a zone.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Tested-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agozbd: reset one zone at a time
Naohiro Aota [Fri, 28 Feb 2020 07:12:44 +0000 (16:12 +0900)]
zbd: reset one zone at a time

zbd_rest_zones() currently scans over device zones and try to reset as much
zones as possible at a time. However, this routine takes all the lock on
the range and causes a lot of lock contentions with other threads.

This commit change the behavior to hold the lock and examine one zone at a
time. While it will increase the number of ioctl() call when it need to
reset contiguous, the overhead of increased number of ioctl()s are anyway
amortized by device side's reset performance.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Tested-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agozbd: avoid initializing swd when unnecessary
Naohiro Aota [Fri, 28 Feb 2020 07:12:43 +0000 (16:12 +0900)]
zbd: avoid initializing swd when unnecessary

When enable_check_swd == false, there is no use to initialize swd. Just
disable it in this case.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Tested-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agogauss.c: correct the stddev initializtion
Feng Tang [Tue, 17 Mar 2020 12:53:31 +0000 (20:53 +0800)]
gauss.c: correct the stddev initializtion

Signed-off-by: Feng Tang <feng.tang@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agot/io_uring: don't use *rand48_r() variants
Jens Axboe [Mon, 16 Mar 2020 14:30:36 +0000 (08:30 -0600)]
t/io_uring: don't use *rand48_r() variants

Not available on all platforms.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoMerge branch 'patch-1' of https://github.com/neheb/fio
Jens Axboe [Mon, 16 Mar 2020 00:59:19 +0000 (18:59 -0600)]
Merge branch 'patch-1' of https://github.com/neheb/fio

* 'patch-1' of https://github.com/neheb/fio:
  configure: fix vasprintf check under musl

2 weeks agoconfigure: fix vasprintf check under musl
Rosen Penev [Sun, 15 Mar 2020 21:43:39 +0000 (14:43 -0700)]
configure: fix vasprintf check under musl

It errors when passing NULL or 0. Passing an empty va_list works.

3 weeks agoFio 3.19 fio-3.19
Jens Axboe [Thu, 12 Mar 2020 17:12:50 +0000 (11:12 -0600)]
Fio 3.19

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 weeks agoengines/io_uring: delete fio_option_is_set() calls when submitting sqes
Xiaoguang Wang [Thu, 12 Mar 2020 11:16:17 +0000 (19:16 +0800)]
engines/io_uring: delete fio_option_is_set() calls when submitting sqes

The fio_option_is_set() call in fio_ioring_prep() is time-consuming,
which will reduce sqe's submit rate drastically. To fix this issue,
add two new variables to record whether ioprio_class or ioprio_set
is set. I use a simple fio job to evaluate the performance:
    fio -name=fiotest -filename=/dev/nvme0n1 -iodepth=4 -thread -rw=read
    -ioengine=io_uring -hipri=0 -sqthread_poll=0 -direct=1 -bs=4k -size=10G
    -numjobs=1 -time_based -runtime=120

Before this patch:
  READ: bw=969MiB/s (1016MB/s), 969MiB/s-969MiB/s (1016MB/s-1016MB/s),
  io=114GiB (122GB), run=120001-120001msec

With this patch:
  READ: bw=1259MiB/s (1320MB/s), 1259MiB/s-1259MiB/s (1320MB/s-1320MB/s),
  io=148GiB (158GB), run=120001-120001msec

Signed-off-by: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 weeks agoMerge branch 'patch-1' of https://github.com/felixonmars/fio
Jens Axboe [Tue, 3 Mar 2020 15:11:08 +0000 (08:11 -0700)]
Merge branch 'patch-1' of https://github.com/felixonmars/fio

* 'patch-1' of https://github.com/felixonmars/fio:
  Correct multiple typos in engines/libhdfs.c

4 weeks agoCorrect multiple typos in engines/libhdfs.c
Felix Yan [Tue, 3 Mar 2020 07:37:54 +0000 (15:37 +0800)]
Correct multiple typos in engines/libhdfs.c

4 weeks agoMerge branch 'filestat3' of https://github.com/kusumi/fio
Jens Axboe [Mon, 2 Mar 2020 16:34:38 +0000 (09:34 -0700)]
Merge branch 'filestat3' of https://github.com/kusumi/fio

* 'filestat3' of https://github.com/kusumi/fio:
  engines/filestat: add statx(2) syscall support

4 weeks agoengines/filestat: add statx(2) syscall support
Tomohiro Kusumi [Mon, 2 Mar 2020 15:31:25 +0000 (00:31 +0900)]
engines/filestat: add statx(2) syscall support

This commit
1) tests existence of statx(2) and libc support on ./confiugre.
2) adds oslib/statx.c and implements statx(2) using above result.
3) adds statx(2) support in filestat ioengine.

Confirmed compilation on Fedora31, FreeBSD, and NetBSD.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
4 weeks agoMerge branch 'genfio-bash' of https://github.com/sitsofe/fio
Jens Axboe [Sun, 1 Mar 2020 22:43:14 +0000 (15:43 -0700)]
Merge branch 'genfio-bash' of https://github.com/sitsofe/fio

* 'genfio-bash' of https://github.com/sitsofe/fio:
  genfio: use /bin/bash hashbang

4 weeks agoMerge branch 'fix-win-raw' of https://github.com/sitsofe/fio
Jens Axboe [Sun, 1 Mar 2020 22:42:48 +0000 (15:42 -0700)]
Merge branch 'fix-win-raw' of https://github.com/sitsofe/fio

* 'fix-win-raw' of https://github.com/sitsofe/fio:
  filesetup: fix win raw disk access and improve dir creation failure msg

4 weeks agoMerge branch 'clean1' of https://github.com/kusumi/fio
Jens Axboe [Sun, 1 Mar 2020 22:41:37 +0000 (15:41 -0700)]
Merge branch 'clean1' of https://github.com/kusumi/fio

* 'clean1' of https://github.com/kusumi/fio:
  Makefile: don't fail to remove conditionally compiled binaries on clean

4 weeks agofilesetup: fix win raw disk access and improve dir creation failure msg
Sitsofe Wheeler [Sun, 1 Mar 2020 18:27:19 +0000 (18:27 +0000)]
filesetup: fix win raw disk access and improve dir creation failure msg

The commit df18600fd06258b96ae6f6b530ecdff541c2a82d ("filesetup: fix
directory creation issues") broke Windows raw/physical disk access
because Windows doesn't consider a path that only consists of a
namespace (such as the device namespace "\\.\" - see
https://docs.microsoft.com/en-gb/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#namespaces
for information on Windows' namespaces) to exist as a directory(!).

Workaround the issue for raw devices by explicitly considering the
device namespace subcomponent to always be valid. Further, intermediate
path components in UNC paths or paths starting with a namespace might
also not "exist" so introduce backwards scanning for the longest
pre-existing directory to sidestep this. The function doing this is made
available for non-windows platforms so a similar code path is used
everywhere.

Tests done:

Windows:
> ./fio.exe --name=dtest --thread --size=16k --rw=write `
  --filename 'fio.tmp'
> ./fio.exe --name=dtest --thread --size=16k --rw=write `
  --filename '\\?\C\:\Windows\Temp\fio\fio.tmp'
> Clear-Disk 1 -RemoveData -Confirm:$false # Destroys partition data!
> ./fio.exe --name=dtest --thread --size=16k --rw=write `
   --filename '\\.\PhysicalDrive1'
> ./fio.exe --name=dtest --thread --size=16k --rw=write `
  --filename '\fio.tmp'
> ./fio.exe --name=dtest --thread --size=16k --rw=write `
  --filename '\\LOCALHOST\Users\User\fio\fio.tmp'

macOS:
$ rm -rf /tmp/fio
$ ./fio --name=dtest --size=16k --filename /tmp/fio/fio.tmp \
  --rw=write

Finally, change the directory creation error message to give a human
error message rather than just an errno.

Fixes: https://github.com/axboe/fio/issues/916
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
4 weeks agogenfio: use /bin/bash hashbang
Sitsofe Wheeler [Sun, 1 Mar 2020 22:01:39 +0000 (22:01 +0000)]
genfio: use /bin/bash hashbang

Not all distros have done usrmerge - Debian
https://salsa.debian.org/debian/fio/-/blob/5a608e04b947aed0d3b49d7ca9a85ed9afe9ec56/debian/patches/genfio-interpreter
and SUSE (https://build.opensuse.org/request/show/541207 ) have been
carrying a patch changing the interpreter for genfio back to /bin/bash
and that's before you get to other OSes such as macOS...

Since /bin/bash works everywhere let's do this upstream too.

Fixes: https://github.com/axboe/fio/pull/883
Inspired-by: Changcheng Liu <changcheng.liu@aliyun.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
5 weeks agoio_uring: we should not need two write barriers for SQ updates
Jens Axboe [Thu, 27 Feb 2020 02:54:12 +0000 (19:54 -0700)]
io_uring: we should not need two write barriers for SQ updates

Just after updating the tail should be fine.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agofio: remove duplicate global definition of tsc_reliable
Eric Sandeen [Fri, 31 Jan 2020 19:19:07 +0000 (13:19 -0600)]
fio: remove duplicate global definition of tsc_reliable

Newer gcc v10 complains about globals being defined twice due to
a new default -fno-common flag. This causes build failures:

  LINK t/fio-dedupe
/usr/bin/ld: t/arch.o:(.bss+0x4): multiple definition of `tsc_reliable'; gettime.o:(.bss+0x4): first defined here
collect2: error: ld returned 1 exit status
make: *** [Makefile:499: t/fio-dedupe] Error 1

Each of the tests which call arch_init() to initialize tsc_reliable:

  File          Function       Line
1 t/dedupe.c    main           536 arch_init(argv);
2 t/lfsr-test.c main            37 arch_init(argv);
3 t/stest.c     main            86 arch_init(argv);

already link with gettime.o which defines tsc_reliable,
so there is no need to provide it from t/arch.c

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoMerge branch 'div-by-zero' of https://github.com/vincentkfu/fio
Jens Axboe [Mon, 24 Feb 2020 15:23:52 +0000 (08:23 -0700)]
Merge branch 'div-by-zero' of https://github.com/vincentkfu/fio

* 'div-by-zero' of https://github.com/vincentkfu/fio:
  t/latency_percentiles: avoid division by zero

5 weeks agot/latency_percentiles: avoid division by zero
Vincent Fu [Mon, 24 Feb 2020 14:20:46 +0000 (09:20 -0500)]
t/latency_percentiles: avoid division by zero

For the smallest latency durations, the actual and approximate values
should be exactly the same. Use this property to avoid a division by
zero when the actual latency value is zero.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
7 weeks agoMakefile: don't fail to remove conditionally compiled binaries on clean
Tomohiro Kusumi [Thu, 13 Feb 2020 15:57:31 +0000 (00:57 +0900)]
Makefile: don't fail to remove conditionally compiled binaries on clean

`rm -f $(T_PROGS) $(T_TEST_PROGS)` in `clean` target is intended to
remove all binaries under t directory, but it actually fails to remove
`t/{fio-btrace2fio,io_uring,read-to-pipe-async}` files for not
evaluating platform specific variables on `make clean`.

 # uname -r
 5.4.17-200.fc31.x86_64
 # make clean
 # file t/* | grep ELF | awk '{print $1,$2,$3,$4,$5}'
 t/fio-btrace2fio: ELF 64-bit LSB executable,
 t/io_uring: ELF 64-bit LSB executable,
 t/read-to-pipe-async: ELF 64-bit LSB executable,

The easiest way to fix this is to hard-code these conditionally
compiled binaries.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
8 weeks agoMerge branch 'master' of https://github.com/vincentkfu/fio
Jens Axboe [Thu, 6 Feb 2020 19:17:25 +0000 (12:17 -0700)]
Merge branch 'master' of https://github.com/vincentkfu/fio

* 'master' of https://github.com/vincentkfu/fio:
  t/run-fio-tests: fix style issues
  gfio: add high/low priority latency results
  .gitignore: add some test programs
  stat: summary statistics for both high/low priority latencies

8 weeks agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Thu, 6 Feb 2020 18:58:09 +0000 (11:58 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Make the JSON code easier to analyze

8 weeks agoUnify architecture io_uring syscall numbers
Jens Axboe [Thu, 6 Feb 2020 14:59:00 +0000 (07:59 -0700)]
Unify architecture io_uring syscall numbers

Only Alpha is the odd one out, apart from that all architectures use
the same system call number. So pull the code out arch headers, and
into the generic arch header.

Fixes: https://github.com/axboe/fio/issues/923
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agoMake the JSON code easier to analyze
Bart Van Assche [Wed, 5 Feb 2020 05:18:31 +0000 (21:18 -0800)]
Make the JSON code easier to analyze

Coverity reports the following false positive for the JSON code in stat.c:

CID 284826 (#1 of 1): Resource leak (RESOURCE_LEAK)
17. leaked_storage: Variable percentile_object going out of scope leaks the storage it points to.

Suppress this false positive by making the JSON code easier to analyze. The
changes in this patch are as follows:
- Change the macros in json.h into inline functions such that the compiler
  performs argument type verification at compile time.
- Pass a struct json_value pointer to json_*_add_value_type() instead of
  using varargs.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
8 weeks agot/run-fio-tests: fix style issues
Vincent Fu [Wed, 5 Feb 2020 21:17:42 +0000 (16:17 -0500)]
t/run-fio-tests: fix style issues

Many style issues were identified by pylint3. Fix most of them.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agogfio: add high/low priority latency results
Vincent Fu [Wed, 5 Feb 2020 19:55:50 +0000 (14:55 -0500)]
gfio: add high/low priority latency results

For the cmdprio_percentage option, add high/low priority summary
statistics and latency percentiles.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks ago.gitignore: add some test programs
Vincent Fu [Wed, 5 Feb 2020 19:12:17 +0000 (14:12 -0500)]
.gitignore: add some test programs

t/io_uring, t/memlock, and t/read-to-pipe-async are now routinely built.
Add them to .gitignore to avoid cluttering up the output of 'git
status'.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agostat: summary statistics for both high/low priority latencies
Vincent Fu [Wed, 5 Feb 2020 19:05:50 +0000 (14:05 -0500)]
stat: summary statistics for both high/low priority latencies

For the sake of completeness, include summary statistics for both high
and low priority latencies in the normal output when cmdprio_percentage
is used.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agoFio 3.18 fio-3.18
Jens Axboe [Wed, 5 Feb 2020 14:59:58 +0000 (07:59 -0700)]
Fio 3.18

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agoMerge branch 'latency-rebase-again' of https://github.com/vincentkfu/fio
Jens Axboe [Tue, 4 Feb 2020 17:05:09 +0000 (10:05 -0700)]
Merge branch 'latency-rebase-again' of https://github.com/vincentkfu/fio

* 'latency-rebase-again' of https://github.com/vincentkfu/fio:
  t/run-fio-tests: increase time allowed for tests to pass
  t/run-fio-tests: add latency_percentiles.py
  t/latency_percentiles: test latency percentile reporting
  stat: fix high/low prio unified rw bug
  fio: better distinguish between high and low priority
  stat: make priority summary statistics consistent with percentiles
  docs: updates for slat, clat, lat percentile reporting
  gfio: display slat, clat, and lat percentiles
  fio: report percentiles for slat, clat, lat
  fio: groundwork for adding slat, lat percentiles

8 weeks agoMerge branch 'filestat2' of https://github.com/kusumi/fio
Jens Axboe [Tue, 4 Feb 2020 17:04:19 +0000 (10:04 -0700)]
Merge branch 'filestat2' of https://github.com/kusumi/fio

* 'filestat2' of https://github.com/kusumi/fio:
  engines/filestat: change "lstat" bool option to "stat_type" str option

8 weeks agoengines/filestat: change "lstat" bool option to "stat_type" str option
Tomohiro Kusumi [Tue, 4 Feb 2020 12:25:47 +0000 (21:25 +0900)]
engines/filestat: change "lstat" bool option to "stat_type" str option

Per suggestion from Jens, change a bool option to str option
to better support stat(2) variants (at this point before 3.18).

https://github.com/axboe/fio/pull/912#issuecomment-577814885

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
8 weeks agot/run-fio-tests: increase time allowed for tests to pass
Vincent Fu [Mon, 3 Feb 2020 16:06:36 +0000 (11:06 -0500)]
t/run-fio-tests: increase time allowed for tests to pass

With a 300s timeout I occasionally see tests fail when they usually
pass. Bump the timeout to 600s to give tests more of an opportunity to
finish successfully.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agot/run-fio-tests: add latency_percentiles.py
Vincent Fu [Tue, 21 Jan 2020 19:10:38 +0000 (14:10 -0500)]
t/run-fio-tests: add latency_percentiles.py

Add the latency percentiles test script to the general test script.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agot/latency_percentiles: test latency percentile reporting
Vincent Fu [Fri, 17 Jan 2020 20:50:45 +0000 (15:50 -0500)]
t/latency_percentiles: test latency percentile reporting

This script runs fio jobs exercising the slat_percentiles,
clat_percentiles, and lat_percentiles options under various conditions.
It carries out several tests comparing write_lat_log data with
percentiles reported by fio.

It also includes limited tests of high/low priority latency reporting
with the cmdprio_percentage option. Only Linux supports the
cmdprio_percentage option. Skips these tests on other platforms.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agostat: fix high/low prio unified rw bug
Vincent Fu [Fri, 31 Jan 2020 22:27:26 +0000 (17:27 -0500)]
stat: fix high/low prio unified rw bug

With unified_rw_reporting enabled, high/low priority summary statistics
were not correctly reported. This patch fixes the problem.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agofio: better distinguish between high and low priority
Vincent Fu [Thu, 30 Jan 2020 22:37:05 +0000 (17:37 -0500)]
fio: better distinguish between high and low priority

Change the variable names for the low priority latency variables to
explicity distinguish them from the high priority latency variables.
Also rename the JSON objects to do the same. This reduces the mental
effort required to read the code and output.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agostat: make priority summary statistics consistent with percentiles
Vincent Fu [Thu, 30 Jan 2020 20:16:52 +0000 (15:16 -0500)]
stat: make priority summary statistics consistent with percentiles

When lat_percentiles=1, reported high/low priority percentiles will
describe total latency values. Make the summary statistics (min, max,
mean, stddev) also describe total latency. Otherwise there will be a
mismatch in the JSON output with completion latency summary statistics
combined with total latency percentile values.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agodocs: updates for slat, clat, lat percentile reporting
Vincent Fu [Thu, 9 Jan 2020 16:15:00 +0000 (11:15 -0500)]
docs: updates for slat, clat, lat percentile reporting

Modify the HOWTO and man page to reflect changes to the slat, clat, and
lat percentile reporting options.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agogfio: display slat, clat, and lat percentiles
Vincent Fu [Tue, 21 Jan 2020 22:00:39 +0000 (17:00 -0500)]
gfio: display slat, clat, and lat percentiles

Display in the results window slat, clat, and lat percentiles if they
were requested by the user.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agofio: report percentiles for slat, clat, lat
Vincent Fu [Wed, 29 Jan 2020 19:08:26 +0000 (14:08 -0500)]
fio: report percentiles for slat, clat, lat

Currently, fio can report percentiles for either completion latencies or
total latencies. This patch adds the ability to simultaneously report
percentiles for submission, completion, and total latencies.

- simplify the JSON lat percentile reporting code
- parsing the JSON output will be unaffected, although now setting
lat_percentiles=1 no longer suppresses clat percentiles
- this may break parsing of the normal output since more latency
percentile blocks may be displayed
- to maintain backward compatibility in the terse output, total
latencies are reported if lat_percentiles=1 even when clat percentiles
have also been computed
- change the gtod_reduce callback to only change option values when
gtod_reduce=1
- for the new cmdprio_percentage latencies, if lat_percentiles=1,
*total* latency percentiles will be tracked. Otherwise, *completion*
latency percentiles will be tracked.
- also make sure we skip the high/low priority latency code if we are
producing output for sync latencies.
- make add_slat_sample use an unsigned long long for the time to be
consistent with other functions

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
8 weeks agofio: groundwork for adding slat, lat percentiles
Vincent Fu [Thu, 23 Jan 2020 19:46:14 +0000 (14:46 -0500)]
fio: groundwork for adding slat, lat percentiles

Expand the data structure used for tabulating completion latency
percentiles so that we will have the ability to tabulate percentiles for
submission and total latency as well. This should not change fio's
behavior.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2 months agoMerge branch 'stat-averaging-interval-start-fix' of https://github.com/maciejsszmigie...
Jens Axboe [Tue, 28 Jan 2020 21:15:35 +0000 (14:15 -0700)]
Merge branch 'stat-averaging-interval-start-fix' of https://github.com/maciejsszmigiero/fio

* 'stat-averaging-interval-start-fix' of https://github.com/maciejsszmigiero/fio:
  stat: fix calculation of bw and iops statistics based on samples

2 months agoMerge branch 'openbsd_swap' of https://github.com/bsdkurt/fio
Jens Axboe [Tue, 28 Jan 2020 21:15:06 +0000 (14:15 -0700)]
Merge branch 'openbsd_swap' of https://github.com/bsdkurt/fio

* 'openbsd_swap' of https://github.com/bsdkurt/fio:
  Use swap16/32/64 on OpenBSD.

2 months agoMerge branch 'no_cpu_clock_no_tls_thread' of https://github.com/bsdkurt/fio
Jens Axboe [Tue, 28 Jan 2020 21:14:30 +0000 (14:14 -0700)]
Merge branch 'no_cpu_clock_no_tls_thread' of https://github.com/bsdkurt/fio

* 'no_cpu_clock_no_tls_thread' of https://github.com/bsdkurt/fio:
  Fix build on architectures that don't have both cpu clock and __thread support.

2 months agoUse swap16/32/64 on OpenBSD.
Kurt Miller [Tue, 28 Jan 2020 20:52:51 +0000 (15:52 -0500)]
Use swap16/32/64 on OpenBSD.

2 months agoFix build on architectures that don't have both cpu clock and __thread
Kurt Miller [Tue, 28 Jan 2020 20:49:03 +0000 (15:49 -0500)]
Fix build on architectures that don't have both cpu clock and __thread
support.

2 months agostat: fix calculation of bw and iops statistics based on samples
Maciej S. Szmigiero [Tue, 28 Jan 2020 20:39:15 +0000 (21:39 +0100)]
stat: fix calculation of bw and iops statistics based on samples

It is obvious on many runs that bw and iops averages from statistics based
on samples are way off (sometimes by an order of magnitude) in comparison
to the main numbers reported for these values.

When sampling these values fio assumes that their each next averaging
interval starts exactly the set averaging interval later from the start of
the previous averaging interval.

However, this isn't necessary true as scheduling delays may lengthen such
intervals, so the next one should start when the previous one has actually
ended, since that's when the collected totals are counted.

On my test system (with fio set to the default averaging interval length of
500 msec) the actual lengths are usually around 650 msec, however
700+ msec intervals aren't unusual.

This causes pretty significant bw and iops measurement discrepancy between
the sample-based values and the overall measurements.

Let's make sure that we use the actual end time of the previous averaging
interval as the beginning of the next averaging interval.

With this change the bw and iops sample-based values become almost the
same as the main numbers reported.

Signed-off-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com>
2 months agoMerge branch 'filestat1' of https://github.com/kusumi/fio
Jens Axboe [Thu, 23 Jan 2020 18:35:23 +0000 (11:35 -0700)]
Merge branch 'filestat1' of https://github.com/kusumi/fio

* 'filestat1' of https://github.com/kusumi/fio:
  engines/filestat: add "lstat" option to use lstat(2)

2 months agoengines/filestat: add "lstat" option to use lstat(2)
Tomohiro Kusumi [Thu, 23 Jan 2020 16:36:32 +0000 (01:36 +0900)]
engines/filestat: add "lstat" option to use lstat(2)

Use stat(2) by default, but lstat(2) if "lstat" bool option specified.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
2 months agostat: ensure we align correctly
Jens Axboe [Thu, 23 Jan 2020 02:53:14 +0000 (19:53 -0700)]
stat: ensure we align correctly

Fixes: b2a432bfbb6d ("Per-command priority: Priority logging and libaio/io_uring cmdprio_percentage")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'priorityQueuing' of https://github.com/Seagate/fio
Jens Axboe [Thu, 23 Jan 2020 02:50:41 +0000 (19:50 -0700)]
Merge branch 'priorityQueuing' of https://github.com/Seagate/fio

* 'priorityQueuing' of https://github.com/Seagate/fio:
  Per-command priority: Priority logging and libaio/io_uring cmdprio_percentage
  Whitespace standardization

2 months agoPer-command priority: Priority logging and libaio/io_uring cmdprio_percentage
Phillip Chen [Wed, 22 Jan 2020 23:28:38 +0000 (16:28 -0700)]
Per-command priority: Priority logging and libaio/io_uring cmdprio_percentage

Add cmdprio_percentage option to libaio and io_uring engines to set
ioprio on a per-command basis. Add tracking of high priority
commands to be displayed separately in human readable and JSON
outputs.

2 months agoWhitespace standardization
Phillip Chen [Tue, 21 Jan 2020 18:00:17 +0000 (11:00 -0700)]
Whitespace standardization

Remove trailing whitespace.

2 months agoengines: add engine for file stat
Su, Friendy [Wed, 22 Jan 2020 15:31:25 +0000 (08:31 -0700)]
engines: add engine for file stat

This engine is to measure performance of accessing file's meta data.
This is for the actual access pattern which does not do real IO, but
just look up the file and get file's attribute.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Sat, 18 Jan 2020 16:35:25 +0000 (09:35 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  client: Fix memory leaks in handle_job_opt()
  pmemblk: Fix a memory leak
  lib/memcpy: Suppress a Coverity leak report for setup_tests()
  stat: Fix another memory leak in add_ddir_status_json()
  stat: Fix a memory leak in add_ddir_status_json()
  stat: Remove several superfluous if-tests
  stat: Remove more superfluous casts
  stat: Remove two superfluous casts

2 months agoclient: Fix memory leaks in handle_job_opt()
Bart Van Assche [Sat, 18 Jan 2020 05:10:41 +0000 (21:10 -0800)]
client: Fix memory leaks in handle_job_opt()

This patch fixes the following Coverity complaint:

CID 169311 (#1 of 1): Resource leak (RESOURCE_LEAK)
9. leaked_storage: Variable p going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agopmemblk: Fix a memory leak
Bart Van Assche [Sat, 18 Jan 2020 05:08:10 +0000 (21:08 -0800)]
pmemblk: Fix a memory leak

This patch fixes the following Coverity complaint:

CID 169312 (#1 of 1): Resource leak (RESOURCE_LEAK)
5. leaked_storage: Variable path going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agolib/memcpy: Suppress a Coverity leak report for setup_tests()
Bart Van Assche [Sat, 18 Jan 2020 05:02:00 +0000 (21:02 -0800)]
lib/memcpy: Suppress a Coverity leak report for setup_tests()

Suppress the following Coverity memory leak complaint:

CID 169324 (#1 of 1): Resource leak (RESOURCE_LEAK)
7. leaked_storage: Variable src going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Fix another memory leak in add_ddir_status_json()
Bart Van Assche [Sat, 18 Jan 2020 05:00:01 +0000 (21:00 -0800)]
stat: Fix another memory leak in add_ddir_status_json()

Only compute the percentile_object and clat_bins_object data structures if
these will be stored inside another JSON object.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Fix a memory leak in add_ddir_status_json()
Bart Van Assche [Sat, 18 Jan 2020 04:42:27 +0000 (20:42 -0800)]
stat: Fix a memory leak in add_ddir_status_json()

This patch fixes the following Coverity complaint:

CID 173000 (#1 of 1): Resource leak (RESOURCE_LEAK)
18. leaked_storage: Variable ovals going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Remove several superfluous if-tests
Bart Van Assche [Sat, 18 Jan 2020 04:41:43 +0000 (20:41 -0800)]
stat: Remove several superfluous if-tests

Passing NULL to free() is allowed. From https://pubs.opengroup.org/onlinepubs/9699919799/functions/free.html#:

If ptr is a null pointer, no action shall occur.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Remove more superfluous casts
Bart Van Assche [Sat, 18 Jan 2020 04:50:19 +0000 (20:50 -0800)]
stat: Remove more superfluous casts

No cast is necessary to pass a char * argument to a function that accepts
a const char * argument.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agostat: Remove two superfluous casts
Bart Van Assche [Sat, 18 Jan 2020 04:40:41 +0000 (20:40 -0800)]
stat: Remove two superfluous casts

The C language supports implicit casting from a non-void to void pointer.
Remove two explicit casts from a non-void to a void pointer.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoMerge branch 'fix_verify_push' of https://github.com/gwendalcr/fio
Jens Axboe [Thu, 16 Jan 2020 22:44:44 +0000 (15:44 -0700)]
Merge branch 'fix_verify_push' of https://github.com/gwendalcr/fio

* 'fix_verify_push' of https://github.com/gwendalcr/fio:
  verify: Fix test to not check for numberio when verify_only is true

2 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Thu, 16 Jan 2020 22:44:16 +0000 (15:44 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  t/read-to-pipe-async: Complain if pthread_detach() fails
  t/read-to-pipe-async: Do not divide by zero
  t/memlock: Verify 'threads' argument
  server: Make it explicit that the setsockopt() return value is ignored
  blktrace: Fix memory leaks in error paths
  blktrace: Check value of 'merge_buf' pointer before using it
  blktrace: Pass a positive error code to td_verror()

2 months agoMerge branch 'issue-825' of https://github.com/LeaflessMelospiza/fio
Jens Axboe [Thu, 16 Jan 2020 22:43:10 +0000 (15:43 -0700)]
Merge branch 'issue-825' of https://github.com/LeaflessMelospiza/fio

* 'issue-825' of https://github.com/LeaflessMelospiza/fio:
  Moved diskutil reporting functions to stat.c

2 months agoverify: Fix test to not check for numberio when verify_only is true
Gwendal Grignou [Tue, 14 Jan 2020 00:35:10 +0000 (16:35 -0800)]
verify: Fix test to not check for numberio when verify_only is true

io->numberio can not be populated when verify_only is true, because
do_dry_run() build and complete IOs immediately, so it can not replicate
the numberio that was produced when the data was layered on the media.

Without this fix, using write_random
[write_stress]
filename=${FILENAME}
size=${FILESIZE}
verify_only=${VERIFY_ONLY}
readwrite=randwrite
bs=4k
ioengine=libaio
iodepth=32
direct=1
do_verify=1
verify=crc32c

'VERIFY_ONLY=1 FILENAME=/dev/sda1 FILESIZE=1M fio write_random' passes,
but
'VERIFY_ONLY=0 FILENAME=/dev/sda1 FILESIZE=1M fio write_random' fails:
"""verify_only option fails with verify: bad header numberio 1, wanted
0""".
The fix addresses the problem by not checking numberio.

Fixes #732

Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
2 months agoUse aux_path, if set, when loading verify state
Andrey Kuzmin [Thu, 16 Jan 2020 15:15:40 +0000 (08:15 -0700)]
Use aux_path, if set, when loading verify state

Fixes: https://github.com/axboe/fio/issues/908
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMoved diskutil reporting functions to stat.c
LeaflessMelospiza [Wed, 15 Jan 2020 10:13:15 +0000 (15:43 +0530)]
Moved diskutil reporting functions to stat.c

2 months agoengines/io_uring: use fixed opcodes for pre-mapped buffers
Keith Busch [Tue, 14 Jan 2020 21:24:24 +0000 (13:24 -0800)]
engines/io_uring: use fixed opcodes for pre-mapped buffers

Use the correct opcode when for reads and writes using the fixedbuf
option, otherwise EINVAL errors will be returned to these requests.

Fixes: b10b1e70a ("io_uring: add option for non-vectored read/write commands")
Signed-off-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agotime: limit usec_sleep() to maximum intervals of 1 second
Jens Axboe [Mon, 13 Jan 2020 21:51:35 +0000 (14:51 -0700)]
time: limit usec_sleep() to maximum intervals of 1 second

The total sleep may be much longer, but wakeup every ~1 second at least
to check if we got signaled to exit.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/read-to-pipe-async: Complain if pthread_detach() fails
Bart Van Assche [Sat, 11 Jan 2020 23:46:57 +0000 (15:46 -0800)]
t/read-to-pipe-async: Complain if pthread_detach() fails

This patch fixes the following Coverity complaint:

CID 280680 (#1 of 1): Unchecked return value (CHECKED_RETURN)
5. check_return: Calling pthread_detach without checking return value (as is done elsewhere 4 out of 5 times).

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agot/read-to-pipe-async: Do not divide by zero
Bart Van Assche [Sat, 11 Jan 2020 23:42:49 +0000 (15:42 -0800)]
t/read-to-pipe-async: Do not divide by zero

This patch fixes the following Coverity complaint:

23. zero_return: Function call utime_since(&s, &re) returns 0.
CID 280732 (#2 of 2): Division or modulo by zero (DIVIDE_BY_ZERO)
24. divide_by_zero: In expression bytes * 1000UL * 1000UL / utime_since(&s, &re), division by expression utime_since(&s, &re) which may be zero has undefined behavior.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agot/memlock: Verify 'threads' argument
Bart Van Assche [Sat, 11 Jan 2020 23:45:32 +0000 (15:45 -0800)]
t/memlock: Verify 'threads' argument

This patch fixes the following Coverity complaint:

CID 280681 (#1 of 1): Untrusted value as argument (TAINTED_SCALAR)
4. tainted_data: Passing tainted variable threads to a tainted sink.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoserver: Make it explicit that the setsockopt() return value is ignored
Bart Van Assche [Sun, 12 Jan 2020 00:31:14 +0000 (16:31 -0800)]
server: Make it explicit that the setsockopt() return value is ignored

This patch fixes the following Coverity complaint:

CID 169316 (#1 of 1): Unchecked return value from library (CHECKED_RETURN)
5. check_return: Calling setsockopt(sk, 1, 15, &opt, 4U) without checking return value. This library function may fail and return an error code.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoblktrace: Fix memory leaks in error paths
Bart Van Assche [Sun, 12 Jan 2020 00:07:39 +0000 (16:07 -0800)]
blktrace: Fix memory leaks in error paths

This patch fixes the following Coverity complaint:

CID 183977 (#1 of 1): Resource leak (RESOURCE_LEAK)
17. leaked_storage: Variable str going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoblktrace: Check value of 'merge_buf' pointer before using it
Bart Van Assche [Sun, 12 Jan 2020 00:10:42 +0000 (16:10 -0800)]
blktrace: Check value of 'merge_buf' pointer before using it

This patch fixes the following Coverity complaint:

CID 183976 (#1 of 1): Dereference null return value (NULL_RETURNS)
5. dereference: Dereferencing a pointer that might be null 'merge_fp' when calling 'setvbuf'.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoblktrace: Pass a positive error code to td_verror()
Bart Van Assche [Sat, 11 Jan 2020 23:58:26 +0000 (15:58 -0800)]
blktrace: Pass a positive error code to td_verror()

This patch fixes the following Coverity complaint:

CID 183980 (#1-2 of 2): Argument cannot be negative (NEGATIVE_RETURNS)
13. negative_returns: ret is passed to a parameter that cannot be negative.

Fixes: e28875637094 ("blktrace support: speedup reading of data")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoio_uring: set sqe iopriority, if prio/prioclass is set
Jens Axboe [Thu, 9 Jan 2020 21:58:51 +0000 (14:58 -0700)]
io_uring: set sqe iopriority, if prio/prioclass is set

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 7 Jan 2020 01:38:02 +0000 (18:38 -0700)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Windows >= 7: Make fio_getaffinity() error reporting more detailed
  Windows: Uninline CPU affinity functions
  Windows: Use snprintf() instead of StringCch*()
  Include "oslib/asprintf.h" where necessary
  Windows: Remove more unused OS dependent code
  Change off64_t into uint64_t
  Fix the build in case FIO_HAVE_DISK_UTIL is not defined
  Windows: Fix multiple configure tests
  configure: Improve the getopt_long_only() test
  configure: Improve the ibverbs test
  Makefile: Use 'tr' if 'fmt' is not available
  Makefile: Avoid duplicating code

2 months agoWindows >= 7: Make fio_getaffinity() error reporting more detailed
Bart Van Assche [Mon, 6 Jan 2020 04:25:36 +0000 (20:25 -0800)]
Windows >= 7: Make fio_getaffinity() error reporting more detailed

Report different error messages for GetProcessGroupAffinity() failures
and also if a process is associated with multiple process groups.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoWindows: Uninline CPU affinity functions
Bart Van Assche [Mon, 6 Jan 2020 04:17:41 +0000 (20:17 -0800)]
Windows: Uninline CPU affinity functions

None of the CPU affinity functions is in the hot path. Reduce the build
time by uninlining these functions.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoWindows: Use snprintf() instead of StringCch*()
Bart Van Assche [Mon, 6 Jan 2020 01:51:19 +0000 (17:51 -0800)]
Windows: Use snprintf() instead of StringCch*()

Use ANSI C functions instead of Windows-specific functions.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoInclude "oslib/asprintf.h" where necessary
Bart Van Assche [Sun, 5 Jan 2020 21:37:48 +0000 (13:37 -0800)]
Include "oslib/asprintf.h" where necessary

The header file "oslib/asprintf.h" must be included by every source file
that calls asprintf().

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoWindows: Remove more unused OS dependent code
Bart Van Assche [Sun, 5 Jan 2020 19:40:36 +0000 (11:40 -0800)]
Windows: Remove more unused OS dependent code

Since commit 46bfd4e5170e ("Remove old OS dependent (unused) random code")
rand_r() is no longer used. Hence remove its definition.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoChange off64_t into uint64_t
Bart Van Assche [Sun, 5 Jan 2020 19:10:30 +0000 (11:10 -0800)]
Change off64_t into uint64_t

off64_t is Linux specific while uint64_t is defined by a C standard. This
change makes fio easier to port.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>