fio.git
12 hours agoMerge branch 'fix-corrupt-hist-log' of https://github.com/sitsofe/fio master
Jens Axboe [Sun, 13 Oct 2019 17:02:00 +0000 (11:02 -0600)]
Merge branch 'fix-corrupt-hist-log' of https://github.com/sitsofe/fio

* 'fix-corrupt-hist-log' of https://github.com/sitsofe/fio:
  stat: fix corruption in histogram logs

12 hours agostat: fix corruption in histogram logs
Sitsofe Wheeler [Sun, 13 Oct 2019 08:55:51 +0000 (09:55 +0100)]
stat: fix corruption in histogram logs

Commit 6cc0e5aa9eddd2487dfa9ac80be1264151058409 ("Fix overflow of
counters incremented on each I/O operation") changed various structures
from [unsigned] int -> [u]int64_t but the size used when memcpy-ing
histogram data was still the older (smaller) size. This led to un-copied
values and usage of uninitialised data. An example job that demonstrates
the issue is:

$ valgrind ./fio --size=256k --thread=1 --rate_iops=5 --direct=1 \
 --number_ios=15 --name=hist_test --filename=fio.test \
 --log_hist_msec=1000 --write_hist_log=fio.test --log_hist_coarseness=6
==1743== Memcheck, a memory error detector
[...]
fio-3.16-8-gf951
Starting 1 thread
==1743== Thread 3: IOPS: [R(1)][23.1%][r=20KiB/s][r=5 IOPS][eta 00m:10s]
==1743== Use of uninitialised value of size 8
==1743==    at 0x609D86B: _itoa_word (_itoa.c:179)
==1743==    by 0x60A0F0D: vfprintf (vfprintf.c:1642)
==1743==    by 0x60A9E53: fprintf (fprintf.c:32)
==1743==    by 0x47FA6C: flush_hist_samples (iolog.c:869)
==1743==    by 0x47F637: flush_log (iolog.c:1212)
==1743==    by 0x4813C5: finish_log (iolog.c:1238)
==1743==    by 0x4812CD: __write_log (iolog.c:1571)
==1743==    by 0x481202: write_clat_hist_log (iolog.c:1625)
==1743==    by 0x480045: td_writeout_logs (iolog.c:1721)
==1743==    by 0x485C7A: thread_main (backend.c:1889)
==1743==    by 0x5C296DA: start_thread (pthread_create.c:463)
==1743==    by 0x616688E: clone (clone.S:95)

This commit fixes the issue by using the new size in the memcpy.

Fixes: https://github.com/axboe/fio/issues/827
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
5 days agoMerge branch 'bumpflocks' of https://github.com/earlephilhower/fio
Jens Axboe [Wed, 9 Oct 2019 02:58:57 +0000 (20:58 -0600)]
Merge branch 'bumpflocks' of https://github.com/earlephilhower/fio

* 'bumpflocks' of https://github.com/earlephilhower/fio:
  Increase MAX_FILELOCKS for highly parallel IO test

5 days agoIncrease MAX_FILELOCKS for highly parallel IO test
Earle F. Philhower, III [Tue, 8 Oct 2019 23:13:02 +0000 (16:13 -0700)]
Increase MAX_FILELOCKS for highly parallel IO test

When running hundreds of threads and collecting CLAT, SLAT, IOS on a
per-thread basis, FIO can need a lot of file locks to complete.  When
it can't get them, it gives the cryptic error message:
> fio: filelock.c:182: __fio_lock_file: Assertion `!trylock' failed.

Bump the maximum from 128 to 1024 to give headroom for high core count,
high thread count testing.

6 days agoMerge branch 'replay-blktrace-fixes' of https://github.com/shimrot/fio
Jens Axboe [Tue, 8 Oct 2019 03:23:26 +0000 (21:23 -0600)]
Merge branch 'replay-blktrace-fixes' of https://github.com/shimrot/fio

* 'replay-blktrace-fixes' of https://github.com/shimrot/fio:
  Fix assert error on blktrace replay containing trims

7 days agoMerge branch 'error-on-implicit-decl' of https://github.com/sitsofe/fio
Jens Axboe [Sun, 6 Oct 2019 15:15:51 +0000 (09:15 -0600)]
Merge branch 'error-on-implicit-decl' of https://github.com/sitsofe/fio

* 'error-on-implicit-decl' of https://github.com/sitsofe/fio:
  configure: stop enabling fdatasync on OSX

7 days agoconfigure: stop enabling fdatasync on OSX
Sitsofe Wheeler [Sat, 5 Oct 2019 11:04:33 +0000 (12:04 +0100)]
configure: stop enabling fdatasync on OSX

Change configure compile probes to fail on implicit declarations. This
(correctly) stops us from enabling fdatasync on OSX which was
problematic because we were actually calling a syscall stub with
arguments different to what was declared in its prototype (see
https://github.com/gbrault/picoc/issues/145#issuecomment-89734655 and
https://gitlab.freedesktop.org/xdg/shared-mime-info/issues/7 ).

Fixes: https://github.com/axboe/fio/issues/834 ("Drop macOS support for
fdatasync")

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
11 days agoWindows: Update URLs to https, and remove mention of WiX version
Rebecca Cran [Mon, 30 Sep 2019 20:57:08 +0000 (14:57 -0600)]
Windows: Update URLs to https, and remove mention of WiX version

Signed-off-by: Jens Axboe <axboe@kernel.dk>
12 days agoUpdate the email and web address for Windows binaries.
Rebecca Cran [Mon, 30 Sep 2019 20:57:07 +0000 (14:57 -0600)]
Update the email and web address for Windows binaries.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoconfigure: add --enable-libaio-uring parameter
Jens Axboe [Fri, 27 Sep 2019 19:51:12 +0000 (13:51 -0600)]
configure: add --enable-libaio-uring parameter

If set, this attempts to link with liburing instead of libaio, and
use the libaio wrapper there.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agot/zbd: Avoid magic number of test case count
Shin'ichiro Kawasaki [Thu, 26 Sep 2019 01:00:44 +0000 (10:00 +0900)]
t/zbd: Avoid magic number of test case count

In the test script t/zbd/test-zbd-support, the number of test cases is
defined as a raw constant. Every time a new test case is added, need to
update the constant but this chore tends to be overlooked. When test
case 47 was added recently, the number was not incremented from 46. Then
the test script does not run the test case 47.

To avoid the constant increment chore, automate the test case number
counting. List bash functions which implement test cases and convert the
list to test case numbers.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agofilesetup: Extend file size for 'null' and 'filecreate' ioengines
Shin'ichiro Kawasaki [Wed, 25 Sep 2019 08:29:24 +0000 (17:29 +0900)]
filesetup: Extend file size for 'null' and 'filecreate' ioengines

It was reported the fio command with following options fails because of
file access beyond the file size specified with size option.

./fio --name=test --rw=randread --runtime=10s --offset=90% --time_based --ioengine=null --size=1T --norandommap --randrepeat=0

One of the options specifies null ioengine which has FIO_DISKLESSIO and
FIO_FAKEIO flags. When FIO_DISKLESSIO flag is set, current fio code does
not extend the file size even when io_size + file_offset is larger than
real_file_size. With this, random offsets are generated within io_size +
file_offset range and fail comparison with real_file_size. This failure
does not happen with other engines such as libaio since the target file
is extended to ensure real_file_size is larger than io_size +
file_offset.

To avoid the failure, extend file size for ioengines with FIO_DISKLESSIO
and FIO_FAKEIO. This changes behavior of two ioengines only: null
ioengine and filecreate engine. It does not change behavior of all other
ioengines including 12 ioengines with FIO_DISKLESSIO flag, such as nbd,
glusterfs, pmemblk, etc.

Signed-off-by: Masato Suzuki <masato.suzuki@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoRevert "filesetup: honor the offset option"
Shin'ichiro Kawasaki [Wed, 25 Sep 2019 08:29:23 +0000 (17:29 +0900)]
Revert "filesetup: honor the offset option"

This reverts commit ad46c5715ffc5d4e70d2e941225b1f504f23e409.

The commit tried to fix a failure observed with --offset=90%, --size=1T
and --ioengine=null options. However, it affected fio behavior not only
for null ioengine but also for other all ioengines. It changed io_size,
then I/O read or written bytes are not same as that specified with
command line 'size' option. This caused t/zbd/test-zbd-support test
script failures for zoned block devices: test cases #17, #28 and #38
failed.

To prepare for other fix with limited ioengine impacts, revert the fix
which affects all ioengines.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoverify: check that the block size is big enough
Jens Axboe [Tue, 24 Sep 2019 08:43:39 +0000 (02:43 -0600)]
verify: check that the block size is big enough

Fixes: https://github.com/axboe/fio/issues/822
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoMerge branch 'master' of https://github.com/anatol/fio
Jens Axboe [Mon, 23 Sep 2019 22:09:42 +0000 (16:09 -0600)]
Merge branch 'master' of https://github.com/anatol/fio

* 'master' of https://github.com/anatol/fio:
  Fix compilation error with gfio

2 weeks agoFix compilation error with gfio
Anatol Pomozov [Mon, 23 Sep 2019 21:12:12 +0000 (14:12 -0700)]
Fix compilation error with gfio

Commit 36833fb04 replaced many usages of strncpy with snprintf.
But there is one place where new arguments were added but the function
name has not been changed. It leads to the following compilation error:

gclient.c:333:2: error: too many arguments to function ‘strncpy’
  333 |  strncpy(message, sizeof(message), "%s", status_message);
      |  ^~~~~~~
In file included from /usr/include/features.h:450,
                 from /usr/include/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:25,
                 from gclient.c:1:
/usr/include/bits/string_fortified.h:103:1: note: declared here
  103 | __NTH (strncpy (char *__restrict __dest, const char *__restrict __src,
      | ^~~~~

Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
3 weeks agoFio 3.16 fio-3.16
Jens Axboe [Fri, 20 Sep 2019 01:01:52 +0000 (19:01 -0600)]
Fio 3.16

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 weeks agoengines/io_uring: remove debug printf
Jens Axboe [Thu, 19 Sep 2019 14:35:29 +0000 (08:35 -0600)]
engines/io_uring: remove debug printf

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 weeks agoMerge branch 'fio_reset_sqe' of https://github.com/anarazel/fio
Jens Axboe [Fri, 13 Sep 2019 20:56:16 +0000 (14:56 -0600)]
Merge branch 'fio_reset_sqe' of https://github.com/anarazel/fio

* 'fio_reset_sqe' of https://github.com/anarazel/fio:
  engines/io_uring: Fully clear out previous SQE contents.

4 weeks agoengines/io_uring: Fully clear out previous SQE contents.
Andres Freund [Fri, 13 Sep 2019 19:10:10 +0000 (12:10 -0700)]
engines/io_uring: Fully clear out previous SQE contents.

Without this change SQEs can contain data set in previous
submissions. E.g. a WRITE following an fdatasync would have still have
IORING_FSYNC_DATASYNC set in sync_flags, which shares storage with the
WRITE's rw_flags. Which was not reset, causing all writes to be
synchronous. Similarly, an fsync following a READ/WRITE would not
reset off/addr/len, which causes errors, because the kernel's
io_prep_fsync returns an error if e.g. addr is not 0.

While this could also be fixed by resetting only the unused fields in
the respective branches, it seems less failure prone to start with a
zeroed out sqe.

4 weeks agoMerge branch 'ioring_add_sync_file_range' of https://github.com/anarazel/fio
Jens Axboe [Thu, 12 Sep 2019 20:24:23 +0000 (14:24 -0600)]
Merge branch 'ioring_add_sync_file_range' of https://github.com/anarazel/fio

* 'ioring_add_sync_file_range' of https://github.com/anarazel/fio:
  engines/io_uring: Add support for sync_file_range.

4 weeks agoMerge branch 'fix_iouring_eintr' of https://github.com/anarazel/fio
Jens Axboe [Thu, 12 Sep 2019 20:23:21 +0000 (14:23 -0600)]
Merge branch 'fix_iouring_eintr' of https://github.com/anarazel/fio

* 'fix_iouring_eintr' of https://github.com/anarazel/fio:
  engines/io_uring: Handle EINTR.

4 weeks agoengines/io_uring: Add support for sync_file_range.
Andres Freund [Thu, 12 Sep 2019 18:17:08 +0000 (11:17 -0700)]
engines/io_uring: Add support for sync_file_range.

Previously sync_file_range() requests were just dropped to the floor.

Signed-off-by: Andres Freund <andres@anarazel.de>
4 weeks agoengines/io_uring: Handle EINTR.
Andres Freund [Thu, 12 Sep 2019 17:51:09 +0000 (10:51 -0700)]
engines/io_uring: Handle EINTR.

Several paths in io_uring_enter can trigger EINTR, but it was not
handled, leading to fio failing with spurious error messages.

An easy way to trigger EINTR is to just strace a running fio using
the io_uring engine and detach again.

Signed-off-by: Andres Freund <andres@anarazel.de>
4 weeks agoengines/io_uring: fix crash with registerfiles=1
Jens Axboe [Thu, 12 Sep 2019 16:15:42 +0000 (10:15 -0600)]
engines/io_uring: fix crash with registerfiles=1

If used with a raw bdev, we're crashing in attempting to open a
registered file before we have actually registered them.

If we're called before files are registered, just open the file
normally. This is done to query sizes etc, and we'll get the file
closed after that anyway. The job open/close will use the right
registered fd.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 weeks agofilesetup: honor the offset option
Vincent Fu [Wed, 11 Sep 2019 14:56:40 +0000 (10:56 -0400)]
filesetup: honor the offset option

Commands like the following do not honor the value given by the offset
option:

./fio --name=test --rw=randread --runtime=10s --offset=90% --time_based --ioengine=null --size=1T --norandommap --randrepeat=0
./fio --name=test --size=8k --offset=4k

In the random case, eventually a random offset will be generated beyond
the 1T file size, leading to a failure.

In the sequential case, a 12k file will be created despite size
specifying the 8k end boundary.

This patch modifies setup_files() so that f->io_size incorporates the
offset for cases like those above.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 weeks agodoc: clarify what --alloc-size does
Vincent Fu [Wed, 11 Sep 2019 14:56:39 +0000 (10:56 -0400)]
doc: clarify what --alloc-size does

The alloc-size option actually directs fio to allocate additional shared
memory pools of the specified size, augmenting the default allocation.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoengines/io_uring: use its own option group
Jens Axboe [Thu, 5 Sep 2019 15:15:41 +0000 (09:15 -0600)]
engines/io_uring: use its own option group

Was under libaio for now, lets move it to a distinct option group name.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoengines/io_uring: add support for registered files
Jens Axboe [Thu, 5 Sep 2019 14:59:29 +0000 (08:59 -0600)]
engines/io_uring: add support for registered files

This feature is exposed as a separate option, like fixedbufs, and
provides a way for fio to register a set of files with the kernel.
This improves IO efficiency.

It is also a requirement to be able to use sqthread_poll, as that
feature requires fixed files on the kernel side.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agosmalloc: use SMALLOC_BPI instead of SMALLOC_BPB in add_pool()
Vincent Fu [Tue, 3 Sep 2019 17:44:45 +0000 (13:44 -0400)]
smalloc: use SMALLOC_BPI instead of SMALLOC_BPB in add_pool()

Change the calculation of free_blocks in add_pool() to use SMALLOC_BPI
instead of SMALLOC_BPB. These two constants are coincidentally the same
on Linux and Windows but SMALLOC_BPI is the correct one to use.
free_blocks is the number of available blocks of size SMALLOC_BPB. It is
the product of the number of unsigned integers in the bitmap
(bitmap_blocks) and the number of bits per unsigned integer
(SMALLOC_BPI).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agosmalloc: allocate struct pool array from shared memory
Vincent Fu [Tue, 3 Sep 2019 17:44:44 +0000 (13:44 -0400)]
smalloc: allocate struct pool array from shared memory

If one process is making smalloc calls and another process is making
sfree calls, pool->free_blocks and pool->next_non_full will not be
synchronized because the two processes each have independent, local
copies of the variables.

This patch allocates space for the array of struct pool instances from
shared storage so that separate processes will be modifying quantities
stored at the same locations.

This issue was discovered on the server side running a client/server job
with --status-interval=1. Such a job encountered an OOM error when only
~50 objects were allocated from the smalloc pool.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agozbd: Improve job zonesize initialization checks
Damien Le Moal [Fri, 30 Aug 2019 02:16:05 +0000 (11:16 +0900)]
zbd: Improve job zonesize initialization checks

For a multijob workload, each job may specify a zonesize option for
access to a zoned block device or regular device with zonemode=zbd.
In such case, make sure that the zone size value specified by each job
matches the device zone size.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agozbd: Fix job zone size initialization
Damien Le Moal [Fri, 30 Aug 2019 01:45:03 +0000 (10:45 +0900)]
zbd: Fix job zone size initialization

For a job accessing a zoned block device, the zone size is automatically
initialized to the device zone size. However, since zone information for
a zoned block device is parsed once only for the first job
initialization of a multi job workload, only the first job has its
zonesize option intialized, causing problem if the zoneskip option is
also used (assert exit).

Fix this by always initializing a job zonesize option using the job
file zbd information when verifying the job ZBD related sizes and
offsets.

Fixes: 4d37720ae029 ("zbd: Add support for zoneskip option")
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agozbd: provide empty setup_zbd_zone_mode()
Jens Axboe [Thu, 29 Aug 2019 12:38:08 +0000 (06:38 -0600)]
zbd: provide empty setup_zbd_zone_mode()

If we don't enable zbd, then provide an empty stub for the setup.
This fixes a build breakage on anything but Linux.

Fixes: 4d37720ae029 ("zbd: Add support for zoneskip option")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agozbd: Add support for zoneskip option
Damien Le Moal [Thu, 29 Aug 2019 07:21:35 +0000 (16:21 +0900)]
zbd: Add support for zoneskip option

To speed up device tests (performance and or quality validation) of very
large capacity block devices such as SMR disks, it is useful to allow
skipping some block ranges for sequential workloads. While the
zonemode=stridded implements such feature, it does not allow controlling
read operations in partially written zones of zoned block devices (i.e.
prevent reads after a zone write pointer) and can result in IO errors
if executed on a zoned block devices with zones already written.

To solve this problem, add support for the zoneskip option with
zonemode=zbd, allowing a sequential workload to skip zoneskip bytes once
a zone has been fully written or its data has been read. The zoneskip
option is ignored for random workloads.

For read workloads, zone skipping takes into account the read_beyond_wp
option to switch zone either when all valid data in the zone is read
(read_beyond_wp=0) or the entire zone has been read (read_beyond_wp=1).

Add test47 to t/zbd/test-zbd-support to test that zoneskip invalid
values are handled correctly.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoman: Improve zonemode=zbd information
Damien Le Moal [Thu, 29 Aug 2019 07:21:34 +0000 (16:21 +0900)]
man: Improve zonemode=zbd information

Clarify the use of the zonerange and zonesize options for zonemode=zbd.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoman page: Fix read_beyond_wp description
Damien Le Moal [Thu, 29 Aug 2019 07:21:33 +0000 (16:21 +0900)]
man page: Fix read_beyond_wp description

Strictly speaking, drive managed disks are not zoned block devices as
they do not provide zone information nor zone commands. So remove
mention of this type of disk in the zoned block device description.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agozbd: Fix error message
Damien Le Moal [Thu, 29 Aug 2019 07:21:32 +0000 (16:21 +0900)]
zbd: Fix error message

Use log_err() instead of log_info() for notifying invalid zonesize
values specified by the user.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agozbd: Fix initialization error message
Damien Le Moal [Thu, 29 Aug 2019 07:21:31 +0000 (16:21 +0900)]
zbd: Fix initialization error message

Fix the error message on zbd_create_zone_info() failures to a more
generic message rather than indicating a BLKREPORTZONE ioctl error.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agozbd: Cleanup zbd_init()
Damien Le Moal [Thu, 29 Aug 2019 07:21:30 +0000 (16:21 +0900)]
zbd: Cleanup zbd_init()

For a job using a zoned device, the zonesize option must always specify
the device zone size. That is checked in the function parse_zone_info().
The zonesize checks in zbd_init() apply only to jobs running with
zonemode=zbd on a regular block device. So move these checks into
init_zone_info() which is used to emulate zone information for regular
block devices.

Fix t/zbd test #43 accordingly.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agooptions: allow offset_increment to understand percentages
Vincent Fu [Wed, 28 Aug 2019 17:48:41 +0000 (13:48 -0400)]
options: allow offset_increment to understand percentages

Add the ability for the offset_increment option to understand
percentages. With this patch offset_increment=10% will, for example,
increase the start offset by 10% of the file size for additional jobs
created by numjobs.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agodocs: small HOWTO fixes
Vincent Fu [Wed, 28 Aug 2019 17:48:40 +0000 (13:48 -0400)]
docs: small HOWTO fixes

Document three debug options and fix a formatting error.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agonbd: Update for libnbd 0.9.8
Richard W.M. Jones [Thu, 15 Aug 2019 15:30:40 +0000 (16:30 +0100)]
nbd: Update for libnbd 0.9.8

As the libnbd API isn't permanently stable until we reach the 1.0
release (expected soon), some code changes are needed to cope with API
changes between 0.9.6 and 0.9.8.  In this case we made changes to
completion handlers after feedback from reviewers.  This fix for fio
incorporates all the changes needed and bumps the minimum version to
libnbd >= 0.9.8.

Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agostat: ensure that struct jobs_eta packs nicely
Jens Axboe [Thu, 15 Aug 2019 15:16:19 +0000 (09:16 -0600)]
stat: ensure that struct jobs_eta packs nicely

It has two holes in it, and some weird mid-struct packing. Let's
clean it up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agoeta: use struct jobs_eta_packed
Jens Axboe [Wed, 14 Aug 2019 21:16:09 +0000 (15:16 -0600)]
eta: use struct jobs_eta_packed

I think this is why the build fails for some, which is odd.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agoMakefile: Add 'fulltest' target
Bart Van Assche [Wed, 14 Aug 2019 20:10:12 +0000 (13:10 -0700)]
Makefile: Add 'fulltest' target

Make it easier to run the zoned block device tests.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agoRestore type checking in calc_thread_status()
Bart Van Assche [Wed, 14 Aug 2019 20:10:11 +0000 (13:10 -0700)]
Restore type checking in calc_thread_status()

Due to a previous patch it is no longer necessary to hide the type of
accesses to the 'rate' and 'iops' members in struct jobs_eta.

This patch reverts commit df0ca15ce2ff ("eta: Fix compiler warning").

Cc: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agoVerify the absence of holes in struct jobs_eta at compile time
Bart Van Assche [Wed, 14 Aug 2019 20:10:10 +0000 (13:10 -0700)]
Verify the absence of holes in struct jobs_eta at compile time

This patch verifies the correctness of the previous patch.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agoRefine packed annotations in stat.h
Bart Van Assche [Wed, 14 Aug 2019 20:10:09 +0000 (13:10 -0700)]
Refine packed annotations in stat.h

Instead of declaring the whole structure packed, only declare non-aligned
members packed. This patch is an alternative way to fix the following gcc 9
compiler warnings:

eta.c: In function 'calc_thread_status':
eta.c:510:7: error: taking address of packed member of 'struct jobs_eta' may result in an unaligned pointer value [-Werror=address-of-packed-member]
  510 |     je->rate);
      |     ~~^~~~~~
eta.c:522:66: error: taking address of packed member of 'struct jobs_eta' may result in an unaligned pointer value [-Werror=address-of-packed-member]
  522 |  calc_rate(unified_rw_rep, disp_time, io_bytes, disp_io_bytes, je->rate);
      |                                                                ~~^~~~~~
eta.c:523:64: error: taking address of packed member of 'struct jobs_eta' may result in an unaligned pointer value [-Werror=address-of-packed-member]
  523 |  calc_iops(unified_rw_rep, disp_time, io_iops, disp_io_iops, je->iops);
      |

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agoOptimize the code that copies strings
Bart Van Assche [Wed, 14 Aug 2019 20:10:08 +0000 (13:10 -0700)]
Optimize the code that copies strings

Using strncpy() to copy strings is suboptimal because strncpy writes a
bunch of additional unnecessary null bytes. Use snprintf() instead of
strncpy(). An additional advantage of snprintf() is that it guarantees
that the output string is '\0'-terminated.

This patch is an improvement for commit 32e31c8c5f7b ("Fix string copy
compilation warnings").

Cc: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agozbd: Improve robustness of unit tests
Bart Van Assche [Wed, 14 Aug 2019 20:10:07 +0000 (13:10 -0700)]
zbd: Improve robustness of unit tests

Give up if creation of the null_blk instance fails.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 weeks agozbd: Declare local functions 'static'
Bart Van Assche [Wed, 14 Aug 2019 20:10:06 +0000 (13:10 -0700)]
zbd: Declare local functions 'static'

This patch fixes two sparse warnings.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/zbd: Fix I/O bytes rounding errors
Shin'ichiro Kawasaki [Fri, 9 Aug 2019 03:28:36 +0000 (12:28 +0900)]
t/zbd: Fix I/O bytes rounding errors

When fio reports write bytes or read bytes, it rounds the number with
units MiB or KiB to fit the number within limited number of digits.
This results in rounding errors of the reported bytes and sometimes
causes test failures for test case #17 in test-zbd-support
which reports incorrect total I/O bytes in case both of write bytes
and read bytes are rounded up.

To avoid the rounding error, increase the number of digits from default
value 4 to 10 to keep precision. For example, a number "256MiB" will be
reported as "267911168B" with this change.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoAdd tests from t/ to the Windows installer
Rebecca Cran [Mon, 5 Aug 2019 20:11:00 +0000 (13:11 -0700)]
Add tests from t/ to the Windows installer

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agonbd: Remove copy and paste error in example
Richard W.M. Jones [Sat, 3 Aug 2019 15:51:06 +0000 (16:51 +0100)]
nbd: Remove copy and paste error in example

Fixes a copy and paste error introduced in commit d643a1e29d
("engines: Add Network Block Device (NBD) support using libnbd.").

Thanks: Sitsofe Wheeler

Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoengines/splice: remove buggy ->mem_align set
Jens Axboe [Sat, 3 Aug 2019 14:33:24 +0000 (08:33 -0600)]
engines/splice: remove buggy ->mem_align set

Two things wrong here:

1) We align buffers by default, so no need for splice to do anything
   extra.
2) ->mem_align is not a true/false setting, it's the alignment itself.
   Hence the current setting to 1 is just buggy.

Fixes: https://github.com/axboe/fio/issues/810
Reported-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agonbd: Document the NBD-specific uri parameter
Richard W.M. Jones [Fri, 2 Aug 2019 17:03:52 +0000 (18:03 +0100)]
nbd: Document the NBD-specific uri parameter

Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoparse: bump max value pairs supported from 24 to 32
Jens Axboe [Fri, 2 Aug 2019 16:53:22 +0000 (10:53 -0600)]
parse: bump max value pairs supported from 24 to 32

The recent addition of the nbd engine overflowed what we support.

Fixes: d643a1e29d31 ("engines: Add Network Block Device (NBD) support using libnbd")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoengines: Add Network Block Device (NBD) support using libnbd.
Richard W.M. Jones [Fri, 2 Aug 2019 16:11:25 +0000 (17:11 +0100)]
engines: Add Network Block Device (NBD) support using libnbd.

This commit adds a new engine for testing Network Block Devices
directly.  It requires libnbd (https://github.com/libguestfs/libnbd).

To see how to test nbdkit or qemu-nbd read examples/nbd.fio.

Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agosmalloc: cleanup firstfree()
Jens Axboe [Wed, 31 Jul 2019 21:27:27 +0000 (15:27 -0600)]
smalloc: cleanup firstfree()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agosmalloc: fix compiler warning on Windows
Vincent Fu [Wed, 31 Jul 2019 20:57:51 +0000 (16:57 -0400)]
smalloc: fix compiler warning on Windows

firstfree() triggers a warning from the Windows compiler used by
AppVeyor because it doesn't return a value if the for loop iterates to
completion. This patch resolves the compiler warning.

AppVeyor Windows build log: https://ci.appveyor.com/project/axboe/fio/builds/26381726

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoRemove unused fio_assert()
Jens Axboe [Wed, 31 Jul 2019 21:15:50 +0000 (15:15 -0600)]
Remove unused fio_assert()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'smalloc-gc' of https://github.com/vincentkfu/fio
Jens Axboe [Wed, 31 Jul 2019 19:58:44 +0000 (13:58 -0600)]
Merge branch 'smalloc-gc' of https://github.com/vincentkfu/fio

* 'smalloc-gc' of https://github.com/vincentkfu/fio:
  smalloc: fix garbage collection problem
  t/stest: make the test more challenging
  smalloc: print debug info on oom error

2 months agosmalloc: fix garbage collection problem
Vincent Fu [Tue, 30 Jul 2019 15:00:56 +0000 (09:00 -0600)]
smalloc: fix garbage collection problem

If a large request arrives when pool->next_non_full points to empty
space that is insufficient to satisfy the request, pool->next_non_full
will be inappropriately advanced when the free space is followed by
lines of fully allocated space. The free space originally pointed to by
pool->next_non_full will be unavailable unless a subsequent sfree() call
frees allocated space above it. Resolve this issue by advancing
pool->next_non_full only outside the search loop and only when it points
to fully allocated space.

2 months agot/stest: make the test more challenging
Vincent Fu [Fri, 26 Jul 2019 17:33:52 +0000 (11:33 -0600)]
t/stest: make the test more challenging

Add large smalloc requests to the sfree phase of the test. This exposes
a smalloc garbage collection issue.

2 months agosmalloc: print debug info on oom error
Vincent Fu [Tue, 30 Jul 2019 16:07:26 +0000 (12:07 -0400)]
smalloc: print debug info on oom error

Provide more details about the request and the state of the memory pools
when smalloc encounters an oom situation.

2 months agoMerge branch 'dev' of https://github.com/smartxworks/fio
Jens Axboe [Wed, 31 Jul 2019 16:55:51 +0000 (10:55 -0600)]
Merge branch 'dev' of https://github.com/smartxworks/fio

* 'dev' of https://github.com/smartxworks/fio:
  libiscsi: continue working when meets EINTR or EAGAIN

2 months agoMerge branch 'gnuplot-tabs' of https://github.com/kapsh/fio
Jens Axboe [Wed, 31 Jul 2019 16:54:43 +0000 (10:54 -0600)]
Merge branch 'gnuplot-tabs' of https://github.com/kapsh/fio

* 'gnuplot-tabs' of https://github.com/kapsh/fio:
  fio2gnuplot: fix TabErrors when running with Python 3

2 months agobssplit: Fix handling of 0 percentage
Damien Le Moal [Wed, 31 Jul 2019 04:32:07 +0000 (13:32 +0900)]
bssplit: Fix handling of 0 percentage

If a block size percentage is ispecified as 0 in bssplit, the block size
defined is not ignored by the loop in get_next_buflen(). In particular,
if the first (smallest) block size specified has a 0 percentage, the
loop is existed and that block size used as the next IO size, resulting
in a behavior equivalent to specifying 100%. E.g. using
--bssplit=64k/0,1024k/100 results in 100% of issued IOs to be 64KB
instead of 1MB.

Fix this by ignoring bssplit entries that have a 0 percentage. This is
safe as the initialization of the bssplit array ensure that the sum of
all percentages is always 100, guaranteeing that a block size will be
chosen for the next IO size.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agofio2gnuplot: fix TabErrors when running with Python 3
Alexander Kapshuna [Sat, 27 Jul 2019 15:45:32 +0000 (18:45 +0300)]
fio2gnuplot: fix TabErrors when running with Python 3

Closes #804

Signed-off-by: Alexander Kapshuna <kapsh@kap.sh>
2 months agoDocument io_uring feature
Jens Axboe [Fri, 26 Jul 2019 15:53:43 +0000 (09:53 -0600)]
Document io_uring feature

This was released with Linux 5.1, so we can document it now as the
API won't change.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoengines/libaio: remove remnants of abandoned aio features
Jens Axboe [Fri, 26 Jul 2019 15:41:06 +0000 (09:41 -0600)]
engines/libaio: remove remnants of abandoned aio features

Before we had io_uring, aio grew support for polled IO and setup
through a new system call. As this never made it into Linux and
was abandoned in favor of io_uring, kill it off.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agolibiscsi: continue working when meets EINTR or EAGAIN
Kyle Zhang [Fri, 19 Jul 2019 07:50:01 +0000 (15:50 +0800)]
libiscsi: continue working when meets EINTR or EAGAIN

When poll meets EINTR or EAGAIN, it should continue working rather
than return error.

Signed-off-by: Kyle Zhang <kyle@smartx.com>
3 months agoMerge branch 'dev' of https://github.com/smartxworks/fio
Jens Axboe [Fri, 12 Jul 2019 16:44:45 +0000 (10:44 -0600)]
Merge branch 'dev' of https://github.com/smartxworks/fio

* 'dev' of https://github.com/smartxworks/fio:
  libiscsi: log reason of error when readcapacity failed

3 months agoFio 3.15 fio-3.15
Jens Axboe [Fri, 12 Jul 2019 16:40:36 +0000 (10:40 -0600)]
Fio 3.15

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agolibiscsi: log reason of error when readcapacity failed
Kyle Zhang [Fri, 12 Jul 2019 14:39:56 +0000 (22:39 +0800)]
libiscsi: log reason of error when readcapacity failed

Signed-off-by: Kyle Zhang <kyle@smartx.com>
3 months agoengines/http: set FIO_SYNCIO flag
Vincent Fu [Tue, 9 Jul 2019 14:54:40 +0000 (10:54 -0400)]
engines/http: set FIO_SYNCIO flag

This ioengine carries out synchronous operations. So the FIO_SYNCIO flag
needs to be set in order for latencies to be calculated correctly.

Fixes: https://github.com/axboe/fio/issues/797
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agofio: fix aio trim completion latencies
Vincent Fu [Mon, 8 Jul 2019 19:23:54 +0000 (15:23 -0400)]
fio: fix aio trim completion latencies

The io_uring, libaio, and posixaio ioengines actually carry out
synchronous trim operations, but latency timestamps are recorded as if
the trims were issued asynchronously. This patch fixes how timestamps
are recorded for trim operations issued by these ioengines.

Fixes: https://github.com/axboe/fio/issues/764
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agoMerge branch 'optlenmax' of https://github.com/powernap/fio
Jens Axboe [Mon, 1 Jul 2019 20:26:29 +0000 (14:26 -0600)]
Merge branch 'optlenmax' of https://github.com/powernap/fio

* 'optlenmax' of https://github.com/powernap/fio:
  Increase maximum length of line in jobs file to 8192

3 months agoIncrease maximum length of line in jobs file to 8192
Nick Principe [Mon, 1 Jul 2019 19:45:21 +0000 (15:45 -0400)]
Increase maximum length of line in jobs file to 8192

Increase OPT_LEN_MAX to 8192, use OPT_LEN_MAX for string in
__parse_jobs_ini(), and document line length limit for job files in man
page.

Signed-off-by: Nick Principe nick@princi.pe
3 months agoengines/rbd: hide rbd_io_u_seen() if not used
Jens Axboe [Mon, 1 Jul 2019 20:09:23 +0000 (14:09 -0600)]
engines/rbd: hide rbd_io_u_seen() if not used

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoeta: Fix compiler warning
Damien Le Moal [Tue, 4 Jun 2019 07:07:32 +0000 (16:07 +0900)]
eta: Fix compiler warning

gcc 9 complains about directly referencing pointer values in the packed
structure jobs_eta. E.g.:

warning: taking address of packed member of ‘struct jobs_eta’ may
result in an unaligned pointer value [-Waddress-of-packed-member]

Remove this by using a local void pointer for the rate and iops array
references that generate the warning.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoFix string copy compilation warnings
Damien Le Moal [Tue, 4 Jun 2019 07:07:31 +0000 (16:07 +0900)]
Fix string copy compilation warnings

Fix the many warnings that gcc 9 spits out.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoMerge branch 'wip-tcmalloc' of https://github.com/dillaman/fio
Jens Axboe [Fri, 31 May 2019 21:37:18 +0000 (15:37 -0600)]
Merge branch 'wip-tcmalloc' of https://github.com/dillaman/fio

* 'wip-tcmalloc' of https://github.com/dillaman/fio:
  configure: attempt to link against tcmalloc by default if available

4 months agoconfigure: attempt to link against tcmalloc by default if available
Jason Dillaman [Fri, 31 May 2019 20:50:33 +0000 (16:50 -0400)]
configure: attempt to link against tcmalloc by default if available

librbd will run up to 20% faster under small IO workloads when fio is linked
against tcmalloc. By automatically linking against tcmalloc, it avoids
the need to use LD_PRELOAD to pull in a faster memory management toolset.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
4 months agoFix assert error on blktrace replay containing trims
krisd [Fri, 3 May 2019 13:33:50 +0000 (08:33 -0500)]
Fix assert error on blktrace replay containing trims

4 months agoconfigure: check for gtk version using pkg-config
Helmut Grohne [Mon, 27 May 2019 19:32:57 +0000 (21:32 +0200)]
configure: check for gtk version using pkg-config

The previous way of running a program was incompatible with cross
compilation. The program is still being compiled as a sanity check, but
no longer run.

Signed-off-by: Helmut Grohne <helmut@subdivi.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoconfigure: apply ${cross_prefix} to pkg-config calls
Helmut Grohne [Mon, 27 May 2019 19:32:49 +0000 (21:32 +0200)]
configure: apply ${cross_prefix} to pkg-config calls

Otherwise, we're searching for build architecture libraries which is not
what we want.

Signed-off-by: Helmut Grohne <helmut@subdivi.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoglusterfs: update for new API
Jens Axboe [Thu, 23 May 2019 20:57:03 +0000 (14:57 -0600)]
glusterfs: update for new API

Apparently glusterfs just changes their API as they see fit. Add
a configure check for the newer version, which adds pre/post stat
variables in a few random spots.

This should fix compilation with v6.0 of the API.

Fixes: https://github.com/axboe/fio/issues/781
Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoconfigure: add gettid() test
Jens Axboe [Wed, 22 May 2019 23:12:55 +0000 (17:12 -0600)]
configure: add gettid() test

Apparently some newer glibcs now have it, decades after the
fact.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoFio 3.14 fio-3.14
Jens Axboe [Wed, 22 May 2019 19:41:41 +0000 (13:41 -0600)]
Fio 3.14

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoio_uring: sync with liburing/kernel
Jens Axboe [Mon, 20 May 2019 14:49:49 +0000 (08:49 -0600)]
io_uring: sync with liburing/kernel

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agostat: remove terse v2 blank lines with description not set
Vincent Fu [Wed, 15 May 2019 20:05:03 +0000 (16:05 -0400)]
stat: remove terse v2 blank lines with description not set

The documentation says that the job description will appear on a second
line if this is set. If it is not set there will be an empty line.
Eliminate the empty line when the description is not set.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agodocs: improve terse output format documentation
Vincent Fu [Wed, 15 May 2019 20:05:02 +0000 (16:05 -0400)]
docs: improve terse output format documentation

Fix up some details and note client/server mode differences.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoclient: add a newline after terse disk util
Vincent Fu [Wed, 15 May 2019 20:05:01 +0000 (16:05 -0400)]
client: add a newline after terse disk util

In client/server mode terse disk utilization output appears at the end
of each blob of terse output. We need a newline so that the next blob of
terse output starts at the beginning of a line.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoclient: handle disk util for all output formats
Vincent Fu [Wed, 15 May 2019 20:05:00 +0000 (16:05 -0400)]
client: handle disk util for all output formats

Since fio allows multiple output formats, it should process the disk
utilization data for each output format that was requested by the user
when it is run in client/server mode.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 months agoclient: do not print disk utilization for terse v2
Vincent Fu [Wed, 15 May 2019 20:04:59 +0000 (16:04 -0400)]
client: do not print disk utilization for terse v2

terse version 2 omits disk utilization data when fio is run locally.
Make this behavior the same when fio is run in client/server mode.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agot/io_uring: improve EOPNOTSUPP message
Jens Axboe [Thu, 9 May 2019 15:56:29 +0000 (09:56 -0600)]
t/io_uring: improve EOPNOTSUPP message

Put in one line and include kernel as well.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agot/io_uring: clarify polled support is fs + device
Jens Axboe [Wed, 8 May 2019 17:56:05 +0000 (11:56 -0600)]
t/io_uring: clarify polled support is fs + device

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 months agoUpdate CFLAGS and LDFLAGS for FreeBSD builds
Rebecca Cran [Mon, 6 May 2019 22:12:41 +0000 (16:12 -0600)]
Update CFLAGS and LDFLAGS for FreeBSD builds

FreeBSD installs third-party headers into /usr/local/include
and libraries into /usr/local/lib.
Update CFLAGS and LDFLAGS to look there.

Signed-off-by: Jens Axboe <axboe@kernel.dk>