fio.git
21 months agoFio 3.5 fio-3.5
Jens Axboe [Tue, 20 Feb 2018 22:30:28 +0000 (15:30 -0700)]
Fio 3.5

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoMerge branch 'wip-ifed-rados' of https://github.com/ifed01/fio
Jens Axboe [Wed, 14 Feb 2018 19:43:04 +0000 (12:43 -0700)]
Merge branch 'wip-ifed-rados' of https://github.com/ifed01/fio

* 'wip-ifed-rados' of https://github.com/ifed01/fio:
  Add support for Ceph Rados benchmarking.

21 months agoAdd support for Ceph Rados benchmarking.
Igor Fedotov [Fri, 9 Feb 2018 14:39:28 +0000 (17:39 +0300)]
Add support for Ceph Rados benchmarking.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
21 months agodebug: fix inverted logic in fio_did_warn() fio-3.4
Jens Axboe [Mon, 12 Feb 2018 17:55:07 +0000 (10:55 -0700)]
debug: fix inverted logic in fio_did_warn()

Fixes: 5ec3bc5401f3 ("init: add global 'warned' state")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoiolog: convert drop warning to fio_did_warn()
Jens Axboe [Mon, 12 Feb 2018 17:19:18 +0000 (10:19 -0700)]
iolog: convert drop warning to fio_did_warn()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoio_u: convert zoned bug warning to fio_did_warn()
Jens Axboe [Mon, 12 Feb 2018 17:18:32 +0000 (10:18 -0700)]
io_u: convert zoned bug warning to fio_did_warn()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoverify: convert verify buf too small warning to fio_did_warn()
Jens Axboe [Mon, 12 Feb 2018 17:14:35 +0000 (10:14 -0700)]
verify: convert verify buf too small warning to fio_did_warn()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofilesetup: convert root flush warning to fio_did_warn()
Jens Axboe [Mon, 12 Feb 2018 17:13:04 +0000 (10:13 -0700)]
filesetup: convert root flush warning to fio_did_warn()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoinit: add global 'warned' state
Jens Axboe [Mon, 12 Feb 2018 17:11:31 +0000 (10:11 -0700)]
init: add global 'warned' state

In various places we spew a warning if some static variable
isn't set, but this still means it can happen once per job.
Add some global state to allow to track this globally instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoMerge branch 'master' of https://github.com/brycepg/fio
Jens Axboe [Mon, 12 Feb 2018 15:24:21 +0000 (08:24 -0700)]
Merge branch 'master' of https://github.com/brycepg/fio

* 'master' of https://github.com/brycepg/fio:
  Make fiologparser_hist compatible with python3

21 months agoinit: fixup some bad style in previous commit
Jens Axboe [Sat, 10 Feb 2018 21:44:49 +0000 (14:44 -0700)]
init: fixup some bad style in previous commit

Make it follow the fio coding style.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoinit: fix broken verify_interval
Damian Yurzola [Fri, 9 Feb 2018 01:29:38 +0000 (17:29 -0800)]
init: fix broken verify_interval

The operands seems to have been inverted which in turn
created the situation whereby the interval was always
changed to match the min_bs

Fixes: https://github.com/axboe/fio/issues/522
Signed-off-by: Damian Yurzola <damian@yurzola.net>
21 months agoInclude 'numjobs' in global options output
Jens Axboe [Thu, 8 Feb 2018 22:46:46 +0000 (15:46 -0700)]
Include 'numjobs' in global options output

Fixes: https://github.com/axboe/fio/issues/237
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agommap: don't include MADV_FREE in fadvise_hint check
Jens Axboe [Wed, 7 Feb 2018 18:30:59 +0000 (11:30 -0700)]
mmap: don't include MADV_FREE in fadvise_hint check

This really should be based on invalidate, leave it as it was
for now.

Fixes: c712c97ab871 ("Let fadvise_hint also apply too mmap engine and madvise")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoLet fadvise_hint also apply too mmap engine and madvise
Jens Axboe [Wed, 7 Feb 2018 18:13:05 +0000 (11:13 -0700)]
Let fadvise_hint also apply too mmap engine and madvise

Fixes: https://github.com/axboe/fio/issues/528
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoMake fiologparser_hist compatible with python3
Bryce Guinta [Tue, 6 Feb 2018 00:36:33 +0000 (17:36 -0700)]
Make fiologparser_hist compatible with python3

In python3, map functions return generators, which caused numpy to break
in this script.

Use list comprehensions instead of map (more pythonic)
They also have the added benefit of behaving the same
between python2 and python3.

22 months agostat: add total fsync ios to json output
Jens Axboe [Thu, 25 Jan 2018 23:04:20 +0000 (16:04 -0700)]
stat: add total fsync ios to json output

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoio_ddir: revert separate ddir count change
Jens Axboe [Thu, 25 Jan 2018 17:28:23 +0000 (10:28 -0700)]
io_ddir: revert separate ddir count change

This essentially reverts the previous commit. We need more
careful changes for looping and indexing, the easiest is just
to move the count values at the end as not to skew the values
of the actual ddir entries.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoio_ddir: move count values out of the enum fio_ddir
Jens Axboe [Thu, 25 Jan 2018 17:16:51 +0000 (10:16 -0700)]
io_ddir: move count values out of the enum fio_ddir

They are not directions, just counts.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agostat: ensure that we align ts->sync_stat appropriately
Jens Axboe [Thu, 25 Jan 2018 16:51:07 +0000 (09:51 -0700)]
stat: ensure that we align ts->sync_stat appropriately

clang correctly complains that we don't know if the alignment is
correct when the structure is packed, so ensure that we align it
to an 8-byte boundary.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoTrack fsync/fdatasync/sync_file_range issue count
Jens Axboe [Thu, 25 Jan 2018 16:27:14 +0000 (09:27 -0700)]
Track fsync/fdatasync/sync_file_range issue count

Also fix an issue where we didn't initialize the sync minimum
latency correctly, so it was always reported as 0.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoSwitch last_was_sync and terminate to bool and pack better
Jens Axboe [Thu, 25 Jan 2018 03:22:50 +0000 (20:22 -0700)]
Switch last_was_sync and terminate to bool and pack better

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoAdd suppor for logging fsync (and friends) latencies
Jens Axboe [Wed, 24 Jan 2018 22:32:07 +0000 (15:32 -0700)]
Add suppor for logging fsync (and friends) latencies

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agonull: drop unneeded casts from void* to non-void*
Tomohiro Kusumi [Mon, 15 Jan 2018 19:52:12 +0000 (21:52 +0200)]
null: drop unneeded casts from void* to non-void*

Some functions in null ioengine are used by C/C++ (e.g. null_init()
which calls malloc(3)), but C specific ones (not __cplusplus) don't
need explicit cast from void* to non-void*, and one usually doesn't.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agonull: make *impl_ private
Tomohiro Kusumi [Mon, 15 Jan 2018 19:52:11 +0000 (21:52 +0200)]
null: make *impl_ private

This can/should be private, given struct is default public in C++.
The null ioengine only needs access to new'd NullData which
encapsulates malloc'd null_data *impl_.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agonull: fix compile time warning on OpenBSD
Tomohiro Kusumi [Mon, 15 Jan 2018 19:52:10 +0000 (21:52 +0200)]
null: fix compile time warning on OpenBSD

Fix the following warning on OpenBSD 6.2 (gcc 4.2.1).
This can probably be silenced with void** argument and memcpy() too,
but returning a non-void pointer is clearer IMO when the code also
targets C++.

Compiles with regular make/gmake as well as
 # g++ -O2 -g -shared -rdynamic -fPIC -o cpp_null null.c -DFIO_EXTERNAL_ENGINE

--
    CC engines/null.o
engines/null.c: In function 'fio_null_init':
engines/null.c:142: warning: dereferencing type-punned pointer will break strict-aliasing rules

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoMerge branch 'diskless_invalidate' of https://github.com/sitsofe/fio
Jens Axboe [Fri, 12 Jan 2018 17:59:08 +0000 (10:59 -0700)]
Merge branch 'diskless_invalidate' of https://github.com/sitsofe/fio

22 months agoMerge branch 'fio-issue-450' of https://github.com/gvkovai/fio
Jens Axboe [Fri, 12 Jan 2018 15:22:41 +0000 (08:22 -0700)]
Merge branch 'fio-issue-450' of https://github.com/gvkovai/fio

22 months agofilesetup: skip fallback invalidation with diskless ioengines
Sitsofe Wheeler [Sat, 30 Dec 2017 20:58:59 +0000 (20:58 +0000)]
filesetup: skip fallback invalidation with diskless ioengines

Since commit 0bcf41cdc22dfee6b3f3b2ba9a533b4b103c70c2 ("io_u:
re-invalidate cache when looping around without file open/close") the
job:

./fio --ioengine=null --size=4k --bs=4k --rate=4k \
 --time_based --runtime=2s --name=badinvalidate

repeatedly produces the following warning while running:

fio: cache invalidation of badinvalidate.0.0 failed: Bad file descriptor

Fix this by skipping cache invalidation and logging a debug message for
diskless ioengines that don't define an explicit invalidation op.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
22 months agoioengines: don't call munmap unless size boundary is exceeded
Robert Elliott [Wed, 10 Jan 2018 15:50:01 +0000 (09:50 -0600)]
ioengines: don't call munmap unless size boundary is exceeded

In ioengines using mmap() and munmap(), don't call munmap() when doing an
IO that accesses the last byte of a file. This affects the dev-dax,
libpmem, and mmap ioengines.

Example:
io       12987 DEBUG fio_libpmem_prep
io       12987  io_u->offset 7516061696 : fdd->libpmem_off 0 : io_u->buflen 131072 : fdd->libpmem_sz 7516192768
io       12987 munmap addr=0x7f5600000000 size=0x1c0000000

That accesses the last 128 KiB of a 7 GiB file, triggering an munmap
because the comparison to 7 GiB fails:
7516061696 + 131072 < 7516192768

Change < to <= so that doesn't occur unless an overflow really
happens.

In one test in linux, this increases performance from 5 GB/s to 20 GB/s
as fio stops spending time in mmap() page table mapping spinlocks and
instead spends its time in memmove().  The perf top results improve from:
  36.81%  [kernel]             [k] queued_spin_lock_slowpath
  24.26%  libc-2.26.so         [.] __memmove_avx_unaligned_erms
   4.09%  [kernel]             [k] __radix_tree_lookup
   3.85%  [kernel]             [k] find_next_iomem_res
   3.70%  [kernel]             [k] _raw_spin_lock_irq
   3.48%  [kernel]             [k] down_read

to:
  79.76%  libc-2.26.so         [.] __memmove_avx_unaligned_erms
   3.23%  libc-2.26.so         [.] vfprintf

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agoio/examples/fio-seq-read: use direct=1
Jens Axboe [Wed, 10 Jan 2018 15:30:07 +0000 (08:30 -0700)]
io/examples/fio-seq-read: use direct=1

It's confusing to use direct=0 with libaio, since that isn't
queued.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
22 months agostat: don't add duplicate clat entries for json
Jens Axboe [Sat, 6 Jan 2018 21:47:01 +0000 (14:47 -0700)]
stat: don't add duplicate clat entries for json

For some reason the initial commit always filled all
entries with zeroes, when it should just stop printing.
I suspect this was carried over from the terse/csv output,
where we do the same. But there it's actually valid, as we
have to have a fixed number of fields. For json we both
don't need it, and it's causing us to create invalid json
when we have duplicate keys.

Fixes: https://github.com/axboe/fio/issues/511
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoverify: don't adjust verification length based on interval when unaligned
Jens Axboe [Fri, 5 Jan 2018 20:38:40 +0000 (13:38 -0700)]
verify: don't adjust verification length based on interval when unaligned

It's perfectly fine to have a verification length that exceeds the
default verify interval when we have unaligned block sizes and
using block size ranges, since the verify interval just defaults to
the smallest blocksize.

Fixes: https://github.com/axboe/fio/issues/509
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoFix client/server "all clients" reporting
Jeff Furlong [Wed, 3 Jan 2018 23:20:38 +0000 (23:20 +0000)]
Fix client/server "all clients" reporting

In commit e883cb3501de0f99297282c18842d50aaa7fa531, the default significant figures was added.  The default value was preserved for local jobs and when client/server was used if jobs=1, but not when client/server was used if jobs > 1.  This patch restores some reporting outputs ("all clients" output for IOPS and BW) when client/server is used if > 1 jobs, for example:

OLD:
All clients: (groupid=0, jobs=2): err= 0: pid=0: Wed Jan  3 14:30:13 2018
   read: IOPS=0, BW=0 (0)(0B/5001msec)                                               <--ZERO VALUES
    slat (nsec): min=762, max=108579, avg=1058.97, stdev=221.30
    clat (nsec): min=254, max=4544.2k, avg=11297.64, stdev=8284.14
     lat (usec): min=9, max=4545, avg=12.41, stdev= 8.30
   bw (  KiB/s): min=290192, max=347160, per=49.35%, avg=303613.33, stdev=11582.87, samples=18
   iops        : min=72548, max=86790, avg=75903.33, stdev=2895.72, samples=18

NEW:
All clients: (groupid=0, jobs=2): err= 0: pid=0: Wed Jan  3 15:08:33 2018
   read: IOPS=147k, BW=575Mi (603M)(2876MiB/5001msec) <--CORRECT VALUES
    slat (nsec): min=841, max=61945, avg=1304.83, stdev=232.75
    clat (nsec): min=544, max=4487.2k, avg=11736.68, stdev=9123.64
     lat (usec): min=10, max=4489, avg=13.10, stdev= 9.13
   bw (  KiB/s): min=260945, max=305976, per=49.99%, avg=294350.22, stdev=9192.53, samples=18
   iops        : min=65236, max=76494, avg=73587.56, stdev=2298.13, samples=18

Regards,
Jeff

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoChange bluestop link to be https
Jens Axboe [Fri, 5 Jan 2018 15:30:32 +0000 (08:30 -0700)]
Change bluestop link to be https

Doesn't answer on http:// anymore.

Also include link to AppVeyor builds for Windows, as they always build
the latest and greatest.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoMerge branch 'percentiles' of https://github.com/sitsofe/fio
Jens Axboe [Tue, 2 Jan 2018 16:05:44 +0000 (09:05 -0700)]
Merge branch 'percentiles' of https://github.com/sitsofe/fio

23 months agoinit: disable percentiles when latency gathering is disabled
Sitsofe Wheeler [Sun, 31 Dec 2017 12:42:59 +0000 (12:42 +0000)]
init: disable percentiles when latency gathering is disabled

When disable_clat/disable_lat is set also disable the corresponding
clat_percentiles/lat_percentiles option.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agostat: make lat_percentiles=1 use sample count from lat_stat
Sitsofe Wheeler [Sun, 31 Dec 2017 11:41:55 +0000 (11:41 +0000)]
stat: make lat_percentiles=1 use sample count from lat_stat

The following job

./fio --name=nolatencies --ioengine=posixaio --direct=1 --rw=randread \
 --iodepth=1 --bs=4k --filename=/tmp/fio.tmp --runtime=2s --size=50m \
 --disable_lat=0 --disable_clat=1 --disable_slat=1 --lat_percentiles=1

outputs the following:

fio-3.3
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=7316: Sun Dec 31 11:55:46 2017
   read: IOPS=41.0k, BW=164MiB/s (172MB/s)(50.0MiB/305msec)
     lat (usec): min=8, max=132, avg=17.58, stdev= 4.36
     lat percentiles (nsec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    0], 50.00th=[    0], 60.00th=[    0],
     | 70.00th=[    0], 80.00th=[    0], 90.00th=[    0], 95.00th=[    0],
     | 99.00th=[    0], 99.50th=[    0], 99.90th=[    0], 99.95th=[    0],
     | 99.99th=[    0]
  cpu          : usr=56.39%, sys=29.18%, ctx=12043, majf=0, minf=85
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=12800,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=164MiB/s (172MB/s), 164MiB/s-164MiB/s (172MB/s-172MB/s), io=50.0MiB (52.4MB), run=305-305msec

which isn't showing total latency percentiles (lat percentiles) when it
should. Additionally there is a segfault when running the following job:

./fio --name=segfault --ioengine=posixaio --direct=1 --rw=randread \
 --iodepth=1 --bs=4k --filename=/tmp/fio.tmp --runtime=2s --size=50m \
 --disable_lat=1 --disable_clat=0 --disable_slat=1 --lat_percentiles=1

Fix these by making latency printing use the number of samples from the
correct latency type (clat_stat for lat_percentiles=0/clat_percentiles=1
and lat_stat for lat_percentiles=1/clat_percentiles=0).

Fixes https://www.spinics.net/lists/fio/msg06628.html ("fio lat reporting
").

Reported-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agolog: fix bad < 0 check for unsigned
Jens Axboe [Fri, 29 Dec 2017 15:45:22 +0000 (08:45 -0700)]
log: fix bad < 0 check for unsigned

Fixes: e5f9a813 ("debug: make debug=io readable with multiple threads")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agodebug: make debug=io readable with multiple threads
Robert Elliott [Fri, 29 Dec 2017 01:13:06 +0000 (01:13 +0000)]
debug: make debug=io readable with multiple threads

When multiple threads are active, debug=io prints are unreadable as the output
is interleaved. Multiple log_info_buf() calls are used to construct each line,
but there is no mutual exclusion covering multiple calls.

Change the dprint call tree to construct the entire line before passing it to
log_info_buf(), rather than make several calls.

Other nits:
* print the thread ID rather than the process ID
* change offset and length from decimal to hex
* separate offset, length, ddir, and file with , rather than / since the
  filename on the same line likely has / of its own
* change "fill_io_u" to "fill" to match the others
* change "io complete" to "complete" to match the others
* change "->prep()=%d" to "prep: io_u %p: ret=%d" to resemble the others
* change offset/buflen in an error print to better resemble the normal prints
* add "file=" prefix for the filename
* check the calloc() return values inside the valist_to_buf functions

Old:
fill_io_u: io_u 0x7feeac010b80: off=720896/len=65536/ddir=1

io       50692io       50692//dev/dax0.0io       50692fill_io_u: io_u 0x7fee98010b80: off=196608/len=65536/ddir=1io       50692io       50692
io       50692->prep(0x7fef10010b80)=0
//dev/dax0.0//dev/dax1.0io       50692io       50692io       50692io       50692->prep(0x7feeec010b80)=0
io       50692prep: io_u 0x7feec4010b80: off=1966080/len=65536/ddir=1io       50692io complete: io_u 0x7feedc010b80: off=393216/len=65536/ddir=1io
50692io       50692prep: io_u 0x7feef4010b80: off=720896/len=65536/ddir=1io       50692io       50692
io       50692io       50692prep: io_u 0x7feef0010b80: off=851968/len=65536/ddir=1//dev/dax0.0io       50692//dev/dax0.0

//dev/dax0.0
io       50692io       50692//dev/dax0.0//dev/dax0.0io       50692

New:
io       71400 queue: io_u 0x7fd0f0010b80: off=0x2f0000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71395 fill: io_u 0x7fd0fc010b80: off=0x80000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71395 prep: io_u 0x7fd0fc010b80: off=0x80000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71395 prep: io_u 0x7fd0fc010b80: ret=0
io       71395 queue: io_u 0x7fd0fc010b80: off=0x80000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71430 complete: io_u 0x7fd05c010b80: off=0x180000,len=0x10000,ddir=1,file=/dev/dax1.0
io       71400 complete: io_u 0x7fd0f0010b80: off=0x2f0000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71430 fill: io_u 0x7fd05c010b80: off=0x190000,len=0x10000,ddir=1,file=/dev/dax1.0
io       71400 fill: io_u 0x7fd0f0010b80: off=0x300000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71400 prep: io_u 0x7fd0f0010b80: off=0x300000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71400 prep: io_u 0x7fd0f0010b80: ret=0
io       71400 queue: io_u 0x7fd0f0010b80: off=0x300000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71430 prep: io_u 0x7fd05c010b80: off=0x190000,len=0x10000,ddir=1,file=/dev/dax1.0
io       71395 complete: io_u 0x7fd0fc010b80: off=0x80000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71430 prep: io_u 0x7fd05c010b80: ret=0
io       71430 queue: io_u 0x7fd05c010b80: off=0x190000,len=0x10000,ddir=1,file=/dev/dax1.0
io       71421 complete: io_u 0x7fd090010b80: off=0x320000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71419 complete: io_u 0x7fd098010b80: off=0x320000,len=0x10000,ddir=1,file=/dev/dax0.0
io       71404 complete: io_u 0x7fd0d0010b80: off=0x0,len=0x10000,ddir=1,file=/dev/dax0.0

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agolib/memcpy: fix warning on FreeBSD
Tomohiro Kusumi [Fri, 29 Dec 2017 13:38:23 +0000 (15:38 +0200)]
lib/memcpy: fix warning on FreeBSD

Include struct timeval via sys/time.h.
http://pubs.opengroup.org/onlinepubs/009604499/basedefs/sys/time.h.html

--
In file included from lib/memcpy.c:7:
lib/../fio_time.h:14:45: warning: declaration of 'struct timeval' will not be visible outside of this function [-Wvisibility]
extern uint64_t mtime_since_tv(const struct timeval *, const struct timeval *);

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoMerge branch 'master' of https://github.com/yashi/fio
Jens Axboe [Thu, 28 Dec 2017 15:35:11 +0000 (08:35 -0700)]
Merge branch 'master' of https://github.com/yashi/fio

23 months agomutex: down_timeout: check against the base time
Yasushi SHOJI [Thu, 28 Dec 2017 01:53:47 +0000 (10:53 +0900)]
mutex: down_timeout: check against the base time

When the commit 8b6a404cdd2c40715885e562416c3db039912773 changed
timeval to timespec, it accsidentally picked timeout time `t' instead
of the current `base' time for checking timewait() really timed out.

t is base + msecs as follows:

      base = gettimeofday()
      t = base + msecs
      pthread_cond_timedwait()
      while (!done)
         if (timedout)
    if (mutex_timed_out(t, msecs))
       done = true

Thus, after a good timedout, it burns the cpu for the given msecs
seconds doing tight loop in the while shown above.  This causes 30 to
40 % CPU load on my i5-3360M 2.80GHz while fio server is running.

23 months agoMerge branch 'eta_display' of https://github.com/sitsofe/fio
Jens Axboe [Wed, 27 Dec 2017 21:05:46 +0000 (14:05 -0700)]
Merge branch 'eta_display' of https://github.com/sitsofe/fio

23 months agoMerge branch 'barak/mmap_verify_only' of https://github.com/barakp/fio
Jens Axboe [Wed, 27 Dec 2017 21:05:27 +0000 (14:05 -0700)]
Merge branch 'barak/mmap_verify_only' of https://github.com/barakp/fio

23 months agoeta: show complete status line with max job states
Sitsofe Wheeler [Sat, 23 Dec 2017 09:55:02 +0000 (09:55 +0000)]
eta: show complete status line with max job states

Make the ETA output buffer big enough to cope with the maximum possible
jobs when their states can't be compressed (i.e. when every job's state
is different to the state of the job that came before it).

Allows the following script to display the complete status line:

rw[0]='read'; rw[1]='write'; \
for i in {1..4096}; do \
  echo -e "[job$i]\nrw=${rw[$((i % 2))]}\n" \
          "runtime=15"; \
done | \
./fio --eta=always --group_reporting=1 --ioengine=null --size=1g \
 --time_based --bs=1k --thread --rate_iops=1 -

Fixes: https://github.com/axboe/fio/issues/500 ("Large number of threads
result in Seg faults")

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoeta: fix previous line length calculation
Sitsofe Wheeler [Sat, 23 Dec 2017 08:43:54 +0000 (08:43 +0000)]
eta: fix previous line length calculation

We work out how many stale characters we haven't yet overprinted from
the previous run by comparing the number of characters that snprintf
added to the buffer in both cases. Unfortunately this doesn't account
for earlier characters that may (or may not) have already been in the
output buffer before snprintf was called...

Change the code to just use the current and previous output buffer
position to calculate the remainder of the line to be cleared.

When fio is compiled under clang with
./configure --extra-cflags="-fsanitize=address -fno-builtin" \
 --disable-optimizations

a buffer overrun is demonstrated by the following script

rw[0]='read'; rw[1]='write'; \
for i in {1..4096}; do \
  echo -e "[job$i]\nrw=${rw[$((i % 2))]}\n" \
  "ramp_time=$(((4096 - i) / 2049 * 10))\n" \
          "runtime=15"; \
done | \
./fio --group_reporting --ioengine=null --size=1g --time_based \
 --bs=512 --thread --rate_iops=2 -

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoeta: skip clearing of remainder of line when starting a new line
Sitsofe Wheeler [Sat, 23 Dec 2017 08:39:11 +0000 (08:39 +0000)]
eta: skip clearing of remainder of line when starting a new line

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoeta: fix run_str_condensed overflow with maximum jobs
Sitsofe Wheeler [Sat, 23 Dec 2017 08:01:01 +0000 (08:01 +0000)]
eta: fix run_str_condensed overflow with maximum jobs

When the maximum number of jobs all have a different state to their
previous job, run_str didn't have space for the terminating null. Fix
this by making run_str one character larger.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoeta: adjust truncation case
Sitsofe Wheeler [Fri, 22 Dec 2017 11:10:35 +0000 (11:10 +0000)]
eta: adjust truncation case

c076de85fa4901683f6ce23ecdc071c17801ffad ("eta: fix buffer overflow in
ETA output") made adjustments to cope with snprintf truncation. However
since we're replacing the null added by snprintf, left only needs to be
one smaller than the output buffer. Make this change and add a comment
about what we're doing.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoMerge branch 'eta_overflow' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 21 Dec 2017 15:22:39 +0000 (08:22 -0700)]
Merge branch 'eta_overflow' of https://github.com/sitsofe/fio

23 months agoeta: fix buffer overflow in ETA output
Sitsofe Wheeler [Thu, 21 Dec 2017 12:23:36 +0000 (12:23 +0000)]
eta: fix buffer overflow in ETA output

When 900 or more jobs all have different states it is possible for the
space required to display the ETA line to be larger than the ETA output
buffer. fio initially truncates what it puts in the output buffer but
incorrectly advances where it thinks the end of the buffer is - when
truncating snprintf returns the number of characters that _would_ have
been converted if there had been enough space...

This patch fixes the post truncation calculation and adjusts the "left"
variable so there is room for the carriage return and a null terminator.

The following script reproduces the problem when fio has been compiled
with -fstack-protector or -fsanitize=address:

rw[0]='read'; rw[1]='write'; \
for i in {1..1000}; do \
 echo -e "[job$i]\nrw=${rw[$((i % 2))]}\n" \
         "ramp_time=$(((1000 - i) / 350 * 5))\n" \
         "runtime=$((60 - (i % 3) * 2))"; \
done | \
./fio --eta-newline=1s --group_reporting --ioengine=null --size=1g \
 --time_based --bs=512 --thread --rate_iops=123456 -

Fixes: https://github.com/axboe/fio/issues/500 ("Large number of threads
result in Seg faults")
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoFio 3.3 fio-3.3
Jens Axboe [Tue, 19 Dec 2017 20:16:36 +0000 (13:16 -0700)]
Fio 3.3

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agobackend: tweaks to missed rate thinktime
Jens Axboe [Tue, 19 Dec 2017 17:17:41 +0000 (10:17 -0700)]
backend: tweaks to missed rate thinktime

shimrot says:

"May I suggest a tweak to the backend.c::handle_thinktime calculation of
"over"? That is, adjust based on the rate per byte similar to what
you've done. That is, if thinktime is greater than one op worth of
bytes, then subtract one op worth of bytes. But, if think time is
greater than one ops worth, instead add the missing the number of bytes
a full op would have exceeded the think time."

See: https://github.com/axboe/fio/issues/497#issuecomment-352816955

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoioengines: clear out ->td_ops_dlhandle if we close it
Jens Axboe [Fri, 15 Dec 2017 20:35:56 +0000 (13:35 -0700)]
ioengines: clear out ->td_ops_dlhandle if we close it

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoinit: fix missing dlhandle reference put
Jens Axboe [Fri, 15 Dec 2017 20:34:22 +0000 (13:34 -0700)]
init: fix missing dlhandle reference put

If we keep using the same engine, put the dlhandle reference that
we now hold twice.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoparse: don't check for < 0 on an unsigned type
Jens Axboe [Fri, 15 Dec 2017 16:13:28 +0000 (09:13 -0700)]
parse: don't check for < 0 on an unsigned type

We do strict errors on some builds, this breaks them. Remove the
<= 0 check, just check for a value that's too large.

Fixes: c26438ad ("parse: dump option type when using --debug=parse")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoioengines: improve "is this the same IO engine" check
Jens Axboe [Fri, 15 Dec 2017 16:08:26 +0000 (09:08 -0700)]
ioengines: improve "is this the same IO engine" check

We can't just compare the name, that assumes that name and filename
match for an IO engine. While that's generally the case for the
engines that fio ships with, it's not a requirement, and it's
definitely not true for external engines.

Fixup the check by re-loading the engine and checking the OPS
instead. That should be bullet proof.

Fixes: 800334d ("Correctly detect whether ioengine_load can exit early")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoparse: dump option type when using --debug=parse
Jens Axboe [Fri, 15 Dec 2017 15:41:03 +0000 (08:41 -0700)]
parse: dump option type when using --debug=parse

Currently we do things like:

parse    8731  __handle_option=dummy, type=10, ptr=1

for the debug parsing, and then you have to look up what that
option type is. Add names to them so we get the below instead:

parse    9170  __handle_option=dummy, type=OPT_STR_SET, ptr=1

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoAllow configurable ETA intervals
Jens Axboe [Thu, 14 Dec 2017 18:51:41 +0000 (11:51 -0700)]
Allow configurable ETA intervals

By default, fio prints ETA output every second. For some client/server
setups, it's desirable to allow a much longer interval, to avoid
spending too much time getting and printing ETA time.

Takes a normal time input, allowing usec/msec/sec etc postfixes.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoclient: respect --eta=never for networked connections
Jens Axboe [Thu, 14 Dec 2017 17:58:03 +0000 (10:58 -0700)]
client: respect --eta=never for networked connections

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoserver: convert more memset to on-stack initialization
Jens Axboe [Thu, 14 Dec 2017 16:47:35 +0000 (09:47 -0700)]
server: convert more memset to on-stack initialization

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoserver: cleanup iolog pdu prep
Jens Axboe [Thu, 14 Dec 2017 16:25:10 +0000 (09:25 -0700)]
server: cleanup iolog pdu prep

We don't have to memset() the whole thing, we just need to ensure
that the members we don't assign are cleared. Remove the memset
and initialize the pdu on the stack.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoFix Windows local time
Jeff Furlong [Thu, 14 Dec 2017 01:19:09 +0000 (01:19 +0000)]
Fix Windows local time

The fio summary output shows when the job completed, but references
system time on Windows.  On Linux, local time (such as with time zone
specifics) is referenced.  This patch converts Windows system time to
local time.

Before:
fio-3.2-72-ge80d-dirty
Starting 1 thread
Jobs: 1 (f=0): [f(1)][100.0%][r=69.6MiB/s,w=0KiB/s][r=17.8k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1360: Wed Dec 13 23:21:16 2017 <--System Time

After:
fio-3.2-72-ge80d-dirty
Starting 1 thread
Jobs: 1 (f=0): [f(1)][100.0%][r=73.2MiB/s,w=0KiB/s][r=18.7k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=8088: Wed Dec 13 15:22:07 2017 <--Local Time

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months ago.gitignore: ignore tags files and additional output binaries
Robert Elliott [Mon, 4 Dec 2017 20:53:16 +0000 (14:53 -0600)]
.gitignore: ignore tags files and additional output binaries

Ignore tags files used by emacs and ctags/vim.

Ignore the additional output binaries alongside fio, such as gfio
and all the programs in t/.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoserver: initialize first iolog header properly
Jens Axboe [Wed, 13 Dec 2017 15:44:34 +0000 (08:44 -0700)]
server: initialize first iolog header properly

We're sending potential stack garbage over the wire, init the
whole pdu to zeroes before filling it in. This shuts up valgrind
about using uninitialized memory.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agofio: kill td->nr_normal_files
Jens Axboe [Fri, 8 Dec 2017 19:50:28 +0000 (12:50 -0700)]
fio: kill td->nr_normal_files

It's unused.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoRemove old emails
Jens Axboe [Fri, 8 Dec 2017 19:38:27 +0000 (12:38 -0700)]
Remove old emails

Just use my private email, it's the one I prefer using anyway.
Remove the old fusionio/oracle emails, they just cause
confusion and people tend to just write to all of them at once
when reporting an issue.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agorate: fix bad math
Jens Axboe [Fri, 8 Dec 2017 18:59:38 +0000 (11:59 -0700)]
rate: fix bad math

To figure out how much data we missed when doing a thinktime sleep,
we're currently dividing by the time slept. This is wrong, it should
be multiplied by the time slept and divided by 1000000 to go from
usec to a second base.

Additionally, don't ever subtract more than a block of data, and
adjust down depending on sleep.

Fixes: 1aa39b0c ("rate: ensure IO issue restarts right after sleep")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agorate: ensure IO issue restarts right after sleep
Jens Axboe [Thu, 7 Dec 2017 16:06:04 +0000 (09:06 -0700)]
rate: ensure IO issue restarts right after sleep

If we have rate_ignore_thinktime set, we account the number of bytes
we would have done in the period of the sleep. However, we need
to subtract one block, otherwise we end up doing a rated sleep
right after the thinktime sleep.

Fixes: 1a9bf814 ("Add option to ignore thinktime for rated IO")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoio_u: rate cleanup and spelling error
Jens Axboe [Wed, 6 Dec 2017 19:30:20 +0000 (12:30 -0700)]
io_u: rate cleanup and spelling error

Fixes: 50a8ce86 ("Implement new Rate Control")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoAdd option to ignore thinktime for rated IO
Jens Axboe [Wed, 6 Dec 2017 19:27:07 +0000 (12:27 -0700)]
Add option to ignore thinktime for rated IO

By default, fio will ignore thinktime when calculating the next
time to issue and IO, if rated IO is specified. This leads to
fio entering a catch-up type of mode after doing the specified
sleep. For some workloads, that may not be useful. If someone
asks for a specific amount of IOPS and sets a thinktime, they
may want to exclude the sleep time.

Fixes: https://github.com/axboe/fio/issues/497
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofix verify_only when using ioengine=mmap
Barak Pinhas [Tue, 5 Dec 2017 12:23:22 +0000 (14:23 +0200)]
fix verify_only when using ioengine=mmap

verify_only was opening the file with O_RDONLY and the mmap
engine was trying to mmap it with PROT_READ | PROTWRITE hence
getting -EACCESS and failing the job

2 years agoMerge branch 'wip-cleanup' of https://github.com/ZVampirEM77/fio
Jens Axboe [Sun, 3 Dec 2017 17:11:53 +0000 (10:11 -0700)]
Merge branch 'wip-cleanup' of https://github.com/ZVampirEM77/fio

2 years agoconfigure: fix typos
Enming Zhang [Sun, 3 Dec 2017 09:50:10 +0000 (17:50 +0800)]
configure: fix typos

check for rbd_invaidate_cache() --> check for rbd_invalidate_cache()

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>
2 years agosteadystate: style cleanup
Jens Axboe [Sat, 2 Dec 2017 23:29:44 +0000 (16:29 -0700)]
steadystate: style cleanup

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agosteadystate: add free helper
Jens Axboe [Sat, 2 Dec 2017 16:18:46 +0000 (09:18 -0700)]
steadystate: add free helper

Also fix a style issue.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agothread_options: drop fadvise_stream from thread_options
Vincent Fu [Fri, 1 Dec 2017 21:52:08 +0000 (16:52 -0500)]
thread_options: drop fadvise_stream from thread_options

ae8e559e got rid of the fadvise_stream option but did
not remove the corresponding member from thread_options

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agomemcpy: add hybrid
Jens Axboe [Fri, 1 Dec 2017 21:49:42 +0000 (14:49 -0700)]
memcpy: add hybrid

Use builtin memcpy for < 64 bytes, use simple variant for larger
ones. libc doesn't seem to like to use SSE, the hand rolled simple
variant uses that for larger copies.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agomemcpy: free buffer in case of failure
Jens Axboe [Fri, 1 Dec 2017 20:56:09 +0000 (13:56 -0700)]
memcpy: free buffer in case of failure

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agomemcpy: use malloc
Jens Axboe [Fri, 1 Dec 2017 18:30:49 +0000 (11:30 -0700)]
memcpy: use malloc

We don't have posix_memalign() everywhere, let's just use malloc.

Also free memory when done.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofio_time: should include time.h
Jens Axboe [Fri, 1 Dec 2017 18:21:58 +0000 (11:21 -0700)]
fio_time: should include time.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoAdd basic memcpy test
Jens Axboe [Fri, 1 Dec 2017 18:13:23 +0000 (11:13 -0700)]
Add basic memcpy test

Research is ongoing in how to improve (and make deterministic) the
memcpy speed, since it's especially applicable to the persistent
memory engines.

Not documented yet.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: don't account io issue blocks for verify backlog
Jens Axboe [Fri, 1 Dec 2017 04:48:12 +0000 (21:48 -0700)]
io_u: don't account io issue blocks for verify backlog

We don't account the bytes, we should not account the blocks
either.

Fixes: ae2fafc8 ("verify: verify bytes should not add to this_io_bytes")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooptions: warn if we exceed the supported number of split entries
Jens Axboe [Fri, 1 Dec 2017 04:29:32 +0000 (21:29 -0700)]
options: warn if we exceed the supported number of split entries

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoDocumentation: add note about how many bssplit and zones fio supports
Jens Axboe [Fri, 1 Dec 2017 04:28:15 +0000 (21:28 -0700)]
Documentation: add note about how many bssplit and zones fio supports

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoioengine: don't account verify bytes
Jens Axboe [Fri, 1 Dec 2017 03:24:54 +0000 (20:24 -0700)]
ioengine: don't account verify bytes

If we do that, we can terminate early for a readwrite+verify
backlog workload.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoclient: fix use-after-free for client timeout
Jens Axboe [Fri, 1 Dec 2017 02:50:34 +0000 (19:50 -0700)]
client: fix use-after-free for client timeout

We free the reply, but if it times out, we dereference it afterwards.
Store the opcode before calling into handle_cmd_timeout().

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoengines/dev-dax: fix leak of 'sfile' in error case
Jens Axboe [Fri, 1 Dec 2017 02:48:25 +0000 (19:48 -0700)]
engines/dev-dax: fix leak of 'sfile' in error case

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agot/verify-state: fix leak in error case
Jens Axboe [Fri, 1 Dec 2017 02:47:42 +0000 (19:47 -0700)]
t/verify-state: fix leak in error case

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agobackend: make it clear that we passed 'fd' to the new thread
Jens Axboe [Fri, 1 Dec 2017 02:45:05 +0000 (19:45 -0700)]
backend: make it clear that we passed 'fd' to the new thread

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agogettime-thread: fix failure to check setaffinity return value
Jens Axboe [Fri, 1 Dec 2017 02:41:22 +0000 (19:41 -0700)]
gettime-thread: fix failure to check setaffinity return value

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoDocumentation cleanup
Jens Axboe [Fri, 1 Dec 2017 02:17:38 +0000 (19:17 -0700)]
Documentation cleanup

- Space at end-of-line for the significant_figures change
- > 80 char line section in HOWTO

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooptions: don't overrun bssplit array
Jens Axboe [Thu, 30 Nov 2017 23:49:23 +0000 (16:49 -0700)]
options: don't overrun bssplit array

Since we now have a different number of splits we support for
zones and block size splits, make sure we pass in the right
max.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoBump support of zones to 256 max
Jens Axboe [Thu, 30 Nov 2017 23:47:27 +0000 (16:47 -0700)]
Bump support of zones to 256 max

This moves it from 64 to 256.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agosteadystate: make flags conform to usual fio standard
Jens Axboe [Thu, 30 Nov 2017 23:43:18 +0000 (16:43 -0700)]
steadystate: make flags conform to usual fio standard

__VAL are usually bit shift values, with VAL being the proper
mask. For some reason, the steadystate code uses both as
masks, fix that up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoserver: process connection list before executing trigger
Jens Axboe [Thu, 30 Nov 2017 20:28:48 +0000 (13:28 -0700)]
server: process connection list before executing trigger

This shrinks the window a bit for allowing the backend to notice
that jobs have gone away, and notify the client.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoclient: ignore a client timeout, if the last thing we saw as a trigger
Jens Axboe [Thu, 30 Nov 2017 20:27:40 +0000 (13:27 -0700)]
client: ignore a client timeout, if the last thing we saw as a trigger

If triggers are used and are fatal to the fio backend (killing power,
etc), then we don't have time to process the stop/quit commands. We
expect the client to go away, but the backend can't let us know.

If the last thing we saw was a trigger command, don't exit in error
if the client times out and goes away.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofio: add check rate flag
Jens Axboe [Thu, 30 Nov 2017 17:59:25 +0000 (10:59 -0700)]
fio: add check rate flag

One of the things we do a lot for each IO, is check if we should be
checking the rate. Add a specific flag for that case, so we can answer
that question very cheaply.

This is good for more than a 5% speedup for a null engine test case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: speed up small_content_scramble()
Jens Axboe [Thu, 30 Nov 2017 16:18:37 +0000 (09:18 -0700)]
io_u: speed up small_content_scramble()

This is a hot path for write workloads, since we don't want to send the
same buffers to the device again and again. The idea is to defeat basic
dedupe/compression, but slightly modifying the buffer for each write.
small_content_scramble() does this by filling in the io_u offset into a
random spot in each 512b chunk of an io buffer, and filling in the start
time (sec,nsec) at the end of each 512b chunk.

With this change, we still do those two things, but we generate a random
cacheline within each 512b chunk, and fill the offset at the beginning
of the cacheline, and the time at the end of it.  This means that
instead of potentially dirtying 2 cachelines for each 512b chunk in an
IO buffer, we dirty just 1.

The results should still be random enough that small_content_scramble()
fullfils the promise to defeat basic dedupe and compression, but it is
lighter to run.

Signed-off-by: Jens Axboe <axboe@kernel.dk>