fio.git
14 months agoFio 3.7 fio-3.7
Jens Axboe [Fri, 1 Jun 2018 19:21:56 +0000 (13:21 -0600)]
Fio 3.7

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoio_u: ensure to invalidate cache on time_based random reads
Jens Axboe [Thu, 31 May 2018 15:05:59 +0000 (09:05 -0600)]
io_u: ensure to invalidate cache on time_based random reads

We need to do this with the file reset and retrieval of a
new offset, not of it fails.

Fixes: 0bcf41cdc22df ("io_u: re-invalidate cache when looping around without file open/close")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoFix typo in bssplit documentation
Jens Axboe [Fri, 25 May 2018 14:14:50 +0000 (08:14 -0600)]
Fix typo in bssplit documentation

One of the examples uses a ',' where it should be a ':' to
behave like described.

Reported-by: Konstantin Ruvinsky <konstantin.ruvinsky@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agomake fio scripts python3-ready (part 2)
Bill O'Donnell [Tue, 22 May 2018 17:49:09 +0000 (12:49 -0500)]
make fio scripts python3-ready (part 2)

NOTE: this patch (part 2) is in addition to commit 5eac3b00238b
(this patch was missed on that submit/commit (part 1)).

Many distributions are moving to python3 by default.  This is
another patch to make the python scripts in fio python3-ready.

Conversion was facilitated with automated tools.

The modified scripts pass rudimentary testing when run under
python2.7 as well as python3.

Signed-off-by: Bill O'Donnell <billodo@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoMerge branch 'sg-verify2' of https://github.com/vincentkfu/fio
Jens Axboe [Fri, 18 May 2018 19:34:36 +0000 (13:34 -0600)]
Merge branch 'sg-verify2' of https://github.com/vincentkfu/fio

* 'sg-verify2' of https://github.com/vincentkfu/fio:
  docs: add documentation for sg ioengine WRITE SAME, WRITE AND VERIFY command support
  engines/sg: add support for WRITE AND VERIFY, WRITE SAME

15 months agodocs: add documentation for sg ioengine WRITE SAME, WRITE AND VERIFY command support
Vincent Fu [Thu, 17 May 2018 14:08:03 +0000 (10:08 -0400)]
docs: add documentation for sg ioengine WRITE SAME, WRITE AND VERIFY command support

15 months agoengines/sg: add support for WRITE AND VERIFY, WRITE SAME
Vincent Fu [Wed, 16 May 2018 23:17:47 +0000 (19:17 -0400)]
engines/sg: add support for WRITE AND VERIFY, WRITE SAME

Add an sg ioengine option to treat write IO as either WRITE AND VERIFY
or WRITE SAME commands

The new ioengine option sg_write_mode can take 3 values:

write (default)
  WRITE opcodes are issued as usual
verify
  WRITE AND VERIFY opcodes are issued
same
  WRITE SAME opcodes are issued

WRITE AND VERIFY opcodes are issued with the BYTCHK bit set to 0. This
directs the device to carry out a medium verification with no data
comparison.

WRITE SAME transfers a single block of data to the device and writes
this same block of data to a contiguous sequence of LBAs beginning at
the specified offset.

fio's block size parameter specifies the amount of data written with
each command. However, the amount of data actually transferred to the
device is equal to the device's block (sector) size. For a device with
512 byte sectors, blocksize=8k will write 16 sectors with each command.
fio will still generate 8k of data for each command but only the first
512 bytes will be used and transferred to the device.

15 months agomake fio scripts python3-ready
Bill O'Donnell [Fri, 4 May 2018 19:43:40 +0000 (14:43 -0500)]
make fio scripts python3-ready

Many distributions are moving to python3 by default.  Here's
an attempt to make the python scripts in fio python3-ready.

Conversion was facilitated with automated tools. A few areas
were hand fixed: remove superfluous parentheses introduced by
2to3 converter in print function calls, shebang modifications
to use environment variable for python version, and byte-string
decode correction in steadystate_tests.py following 2to3
conversion.

The modified scripts pass rudimentary testing when run under
python2.7 as well as python3.

Signed-off-by: Bill O'Donnell <billodo@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoMerge branch 'master' of https://github.com/majianpeng/fio
Jens Axboe [Tue, 15 May 2018 01:40:32 +0000 (19:40 -0600)]
Merge branch 'master' of https://github.com/majianpeng/fio

* 'master' of https://github.com/majianpeng/fio:
  This partly revert 97bb54c9606c(add __load_ioengine() to separate ioengine loading from td context)

15 months agoThis partly revert 97bb54c9606c(add __load_ioengine() to separate ioengine loading...
Jianpeng Ma [Mon, 14 May 2018 16:14:01 +0000 (00:14 +0800)]
This partly revert 97bb54c9606c(add __load_ioengine() to separate ioengine loading from td context)

In commit 97bb54c9606c, it change load_ioengine to __load_ioengine in
func fio_show_ioengine_help. The different between load_ioengine and
__load_ioengine is load_ioengine include __load_ioengine and if
__load_ioengine failed it will do dlopen_ioengine to find ioengine.
For external ioengine, if no dlopen_ioengine it can't find, etc
libfio_ceph_objectstore.so in Ceph project.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
15 months agoiolog: default to good return
Jens Axboe [Fri, 27 Apr 2018 04:45:04 +0000 (22:45 -0600)]
iolog: default to good return

We should only return failure if we attempted to load a log
and failed.

Fixes: b153f94a22d8 ("iolog/blktrace: boolean conversion")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoiolog/blktrace: boolean conversion
Jens Axboe [Fri, 27 Apr 2018 04:41:03 +0000 (22:41 -0600)]
iolog/blktrace: boolean conversion

They don't return error numbers, so switch to true/false for good/bad
return.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: make sure to account SYNC/TRIM at load time
Jens Axboe [Fri, 27 Apr 2018 04:19:23 +0000 (22:19 -0600)]
blktrace: make sure to account SYNC/TRIM at load time

We currently only check for reads or writes, but it's OK to have
a workload that doesn't contain reads or writes. Only error if
we found no reads/writes/trims/syncs at all.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoUpdate documentation for 'replay_skip'
Jens Axboe [Thu, 26 Apr 2018 21:50:10 +0000 (15:50 -0600)]
Update documentation for 'replay_skip'

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoMakefile: ensure we kill all object files
Jens Axboe [Thu, 26 Apr 2018 21:46:08 +0000 (15:46 -0600)]
Makefile: ensure we kill all object files

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: add 'reply_skip' option
Jens Axboe [Thu, 26 Apr 2018 21:27:05 +0000 (15:27 -0600)]
blktrace: add 'reply_skip' option

Allows the user to skip certain IO types. Useful for replaying
workloads with trim on a device that doesn't support it, for
instance. Or just replaying parts of a workload (eg just writes,
or just reads).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: kill zero sized write test
Jens Axboe [Thu, 26 Apr 2018 21:10:42 +0000 (15:10 -0600)]
blktrace: kill zero sized write test

Turns out they were flushes. Get rid of the check and turn it into
an assert.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: handle flush/sync replay
Jens Axboe [Thu, 26 Apr 2018 21:01:40 +0000 (15:01 -0600)]
blktrace: handle flush/sync replay

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: change barrier to a flush
Jens Axboe [Thu, 26 Apr 2018 20:52:03 +0000 (14:52 -0600)]
blktrace: change barrier to a flush

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoiolog: always use calloc() and always init both lists
Jens Axboe [Thu, 26 Apr 2018 20:44:18 +0000 (14:44 -0600)]
iolog: always use calloc() and always init both lists

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: ignore 0 byte writes
Jens Axboe [Thu, 26 Apr 2018 19:23:10 +0000 (13:23 -0600)]
blktrace: ignore 0 byte writes

They cause fio to exit replay, but then retry since we didn't meet
our IO limit. When we retry, we attempt to close files that the
trace already closed, and we crash.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: don't re-clear ipo
Jens Axboe [Thu, 26 Apr 2018 19:22:15 +0000 (13:22 -0600)]
blktrace: don't re-clear ipo

We overwrite the init of the list entry.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agot/*: missing statics
Jens Axboe [Tue, 24 Apr 2018 20:03:02 +0000 (14:03 -0600)]
t/*: missing statics

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoengines/libaio: don't sleep for 0 reap return for 0 event check
Jens Axboe [Tue, 24 Apr 2018 14:00:23 +0000 (08:00 -0600)]
engines/libaio: don't sleep for 0 reap return for 0 event check

If we have set iodepth_batch_complete_min to 0, then we should
not sleep for completion checking. If we do sleep, reduce the
sleep to 10usec, 100usec is a lot on modern devices.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoconfigure: use proper CONFIG_ prefix for asprintf/vasprintf
Jens Axboe [Sat, 21 Apr 2018 01:13:31 +0000 (19:13 -0600)]
configure: use proper CONFIG_ prefix for asprintf/vasprintf

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoengines: fixup fio_q_status style violations
Jens Axboe [Fri, 20 Apr 2018 15:46:19 +0000 (09:46 -0600)]
engines: fixup fio_q_status style violations

Commit d3b07186b1d4 broke lines in an unconventional manner, and
not consistently. Fix those up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Fri, 20 Apr 2018 15:37:18 +0000 (09:37 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Introduce enum fio_q_status
  Remove dead code from fio_io_sync()
  gfapi: Make fio_gf_queue() set the I/O unit error status instead of returning -EINVAL
  Change return type of td_io_commit() into void
  Simplify num2str()
  Introduce enum n2s_unit
  Declare stat_calc_lat_nu() static

16 months agoIntroduce enum fio_q_status
Bart Van Assche [Tue, 17 Apr 2018 17:04:15 +0000 (10:04 -0700)]
Introduce enum fio_q_status

Introduce a new enumeration type fio_q_status. This patch does not
change any functionality but makes it clear which variables and
return values represent a queuing status value. Bump FIO_IOOPS_VERSION
because this patch changes the I/O engine API.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoRemove dead code from fio_io_sync()
Bart Van Assche [Wed, 18 Apr 2018 20:06:59 +0000 (13:06 -0700)]
Remove dead code from fio_io_sync()

Due to the previous patch it is now guaranteed that the I/O engine
queue function returns a FIO_Q_* value. These values are >= 0.
Hence remove the code that depends on the queue function returning
a negative value.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agogfapi: Make fio_gf_queue() set the I/O unit error status instead of returning -EINVAL
Bart Van Assche [Wed, 18 Apr 2018 15:30:25 +0000 (08:30 -0700)]
gfapi: Make fio_gf_queue() set the I/O unit error status instead of returning -EINVAL

This makes the gfapi I/O engine more uniform with other I/O engines.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoChange return type of td_io_commit() into void
Bart Van Assche [Tue, 17 Apr 2018 17:05:56 +0000 (10:05 -0700)]
Change return type of td_io_commit() into void

Since td_io_commit() always returns 0, change its return type from
int into void. This patch does not change any functionality.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoSimplify num2str()
Bart Van Assche [Wed, 4 Apr 2018 22:14:18 +0000 (15:14 -0700)]
Simplify num2str()

Instead of translating the N2S_* constants into an array index inside
num2str(), make the N2S_* constants identical to the array index used
inside num2str(). This patch does not change the behavior of num2str().

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoIntroduce enum n2s_unit
Bart Van Assche [Wed, 4 Apr 2018 21:36:40 +0000 (14:36 -0700)]
Introduce enum n2s_unit

This patch does not change any functionality but makes num2str() slightly
easier to read.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoDeclare stat_calc_lat_nu() static
Bart Van Assche [Tue, 17 Apr 2018 23:08:54 +0000 (16:08 -0700)]
Declare stat_calc_lat_nu() static

This was suggested by the sparse static analyzer.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoRemove verifysort/verifysort_nr from documentation
Jens Axboe [Wed, 18 Apr 2018 16:52:00 +0000 (10:52 -0600)]
Remove verifysort/verifysort_nr from documentation

They have been deprecated.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoiolog: update stale comment
Jens Axboe [Wed, 18 Apr 2018 14:12:07 +0000 (08:12 -0600)]
iolog: update stale comment

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoDeprecate verifysort and verifysort_nr
Jens Axboe [Wed, 18 Apr 2018 03:50:55 +0000 (21:50 -0600)]
Deprecate verifysort and verifysort_nr

It was an optimization to read back verifies in a sorted order,
for rotational storage. But I don't think the option makes much
sense, and I've never heard of anyone using it. Mark it as
deprecated, and always verify in the same order that IO was
written.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoparse: add support for soft deprecated options
Jens Axboe [Wed, 18 Apr 2018 03:49:51 +0000 (21:49 -0600)]
parse: add support for soft deprecated options

The currently deprecated option types will fail parsing. But we
can also have options where we don't care, log an error for those
but continue executing the job.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoinit: ensure that read/write use the same random seed for verify
Jens Axboe [Tue, 17 Apr 2018 23:48:43 +0000 (17:48 -0600)]
init: ensure that read/write use the same random seed for verify

If we are verifying, then reads need to use the same blockrange
random trim as writes. Otherwise doing a deferred verify of
a write job may not yield the same block size ranges, causing
verify to fail.

Example of failed job:

$ cat job.fio
[global]
ioengine=sync
verify=md5
bsrange=1k-16k
rw=randwrite
randseed=50
size=10m
filename=foo

[write]
rw=randwrite
do_verify=0
write_iolog=wlog

[read]
rw=randread
do_verify=1
verify_fatal=1
verify_dump=1
write_iolog=rlog

$ fio --section=write job.fio
$ fio --section=read job.fio

Fixes: https://github.com/axboe/fio/issues/582
Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoMerge branch 'nvml-to-pmdk' of https://github.com/sscargal/fio
Jens Axboe [Tue, 17 Apr 2018 01:40:46 +0000 (19:40 -0600)]
Merge branch 'nvml-to-pmdk' of https://github.com/sscargal/fio

* 'nvml-to-pmdk' of https://github.com/sscargal/fio:
  NVML references renamed to PMDK

16 months agoNVML references renamed to PMDK
sscargal [Mon, 16 Apr 2018 22:47:47 +0000 (16:47 -0600)]
NVML references renamed to PMDK

16 months agoFio 3.6 fio-3.6
Jens Axboe [Mon, 16 Apr 2018 15:22:37 +0000 (09:22 -0600)]
Fio 3.6

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agocconv: add conversion for 'replay_time_scale'
Jens Axboe [Sat, 14 Apr 2018 22:26:17 +0000 (16:26 -0600)]
cconv: add conversion for 'replay_time_scale'

This was forgotten in the previous commit.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoAdd 'replay_time_scale' option
Jens Axboe [Sat, 14 Apr 2018 22:23:33 +0000 (16:23 -0600)]
Add 'replay_time_scale' option

This allows the user to scale the replay speed of an IO trace. It
defaults to 100, meaning run at 100% the original rate. If set to
50, fio will replay at 50% the original IO rate. If set to 400,
fio will replay at 4x the original rate. And so on.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoiolog: fix issue with replay rate
Jens Axboe [Sat, 14 Apr 2018 21:22:10 +0000 (15:22 -0600)]
iolog: fix issue with replay rate

We attempt to keep the replay rate accurate, but we mistakenly use
the 'delay' value which we have decremented along the way. Store
the original and use that for the stall detection.

This fixes replays going quicker than they should.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoMerge branch 'proc_group' of https://github.com/sitsofe/fio
Jens Axboe [Fri, 13 Apr 2018 23:25:35 +0000 (17:25 -0600)]
Merge branch 'proc_group' of https://github.com/sitsofe/fio

* 'proc_group' of https://github.com/sitsofe/fio:
  doc: add cpus_allowed reference to log_compression_cpus
  appveyor: make 32 bit build target XP + minor fixes
  configure/Makefile: make Cygwin force less
  doc: add Windows processor group behaviour and Windows target option
  windows: target Windows 7 and add support for more than 64 CPUs
  windows: prepare for Windows build split
  windows: update EULA

16 months agoos/os-dragonfly: sync with header file changes in upstream
Tomohiro Kusumi [Tue, 10 Apr 2018 15:35:32 +0000 (00:35 +0900)]
os/os-dragonfly: sync with header file changes in upstream

IOCTLTRIM was moved and renamed to a different header file,
so conditionally include previous or current header depending on
__DragonFly_version.

Some details in below commit messages.
In short, they have had IOCTLTRIM in a wrong header, and they
decided to fix/change it since fio was the only application using
IOCTLTRIM within their userspace package collections.

<sys/ioctl_compat.h>: Add a reminder comment about IOCTLTRIM.
http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/7226e3530be4df743a5cfe61f602b0c4bd436ee9

Move the IOCTLTRIM ioctl to a better header and rename it to DAIOCTRIM.
http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/32506cfa691f80faa47486d6f94a98ae12387eeb

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agostat: remove dead 'nr_uninit' assignment
Jens Axboe [Mon, 9 Apr 2018 14:10:40 +0000 (08:10 -0600)]
stat: remove dead 'nr_uninit' assignment

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoinit: fix memory leak in error handling
Jens Axboe [Mon, 9 Apr 2018 14:07:26 +0000 (08:07 -0600)]
init: fix memory leak in error handling

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agosteadystate: check for division by zero in mean calculation
Jens Axboe [Sun, 8 Apr 2018 21:54:26 +0000 (15:54 -0600)]
steadystate: check for division by zero in mean calculation

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoeta: fix dead variable assignments
Jens Axboe [Sun, 8 Apr 2018 21:50:45 +0000 (15:50 -0600)]
eta: fix dead variable assignments

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoparse: fix dead 'org' assignment
Jens Axboe [Sun, 8 Apr 2018 21:48:35 +0000 (15:48 -0600)]
parse: fix dead 'org' assignment

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agofilesetup: fix dead assignment of 'ret'
Jens Axboe [Sun, 8 Apr 2018 21:47:55 +0000 (15:47 -0600)]
filesetup: fix dead assignment of 'ret'

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoclient: fix bad shadowing of 'ret'
Jens Axboe [Sun, 8 Apr 2018 21:46:17 +0000 (15:46 -0600)]
client: fix bad shadowing of 'ret'

Different scopes for ret, introduce 'wrote' to check the
return value of write(2).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoserver: fix dead assignment of variable
Jens Axboe [Sun, 8 Apr 2018 21:46:07 +0000 (15:46 -0600)]
server: fix dead assignment of variable

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoaxmap: use calloc() for level alloc
Jens Axboe [Sun, 8 Apr 2018 21:42:50 +0000 (15:42 -0600)]
axmap: use calloc() for level alloc

We rely on it being set to NULL for error handling.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoRemove binject engine
Jens Axboe [Sun, 8 Apr 2018 21:42:20 +0000 (15:42 -0600)]
Remove binject engine

It was an experiment and never submitted upstream, let's just
drop it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Fri, 6 Apr 2018 23:59:20 +0000 (17:59 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Fix floating point option range formatting

16 months agoFix floating point option range formatting
Bart Van Assche [Fri, 6 Apr 2018 22:49:44 +0000 (15:49 -0700)]
Fix floating point option range formatting

Ensure that floating point option ranges are properly formatted if
only one of the two boundaries is specified. A few examples of how
option ranges are formatted with this patch applied:

               range: max=100.000000
               range: min=0.000000, max=100.000000
               range: min=0.000000

Reported-by: Sitsofe Wheeler <sitsofe@gmail.com>
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoFix return value checking of fread() in iolog.c
Rebecca Cran [Wed, 4 Apr 2018 23:18:42 +0000 (17:18 -0600)]
Fix return value checking of fread() in iolog.c

According to http://pubs.opengroup.org/onlinepubs/7908799/xsh/fread.html
fread() returns a size_t, not ssize_t, and returns a value of 0 on
both eof and an error. Therefore, check both feof() and ferror() to
determine whether the call succeeded.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Wed, 4 Apr 2018 21:36:17 +0000 (15:36 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Ensure that .minfp and .maxfp are respected for FIO_OPT_FLOAT_LIST
  option parsing: Mark arguments that are not modified as 'const'
  parse.h: Remove a superfluous cast
  Only populate the write buffer if necessary
  Rename TD_F_VER_NONE into TD_F_DO_VERIFY
  engines/sg: Make I/O error messages more informative

16 months agoEnsure that .minfp and .maxfp are respected for FIO_OPT_FLOAT_LIST
Bart Van Assche [Tue, 3 Apr 2018 18:23:11 +0000 (11:23 -0700)]
Ensure that .minfp and .maxfp are respected for FIO_OPT_FLOAT_LIST

option_init() overwrites the .minfp and .maxfp members defined in
fio_options[] which is wrong. Remove the code that overwrites these two
members and change the convention for no FIO_OPT_FLOAT_LIST limits from
(minfp == DBL_MIN && maxfp == DBL_MAX) into (minfp != 0 || maxfp != 0).

Fixes: 833491908a1a ("stats: Add a function to report completion latency percentiles")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agooption parsing: Mark arguments that are not modified as 'const'
Bart Van Assche [Tue, 3 Apr 2018 20:04:02 +0000 (13:04 -0700)]
option parsing: Mark arguments that are not modified as 'const'

This patch does not modify any functionality but makes the option
parsing code slightly easier to read by marking arguments that are
not modified explicitly as 'const'. A new function has been added,
find_option_c(), that behaves identically to find_option(). Only
the function signature differs between these two functions.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoparse.h: Remove a superfluous cast
Bart Van Assche [Tue, 3 Apr 2018 20:04:25 +0000 (13:04 -0700)]
parse.h: Remove a superfluous cast

Adding an offset to a void pointer has the same effect as adding an
offset to a char pointer. Hence removing the (char *) cast from
td_var() does not change its behavior.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoOnly populate the write buffer if necessary
Bart Van Assche [Mon, 26 Mar 2018 16:00:28 +0000 (09:00 -0700)]
Only populate the write buffer if necessary

This patch moves the populate_verify_io_u() call from inside
get_io_u() into all its callers except do_dry_run() and thereby skips
write buffer population for dry runs. This patch does not change the
behavior of fio but is necessary because the ZBC patch will insert
code in fill_io_u() after the get_io_u() call that may modify the
write offset.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoRename TD_F_VER_NONE into TD_F_DO_VERIFY
Bart Van Assche [Mon, 26 Mar 2018 15:50:09 +0000 (08:50 -0700)]
Rename TD_F_VER_NONE into TD_F_DO_VERIFY

Rename TD_F_VER_NONE into TD_F_DO_VERIFY to make it clear that this flag means
that data verification has to be performed. See also commit d72be5454c8c ("Cache
layout improvements").

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoengines/sg: Make I/O error messages more informative
Bart Van Assche [Wed, 4 Apr 2018 20:45:54 +0000 (13:45 -0700)]
engines/sg: Make I/O error messages more informative

Ensure that if an I/O error is encountered that not only the error
number but also a textual error message and the engine name are
reported. Without this patch, SG error messages look like this:

fio: pid=12803, err=5/

With this patch applied SG I/O error messages change into:

fio: pid=12803, err=5/file:engines/sg.c:290, func=fio_sgio_doio, error=Input/output error

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoMerge branch 'fixbug_glfs' of https://github.com/simon-rock/fio
Jens Axboe [Tue, 3 Apr 2018 14:19:38 +0000 (08:19 -0600)]
Merge branch 'fixbug_glfs' of https://github.com/simon-rock/fio

* 'fixbug_glfs' of https://github.com/simon-rock/fio:
  glusterfs: always allocate io_u->engine_data

16 months agoglusterfs: always allocate io_u->engine_data
simon [Tue, 3 Apr 2018 12:16:36 +0000 (20:16 +0800)]
glusterfs: always allocate io_u->engine_data

When the iodepth is greater then 1 some io_us in fio_gf_io_u_init() can
be fooled into thinking their .engine_data has already been set when it
hasn't. This happens because .index and .engine_data are part of the
same union and .index can be set to non-zero values:

static int init_io_u(struct thread_data *td)
...
        for (i = 0; i < max_units; i++) {
            ...
            io_u->index = i;
            ...
            if (td->io_ops->io_u_init) {
                int ret = td->io_ops->io_u_init(td, io_u);

which means io_u->engine_data != 0 in fio_gf_io_u_init().

Fix the issue by removing the initial check of io_u->engine_data.

Signed-off-by: Simon Gao <simon29rock@gmail.com>
16 months agoMerge branch 'aarch64-crc32c' of https://github.com/sitsofe/fio
Jens Axboe [Fri, 30 Mar 2018 16:16:27 +0000 (10:16 -0600)]
Merge branch 'aarch64-crc32c' of https://github.com/sitsofe/fio

* 'aarch64-crc32c' of https://github.com/sitsofe/fio:
  Minor style changes
  aarch64: refactor HW ARM CRC32c detection

16 months agoMinor style changes
Sitsofe Wheeler [Wed, 28 Mar 2018 15:09:22 +0000 (16:09 +0100)]
Minor style changes

Change some #endif comments to exactly match the starting #ifdef and
in crc files and make the second posixaio configure test more
descriptive.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
16 months agoaarch64: refactor HW ARM CRC32c detection
Sitsofe Wheeler [Wed, 28 Mar 2018 14:49:14 +0000 (15:49 +0100)]
aarch64: refactor HW ARM CRC32c detection

- Tweak how fio does the configure probe for hardware accelerated ARM
  CRC32c
- Move the Linux specific runtime detection into os-linux.h making it
  cleaner to implement runtime probing for OSes other than Linux in the
  future
- Make the ARCH_HAVE_CRC_CRYPTO define go into the config-host.* files
  the same way as most other configure defines
- Set/use only the ARCH_HAVE_CRC_CRYPTO define rather than also
  creating/using ARCH_HAVE_ARM64_CRC_CRYPTO

This commit should silence the "Fails to compile: error: sys/auxv.h file
not found" compilation message that was seen on FreeBSD 12/arm64 (but
note a future commit will be required to implement runtime probing and
actually enable the accelerated path on that platform).

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
16 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Thu, 29 Mar 2018 16:02:25 +0000 (10:02 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  switch_ioscheduler(): only remove the last character if it's a newline
  Make it clear to Coverity that the tmp buffer in switch_ioscheduler() is \0-terminated

16 months agoswitch_ioscheduler(): only remove the last character if it's a newline
Bart Van Assche [Thu, 29 Mar 2018 15:30:38 +0000 (08:30 -0700)]
switch_ioscheduler(): only remove the last character if it's a newline

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoMake it clear to Coverity that the tmp buffer in switch_ioscheduler() is \0-terminated
Bart Van Assche [Thu, 29 Mar 2018 15:24:23 +0000 (08:24 -0700)]
Make it clear to Coverity that the tmp buffer in switch_ioscheduler() is \0-terminated

The 'tmp' buffer used by switch_ioscheduler() will be '\0'-terminated
because the I/O scheduler name read from sysfs has much fewer characters
than the buffer size (256). Make it clear to Coverity that it is
guaranteed that the 'tmp' buffer is '\0'-terminated. Additionally, fix
the order of the fread() arguments. This patch fixes Coverity ID #24132.

Fixes: b44b9e45cf38 ("Clear sysfs path before reading current ioscheduler from sysfs")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agodoc: add cpus_allowed reference to log_compression_cpus
Sitsofe Wheeler [Tue, 27 Mar 2018 16:45:33 +0000 (17:45 +0100)]
doc: add cpus_allowed reference to log_compression_cpus

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
16 months agoappveyor: make 32 bit build target XP + minor fixes
Sitsofe Wheeler [Mon, 26 Mar 2018 15:06:46 +0000 (16:06 +0100)]
appveyor: make 32 bit build target XP + minor fixes

To get some coverage of old XP paths make the 32 bit CI build target XP but
leave the 64 bit CI build targeting Windows 7.

Also fix Cygwin output being redirected into a file called NULL (rather
than going to NUL) while we're here.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
16 months agoconfigure/Makefile: make Cygwin force less
Sitsofe Wheeler [Mon, 26 Mar 2018 14:50:03 +0000 (15:50 +0100)]
configure/Makefile: make Cygwin force less

Continue the work started in commit
ca205a752c3d6ebe7de74a3dfe81808e48a502e3 ("configure: Make Cygwin take
regular configure path") by removing the forceful setting of flags that
are automatically detected anyway.

Even CONFIG_64BIT_LLP64 is unneeded because despite their names
CONFIG_32BIT/CONFIG_64BIT are only used to control the size's of words
and the regular configure path correctly detects CONFIG_32BIT for 64 bit
Windows even though it's LLP64.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
16 months agodoc: add Windows processor group behaviour and Windows target option
Sitsofe Wheeler [Tue, 27 Mar 2018 16:28:27 +0000 (17:28 +0100)]
doc: add Windows processor group behaviour and Windows target option

- Add information on targeting the build against different versions of
  Windows to the README
- Stop listing -i686 packages in the 64 bit section of the README and
  add a note to about them to the 32 bit build section
- Rejig cpus_allowed to be before cpumask in the documentation to make
  people more likely to read that section first
- Make it clear that CPUs specified in the cpus_allowed section are
  indexed from 0 (and not 1)
- Add information about how processor group awareness when setting CPUs
  is only available when using a build targeted at Windows 7 (because
  only then do we have the processor group APIs available to us) and how
  selecting a CPU sets the processor group too (because we concatanate
  the CPUs from all the processor groups together)

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
16 months agowindows: target Windows 7 and add support for more than 64 CPUs
Sitsofe Wheeler [Sun, 25 Mar 2018 19:56:07 +0000 (20:56 +0100)]
windows: target Windows 7 and add support for more than 64 CPUs

Introduce support for targeting the build to either Windows XP or
Windows 7 (the default). When targeting Windows 7 this allows us to use
the process group APIs which are required are required to support more
than 64 CPUs.

These changes mean if you want a Windows binary that supports versions
of Windows below Windows 7/Windows Server 2008 R2 you will need to
explicitly use the --target-win-ver=xp configure option at build time.
Such builds will lack features (such as the ability to access CPUs
beyond those in fio's default process group) that are dependent on
recent Windows APIs.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
16 months agowindows: prepare for Windows build split
Sitsofe Wheeler [Sun, 25 Mar 2018 19:15:29 +0000 (20:15 +0100)]
windows: prepare for Windows build split

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
16 months agowindows: update EULA
Sitsofe Wheeler [Sun, 25 Mar 2018 17:50:53 +0000 (18:50 +0100)]
windows: update EULA

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
17 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Fri, 23 Mar 2018 15:58:51 +0000 (09:58 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  smalloc: Remove Valgrind instrumentation

17 months agosmalloc: Remove Valgrind instrumentation
Jens Axboe [Thu, 22 Mar 2018 17:29:25 +0000 (11:29 -0600)]
smalloc: Remove Valgrind instrumentation

Since it can happen that memory allocated with smalloc() is freed by
a child process and since Valgrind's memory tracking is per process
the smalloc()/sfree() instrumentation leads to incorrect Valgrind
reports. Hence remove that instrumentation again.

Fixes: 0ffccc21fcd6 ("Improve Valgrind instrumentation of memory allocations")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
17 months agoserver: use scalloc() for sk_out allocation
Jens Axboe [Thu, 22 Mar 2018 17:29:25 +0000 (11:29 -0600)]
server: use scalloc() for sk_out allocation

This informs valgrind that it's properly initialized.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
17 months agoMerge branch 'gcc' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 22 Mar 2018 17:02:52 +0000 (11:02 -0600)]
Merge branch 'gcc' of https://github.com/sitsofe/fio

* 'gcc' of https://github.com/sitsofe/fio:
  compiler: set minimum compiler version to GCC 4.1.0

17 months agocompiler: set minimum compiler version to GCC 4.1.0
Sitsofe Wheeler [Wed, 21 Mar 2018 22:35:21 +0000 (22:35 +0000)]
compiler: set minimum compiler version to GCC 4.1.0

fio hasn't compiled under GCC 3 for a while due to unguarded calls to
GCC's __sync intrinsics in critical paths. Since those intrinsics didn't
come along until GCC 4.1.0 change fio to explicitly error out on older
compilers.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
17 months agosmalloc: oom cleanups
Jens Axboe [Thu, 22 Mar 2018 14:34:48 +0000 (08:34 -0600)]
smalloc: oom cleanups

Signed-off-by: Jens Axboe <axboe@kernel.dk>
17 months agosg: fix sign extension
Jens Axboe [Thu, 22 Mar 2018 02:09:36 +0000 (20:09 -0600)]
sg: fix sign extension

Signed-off-by: Jens Axboe <axboe@kernel.dk>
17 months agoserver: fix error handling for shared memory handling
Jens Axboe [Thu, 22 Mar 2018 01:19:58 +0000 (19:19 -0600)]
server: fix error handling for shared memory handling

Signed-off-by: Jens Axboe <axboe@kernel.dk>
17 months agoMerge branch 'include_refactor' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 22 Mar 2018 01:01:33 +0000 (19:01 -0600)]
Merge branch 'include_refactor' of https://github.com/sitsofe/fio

* 'include_refactor' of https://github.com/sitsofe/fio:
  Add include-what-you-use pragmas
  oslib: make str* compat functions more uniform
  Use POSIX path for poll.h and fcntl.h headers
  Refactor #includes and headers

17 months agoserver: handle shared mem pool allocation failures
Jens Axboe [Thu, 22 Mar 2018 00:58:22 +0000 (18:58 -0600)]
server: handle shared mem pool allocation failures

Signed-off-by: Jens Axboe <axboe@kernel.dk>
17 months agoAdd include-what-you-use pragmas
Sitsofe Wheeler [Tue, 20 Mar 2018 15:18:43 +0000 (15:18 +0000)]
Add include-what-you-use pragmas

Add IWYU pragmas to improve its analysis.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
17 months agooslib: make str* compat functions more uniform
Sitsofe Wheeler [Tue, 20 Mar 2018 10:46:32 +0000 (10:46 +0000)]
oslib: make str* compat functions more uniform

- Ensure there are CONFIG_ guards in the headers and .c files because
  sometime utilities include oslib headers directly and reference oslib
  .c files in the Makefile
- Ensure all the oslib/str*.h files have redefinition guards
- Shuffle some #include lines from the .h files to the .c ones
- Ensure the <name>.c files have #include "<name>.h" in them so the
  compiler can check the declaration and definition match

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
17 months agoUse POSIX path for poll.h and fcntl.h headers
Sitsofe Wheeler [Mon, 19 Mar 2018 05:43:41 +0000 (05:43 +0000)]
Use POSIX path for poll.h and fcntl.h headers

musl complains when using include paths like <sys/poll.h> rather than
POSIX's <poll.h> so change the poll.h include path and update the
windows poll.h compatibility shim to match.

Allegedly sys/poll.h was needed for GLIBCs that were older than 2.3
according to
https://www.winehq.org/pipermail/wine-patches/2014-June/132877.html but
I can still see that path in an ancient Red Hat 5.0
glibc-devel-2.0.7-32.i386.rpm ...

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
17 months agoRefactor #includes and headers
Sitsofe Wheeler [Mon, 19 Mar 2018 05:33:13 +0000 (05:33 +0000)]
Refactor #includes and headers

- Try and remove unneeded #include lines
- Try and add #include lines that would allow the files to be built in a
  more standalone manner

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
17 months agoserver: process STOP/QUIT commands out-of-line
Jens Axboe [Wed, 21 Mar 2018 21:48:17 +0000 (15:48 -0600)]
server: process STOP/QUIT commands out-of-line

At the end of a job, the stats entries are queued up for
out-of-line transmission, but we immediately queue a client
STOP+QUIT command right after that. If the command handling
loop doesn't wakeup and process the stats commands before we
process the STOP+QUIT inline, then we lose the stats output
from those commands. The result is that the client won't see
the job output from that job, or the "All clients" output.

We should impose proper ordering for all commands, but this
is a quick fix to ensure that we don't hit the above mentioned
but. For other commands we don't care about ordering between
inline and out-of-line commands, so it's likely good enough
to carry us forward.

Reported-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
17 months agoMerge branch 'asprintf' of https://github.com/bvanassche/fio
Jens Axboe [Wed, 21 Mar 2018 14:26:28 +0000 (08:26 -0600)]
Merge branch 'asprintf' of https://github.com/bvanassche/fio

* 'asprintf' of https://github.com/bvanassche/fio:
  verify: Simplify dump_buf()
  log: Modify the implementation such that it uses asprintf()
  Add an asprintf() implementation

17 months agodebug: remove extra parens
Jens Axboe [Wed, 21 Mar 2018 14:23:50 +0000 (08:23 -0600)]
debug: remove extra parens

Signed-off-by: Jens Axboe <axboe@kernel.dk>
17 months agooptgroup: move debug code into function
Jens Axboe [Tue, 20 Mar 2018 17:19:19 +0000 (11:19 -0600)]
optgroup: move debug code into function

The current global construct breaks on some compilers:

optgroup.c:208: error: expected identifier or '(' before 'do'
optgroup.c:208: error: expected identifier or '(' before 'while'

Reported-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Fixes: 96344ff00349 ("optgroup: add check for optgroup bit numbers being within range")
Signed-off-by: Jens Axboe <axboe@kernel.dk>