fio.git
13 months agoFio 3.8 fio-3.8
Jens Axboe [Mon, 23 Jul 2018 14:55:32 +0000 (08:55 -0600)]
Fio 3.8

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agogfio: cleanup includes
Jens Axboe [Mon, 23 Jul 2018 14:29:14 +0000 (08:29 -0600)]
gfio: cleanup includes

Use stdlib.h, not malloc.h. And ensure that we pull in the
right header for basename(3).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoaxmap: optimize ulog64 usage in axmap_handler()
Jens Axboe [Thu, 12 Jul 2018 14:33:14 +0000 (08:33 -0600)]
axmap: optimize ulog64 usage in axmap_handler()

We can do this incrementally, no need to compute from scratch
for every iteration.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agot/axmap: add longer overlap test case
Jens Axboe [Wed, 11 Jul 2018 21:32:31 +0000 (15:32 -0600)]
t/axmap: add longer overlap test case

Also fix bad test stop termination.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agot/axmap: add regression case for recent overlap failure case
Jens Axboe [Wed, 11 Jul 2018 21:17:32 +0000 (15:17 -0600)]
t/axmap: add regression case for recent overlap failure case

Catches the issue that bd71edcedc3d fixes.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoaxmap: fix continued sequential bit setting
Jens Axboe [Wed, 11 Jul 2018 21:16:58 +0000 (15:16 -0600)]
axmap: fix continued sequential bit setting

We need to remember to clear ->set_bits if we don't set new bits,
or the caller will think we re-set the previously set bits.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoaxmap: a few more cleanups
Jens Axboe [Wed, 11 Jul 2018 19:59:32 +0000 (13:59 -0600)]
axmap: a few more cleanups

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoaxmap: remove unused 'data' argument to topdown handler
Jens Axboe [Wed, 11 Jul 2018 19:48:32 +0000 (13:48 -0600)]
axmap: remove unused 'data' argument to topdown handler

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agofio: should_fsync() returns bool
Jens Axboe [Wed, 11 Jul 2018 19:26:19 +0000 (13:26 -0600)]
fio: should_fsync() returns bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoaxmap: code cleanups
Jens Axboe [Wed, 11 Jul 2018 18:25:43 +0000 (12:25 -0600)]
axmap: code cleanups

- Move the const bit masks to the start of the file
- Get rid of a useless variable in axmap_handler_topdown()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agot/axmap: add zero return overlap cases
Jens Axboe [Wed, 11 Jul 2018 15:11:15 +0000 (09:11 -0600)]
t/axmap: add zero return overlap cases

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoaxmap: clean up 'no bits to set' case
Jens Axboe [Wed, 11 Jul 2018 14:34:06 +0000 (08:34 -0600)]
axmap: clean up 'no bits to set' case

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agot/axmap: don't print 'pass' on failure
Jens Axboe [Wed, 11 Jul 2018 14:32:37 +0000 (08:32 -0600)]
t/axmap: don't print 'pass' on failure

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agot/axmap: a few more overlap cases
Jens Axboe [Wed, 11 Jul 2018 14:29:22 +0000 (08:29 -0600)]
t/axmap: a few more overlap cases

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agot/axmap: clean up overlap tests
Jens Axboe [Wed, 11 Jul 2018 03:51:16 +0000 (21:51 -0600)]
t/axmap: clean up overlap tests

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoMakefile: lib/axmap no longer needs hweight
Jens Axboe [Tue, 10 Jul 2018 23:02:25 +0000 (17:02 -0600)]
Makefile: lib/axmap no longer needs hweight

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agot/axmap: add a few more overlap test cases
Jens Axboe [Tue, 10 Jul 2018 23:02:03 +0000 (17:02 -0600)]
t/axmap: add a few more overlap test cases

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoaxmap: ensure that overlaps are handled strictly sequential
Jens Axboe [Tue, 10 Jul 2018 22:52:16 +0000 (16:52 -0600)]
axmap: ensure that overlaps are handled strictly sequential

We must terminate at the first overlapping bit, we can't
just mask off non-sequential ranges.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agot/axmap: add overlap test cases
Jens Axboe [Tue, 10 Jul 2018 22:51:24 +0000 (16:51 -0600)]
t/axmap: add overlap test cases

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoio_u: ensure we generate the full length of block sizes
Jens Axboe [Tue, 10 Jul 2018 21:10:17 +0000 (15:10 -0600)]
io_u: ensure we generate the full length of block sizes

Since we round down, we can miss the last entry. This ensures that
if we do:

bsrange=4k-16k

we actually get an even split of 4,8,12,16k ios.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoio_u: fix negative offset due to wrap
Jens Axboe [Tue, 10 Jul 2018 19:06:38 +0000 (13:06 -0600)]
io_u: fix negative offset due to wrap

If we do wrap, the math is off and we end up wrapping a 64-bit
value. Instead reset to the initial offset.

Reported-by: Bart Van Assche <Bart.VanAssche@wdc.com>
Fixes: 4c8be5b1569f ("Fix bug with zone and zone skipping and io_limit")
Fixes: 224b3093cc21 ("Fix zoning issue with seq-io and randommap issue")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoblktrace: just ignore zero byte traces
Jens Axboe [Wed, 4 Jul 2018 20:59:18 +0000 (14:59 -0600)]
blktrace: just ignore zero byte traces

Apparently they do happen. Log a warning (once) and just discard
them.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoFix compilation without cgroups
Jens Axboe [Fri, 29 Jun 2018 14:00:29 +0000 (08:00 -0600)]
Fix compilation without cgroups

Fixes: c3dc516ad75e ("fio: work with cgroup2 as well")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agofio: add job_runtime to the thread json output
Josef Bacik [Fri, 29 Jun 2018 13:49:03 +0000 (09:49 -0400)]
fio: add job_runtime to the thread json output

I discovered that the 'elapsed' value of each job is the overall time
the whole fio run took, not the per-job elapsed time, despite it being
in the per-job section of the json output.  Rather than break everybody
by fixing this, just add a 'job_runtime' value to the job section and
include the time the thread ran in milliseconds.  This is handy for jobs
that want to compare runtime of two threads in different cgroups.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agofio: work with cgroup2 as well
Josef Bacik [Fri, 29 Jun 2018 13:33:34 +0000 (09:33 -0400)]
fio: work with cgroup2 as well

The current code only works with cgroupv1, but as long as you don't want
to set blkio.weight or anything we can use the 'cgroup=' setting with
cgroupv2 easily with cgroupv1, we just need to write the thread pids to
a different file for cgroup2.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoMerge branch 'doc-norandommap' of https://github.com/larrystevenwise/fio
Jens Axboe [Thu, 21 Jun 2018 17:22:41 +0000 (11:22 -0600)]
Merge branch 'doc-norandommap' of https://github.com/larrystevenwise/fio

* 'doc-norandommap' of https://github.com/larrystevenwise/fio:
  doc: add text about possibly verify errors with norandommap

14 months agodoc: add text about possibly verify errors with norandommap
Steve Wise [Thu, 21 Jun 2018 17:16:50 +0000 (10:16 -0700)]
doc: add text about possibly verify errors with norandommap

In both the norandommap option section, and the verify option section,
add text warning that norandommap can cause data verification errors
when using async io engines and io depths > 1.

Signed-off-by: Steve Wise <swise@opengridcomputing.com>
14 months agoMerge branch 'wip-single-glfs-instance' of https://github.com/zhanghuan/fio
Jens Axboe [Wed, 20 Jun 2018 14:14:08 +0000 (08:14 -0600)]
Merge branch 'wip-single-glfs-instance' of https://github.com/zhanghuan/fio

* 'wip-single-glfs-instance' of https://github.com/zhanghuan/fio:
  glusterfs: capable to test with one single glfs instance

14 months agoglusterfs: capable to test with one single glfs instance
Zhang Huan [Wed, 6 Jun 2018 03:52:14 +0000 (11:52 +0800)]
glusterfs: capable to test with one single glfs instance

Current multi-thread test creates one glfs instance per job. However,
there is requirement to run all jobs on one single instance, and that
all jobs would share the same set of underlying working threads.

Add a new option "single-instance" to control whether to create one
global instance or one instance per job. For testing with mutilple
gluster volumes, use host name and volume name to filter out same
volume, one specific volume will have one instance.

Signed-off-by: Zhang Huan <zhanghuan@open-fs.com>
14 months agoMerge branch 'readonly-trim' of https://github.com/vincentkfu/fio
Jens Axboe [Mon, 18 Jun 2018 19:58:26 +0000 (13:58 -0600)]
Merge branch 'readonly-trim' of https://github.com/vincentkfu/fio

* 'readonly-trim' of https://github.com/vincentkfu/fio:
  testing: add test script for readonly parameter
  doc: improve readonly option description
  options: check for conflict between trims and readonly option
  init: abort write and trim jobs when --readonly option is present
  init: ensure that fatal errors in fixup_options are always propogated to caller
  filesetup: make trim jobs respect --readonly during file open
  fio.h: also check trim operations in fio_ro_check

14 months agotesting: add test script for readonly parameter
Vincent Fu [Mon, 18 Jun 2018 16:55:21 +0000 (10:55 -0600)]
testing: add test script for readonly parameter

Add a shell script and jobs for testing the readonly parameter

14 months agodoc: improve readonly option description
Vincent Fu [Mon, 18 Jun 2018 15:48:17 +0000 (09:48 -0600)]
doc: improve readonly option description

Make explicit that --readonly also prevents trim workloads
Drop the mention of the check in the I/O engine core because
fio_ro_check() is run even without the readonly option

14 months agooptions: check for conflict between trims and readonly option
Vincent Fu [Fri, 15 Jun 2018 21:52:43 +0000 (15:52 -0600)]
options: check for conflict between trims and readonly option

When processing the readwrite option, make sure that trim operations
are not requested if readonly is set.

14 months agoinit: abort write and trim jobs when --readonly option is present
Vincent Fu [Fri, 15 Jun 2018 21:17:38 +0000 (15:17 -0600)]
init: abort write and trim jobs when --readonly option is present

Check for conflicts between --readonly and the job's data direction
in fixup_options(). rw_verify() in options.c only detects this
conflict if --readonly precedes --readwrite on the command line.

14 months agoinit: ensure that fatal errors in fixup_options are always propogated to caller
Vincent Fu [Fri, 15 Jun 2018 21:11:10 +0000 (15:11 -0600)]
init: ensure that fatal errors in fixup_options are always propogated to caller

When warnings_fatal = 0, fatal problems detected in fixup_options
may not always be propogated back to the caller. This patch ensures
that the caller will always be notified about fatal problems.

14 months agofilesetup: make trim jobs respect --readonly during file open
Vincent Fu [Fri, 15 Jun 2018 21:05:28 +0000 (15:05 -0600)]
filesetup: make trim jobs respect --readonly during file open

Open files for trim workloads with the O_RDWR access mode only when
the --readonly option is absent.

14 months agofio.h: also check trim operations in fio_ro_check
Vincent Fu [Fri, 15 Jun 2018 21:00:25 +0000 (15:00 -0600)]
fio.h: also check trim operations in fio_ro_check

IO engines call fio_ro_check to ensure that the data direction
set in the thread data structure permits writes before proceeding
with a write io_u. This patch makes fio_ro_check ensure that the
data direction also permits trims before proceeding with a trim
io_u.

14 months agoclient: close dup'ed descriptor if fdopen() fails
Jens Axboe [Fri, 15 Jun 2018 15:18:04 +0000 (09:18 -0600)]
client: close dup'ed descriptor if fdopen() fails

The saga continues... Ensure that we close the dup'ed descriptor,
if we give up due to fdopen() failure.

Reported-by: Bart Van Assche <bart.vanassche@wdc.com>
Fixes: ec9e13345f3f ("client: check return of dup(2)")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoclient: check return of dup(2)
Jens Axboe [Fri, 15 Jun 2018 15:10:48 +0000 (09:10 -0600)]
client: check return of dup(2)

If dup(2) fails, we can't pass it to fdopen.

Fixes: b4f5e72f1383 ("client: parse env variables before sending job-file contents to server")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoclient: parse env variables before sending job-file contents to server
Tomohiro Kusumi [Fri, 15 Jun 2018 14:58:22 +0000 (08:58 -0600)]
client: parse env variables before sending job-file contents to server

Fixes "fio environment var bug".
https://www.spinics.net/lists/fio/msg07093.html

Add read_ini_data() to parse and expand env variables within job-file
before sending to server. By doing this, clients can control parameters
embedded within the job-file, without server side having to set them
separately.

Reported-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Minor fixups from Jens.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agorand: make randX_upto() do the end value increment
Jens Axboe [Tue, 12 Jun 2018 17:43:29 +0000 (11:43 -0600)]
rand: make randX_upto() do the end value increment

We should not do it in the caller, the functions that need
fixing are really rand32/64_upto() instead.

Also move a (now) misplaced comment.

Fixes: c6fc6d2ab2c2 ("rand: ensure that rand_between() can reach max value")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agorand: ensure that rand_between() can reach max value
Jens Axboe [Tue, 12 Jun 2018 17:37:16 +0000 (11:37 -0600)]
rand: ensure that rand_between() can reach max value

We need to add 1, otherwise the maximum generated value will
be end -1. The API is both inclusive.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoinit: use o-> instead of td->o
Jens Axboe [Tue, 12 Jun 2018 14:51:07 +0000 (08:51 -0600)]
init: use o-> instead of td->o

We already have the options pointer, just use it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoinit: kill get_rand_start_delay()
Jens Axboe [Tue, 12 Jun 2018 14:47:08 +0000 (08:47 -0600)]
init: kill get_rand_start_delay()

It's now useless, just call rand_between().

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agorand: cleanup rand_between() and helpers
Jens Axboe [Tue, 12 Jun 2018 14:42:49 +0000 (08:42 -0600)]
rand: cleanup rand_between() and helpers

Make the 32/64-bit helper just return a random number up to a certain
value, and let the generic helper handle the range part.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agorand: add rand64_between()
Jens Axboe [Tue, 12 Jun 2018 14:10:47 +0000 (08:10 -0600)]
rand: add rand64_between()

For some reason we only had the 32-bit variant, and there's
a use case for the 64-bit version. Add that, and add a
32/64 agnostic helper that can be called.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoFix start delay being the same across threads
Jens Axboe [Tue, 12 Jun 2018 02:02:10 +0000 (20:02 -0600)]
Fix start delay being the same across threads

Two issues here:

1) We copy the start_delay into the original thread, which ends up
   offsetting job 2..N off the thread 1 start delay
2) We don't initialize the start delay random seed prior to
   generating the offsets

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agostat: fix --bandwidth-log segfault
Igor Konopko [Fri, 8 Jun 2018 08:26:25 +0000 (10:26 +0200)]
stat: fix --bandwidth-log segfault

Since bandwidth log has no corresponding thread data
and pending struct, we cannot access any fields inside it.

Signed-off-by: Igor Konopko <igor.j.konopko@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoidle-prof: Fix segment fault issue when run with '--idle-prof' and multiple output...
Friendy.Su@sony.com [Fri, 8 Jun 2018 02:27:25 +0000 (20:27 -0600)]
idle-prof: Fix segment fault issue when run with '--idle-prof' and multiple output format normal,json

Original issue:

./fio --output-format=normal,json --idle-prof=system jobfile.job

job1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.7-19-ga43f
Starting 1 process
Segmentation fault (core dumped)

Fix solution:

do fio_idle_prof_cleanup() after all formats of output shown.

Signed-off-by: friendy-su <friendy.su@sony.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoFix variable shadowing
Jens Axboe [Tue, 5 Jun 2018 19:44:47 +0000 (13:44 -0600)]
Fix variable shadowing

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoiolog: remove 'td' from trim_io_piece()
Jens Axboe [Tue, 5 Jun 2018 19:34:47 +0000 (13:34 -0600)]
iolog: remove 'td' from trim_io_piece()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoclient: remove 'cmd' argument to stop handler
Jens Axboe [Tue, 5 Jun 2018 19:33:50 +0000 (13:33 -0600)]
client: remove 'cmd' argument to stop handler

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agooptions: remove unused function parameters
Jens Axboe [Tue, 5 Jun 2018 19:30:57 +0000 (13:30 -0600)]
options: remove unused function parameters

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agolog: fix signedness issue
Jens Axboe [Tue, 5 Jun 2018 19:37:56 +0000 (13:37 -0600)]
log: fix signedness issue

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agogettime: remove 'is_thread' variable to local clock init
Jens Axboe [Tue, 5 Jun 2018 19:19:33 +0000 (13:19 -0600)]
gettime: remove 'is_thread' variable to local clock init

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agox86: don't need 'level' passed to amd/intel init functions
Jens Axboe [Tue, 5 Jun 2018 19:18:02 +0000 (13:18 -0600)]
x86: don't need 'level' passed to amd/intel init functions

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoCleanup should_check_rate()
Jens Axboe [Tue, 5 Jun 2018 19:21:36 +0000 (13:21 -0600)]
Cleanup should_check_rate()

Don't need to check direction for __should_check_rate(), so
clean up the logic.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agofallocate: use 'offset' parameter
Jens Axboe [Tue, 5 Jun 2018 19:15:27 +0000 (13:15 -0600)]
fallocate: use 'offset' parameter

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoMerge branch 'make-nowarn-snprintf-call-va-end' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 5 Jun 2018 17:20:45 +0000 (11:20 -0600)]
Merge branch 'make-nowarn-snprintf-call-va-end' of https://github.com/bvanassche/fio

* 'make-nowarn-snprintf-call-va-end' of https://github.com/bvanassche/fio:
  Make nowarn_snprintf() call va_end()

14 months agoMake nowarn_snprintf() call va_end()
Bart Van Assche [Tue, 5 Jun 2018 17:15:18 +0000 (19:15 +0200)]
Make nowarn_snprintf() call va_end()

This patch suppresses the following Coverity complaint:

CID 175867:  API usage errors  (VARARGS)
    va_end was not called for "args".

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
14 months agoMove nowarn_snprintf.h to lib/
Jens Axboe [Tue, 5 Jun 2018 17:11:03 +0000 (11:11 -0600)]
Move nowarn_snprintf.h to lib/

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoMerge branch 'suppress-gcc-8-warnings' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 5 Jun 2018 16:44:51 +0000 (10:44 -0600)]
Merge branch 'suppress-gcc-8-warnings' of https://github.com/bvanassche/fio

* 'suppress-gcc-8-warnings' of https://github.com/bvanassche/fio:
  Suppress gcc 8 compiler warnings

14 months agoSuppress gcc 8 compiler warnings
Bart Van Assche [Tue, 5 Jun 2018 16:16:30 +0000 (18:16 +0200)]
Suppress gcc 8 compiler warnings

Introduce a new function nowarn_snprintf() that behaves identically
to snprintf() except that it does not make gcc complain about potential
truncation of the output buffer.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
14 months agoMerge branch 'appveyor' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 5 Jun 2018 14:32:20 +0000 (08:32 -0600)]
Merge branch 'appveyor' of https://github.com/bvanassche/fio

* 'appveyor' of https://github.com/bvanassche/fio:
  Rename appveyor.yml into .appveyor.yml

14 months agoMerge branch 'remove-show-run-stats' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 5 Jun 2018 14:29:21 +0000 (08:29 -0600)]
Merge branch 'remove-show-run-stats' of https://github.com/bvanassche/fio

* 'remove-show-run-stats' of https://github.com/bvanassche/fio:
  Remove show_run_stats() because it has no callers

14 months agoRename appveyor.yml into .appveyor.yml
Bart Van Assche [Tue, 5 Jun 2018 08:27:23 +0000 (10:27 +0200)]
Rename appveyor.yml into .appveyor.yml

Make the appveyor.yml file less visible because it is not a fio source
code file.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
14 months agoRemove show_run_stats() because it has no callers
Bart Van Assche [Tue, 5 Jun 2018 07:08:50 +0000 (09:08 +0200)]
Remove show_run_stats() because it has no callers

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
14 months agoFix issue with rate_process=poisson and ramp time
Jens Axboe [Mon, 4 Jun 2018 19:51:47 +0000 (13:51 -0600)]
Fix issue with rate_process=poisson and ramp time

Fix from shimrot

Fixes: https://github.com/axboe/fio/issues/596
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoFio 3.7 fio-3.7
Jens Axboe [Fri, 1 Jun 2018 19:21:56 +0000 (13:21 -0600)]
Fio 3.7

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoio_u: ensure to invalidate cache on time_based random reads
Jens Axboe [Thu, 31 May 2018 15:05:59 +0000 (09:05 -0600)]
io_u: ensure to invalidate cache on time_based random reads

We need to do this with the file reset and retrieval of a
new offset, not of it fails.

Fixes: 0bcf41cdc22df ("io_u: re-invalidate cache when looping around without file open/close")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoFix typo in bssplit documentation
Jens Axboe [Fri, 25 May 2018 14:14:50 +0000 (08:14 -0600)]
Fix typo in bssplit documentation

One of the examples uses a ',' where it should be a ':' to
behave like described.

Reported-by: Konstantin Ruvinsky <konstantin.ruvinsky@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agomake fio scripts python3-ready (part 2)
Bill O'Donnell [Tue, 22 May 2018 17:49:09 +0000 (12:49 -0500)]
make fio scripts python3-ready (part 2)

NOTE: this patch (part 2) is in addition to commit 5eac3b00238b
(this patch was missed on that submit/commit (part 1)).

Many distributions are moving to python3 by default.  This is
another patch to make the python scripts in fio python3-ready.

Conversion was facilitated with automated tools.

The modified scripts pass rudimentary testing when run under
python2.7 as well as python3.

Signed-off-by: Bill O'Donnell <billodo@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoMerge branch 'sg-verify2' of https://github.com/vincentkfu/fio
Jens Axboe [Fri, 18 May 2018 19:34:36 +0000 (13:34 -0600)]
Merge branch 'sg-verify2' of https://github.com/vincentkfu/fio

* 'sg-verify2' of https://github.com/vincentkfu/fio:
  docs: add documentation for sg ioengine WRITE SAME, WRITE AND VERIFY command support
  engines/sg: add support for WRITE AND VERIFY, WRITE SAME

15 months agodocs: add documentation for sg ioengine WRITE SAME, WRITE AND VERIFY command support
Vincent Fu [Thu, 17 May 2018 14:08:03 +0000 (10:08 -0400)]
docs: add documentation for sg ioengine WRITE SAME, WRITE AND VERIFY command support

15 months agoengines/sg: add support for WRITE AND VERIFY, WRITE SAME
Vincent Fu [Wed, 16 May 2018 23:17:47 +0000 (19:17 -0400)]
engines/sg: add support for WRITE AND VERIFY, WRITE SAME

Add an sg ioengine option to treat write IO as either WRITE AND VERIFY
or WRITE SAME commands

The new ioengine option sg_write_mode can take 3 values:

write (default)
  WRITE opcodes are issued as usual
verify
  WRITE AND VERIFY opcodes are issued
same
  WRITE SAME opcodes are issued

WRITE AND VERIFY opcodes are issued with the BYTCHK bit set to 0. This
directs the device to carry out a medium verification with no data
comparison.

WRITE SAME transfers a single block of data to the device and writes
this same block of data to a contiguous sequence of LBAs beginning at
the specified offset.

fio's block size parameter specifies the amount of data written with
each command. However, the amount of data actually transferred to the
device is equal to the device's block (sector) size. For a device with
512 byte sectors, blocksize=8k will write 16 sectors with each command.
fio will still generate 8k of data for each command but only the first
512 bytes will be used and transferred to the device.

15 months agomake fio scripts python3-ready
Bill O'Donnell [Fri, 4 May 2018 19:43:40 +0000 (14:43 -0500)]
make fio scripts python3-ready

Many distributions are moving to python3 by default.  Here's
an attempt to make the python scripts in fio python3-ready.

Conversion was facilitated with automated tools. A few areas
were hand fixed: remove superfluous parentheses introduced by
2to3 converter in print function calls, shebang modifications
to use environment variable for python version, and byte-string
decode correction in steadystate_tests.py following 2to3
conversion.

The modified scripts pass rudimentary testing when run under
python2.7 as well as python3.

Signed-off-by: Bill O'Donnell <billodo@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoMerge branch 'master' of https://github.com/majianpeng/fio
Jens Axboe [Tue, 15 May 2018 01:40:32 +0000 (19:40 -0600)]
Merge branch 'master' of https://github.com/majianpeng/fio

* 'master' of https://github.com/majianpeng/fio:
  This partly revert 97bb54c9606c(add __load_ioengine() to separate ioengine loading from td context)

15 months agoThis partly revert 97bb54c9606c(add __load_ioengine() to separate ioengine loading...
Jianpeng Ma [Mon, 14 May 2018 16:14:01 +0000 (00:14 +0800)]
This partly revert 97bb54c9606c(add __load_ioengine() to separate ioengine loading from td context)

In commit 97bb54c9606c, it change load_ioengine to __load_ioengine in
func fio_show_ioengine_help. The different between load_ioengine and
__load_ioengine is load_ioengine include __load_ioengine and if
__load_ioengine failed it will do dlopen_ioengine to find ioengine.
For external ioengine, if no dlopen_ioengine it can't find, etc
libfio_ceph_objectstore.so in Ceph project.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
15 months agoiolog: default to good return
Jens Axboe [Fri, 27 Apr 2018 04:45:04 +0000 (22:45 -0600)]
iolog: default to good return

We should only return failure if we attempted to load a log
and failed.

Fixes: b153f94a22d8 ("iolog/blktrace: boolean conversion")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoiolog/blktrace: boolean conversion
Jens Axboe [Fri, 27 Apr 2018 04:41:03 +0000 (22:41 -0600)]
iolog/blktrace: boolean conversion

They don't return error numbers, so switch to true/false for good/bad
return.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: make sure to account SYNC/TRIM at load time
Jens Axboe [Fri, 27 Apr 2018 04:19:23 +0000 (22:19 -0600)]
blktrace: make sure to account SYNC/TRIM at load time

We currently only check for reads or writes, but it's OK to have
a workload that doesn't contain reads or writes. Only error if
we found no reads/writes/trims/syncs at all.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoUpdate documentation for 'replay_skip'
Jens Axboe [Thu, 26 Apr 2018 21:50:10 +0000 (15:50 -0600)]
Update documentation for 'replay_skip'

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoMakefile: ensure we kill all object files
Jens Axboe [Thu, 26 Apr 2018 21:46:08 +0000 (15:46 -0600)]
Makefile: ensure we kill all object files

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: add 'reply_skip' option
Jens Axboe [Thu, 26 Apr 2018 21:27:05 +0000 (15:27 -0600)]
blktrace: add 'reply_skip' option

Allows the user to skip certain IO types. Useful for replaying
workloads with trim on a device that doesn't support it, for
instance. Or just replaying parts of a workload (eg just writes,
or just reads).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: kill zero sized write test
Jens Axboe [Thu, 26 Apr 2018 21:10:42 +0000 (15:10 -0600)]
blktrace: kill zero sized write test

Turns out they were flushes. Get rid of the check and turn it into
an assert.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: handle flush/sync replay
Jens Axboe [Thu, 26 Apr 2018 21:01:40 +0000 (15:01 -0600)]
blktrace: handle flush/sync replay

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: change barrier to a flush
Jens Axboe [Thu, 26 Apr 2018 20:52:03 +0000 (14:52 -0600)]
blktrace: change barrier to a flush

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoiolog: always use calloc() and always init both lists
Jens Axboe [Thu, 26 Apr 2018 20:44:18 +0000 (14:44 -0600)]
iolog: always use calloc() and always init both lists

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: ignore 0 byte writes
Jens Axboe [Thu, 26 Apr 2018 19:23:10 +0000 (13:23 -0600)]
blktrace: ignore 0 byte writes

They cause fio to exit replay, but then retry since we didn't meet
our IO limit. When we retry, we attempt to close files that the
trace already closed, and we crash.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoblktrace: don't re-clear ipo
Jens Axboe [Thu, 26 Apr 2018 19:22:15 +0000 (13:22 -0600)]
blktrace: don't re-clear ipo

We overwrite the init of the list entry.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agot/*: missing statics
Jens Axboe [Tue, 24 Apr 2018 20:03:02 +0000 (14:03 -0600)]
t/*: missing statics

Signed-off-by: Jens Axboe <axboe@kernel.dk>
15 months agoengines/libaio: don't sleep for 0 reap return for 0 event check
Jens Axboe [Tue, 24 Apr 2018 14:00:23 +0000 (08:00 -0600)]
engines/libaio: don't sleep for 0 reap return for 0 event check

If we have set iodepth_batch_complete_min to 0, then we should
not sleep for completion checking. If we do sleep, reduce the
sleep to 10usec, 100usec is a lot on modern devices.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoconfigure: use proper CONFIG_ prefix for asprintf/vasprintf
Jens Axboe [Sat, 21 Apr 2018 01:13:31 +0000 (19:13 -0600)]
configure: use proper CONFIG_ prefix for asprintf/vasprintf

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoengines: fixup fio_q_status style violations
Jens Axboe [Fri, 20 Apr 2018 15:46:19 +0000 (09:46 -0600)]
engines: fixup fio_q_status style violations

Commit d3b07186b1d4 broke lines in an unconventional manner, and
not consistently. Fix those up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
16 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Fri, 20 Apr 2018 15:37:18 +0000 (09:37 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  Introduce enum fio_q_status
  Remove dead code from fio_io_sync()
  gfapi: Make fio_gf_queue() set the I/O unit error status instead of returning -EINVAL
  Change return type of td_io_commit() into void
  Simplify num2str()
  Introduce enum n2s_unit
  Declare stat_calc_lat_nu() static

16 months agoIntroduce enum fio_q_status
Bart Van Assche [Tue, 17 Apr 2018 17:04:15 +0000 (10:04 -0700)]
Introduce enum fio_q_status

Introduce a new enumeration type fio_q_status. This patch does not
change any functionality but makes it clear which variables and
return values represent a queuing status value. Bump FIO_IOOPS_VERSION
because this patch changes the I/O engine API.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoRemove dead code from fio_io_sync()
Bart Van Assche [Wed, 18 Apr 2018 20:06:59 +0000 (13:06 -0700)]
Remove dead code from fio_io_sync()

Due to the previous patch it is now guaranteed that the I/O engine
queue function returns a FIO_Q_* value. These values are >= 0.
Hence remove the code that depends on the queue function returning
a negative value.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agogfapi: Make fio_gf_queue() set the I/O unit error status instead of returning -EINVAL
Bart Van Assche [Wed, 18 Apr 2018 15:30:25 +0000 (08:30 -0700)]
gfapi: Make fio_gf_queue() set the I/O unit error status instead of returning -EINVAL

This makes the gfapi I/O engine more uniform with other I/O engines.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoChange return type of td_io_commit() into void
Bart Van Assche [Tue, 17 Apr 2018 17:05:56 +0000 (10:05 -0700)]
Change return type of td_io_commit() into void

Since td_io_commit() always returns 0, change its return type from
int into void. This patch does not change any functionality.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
16 months agoSimplify num2str()
Bart Van Assche [Wed, 4 Apr 2018 22:14:18 +0000 (15:14 -0700)]
Simplify num2str()

Instead of translating the N2S_* constants into an array index inside
num2str(), make the N2S_* constants identical to the array index used
inside num2str(). This patch does not change the behavior of num2str().

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>