fio.git
13 months agotarget: fixes latency-probe
Jens Axboe [Wed, 24 Oct 2018 21:43:29 +0000 (15:43 -0600)]
target: fixes

1) Use parent for should_account(), if we have a parent
2) Only sum step stats if src has them, to prevent
   overwriting destination stats in the parent.
3) Pretty up the normal output a bit

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoUpdate iodepth_mode sample job file
Jens Axboe [Wed, 24 Oct 2018 11:03:13 +0000 (05:03 -0600)]
Update iodepth_mode sample job file

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoAdd support for latency probing over an interval of load
Jens Axboe [Wed, 24 Oct 2018 11:01:43 +0000 (05:01 -0600)]
Add support for latency probing over an interval of load

Provide a way to easily run a latency probe on the device.
You define a job with peak parameters, and then probe settings
for generating iops/latency numbers based on that workload.
The latter looks something like this:

iodepth_mode=stepped:10-130/10,5/10

which has the format of:

low_percentage-high_percentage/step,ramp_time/run_time

The above would probe from 10% of peak performance to 130%,
in steps of 10%. For each step, it would run a 5 second ramp,
then do 10 seconds of testing. For percentages <= 100%,
fio will limit the IOPS. For percentages above, it'll ramp up
the queue depth. For each section run, it'll look the avg
completion latency associated with that queue depth / iops
setting.

Has normal output (which sucks), and json output. Still
experimenting, not final form yet.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoio_u: move trim error notification out-of-line
Jens Axboe [Wed, 24 Oct 2018 00:05:41 +0000 (18:05 -0600)]
io_u: move trim error notification out-of-line

Also kill dead code, we never touch BLOCK_STATE_WRITTEN as
it was guarded by a check for ddir == DDIR_TRIM.

The latter should probably be double checked...

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agostat: use helper for IO direction name
Jens Axboe [Tue, 23 Oct 2018 20:43:36 +0000 (14:43 -0600)]
stat: use helper for IO direction name

Better than redefining the same thing in two locations...

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agodocs: serialize_overlap=1 with io_submit_mode=offload no longer requires threads
Vincent Fu [Wed, 17 Oct 2018 16:03:25 +0000 (12:03 -0400)]
docs: serialize_overlap=1 with io_submit_mode=offload no longer requires threads

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agofio: document locking for overlap checking in offload mode
Vincent Fu [Wed, 17 Oct 2018 16:03:24 +0000 (12:03 -0400)]
fio: document locking for overlap checking in offload mode

Overlap checking in offload submission mode uses locks in an unusual
manner. Add some comments in the code to clarify how locks are used.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agofio: enable cross-thread overlap checking with processes
Vincent Fu [Wed, 17 Oct 2018 16:03:23 +0000 (12:03 -0400)]
fio: enable cross-thread overlap checking with processes

Overlap checking with io_submit_mode=offload requires relevant jobs to
access each other's io_u's and io_u_all members. This patch modifies the
fio_memalign and io_u_queue helpers to include an indicator signifying
whether operations should use the shared memory pool. When fio is
carrying out cross-job overlap checking in offload submission mode,
these variables will be allocated from shared memory so that processes
can be used and threads will no longer be required.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agofio: add function to check for serialize_overlap with offload submission
Vincent Fu [Wed, 17 Oct 2018 16:03:22 +0000 (12:03 -0400)]
fio: add function to check for serialize_overlap with offload submission

In multiple places fio needs to check whether it is carrying out overlap
checking in offload submission mode. Make this check a function to
improve code readability.

Reviewed-by: Sebastien Boisvert <sboisvert@gydle.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoMerge branch 'samples-colnames' of https://github.com/parallel-fs-utils/fio
Jens Axboe [Fri, 19 Oct 2018 17:07:49 +0000 (11:07 -0600)]
Merge branch 'samples-colnames' of https://github.com/parallel-fs-utils/fio

* 'samples-colnames' of https://github.com/parallel-fs-utils/fio:
  add rsp. time samples as column 2, use meaningful pctile names

13 months agofilesetup: fix whitespace damage introduced by previous patch
Jens Axboe [Fri, 19 Oct 2018 17:07:06 +0000 (11:07 -0600)]
filesetup: fix whitespace damage introduced by previous patch

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoMerge branch 'fix-init-read-iolog' of https://github.com/aclamk/fio
Jens Axboe [Fri, 19 Oct 2018 17:06:51 +0000 (11:06 -0600)]
Merge branch 'fix-init-read-iolog' of https://github.com/aclamk/fio

* 'fix-init-read-iolog' of https://github.com/aclamk/fio:
  iolog: Fix problem with setup() not invoked when read_iolog is used.

13 months agofio: reset more counters when ramp time has elapsed
Vincent Fu [Thu, 18 Oct 2018 20:42:54 +0000 (16:42 -0400)]
fio: reset more counters when ramp time has elapsed

Currently,

fio --name=test --ramp_time=2 --time_based=1 --runtime=2s --size=1M

will produce an output line like

  IO depths    : 1=184.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

184.7% appears because the denominator used in the IO depth calculation
is reset at the conclusion of ramp_time but the numerator is not.
Latency and other IO depth statistics also include data from IOs
completed during ramp time. This patch resets these counters when ramp
time is over.

Tested-by: Sebastien Boisvert <sboisvert@gydle.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoadd rsp. time samples as column 2, use meaningful pctile names
Ben England [Tue, 16 Oct 2018 16:19:39 +0000 (12:19 -0400)]
add rsp. time samples as column 2, use meaningful pctile names

percentiles on a small sample count are suspect.
0th percentile is min, 50th is median, 100th is max.
sample count also lets you know which fio jobs
are contributing the most.

13 months agoMerge branch 'offload-serialize-overlap2' of https://github.com/vincentkfu/fio
Jens Axboe [Mon, 15 Oct 2018 20:09:18 +0000 (14:09 -0600)]
Merge branch 'offload-serialize-overlap2' of https://github.com/vincentkfu/fio

* 'offload-serialize-overlap2' of https://github.com/vincentkfu/fio:
  rate-submit: remove code that can never be executed
  docs: enable serialize_overlap with io_submit_mode=offload
  fio: enable overlap checking with offload submission
  init: loosen serialize_overlap restrictions

13 months agorate-submit: remove code that can never be executed
Vincent Fu [Mon, 15 Oct 2018 14:44:47 +0000 (10:44 -0400)]
rate-submit: remove code that can never be executed

The do-while loop above can only be exited when ret != FIO_Q_BUSY. So
the if (ret == FIO_Q_BUSY) branch will never be taken.

13 months agodocs: enable serialize_overlap with io_submit_mode=offload
Vincent Fu [Mon, 1 Oct 2018 18:42:28 +0000 (14:42 -0400)]
docs: enable serialize_overlap with io_submit_mode=offload

Note that it is now possible to enable overlap checking in offload
submission mode and that threads are required.

13 months agofio: enable overlap checking with offload submission
Vincent Fu [Mon, 1 Oct 2018 18:22:10 +0000 (14:22 -0400)]
fio: enable overlap checking with offload submission

Enable use of the serialize_overlap option with
io_submit_mode=offload. Threads with both of these options enabled will
check each other's in-flight IOs for overlap. If overlap is found,
submission of the new IO will be held back until the overlapping IO has
completed.

Threads must be used when fio is run in this mode because jobs need to
access each other's io_u's. io_u's are not allocated from the shared memory
area.

This fixes "fio serialize across jobs"
https://www.spinics.net/lists/fio/msg07133.html

Tested-by: Jeff Furlong <jeff.furlong@wdc.com>
13 months agoinit: loosen serialize_overlap restrictions
Vincent Fu [Thu, 11 Oct 2018 15:32:21 +0000 (11:32 -0400)]
init: loosen serialize_overlap restrictions

When io_submit_mode=offload, allow overlap checking including for jobs
where no data is being changed and when the iodepth is one. These cases
can occur when checking for overlap across different jobs.

13 months agooptions: kill 'use_os_rand'
Jens Axboe [Mon, 8 Oct 2018 19:43:23 +0000 (13:43 -0600)]
options: kill 'use_os_rand'

It's been deprecated for a very long time, it's time to just kill
it off.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 months agoRemove old OS dependent (unused) random code
Jens Axboe [Mon, 8 Oct 2018 19:36:43 +0000 (13:36 -0600)]
Remove old OS dependent (unused) random code

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoMerge branch 'patch-1' of https://github.com/joaomlneto/fio
Jens Axboe [Sat, 6 Oct 2018 01:11:19 +0000 (19:11 -0600)]
Merge branch 'patch-1' of https://github.com/joaomlneto/fio

* 'patch-1' of https://github.com/joaomlneto/fio:
  Be careful when defining `MPOL_LOCAL`

14 months agoBe careful when defining `MPOL_LOCAL`
João Neto [Fri, 5 Oct 2018 23:34:03 +0000 (00:34 +0100)]
Be careful when defining `MPOL_LOCAL`

`MPOL_LOCAL` will be defined in `numaif.h` in the future. Also, `MPOL_MAX` will be changed to match the kernel values.

14 months agoAdd cross-stripe intel sample verify job
Jens Axboe [Fri, 5 Oct 2018 15:08:58 +0000 (09:08 -0600)]
Add cross-stripe intel sample verify job

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoBool conversions
Jens Axboe [Thu, 4 Oct 2018 15:07:01 +0000 (09:07 -0600)]
Bool conversions

Just various low hanging fruits in terms of local or global
variables that are only 0/1, switch them to boolean.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoFio 3.11 fio-3.11
Jens Axboe [Wed, 3 Oct 2018 18:30:40 +0000 (12:30 -0600)]
Fio 3.11

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoiolog: fix leak for unsupported iolog version
Jens Axboe [Wed, 3 Oct 2018 18:29:46 +0000 (12:29 -0600)]
iolog: fix leak for unsupported iolog version

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoupdate replay_align and replay_scale documentation
Dennis Zhou [Wed, 3 Oct 2018 02:14:51 +0000 (22:14 -0400)]
update replay_align and replay_scale documentation

Options replay_align and replay_scale both happen after sector to byte
translation. This can cause the offset scaling to be unaligned. Make a
note that both options operate on the byte level and replay_align should
most likely be used when replay_scale is used.

Signed-off-by: Dennis Zhou <dennis@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoRevert "blktrace: support for non-512b sector sizes"
Dennis Zhou [Wed, 3 Oct 2018 02:14:50 +0000 (22:14 -0400)]
Revert "blktrace: support for non-512b sector sizes"

This reverts commit 26e616d054f238bf6b7ada351e17b72fdfbbb8e0.

An issue appeared when replaying blktraces using fio on non-512b sector
size devices where fio would report the following when replaying a trace
back on the same device.

  fio: io_u error on file /dev/sdb: No space left on device:

This turns out to be due to a match of how blktrace exports trace
information and how fio interprets it. The kernel considers all IO to be
relative to 512b sectors, so the sector offset is always in these units.
However, when replaying fio was adapting to the hardware sector. This
caused offset calculation issues resulting in the above error.

Reverting this keeps fio more in line with expectations, but will force
users who want to have fio run workloads be cognizant of aligning IO
properly when running on specific media to get expected performance.

Reported-by: Vernon Miller <vwm@fb.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoiolog: fix up some style issues
Jens Axboe [Wed, 3 Oct 2018 14:45:19 +0000 (08:45 -0600)]
iolog: fix up some style issues

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoengines/rados: use fio provided file names
Adam Kupczyk [Mon, 1 Oct 2018 10:48:54 +0000 (12:48 +0200)]
engines/rados: use fio provided file names

Use the regular fio generated names as names of objects to operate on.
This is necessary when using multiple, consecutive --read-iolog. Added
check if --pool parameter is set.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoengines/cpu: fix potential overflow in thinktime calculation
Jens Axboe [Sun, 30 Sep 2018 14:41:40 +0000 (08:41 -0600)]
engines/cpu: fix potential overflow in thinktime calculation

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agofix hung fio process with large I/O sizes and verify= option
Jeff Moyer [Fri, 28 Sep 2018 21:27:58 +0000 (17:27 -0400)]
fix hung fio process with large I/O sizes and verify= option

Hi,

If you use a very large I/O size (1G in this instance), and also specify
the verify option, fio will hang, using up 100% cpu, for ever and ever.
The problem is that __fill_random_buf_percentage does math that
overflows the size of an unsigned int.  Fixing that gets fio to
terminate as expected.

Here is the job file which showed the problem:

[global]
ioengine=dev-dax
direct=0
filename=/dev/dax0.0
verify=crc32c
bs=1G
[write]
rw=write
runtime=5
[read]
stonewall
rw=read
runtime=5

Reported-by: sujith_pandel@dell.com
Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agozbd: Avoid duplicating the code for calculating the number of sectors with data
Bart Van Assche [Fri, 28 Sep 2018 21:51:33 +0000 (14:51 -0700)]
zbd: Avoid duplicating the code for calculating the number of sectors with data

Introduce a new function process_swd() to avoid that almost identical code
occurs in check_swd() and zbd_file_reset(). Change the #if 0 / #endif pair
in check_swd() into if (false) to make sure that the compiler verifies the
syntactical correctness of that code.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agozbd: Restore zbd_check_swd()
Bart Van Assche [Fri, 28 Sep 2018 21:51:32 +0000 (14:51 -0700)]
zbd: Restore zbd_check_swd()

A recent patch removed the check_swd() function. Restore it since this function
is useful when debugging fio. Rename the function into zbd_check_swd(). Insert
the calls to these function where no locks are held to avoid triggering a
deadlock. See also commit d60be7d51cbb ("zbd: Remove unused function and
variable").

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agot/zbd/test-zbd-support: Report a test summary when finished
Bart Van Assche [Fri, 28 Sep 2018 21:51:31 +0000 (14:51 -0700)]
t/zbd/test-zbd-support: Report a test summary when finished

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agot/zbd/test-zbd-support: Set fio aux path and forbid file creation
Bart Van Assche [Fri, 28 Sep 2018 21:51:30 +0000 (14:51 -0700)]
t/zbd/test-zbd-support: Set fio aux path and forbid file creation

Set the fio aux path to /tmp to avoid cluttering the directory from which the
test scripts are run. Disallow file creation to make tests fail if the block
device against which these are run does not exist.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agot/zbd/test-zbd-support: Ensure that an assertion failure causes this test to fail
Bart Van Assche [Fri, 28 Sep 2018 21:51:29 +0000 (14:51 -0700)]
t/zbd/test-zbd-support: Ensure that an assertion failure causes this test to fail

Switch to threaded mode to make sure that an assertion failure results in
a non-zero exit status.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agot/zbd/functions: Make fio_reset_count() return 0 if no resets occurred
Bart Van Assche [Fri, 28 Sep 2018 21:51:28 +0000 (14:51 -0700)]
t/zbd/functions: Make fio_reset_count() return 0 if no resets occurred

This patch avoids that certain bash versions report the following complaint:

t/zbd/test-zbd-support: line 69: [: : integer expression expected

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agozbd: Fix incorrect comments
Damien Le Moal [Fri, 28 Sep 2018 05:53:53 +0000 (14:53 +0900)]
zbd: Fix incorrect comments

Commit ee3696bdfd84 ("zbd: Use bytes unit") changed the zone info
structure zone start and zone wp location unit to bytes. Reflect this
change in the fio_zone_info structure comments.

Fixes: ee3696bdfd84 ("zbd: Use bytes unit")
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agogettime: slightly improve CPU clock calibration
Jens Axboe [Thu, 27 Sep 2018 23:38:21 +0000 (17:38 -0600)]
gettime: slightly improve CPU clock calibration

We should not factor the utime_since() runtime into this, so
do the get_cpu_clock() before. In reality this probably won't
make any difference, as we're doing this tight loop for more
than a milisecond.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoserver: be locally vocal about communication issues
Jens Axboe [Thu, 27 Sep 2018 18:26:47 +0000 (12:26 -0600)]
server: be locally vocal about communication issues

If we fail the CRC check or if we have a mismatched client/server,
then we can't reliably transmit anything across the wire. For that
case, in addition to attempting to relay this information across
the wire, also print a local warning.

This fixes a case where we silently exit when the client/server
versions aren't properly matched.

Fixes: https://github.com/axboe/fio/issues/620
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Thu, 27 Sep 2018 15:42:21 +0000 (09:42 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  zbd: Fix zbd_zone_nr()

14 months agozbd: Fix zbd_zone_nr()
Bart Van Assche [Thu, 27 Sep 2018 15:12:19 +0000 (08:12 -0700)]
zbd: Fix zbd_zone_nr()

Instead of returning sizeof(struct fio_zone_info) * (zone number),
return the zone number.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
14 months agozbd: Remove unused function and variable
Damien Le Moal [Wed, 26 Sep 2018 06:42:26 +0000 (15:42 +0900)]
zbd: Remove unused function and variable

check_swd() was defined to nothing. Remove it.
additionally, the variable ret in zbd_reset_zonei() is unused and not
necessary. Remove it too.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoeta: Avoid adjustements to a negative value
Damien Le Moal [Wed, 26 Sep 2018 06:42:10 +0000 (15:42 +0900)]
eta: Avoid adjustements to a negative value

In the case of zonemode=strided job with both zone_size and zone_skip
specified, thread_eta() calculates the upper bound of the number of
zones that will be processed, including skipped bytes between zones
(zone_skip). Adjusting bytes_total (i.e. total_io_size) by substracting
this number of zones times the amount of skipped bytes can result in
a negative value for bytes_total (a large number of bytes) when for
example the job operates on an entire disk with a capacity that is not
divisible exactly by zone_size+zone_skip or if the options --size is
used to limit the I/O range. In such case, use the lower bound of the
number of zones to obtain a better approximation of the job eta.

Additionnally, if --io_size was specified, bytes_total will indicate
this exact value, so adjusting that value for zonemode != none is not
necessary.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoparse: fix negative FIO_OPT_INT too-large check
Jens Axboe [Wed, 26 Sep 2018 02:15:05 +0000 (20:15 -0600)]
parse: fix negative FIO_OPT_INT too-large check

If we have an option, like nice, that has a minval that is negative,
then we need to ensure we cast properly for the check.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoparse: fix minval checking
Jens Axboe [Wed, 26 Sep 2018 01:44:35 +0000 (19:44 -0600)]
parse: fix minval checking

For values that allow negative min values, don't cast to an uint.

Fixes: 5d2788d5b798 ("parse: fix min/max val checking for FIO_OPT_INT")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoAdd regression test for flow/negative option parser breakage
Jens Axboe [Wed, 26 Sep 2018 01:40:59 +0000 (19:40 -0600)]
Add regression test for flow/negative option parser breakage

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoparse: fix min/max val checking for FIO_OPT_INT
Jens Axboe [Wed, 26 Sep 2018 01:24:36 +0000 (19:24 -0600)]
parse: fix min/max val checking for FIO_OPT_INT

We compare an unsigned long long to an int max/min value, this
is going to break for negative numbers, for instance.

Fixes a case where flow=-1 doesn't work properly because of that.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoparse: print option name for out-of-range settings
Jens Axboe [Wed, 26 Sep 2018 01:14:22 +0000 (19:14 -0600)]
parse: print option name for out-of-range settings

Right now we just say we're out of range:

max value out of range: 18446744073709551612 (4294967295 max)

but we don't actually say which option this is. Add that.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoiolog: Fix problem with setup() not invoked when read_iolog is used.
Adam Kupczyk [Mon, 24 Sep 2018 11:29:39 +0000 (13:29 +0200)]
iolog: Fix problem with setup() not invoked when read_iolog is used.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
14 months agostat: print the right percentile variant for json output
Jens Axboe [Sat, 22 Sep 2018 21:02:04 +0000 (15:02 -0600)]
stat: print the right percentile variant for json output

We always show the clat percentiles for the json output, regardless
of whether lat_percentiles or clat_percentiles was defined in the
job. Correct that so we show the right output depending on the
job setting.

Fixes: https://github.com/axboe/fio/issues/691
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoMerge branch 'steadystate-doc' of https://github.com/vincentkfu/fio
Jens Axboe [Fri, 21 Sep 2018 15:50:07 +0000 (09:50 -0600)]
Merge branch 'steadystate-doc' of https://github.com/vincentkfu/fio

* 'steadystate-doc' of https://github.com/vincentkfu/fio:
  docs: clarify usage of steadystate detection feature

14 months agodocs: clarify usage of steadystate detection feature
Vincent Fu [Fri, 21 Sep 2018 15:37:17 +0000 (11:37 -0400)]
docs: clarify usage of steadystate detection feature

Note that most intended uses of the steadystate detection feature
likely also need the --time_based and --runtime options or the
--loops option so that fio does not stop running after accessing
the full size of the device(s) or file(s).

14 months agoengines/http: work-around crash in certain libssl versions
Jens Axboe [Fri, 21 Sep 2018 15:17:38 +0000 (09:17 -0600)]
engines/http: work-around crash in certain libssl versions

Fixes: https://github.com/axboe/fio/issues/684
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoblktrace: fix leak of 'merge_buf'
Jens Axboe [Thu, 20 Sep 2018 20:25:56 +0000 (14:25 -0600)]
blktrace: fix leak of 'merge_buf'

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoblktrace: provide empty merge_blktrace_iologs()
Jens Axboe [Thu, 20 Sep 2018 19:14:11 +0000 (13:14 -0600)]
blktrace: provide empty merge_blktrace_iologs()

For platforms that don't suppor blktrace, we need an empty
helper that just fails it.

Fixes: b9921d1a56d9 ("blktrace: add support to interleave blktrace files")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoblktrace: add option to iterate over a trace multiple times
Dennis Zhou [Thu, 20 Sep 2018 18:08:10 +0000 (14:08 -0400)]
blktrace: add option to iterate over a trace multiple times

Scaling a particular trace may result in different runtimes among the
merging traces. By knowing the approximate length of each trace as a
user, the overall runtime of each can be tuned to line up by letting
certain traces loop multiple times.

First, the last timestamp of a trace is recorded at the end of the first
iteration to denote the length of a trace. This value is then used to
offset subsequent iterations of a trace.

Next, the "--merge_blktrace_iters" option is introduced to let the user
specify the number of times to loop over each specific trace. This is
done by passing a comma separated list that index-wise pairs with the
passed files in "--read_iolog". Iteration counts are introduced as well
as keeping track of the length of each trace.

In an example, given two traces, A and B, each 60s long. If we want to
see the impact of trace A issuing IOs twice as fast, the
--merge_blktrace_scalars="50:100" can be set and then
--merge_blktrace_iters="2:1". This runs trace A at 2x the speed twice for
approximately the same runtime as a single run of trace B.

Signed-off-by: Dennis Zhou <dennis@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoblktrace: add option to scale a trace
Dennis Zhou [Thu, 20 Sep 2018 18:08:09 +0000 (14:08 -0400)]
blktrace: add option to scale a trace

As we explore stacking traces, it is nice to be able to scale a trace to
understand how the traces end up interacting.

This patch adds scaling by letting the user pass in percentages to scale
a trace by. When passed '--merge_blktrace_scalars="100"', the trace is
ran at 100% speed. If passed 50%, this will halve the trace timestamps.
The new option takes in a comma separated list that index-wise pairs
with the passed files in "--read_iolog".

This option differs from "--replay_time_scale" which scales the trace
during runtime and will not change the output unlike this option.

Signed-off-by: Dennis Zhou <dennis@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoblktrace: add support to interleave blktrace files
Dennis Zhou [Thu, 20 Sep 2018 18:08:08 +0000 (14:08 -0400)]
blktrace: add support to interleave blktrace files

Running concurrent workloads via multiple jobs can lead to
nondeterministic results as we are at the scheduler's mercy. While the
actual performance of the workload may vary regardless, this makes the
order of events consistent.

This patch introduces two flags: --merge_blktrace_file=<file> and
--merge-blktrace-only. When the first is specified, files that are ':'
separated in --read_iolog are passed to a merge function wich then
produces a single sorted trace file. This file is then passed on in
place of the sorted list or fio exists if --merge-blktrace-only is
specified.

During merging, events are filtered based on what fio cares about and
the pdu is discarded as well.

Signed-off-by: Dennis Zhou <dennis@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agooptions: rename name string operations for more general use
Dennis Zhou [Thu, 20 Sep 2018 18:08:07 +0000 (14:08 -0400)]
options: rename name string operations for more general use

get_next_name() and get_max_str_idx() are helpers that iterate and split
a string separated by ':'. s/name/str/g to make this more generic which
will be used to parse file paths for blktraces to merge.

Signed-off-by: Dennis Zhou <dennis@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoMerge branch 'epoch-time-hist-logs' of https://github.com/parallel-fs-utils/fio
Jens Axboe [Wed, 19 Sep 2018 15:42:47 +0000 (09:42 -0600)]
Merge branch 'epoch-time-hist-logs' of https://github.com/parallel-fs-utils/fio

* 'epoch-time-hist-logs' of https://github.com/parallel-fs-utils/fio:
  raise exception if test start time can't be estimated
  handle log_unix_epoch=1

14 months agofilesetup: use 64-bit type for file size generation
Jens Axboe [Mon, 17 Sep 2018 19:46:51 +0000 (13:46 -0600)]
filesetup: use 64-bit type for file size generation

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoRandom distribution 32-bit fixes
Jens Axboe [Mon, 17 Sep 2018 17:00:50 +0000 (11:00 -0600)]
Random distribution 32-bit fixes

We calculate and use nranges as an unsigned long, but that can
be 32-bit on some platforms and hence overflow.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agozbd: fix various 32-bit compilation warnings
Jens Axboe [Mon, 17 Sep 2018 14:39:22 +0000 (08:39 -0600)]
zbd: fix various 32-bit compilation warnings

Just dprint() string types.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoaxmap: isset_fn() should use 1ULL, not 1UL
Jens Axboe [Mon, 17 Sep 2018 03:58:26 +0000 (21:58 -0600)]
axmap: isset_fn() should use 1ULL, not 1UL

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agot/axmap: use a 64-bit type (not size_t) for axmap tests
Jens Axboe [Mon, 17 Sep 2018 03:55:45 +0000 (21:55 -0600)]
t/axmap: use a 64-bit type (not size_t) for axmap tests

Now we actually pass on 32-bit...

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agolfsr: use unsigned long long for 64-bit values
Jens Axboe [Mon, 17 Sep 2018 03:52:22 +0000 (21:52 -0600)]
lfsr: use unsigned long long for 64-bit values

1UL is bit 64-bit on Windows builds.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoaxmap: use 64-bit index for the handlers
Jens Axboe [Mon, 17 Sep 2018 03:43:56 +0000 (21:43 -0600)]
axmap: use 64-bit index for the handlers

Most important one here is axmap_handler_topdown(), in which the
first iteration could end up truncating the index calculation.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoaxmap: use 64-bit type for number of bits
Jens Axboe [Mon, 17 Sep 2018 03:30:39 +0000 (21:30 -0600)]
axmap: use 64-bit type for number of bits

For huge devices, we can overflow on a 32-bit build of fio.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoMerge branch 'windows_static' of https://github.com/sitsofe/fio
Jens Axboe [Sun, 16 Sep 2018 16:13:23 +0000 (10:13 -0600)]
Merge branch 'windows_static' of https://github.com/sitsofe/fio

* 'windows_static' of https://github.com/sitsofe/fio:
  build: change where we set -static for Windows

14 months agobuild: change where we set -static for Windows
Sitsofe Wheeler [Sat, 15 Sep 2018 05:58:33 +0000 (06:58 +0100)]
build: change where we set -static for Windows

The Makefile forced static Windows builds by adding -static to the
CFLAGS but this is long after the configure tests had run. This could
lead to problems (such as https://github.com/axboe/fio/issues/683 "Fio
fails to link libcurl on windows (undefined references to
__imp_curl_easy_cleanup)") where the availability of a dynamic version
of library causes a feature to be enabled but when it comes to final
linking time we find that there's no static version of the library
available and fall over.

Fix the above by asking for a static build via the "Forcing some known
good options on Windows" section of the configure file which occurs
prior to configure tests. This also means configure will correctly
display that we are doing a static build.

Fixes https://github.com/axboe/fio/issues/683

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
14 months agoMerge branch 'histo-log-dup-timestamp' of https://github.com/parallel-fs-utils/fio
Jens Axboe [Wed, 12 Sep 2018 20:33:04 +0000 (14:33 -0600)]
Merge branch 'histo-log-dup-timestamp' of https://github.com/parallel-fs-utils/fio

* 'histo-log-dup-timestamp' of https://github.com/parallel-fs-utils/fio:
  filter out records with duplicate timestamps

14 months agofilter out records with duplicate timestamps
Ben England [Wed, 12 Sep 2018 17:14:50 +0000 (13:14 -0400)]
filter out records with duplicate timestamps

fio should not be outputting records with duplicate timestamps
into the histogram log, but because it does do this,
this script was getting divide-by-zero errors.
This workaround will filter out the duplicates,
which allows the script to operate on existing fio logs

14 months agowindows: make win_to_posix_error() more resilient
Jens Axboe [Wed, 12 Sep 2018 14:50:16 +0000 (08:50 -0600)]
windows: make win_to_posix_error() more resilient

If we pass in ERROR_SUCCESS, we should get a zero back.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agowindows: fix the most egregious posix.c style errors
Jens Axboe [Tue, 11 Sep 2018 22:54:39 +0000 (16:54 -0600)]
windows: fix the most egregious posix.c style errors

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agowindows: handle ERROR_NOT_READY
Jens Axboe [Tue, 11 Sep 2018 22:29:25 +0000 (16:29 -0600)]
windows: handle ERROR_NOT_READY

The windows-to-posix error translation doesn't include
ERROR_NOT_READY, add that.

Also add a default case that catches if we don't know what
an error is, and return EIO. That seems much safer than just
returning the Windows error directly, since the caller expects
an errno.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoFio 3.10 fio-3.10
Jens Axboe [Tue, 11 Sep 2018 15:08:07 +0000 (09:08 -0600)]
Fio 3.10

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoAdd regression test for recent offload locking bug
Jens Axboe [Tue, 11 Sep 2018 14:27:41 +0000 (08:27 -0600)]
Add regression test for recent offload locking bug

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoMerge branch 'offload-flags-fix' of https://github.com/vincentkfu/fio
Jens Axboe [Tue, 11 Sep 2018 14:25:57 +0000 (08:25 -0600)]
Merge branch 'offload-flags-fix' of https://github.com/vincentkfu/fio

* 'offload-flags-fix' of https://github.com/vincentkfu/fio:
  rate_submit: synchronize accesses to io_u_queue->nr

14 months agorate_submit: synchronize accesses to io_u_queue->nr
Vincent Fu [Tue, 11 Sep 2018 13:01:56 +0000 (09:01 -0400)]
rate_submit: synchronize accesses to io_u_queue->nr

Accesses to io_u_queue->nr are not properly synchronized in offload
submission mode. put_io_u locks td but the parent td flags reflecting
the need to lock are not propogated to child threads when the child
threads are intialized.

The main thread accesses io_u_queue->nr via io_u_qpop() as it prepares
io_u's for handing off to the worker threads. The worker threads access
io_u_queue->nr via io_u_qpush() as they complete io_u's. When these
accesses are not protected by locks, io_u_qpop() will return NULL when
it means to provide a valid io_u pointer. This occurs in offload
submission mode with iodepth > 1.

Fixes: 26b3a18 ("Make td_io_u_lock/unlock() explicit")

14 months agoMerge branch 'no-unittest-dep' of https://github.com/parallel-fs-utils/fio
Jens Axboe [Mon, 10 Sep 2018 15:56:11 +0000 (09:56 -0600)]
Merge branch 'no-unittest-dep' of https://github.com/parallel-fs-utils/fio

* 'no-unittest-dep' of https://github.com/parallel-fs-utils/fio:
  remove dependency on unittest2 module

14 months agoraise exception if test start time can't be estimated
Ben England [Mon, 10 Sep 2018 12:48:05 +0000 (08:48 -0400)]
raise exception if test start time can't be estimated

do not allow log_unix_epoch=1 with single-record histogram log
and no log_hist_msec parameter, because then we have no way of
estimating when the test started in absolute time.

14 months agohandle log_unix_epoch=1
Ben England [Mon, 10 Sep 2018 12:13:35 +0000 (08:13 -0400)]
handle log_unix_epoch=1

previously this program only handled log_unix_epoch=0
(time since start of test in histogram log column 1)
supports distributed workload generation
assumes hosts' clocks are tightly synced

14 months agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Sun, 9 Sep 2018 23:42:10 +0000 (17:42 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  num2str(): Avoid an out-of-bounds array access
  Micro-optimize num2str()
  iolog: Ensure that sockaddr_un.sun_path is '\0'-terminated

14 months agoKill fusion atomic write engine
Jens Axboe [Sat, 8 Sep 2018 14:06:57 +0000 (08:06 -0600)]
Kill fusion atomic write engine

Never really went anywhere, kill it with fire.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoRename example job files (*.job -> *.fio)
Jens Axboe [Sat, 8 Sep 2018 14:09:35 +0000 (08:09 -0600)]
Rename example job files (*.job -> *.fio)

Make it consistent.

Sugested-by: Rebecca Cran <rebecca@bluestop.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoclient: cleanup output types
Jens Axboe [Fri, 7 Sep 2018 21:59:51 +0000 (15:59 -0600)]
client: cleanup output types

- Only print hostname= etc info for the NORMAL output, not for
  minimal/terse.

- Only print disk util header for NORMAL output.

- Cleanup json finish check

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoclient: switch to per-client buffer
Jens Axboe [Fri, 7 Sep 2018 20:25:30 +0000 (14:25 -0600)]
client: switch to per-client buffer

Even with the local storage, for some operations we still end up
having to do split dumps. Include the buffer in the actual client,
and flush when the client is done.

Hopefully this fixes the remaining buffer output split for
client/server.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoclient: use temp buffer for single output flush for json/disk util
Jens Axboe [Fri, 7 Sep 2018 19:52:18 +0000 (13:52 -0600)]
client: use temp buffer for single output flush for json/disk util

Similar to previous fixes, ensure that we log everything locally
before flushing it out.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoRevert "client: respect terse output on client <--> backend relationship"
Jens Axboe [Fri, 7 Sep 2018 19:06:32 +0000 (13:06 -0600)]
Revert "client: respect terse output on client <--> backend relationship"

This breaks a standard use case, which is a server without arguments
and a client with the --minimal option.

This reverts commit 1d1b65dc17c39f43dbe9d66062ca61fc3e90ddfc.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agolog: remember to free output buffer when done
Jens Axboe [Fri, 7 Sep 2018 18:57:52 +0000 (12:57 -0600)]
log: remember to free output buffer when done

We need to ensure we call buf_output_free() after using the
temp storage, otherwise we leaked the memory that we allocated
while filling the buffer.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoinit: use __log_buf() if we know buf != NULL
Jens Axboe [Fri, 7 Sep 2018 18:19:43 +0000 (12:19 -0600)]
init: use __log_buf() if we know buf != NULL

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agolog: use __log_buf() if we know buf != NULL
Jens Axboe [Fri, 7 Sep 2018 18:09:24 +0000 (12:09 -0600)]
log: use __log_buf() if we know buf != NULL

This apparently causes complaints on cygwin:

    CC stat.o
In file included from os/os-windows.h:18:0,
                 from os/os.h:54,
                 from thread_options.h:5,
                 from fio.h:18,
                 from init.c:19:
init.c: In function ‘add_job’:
os/../log.h:24:12: error: the comparison will always evaluate as ‘true’ for the address of ‘out’ will never be NULL [-Werror=address]
  if ((buf) != NULL)    \
            ^
os/../log.h:24:12: note: in definition of macro ‘log_buf’
  if ((buf) != NULL)    \

which is utterly stupid, but hey...

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoclient: use temp buffer for client text output
Jens Axboe [Fri, 7 Sep 2018 18:04:14 +0000 (12:04 -0600)]
client: use temp buffer for client text output

Don't split it into pieces, log it into a local buffer and then
flush it all at once.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoCollect startup output before logging it
Jens Axboe [Fri, 7 Sep 2018 18:02:39 +0000 (12:02 -0600)]
Collect startup output before logging it

We currently do the startup logging in separate chunks. This is
inefficient in terms of network trafic for client/server, and
it also causes output to be potentially mingled together:

<x1> Starting <x1> Starting 1 process1 process

Fix this by logging to a buffer, then transmitting that buffer
in one piece.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 months agoremove dependency on unittest2 module
Ben England [Fri, 7 Sep 2018 16:43:21 +0000 (12:43 -0400)]
remove dependency on unittest2 module

The dependency of fio-histo-log-pctiles.py on unittest2 module was causing heartburn
for pbench community so I removed the dependency.
If unittest2 python package is installed and unittest2 module can be imported,
that's fine, we'll use it if requested,
but if it's not, that's fine too, you can still use the program.

14 months agonum2str(): Avoid an out-of-bounds array access
Bart Van Assche [Fri, 7 Sep 2018 14:19:34 +0000 (07:19 -0700)]
num2str(): Avoid an out-of-bounds array access

This patch fixes Coverity ID 169307.

Fixes: 1ec3d69b0ed8 ("Implement a better num2str()")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
14 months agoMicro-optimize num2str()
Bart Van Assche [Sun, 26 Aug 2018 15:54:19 +0000 (08:54 -0700)]
Micro-optimize num2str()

Instead of selecting an element from the thousand[] array several times,
perform the choice between 1000 and 1024 once. This patch does not change
any functionality.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>