Jens Axboe [Tue, 14 Aug 2018 02:21:05 +0000 (20:21 -0600)]
Fix double free of zone cache data
We can't set it up in the parsing callback, as that happens before
we fork off the thread. Fixes a segfault with:
./fio --ioengine=libaio --randrepeat=0 --norandommap --thread --direct=1 --name=pre_test --rw=randwrite --bssplit=4k/67:8k/10:16k/7:32k/3:64k/13 --random_distribution=zoned:50/5:30/15:20/80 --iodepth=32 --runtime=60 --time_based --numjobs=2 --filename=/dev/nvme0n1p9 --group_reporting=1
Fixes: https://github.com/axboe/fio/issues/650
Fixes:
e0a04ac15f61 ("Add support for zones of random IO, with varying frequency of access")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 10 Aug 2018 00:02:59 +0000 (17:02 -0700)]
Improve zone support documentation
Improve the documentation of the zonesize, zoneskip and zonerange job
options such that this documentation becomes easier to comprehend.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Mon, 13 Aug 2018 17:18:09 +0000 (11:18 -0600)]
asprintf: fix indentation
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 13 Aug 2018 15:47:45 +0000 (09:47 -0600)]
gclient: bump output time buf size
We could potentially write a few bytes too many, just bump the
timebuf size up into a safe region.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 12 Aug 2018 21:51:34 +0000 (15:51 -0600)]
Merge branch 'minor_fixes' of https://github.com/sitsofe/fio
* 'minor_fixes' of https://github.com/sitsofe/fio:
man: fix missing/too many backslashes
doc: rewording and add reference to --aux-path
doc: update Log File Formats and write_iops_log sections
minor fio.service cleanups
Sitsofe Wheeler [Sun, 12 Aug 2018 19:58:56 +0000 (20:58 +0100)]
man: fix missing/too many backslashes
When viewing the man page some backslashes were AWOL whereas in other
places there were too many so fix them all.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 12 Aug 2018 19:38:18 +0000 (20:38 +0100)]
doc: rewording and add reference to --aux-path
- Reword --aux-path definition
- Reword "filename semantic" text in directory definition
- Add reference to --aux-path from --filename definition
Closes: https://github.com/axboe/fio/issues/530
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sat, 2 Sep 2017 11:14:19 +0000 (12:14 +0100)]
doc: update Log File Formats and write_iops_log sections
Make it clearer when various log entries will be always set to 0, move
"IOPS value is 1 without windowed logging" warning to write_iops_log
option description, change offset definition to position, add some
formatting.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Thu, 14 Sep 2017 02:08:35 +0000 (03:08 +0100)]
minor fio.service cleanups
- Remove PIDFile line because the service Type=simple and nothing will
write the pidfile (we aren't launching fio with -daemonize=pathtopid)
- Change some capitalisation
- [Install] section
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sun, 12 Aug 2018 16:55:53 +0000 (10:55 -0600)]
travis: include new xcode 9.4
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 12 Aug 2018 15:09:16 +0000 (09:09 -0600)]
Merge branch 'armv6' of https://github.com/sitsofe/fio
* 'armv6' of https://github.com/sitsofe/fio:
arch: fix build breakage on armv6
Sitsofe Wheeler [Sun, 12 Aug 2018 08:42:03 +0000 (09:42 +0100)]
arch: fix build breakage on armv6
The FreeBSD folks found fio didn't build on their armv6 systems
(https://svnweb.freebsd.org/ports/head/benchmarks/fio/Makefile?revision=432897&view=markup
) but Mikaƫl Urankar fixed the issue in
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228042 so copy the fix
to mainline.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Fri, 10 Aug 2018 15:33:44 +0000 (09:33 -0600)]
eta: clean up ETA string printing
No need to have a separate case for trims, we just need to look
at if we have reads, writes, trims.
This also eliminates printing read or write strings if we are not
doing that particular type of IO.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 9 Aug 2018 15:31:26 +0000 (09:31 -0600)]
Merge branch 'realloc-io_u-buffers' of https://github.com/aclamk/fio
* 'realloc-io_u-buffers' of https://github.com/aclamk/fio:
iolog replay: Realloc io_u buffers to adapt to operation size.
iolog replay: Treat 'open' on file that is scheduled to close as cancel of 'close' operation.
Udit agarwal [Thu, 9 Aug 2018 15:20:12 +0000 (09:20 -0600)]
xxhash: fix function definition and declaration mismatch
During the declaration of, let's say XXH32_init parameter type is
'unsigned int' while during the definition it is 'uint32_t'. Now, on
most desktop os, they are are usually same, however on some embedded
platforms, for example RTEMS which uses newlib as it's standard c
library, they both are different.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Adam Kupczyk [Thu, 9 Aug 2018 14:05:22 +0000 (16:05 +0200)]
iolog replay: Realloc io_u buffers to adapt to operation size.
When iolog is read in chunked mode, some prefetching is done. Based on this size of buffers are determined.
Further operations may need larger buffers. If so, io_u buffers are reallocated.
Example that cause problem when read_iolog_chunked=1:
fio version 2 iolog
rbd_data.0 add
rbd_data.0 open
rbd_data.0 write 0 4096 (x20)
rbd_data.0 write 0
4096000 (x10)
rbd_data.0 close
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Adam Kupczyk [Thu, 9 Aug 2018 10:53:29 +0000 (12:53 +0200)]
iolog replay: Treat 'open' on file that is scheduled to close as cancel of 'close' operation.
Problem occurs when processing generated iolog files, such as this:
fio version 2 iolog
rbd_data.0 add
rbd_data.0 open
rbd_data.0 write 0 4096
rbd_data.0 close
rbd_data.0 open
rbd_data.0 write 0 4096
rbd_data.0 close
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Jens Axboe [Tue, 7 Aug 2018 14:09:47 +0000 (08:09 -0600)]
libpmem: fix type print
Fixes:
5fff95436922 ("Add support for >= 4G block sizes")
Fixes: https://github.com/axboe/fio/issues/639
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Tomohiro Kusumi [Sun, 5 Aug 2018 16:30:14 +0000 (09:30 -0700)]
client: support --status-interval option in client/server mode
This fixes "FIO client/server periodic stats" for non JSON output
(the reporter is using JSON output).
https://www.spinics.net/lists/fio/msg07204.html
Mark -L/--status-interval option with FIO_CLIENT_FLAG, so that the
command line string is sent to the server side.
The client can now control the server to periodically show status,
and each status is sent to the client via fio_server_text_output()
(FIO_NET_CMD_TEXT socket event) in log_info_buf().
Reported-by: Mark Beierl <Mark.Beierl@dell.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 3 Aug 2018 20:40:17 +0000 (14:40 -0600)]
iolog: move the chunked items-to-fetch logic into separate function
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 3 Aug 2018 20:35:57 +0000 (14:35 -0600)]
iolog: fix potential div-by-zero
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 3 Aug 2018 15:17:06 +0000 (09:17 -0600)]
Merge branch 'chunked-iolog-reading' of https://github.com/aclamk/fio
* 'chunked-iolog-reading' of https://github.com/aclamk/fio:
iolog: Added new option description to HOWTO
iolog: Added option read_iolog_chunked. Used to avoid reading large iologs at once. Allows iologs to be infinite, generated.
Jens Axboe [Fri, 3 Aug 2018 15:16:10 +0000 (09:16 -0600)]
Merge branch 'read_iolog-from-unix-socket' of https://github.com/aclamk/fio
* 'read_iolog-from-unix-socket' of https://github.com/aclamk/fio:
iolog: allow to read_iolog from unix socket
Jens Axboe [Fri, 3 Aug 2018 15:15:30 +0000 (09:15 -0600)]
Merge branch 'windows-s_issock' of https://github.com/aclamk/fio
* 'windows-s_issock' of https://github.com/aclamk/fio:
platforms/windows: Add S_ISSOCK macro.
Adam Kupczyk [Mon, 30 Jul 2018 10:02:55 +0000 (12:02 +0200)]
iolog: allow to read_iolog from unix socket
Checks is file provided via --read_iolog parameter is a unix socket.
In such case connect to it and fetch data from there.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Adam Kupczyk [Fri, 3 Aug 2018 10:48:28 +0000 (12:48 +0200)]
platforms/windows: Add S_ISSOCK macro.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Adam Kupczyk [Fri, 3 Aug 2018 10:41:59 +0000 (12:41 +0200)]
iolog: Added new option description to HOWTO
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Jens Axboe [Thu, 2 Aug 2018 22:20:24 +0000 (16:20 -0600)]
Merge branch 'szaydel/solaris-Wincompatible-pointer-types' of https://github.com/szaydel/fio
* 'szaydel/solaris-Wincompatible-pointer-types' of https://github.com/szaydel/fio:
Fix incompatible pointer types warning on Solaris with gcc 6.4
Sam Zaydel [Thu, 2 Aug 2018 22:01:27 +0000 (15:01 -0700)]
Fix incompatible pointer types warning on Solaris with gcc 6.4
Adam Kupczyk [Wed, 1 Aug 2018 11:45:27 +0000 (13:45 +0200)]
iolog: Added option read_iolog_chunked. Used to avoid reading large iologs at once.
Allows iologs to be infinite, generated.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Jens Axboe [Mon, 30 Jul 2018 14:24:20 +0000 (08:24 -0600)]
Merge branch 'fio-histo-fix' of https://github.com/parallel-fs-utils/fio
* 'fio-histo-fix' of https://github.com/parallel-fs-utils/fio:
clean up argparse usage
Ben England [Sun, 29 Jul 2018 17:14:59 +0000 (13:14 -0400)]
clean up argparse usage
Jens Axboe [Thu, 26 Jul 2018 17:47:28 +0000 (11:47 -0600)]
Merge branch 'sgunmap2' of https://github.com/vincentkfu/fio
* 'sgunmap2' of https://github.com/vincentkfu/fio:
docs: update HOWTO and manpage for sg trim support
testing: add test scripts for sg ioengine
engines/sg: move asserts and debug statements behind a debug flag
engines/sg: add cmdp and dxferp for trims to sg error string
engines/sg: support trim operations via the UNMAP command
stat: add IO submit and complete depths to JSON output
ioengines: have ioengines with commit do own io accounting for trims
Vincent Fu [Thu, 26 Jul 2018 16:27:21 +0000 (09:27 -0700)]
docs: update HOWTO and manpage for sg trim support
Also fix HOWTO sg_write_mode formatting
Vincent Fu [Fri, 20 Jul 2018 16:54:49 +0000 (09:54 -0700)]
testing: add test scripts for sg ioengine
t/sgunmap-perf.py carries out basic performance testing using
the sg ioengine
t/sgunmap-test.py checks that the IO depths reported for sg
trim workloads are sensible
Vincent Fu [Wed, 20 Jun 2018 19:47:16 +0000 (12:47 -0700)]
engines/sg: move asserts and debug statements behind a debug flag
The sg ioengine modifications include some debug statements and
integrity checks that could hurt performance. Put these behind
a debug flag.
Vincent Fu [Wed, 25 Jul 2018 20:00:26 +0000 (13:00 -0700)]
engines/sg: add cmdp and dxferp for trims to sg error string
Vincent Fu [Thu, 24 May 2018 21:00:36 +0000 (17:00 -0400)]
engines/sg: support trim operations via the UNMAP command
Multiple trim ranges can be submitted with a single UNMAP command
for character devices (e.g., /dev/sg1) using iodepth_batch. For
example, with iodepth_batch=4, each UNMAP command will include
four ranges. IOPS will still be reported as if four IO operations
were carried out.
Note that currently sg only supports a maximum of 16 commands
in flight per file. So be sure to maintain (iodepth/iodepth_batch) <= 16.
Vincent Fu [Tue, 19 Jun 2018 19:25:58 +0000 (12:25 -0700)]
stat: add IO submit and complete depths to JSON output
The IO submit and IO complete depths were missing from the JSON
output. This patch adds JSON objects for these distributions.
Vincent Fu [Wed, 25 Jul 2018 19:37:06 +0000 (12:37 -0700)]
ioengines: have ioengines with commit do own io accounting for trims
ioengines.c:td_io_queue has a special case where it calls
io_u_mark_submit() and io_u_mark_complete() when the io_u
is a trim operation or when the ioengine does not have a
commit function. This patch makes these two calls the
responsibility of the ioengine for trim io_u's when the
ioengine does have a commit function.
This only applies to libaio because it is the only ioengine
supporting trims with a commit function. All other ioengines that
support trim operations do not have a commit function. These
ioengines are: falloc, glusterfs_async, mmap, mtd, posixaio,
rados, rbd, splice, sync, psync, pvsync, pvsync2.
Jens Axboe [Wed, 25 Jul 2018 20:42:57 +0000 (14:42 -0600)]
Merge branch 'fio-histo-log-pctiles' of https://github.com/parallel-fs-utils/fio
* 'fio-histo-log-pctiles' of https://github.com/parallel-fs-utils/fio:
design document for tools/hist/fio-histo-log-pctiles.py
switch to argparse module for CLI parsing
use interpolation for more accurate percentile calculation
get latency percentiles over time from fio histo logs
Ben England [Wed, 25 Jul 2018 19:40:04 +0000 (15:40 -0400)]
design document for tools/hist/fio-histo-log-pctiles.py
Jens Axboe [Wed, 25 Jul 2018 14:41:38 +0000 (08:41 -0600)]
Merge branch 'fio-c++-engine' of https://github.com/tchaikov/fio
* 'fio-c++-engine' of https://github.com/tchaikov/fio:
replace typeof with __typeof__
Kefu Chai [Wed, 25 Jul 2018 14:02:09 +0000 (22:02 +0800)]
replace typeof with __typeof__
so we are able to extend fio with C++ .
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Jens Axboe [Tue, 24 Jul 2018 21:23:28 +0000 (15:23 -0600)]
init: unify 't' time period
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 23 Jul 2018 16:12:54 +0000 (10:12 -0600)]
libpmem: update print statement for bs now being ULL
Fixes:
5fff95436922 ("Add support for >= 4G block sizes")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 23 Jul 2018 16:12:40 +0000 (10:12 -0600)]
parse: mark another fall-through switch case
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 23 Jul 2018 16:10:12 +0000 (10:10 -0600)]
parse: mark fall-through switch case
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 23 Jul 2018 16:05:49 +0000 (10:05 -0600)]
Use stdlib.h instead of malloc.h
A few more conversions.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jeff Furlong [Mon, 23 Jul 2018 15:15:40 +0000 (09:15 -0600)]
Add support for >= 4G block sizes
For trims, it's useful to be able to support larger than (or equal
to) 4GB. This extends the block sizes support for that.
Change from Jeff, various little fixups from me.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 23 Jul 2018 14:55:32 +0000 (08:55 -0600)]
Fio 3.8
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 23 Jul 2018 14:29:14 +0000 (08:29 -0600)]
gfio: cleanup includes
Use stdlib.h, not malloc.h. And ensure that we pull in the
right header for basename(3).
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Ben England [Fri, 20 Jul 2018 17:49:37 +0000 (13:49 -0400)]
switch to argparse module for CLI parsing
Ben England [Thu, 19 Jul 2018 22:05:52 +0000 (18:05 -0400)]
use interpolation for more accurate percentile calculation
Ben England [Tue, 17 Jul 2018 18:40:01 +0000 (14:40 -0400)]
get latency percentiles over time from fio histo logs
Jens Axboe [Thu, 12 Jul 2018 14:33:14 +0000 (08:33 -0600)]
axmap: optimize ulog64 usage in axmap_handler()
We can do this incrementally, no need to compute from scratch
for every iteration.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 21:32:31 +0000 (15:32 -0600)]
t/axmap: add longer overlap test case
Also fix bad test stop termination.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 21:17:32 +0000 (15:17 -0600)]
t/axmap: add regression case for recent overlap failure case
Catches the issue that
bd71edcedc3d fixes.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 21:16:58 +0000 (15:16 -0600)]
axmap: fix continued sequential bit setting
We need to remember to clear ->set_bits if we don't set new bits,
or the caller will think we re-set the previously set bits.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 19:59:32 +0000 (13:59 -0600)]
axmap: a few more cleanups
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 19:48:32 +0000 (13:48 -0600)]
axmap: remove unused 'data' argument to topdown handler
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 19:26:19 +0000 (13:26 -0600)]
fio: should_fsync() returns bool
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 18:25:43 +0000 (12:25 -0600)]
axmap: code cleanups
- Move the const bit masks to the start of the file
- Get rid of a useless variable in axmap_handler_topdown()
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 15:11:15 +0000 (09:11 -0600)]
t/axmap: add zero return overlap cases
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 14:34:06 +0000 (08:34 -0600)]
axmap: clean up 'no bits to set' case
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 14:32:37 +0000 (08:32 -0600)]
t/axmap: don't print 'pass' on failure
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 14:29:22 +0000 (08:29 -0600)]
t/axmap: a few more overlap cases
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Jul 2018 03:51:16 +0000 (21:51 -0600)]
t/axmap: clean up overlap tests
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 10 Jul 2018 23:02:25 +0000 (17:02 -0600)]
Makefile: lib/axmap no longer needs hweight
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 10 Jul 2018 23:02:03 +0000 (17:02 -0600)]
t/axmap: add a few more overlap test cases
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 10 Jul 2018 22:52:16 +0000 (16:52 -0600)]
axmap: ensure that overlaps are handled strictly sequential
We must terminate at the first overlapping bit, we can't
just mask off non-sequential ranges.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 10 Jul 2018 22:51:24 +0000 (16:51 -0600)]
t/axmap: add overlap test cases
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 10 Jul 2018 21:10:17 +0000 (15:10 -0600)]
io_u: ensure we generate the full length of block sizes
Since we round down, we can miss the last entry. This ensures that
if we do:
bsrange=4k-16k
we actually get an even split of 4,8,12,16k ios.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 10 Jul 2018 19:06:38 +0000 (13:06 -0600)]
io_u: fix negative offset due to wrap
If we do wrap, the math is off and we end up wrapping a 64-bit
value. Instead reset to the initial offset.
Reported-by: Bart Van Assche <Bart.VanAssche@wdc.com>
Fixes:
4c8be5b1569f ("Fix bug with zone and zone skipping and io_limit")
Fixes:
224b3093cc21 ("Fix zoning issue with seq-io and randommap issue")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 4 Jul 2018 20:59:18 +0000 (14:59 -0600)]
blktrace: just ignore zero byte traces
Apparently they do happen. Log a warning (once) and just discard
them.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 29 Jun 2018 14:00:29 +0000 (08:00 -0600)]
Fix compilation without cgroups
Fixes:
c3dc516ad75e ("fio: work with cgroup2 as well")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Josef Bacik [Fri, 29 Jun 2018 13:49:03 +0000 (09:49 -0400)]
fio: add job_runtime to the thread json output
I discovered that the 'elapsed' value of each job is the overall time
the whole fio run took, not the per-job elapsed time, despite it being
in the per-job section of the json output. Rather than break everybody
by fixing this, just add a 'job_runtime' value to the job section and
include the time the thread ran in milliseconds. This is handy for jobs
that want to compare runtime of two threads in different cgroups.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Josef Bacik [Fri, 29 Jun 2018 13:33:34 +0000 (09:33 -0400)]
fio: work with cgroup2 as well
The current code only works with cgroupv1, but as long as you don't want
to set blkio.weight or anything we can use the 'cgroup=' setting with
cgroupv2 easily with cgroupv1, we just need to write the thread pids to
a different file for cgroup2.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 21 Jun 2018 17:22:41 +0000 (11:22 -0600)]
Merge branch 'doc-norandommap' of https://github.com/larrystevenwise/fio
* 'doc-norandommap' of https://github.com/larrystevenwise/fio:
doc: add text about possibly verify errors with norandommap
Steve Wise [Thu, 21 Jun 2018 17:16:50 +0000 (10:16 -0700)]
doc: add text about possibly verify errors with norandommap
In both the norandommap option section, and the verify option section,
add text warning that norandommap can cause data verification errors
when using async io engines and io depths > 1.
Signed-off-by: Steve Wise <swise@opengridcomputing.com>
Jens Axboe [Wed, 20 Jun 2018 14:14:08 +0000 (08:14 -0600)]
Merge branch 'wip-single-glfs-instance' of https://github.com/zhanghuan/fio
* 'wip-single-glfs-instance' of https://github.com/zhanghuan/fio:
glusterfs: capable to test with one single glfs instance
Zhang Huan [Wed, 6 Jun 2018 03:52:14 +0000 (11:52 +0800)]
glusterfs: capable to test with one single glfs instance
Current multi-thread test creates one glfs instance per job. However,
there is requirement to run all jobs on one single instance, and that
all jobs would share the same set of underlying working threads.
Add a new option "single-instance" to control whether to create one
global instance or one instance per job. For testing with mutilple
gluster volumes, use host name and volume name to filter out same
volume, one specific volume will have one instance.
Signed-off-by: Zhang Huan <zhanghuan@open-fs.com>
Jens Axboe [Mon, 18 Jun 2018 19:58:26 +0000 (13:58 -0600)]
Merge branch 'readonly-trim' of https://github.com/vincentkfu/fio
* 'readonly-trim' of https://github.com/vincentkfu/fio:
testing: add test script for readonly parameter
doc: improve readonly option description
options: check for conflict between trims and readonly option
init: abort write and trim jobs when --readonly option is present
init: ensure that fatal errors in fixup_options are always propogated to caller
filesetup: make trim jobs respect --readonly during file open
fio.h: also check trim operations in fio_ro_check
Vincent Fu [Mon, 18 Jun 2018 16:55:21 +0000 (10:55 -0600)]
testing: add test script for readonly parameter
Add a shell script and jobs for testing the readonly parameter
Vincent Fu [Mon, 18 Jun 2018 15:48:17 +0000 (09:48 -0600)]
doc: improve readonly option description
Make explicit that --readonly also prevents trim workloads
Drop the mention of the check in the I/O engine core because
fio_ro_check() is run even without the readonly option
Vincent Fu [Fri, 15 Jun 2018 21:52:43 +0000 (15:52 -0600)]
options: check for conflict between trims and readonly option
When processing the readwrite option, make sure that trim operations
are not requested if readonly is set.
Vincent Fu [Fri, 15 Jun 2018 21:17:38 +0000 (15:17 -0600)]
init: abort write and trim jobs when --readonly option is present
Check for conflicts between --readonly and the job's data direction
in fixup_options(). rw_verify() in options.c only detects this
conflict if --readonly precedes --readwrite on the command line.
Vincent Fu [Fri, 15 Jun 2018 21:11:10 +0000 (15:11 -0600)]
init: ensure that fatal errors in fixup_options are always propogated to caller
When warnings_fatal = 0, fatal problems detected in fixup_options
may not always be propogated back to the caller. This patch ensures
that the caller will always be notified about fatal problems.
Vincent Fu [Fri, 15 Jun 2018 21:05:28 +0000 (15:05 -0600)]
filesetup: make trim jobs respect --readonly during file open
Open files for trim workloads with the O_RDWR access mode only when
the --readonly option is absent.
Vincent Fu [Fri, 15 Jun 2018 21:00:25 +0000 (15:00 -0600)]
fio.h: also check trim operations in fio_ro_check
IO engines call fio_ro_check to ensure that the data direction
set in the thread data structure permits writes before proceeding
with a write io_u. This patch makes fio_ro_check ensure that the
data direction also permits trims before proceeding with a trim
io_u.
Jens Axboe [Fri, 15 Jun 2018 15:18:04 +0000 (09:18 -0600)]
client: close dup'ed descriptor if fdopen() fails
The saga continues... Ensure that we close the dup'ed descriptor,
if we give up due to fdopen() failure.
Reported-by: Bart Van Assche <bart.vanassche@wdc.com>
Fixes:
ec9e13345f3f ("client: check return of dup(2)")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 15 Jun 2018 15:10:48 +0000 (09:10 -0600)]
client: check return of dup(2)
If dup(2) fails, we can't pass it to fdopen.
Fixes:
b4f5e72f1383 ("client: parse env variables before sending job-file contents to server")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Tomohiro Kusumi [Fri, 15 Jun 2018 14:58:22 +0000 (08:58 -0600)]
client: parse env variables before sending job-file contents to server
Fixes "fio environment var bug".
https://www.spinics.net/lists/fio/msg07093.html
Add read_ini_data() to parse and expand env variables within job-file
before sending to server. By doing this, clients can control parameters
embedded within the job-file, without server side having to set them
separately.
Reported-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Minor fixups from Jens.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 12 Jun 2018 17:43:29 +0000 (11:43 -0600)]
rand: make randX_upto() do the end value increment
We should not do it in the caller, the functions that need
fixing are really rand32/64_upto() instead.
Also move a (now) misplaced comment.
Fixes:
c6fc6d2ab2c2 ("rand: ensure that rand_between() can reach max value")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 12 Jun 2018 17:37:16 +0000 (11:37 -0600)]
rand: ensure that rand_between() can reach max value
We need to add 1, otherwise the maximum generated value will
be end -1. The API is both inclusive.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 12 Jun 2018 14:51:07 +0000 (08:51 -0600)]
init: use o-> instead of td->o
We already have the options pointer, just use it.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 12 Jun 2018 14:47:08 +0000 (08:47 -0600)]
init: kill get_rand_start_delay()
It's now useless, just call rand_between().
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 12 Jun 2018 14:42:49 +0000 (08:42 -0600)]
rand: cleanup rand_between() and helpers
Make the 32/64-bit helper just return a random number up to a certain
value, and let the generic helper handle the range part.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 12 Jun 2018 14:10:47 +0000 (08:10 -0600)]
rand: add rand64_between()
For some reason we only had the 32-bit variant, and there's
a use case for the 64-bit version. Add that, and add a
32/64 agnostic helper that can be called.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 12 Jun 2018 02:02:10 +0000 (20:02 -0600)]
Fix start delay being the same across threads
Two issues here:
1) We copy the start_delay into the original thread, which ends up
offsetting job 2..N off the thread 1 start delay
2) We don't initialize the start delay random seed prior to
generating the offsets
Signed-off-by: Jens Axboe <axboe@kernel.dk>