fio.git
4 years agoStaticize pmemblk ioengine_ops
Tomohiro Kusumi [Mon, 9 Jan 2017 14:41:59 +0000 (23:41 +0900)]
Staticize pmemblk ioengine_ops

Inbox engines can have static ops.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoBring in additional sg error cases from Linux kernel (and sg3) header
Tomohiro Kusumi [Mon, 9 Jan 2017 14:41:58 +0000 (23:41 +0900)]
Bring in additional sg error cases from Linux kernel (and sg3) header

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix comment on SCSI commands
Tomohiro Kusumi [Fri, 6 Jan 2017 18:15:18 +0000 (03:15 +0900)]
Fix comment on SCSI commands

[0] is service action (unless this is intentional).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoRemove doubled ; in .c
Tomohiro Kusumi [Fri, 6 Jan 2017 18:15:17 +0000 (03:15 +0900)]
Remove doubled ; in .c

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoinit: cleaner gcd()
Jens Axboe [Thu, 5 Jan 2017 17:38:41 +0000 (10:38 -0700)]
init: cleaner gcd()

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoDon't malloc ioengine_ops for cpp_null
Tomohiro Kusumi [Thu, 5 Jan 2017 17:32:17 +0000 (02:32 +0900)]
Don't malloc ioengine_ops for cpp_null

fio assumes opses aren't dynamically allocated which means malloc'd
engine here never gets freed.

According to 46a67478 and a8075704, the point of get_ioengine() is
how non-static ops gets initialized by g++, but not how space for ops
gets allocated, thus it should just define a static global variable
within "C" scope and pass the address of ops to a caller via **.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix invalid ioengine initialization for cpp_null
Tomohiro Kusumi [Thu, 5 Jan 2017 17:32:16 +0000 (02:32 +0900)]
Fix invalid ioengine initialization for cpp_null

->name is a pointer now, so it can't be strcpy'd without a valid
address set to ->name.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: fill in vc->name for pattern verify
Jens Axboe [Thu, 5 Jan 2017 17:33:23 +0000 (10:33 -0700)]
verify: fill in vc->name for pattern verify

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: ensure that verify_interval is always a factor of min/max bs
Jens Axboe [Thu, 5 Jan 2017 17:26:08 +0000 (10:26 -0700)]
verify: ensure that verify_interval is always a factor of min/max bs

A previous commit force aligned the buffer length to the
verify_interval, but that breaks if max_bs isn't a multiple
of min_bs. If that isn't the case, we can align the block size
bigger than the IO unit really is. This causes corruption and
failures.

Fixes: a9f70b1f5087 ("Make sure io_u->buflen is aligned to the verify_interval")
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoRemove '--runtime' command line option
Jens Axboe [Thu, 5 Jan 2017 16:49:06 +0000 (09:49 -0700)]
Remove '--runtime' command line option

It conflicts with the job file option, and we (in most cases) set
the default before we have parsed it, depending on where in the
command line the --runtime option is added.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: use log_verify_failure() for pattern verifies
Jens Axboe [Thu, 5 Jan 2017 16:38:50 +0000 (09:38 -0700)]
verify: use log_verify_failure() for pattern verifies

Similar to the other verify types, otherwise we don't get
the absolute offset logged.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoStyle cleanups for arm crc32c hw support
Jens Axboe [Thu, 5 Jan 2017 02:44:35 +0000 (19:44 -0700)]
Style cleanups for arm crc32c hw support

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd arm64 hardware assisted crc32c support
wei xiao [Thu, 5 Jan 2017 02:32:40 +0000 (10:32 +0800)]
Add arm64 hardware assisted crc32c support

HW assisted crc32c get ~ x39.6 speedups on ARM Cortex-A57@2GHz

Auto-detect whether to use hw assisted crc32c.
If the hardware assisted crypto is available, always use it.
Otherwise, fallback to software.

4 years agoAdd missing .help string for io_size option
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:11 +0000 (23:40 +0900)]
Add missing .help string for io_size option

fio --cmdhelp currently prints "(null)" for io_size.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix README - change just type "configure" to "./configure"
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:10 +0000 (23:40 +0900)]
Fix README - change just type "configure" to "./configure"

Literally typing "configure" may or may not work depending on visible paths.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd BSD package/building info to README
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:09 +0000 (23:40 +0900)]
Add BSD package/building info to README

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix canonical name for runtime/timeout option
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:08 +0000 (23:40 +0900)]
Fix canonical name for runtime/timeout option

.name should be set to "runtime" than "timeout".
fio_options[] as well as the existing documentation consider
"runtime" as an option name, and "timeout" as an alias.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoSync README with fio usage output
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:07 +0000 (23:40 +0900)]
Sync README with fio usage output

Some of the commits only update one of these.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd missing trailing \n in log_err/info()
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:06 +0000 (23:40 +0900)]
Add missing trailing \n in log_err/info()

Callers need trailing \n in case of fwrite(3) (if not syslog(3)).

These are the ones that seem to need trailing \n not followed
by another log function call as consequence.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix trivial calloc argument order
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:05 +0000 (23:40 +0900)]
Fix trivial calloc argument order

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/stest: remove old test
Jens Axboe [Tue, 3 Jan 2017 01:21:14 +0000 (18:21 -0700)]
t/stest: remove old test

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoDocumentation for IEC binary and SI decimal prefix handling
Robert Elliott [Thu, 22 Dec 2016 20:50:51 +0000 (14:50 -0600)]
Documentation for IEC binary and SI decimal prefix handling

Documentation for the previous patch improving IEC and SI
decimal prefix handling (e.g., KiB, MiB, GiB vs. KB, MB, GB).

Includes updates to int, kb_base, and the parameters using
int to express quantities of data (e.g., bs, ba, bsrange, bssplit).

Includes updates to parameters that accept comma-separated
values (usually ints) for reads, writes, and trims; trims were
often not mentioned.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoImprove IEC binary and SI decimal prefix handling
Robert Elliott [Thu, 22 Dec 2016 20:50:50 +0000 (14:50 -0600)]
Improve IEC binary and SI decimal prefix handling

Use kb_base=1000 to follow international standards for unit prefixes.

To specify power-of-10 decimal values defined in the International
System of Units (SI):
Ki means kilo (K) or 1000
Mi means mega (M) or 1000**2
Gi means giga (G) or 1000**3
Ti means tera (T) or 1000**4
Pi means peta (P) or 1000**5

To specify power-of-2 binary values defined in IEC 80000-13:
k means kibi (Ki) or 1024
M means mebi (Mi) or 1024**2
G means gibi (Gi) or 1024**3
T means tebi (Ti) or 1024**4
P means pebi (Pi) or 1024**5

For example, this specifies a blocksize of 4096 bytes:
kb_base=1000
bs=4KiB

With kb_base=1024 (the default), the unit prefixes are opposite from
those specified in the SI and IEC 80000-13 standards to provide
compatibility with old scripts.  For example, this specifies a
blocksize of 4096 bytes:
kb_base=1024
bs=4K

For outputs printing quantities and bandwidths:
* eta stats only use the preferred prefix
* final stats include both (non-preferred prefix in parenthesis)
* in gfio, all windows include both

Text outputs are rearranged to try to obviously break any scripts
parsing the output rather than silently confuse them.  The terse
and json outputs, which are intended for parsing, are unchanged.

Old:
Jobs: 576 (f=576), CR=86.4GB/576MB KB/s: [w(288),r(288)] [0.0% done] [33884MB/27114MB/0KB /s] [8471K/6778K/0 iops] [eta 06h:59m:57s]
   read: io=363338MB, bw=34014MB/s, iops=8304.3K, runt= 10682msec
  write: io=282447MB, bw=26225MB/s, iops=6402.7K, runt= 10770msec
   READ: io=363338MB, aggrb=34014MB/s, minb=34014MB/s, maxb=34014MB/s, mint=10682msec, maxt=10682msec
  WRITE: io=282447MB, aggrb=26225MB/s, minb=26225MB/s, maxb=26225MB/s, mint=10770msec, maxt=10770msec

New:
Jobs: 576 (f=576), 590MB/s-89.2GB/s: [w(288),r(288)][0.0%][r=34.2GB/s,w=26.3GB/s][r=8542k,w=6572k IOPS][eta 06h:59m:55s]
   read: IOPS=8362k, BW=34.3GB/s (31.1GiB/s)(665GB/19417msec)
  write: IOPS=6423k, BW=26.4GB/s (24.6GiB/s)(511GB/19431msec)
   READ: bw=34.3GB/s (31.1GiB/s), 34.3GB/s-34.3GB/s (31.1GiB/s-31.1GiB/s), io=665GB (619GiB), run=19417-19417msec
  WRITE: bw=26.4GB/s (24.6GiB/s), 26.4GB/s-26.4GB/s (24.6GiB/s-24.6GiB/s), io=511GB (476GiB), run=19431-19431msec

Documentation changes are in a subsequent patch.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoClean up unit prefixes for binary multiples in comments and prints
Robert Elliott [Thu, 22 Dec 2016 20:50:49 +0000 (14:50 -0600)]
Clean up unit prefixes for binary multiples in comments and prints

Change comments and prints that are referring to values based on
binary multiples to use IEC 80000-13 prefixes for binary
multiples rather than SI prefixes for decimal multiples (e.g.,
KiB, MiB, and GiB rather than KB, MB, and GB).

Includes a fix for the rate_min print that was printing "KB/s"
although it was really printing bytes per second.

Includes a fix for goptions confusing the PB and TB units (while
changing them to PiB and TiB).

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoConvert group_run_stats to use bytes instead of KiB/KB
Robert Elliott [Thu, 22 Dec 2016 20:50:48 +0000 (14:50 -0600)]
Convert group_run_stats to use bytes instead of KiB/KB

In group_run_stats, convert these fields:
* io_kb (bytes transferred)
* agg (aggregate average bandwidth)
* min_bw
* max_bw

from tracking either KiB or KB depending on kb_base to simply tracking bytes.
Rename io_kb to iobytes to match this new meaning.

This will simplify computing quantities and bandwidth rates with both IEC
binary and SI decimal unit prefixes (e.g., GiB/s and GB/s) in future patches.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogclient: Use proper time units in latency buckets chart
Robert Elliott [Thu, 22 Dec 2016 20:50:47 +0000 (14:50 -0600)]
gclient: Use proper time units in latency buckets chart

In the Results page Latency buckets chart, use us and ms rather
than u and s for the time units.  This matches the use of s for
seconds that is already being done.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogclient: Delete unused code
Robert Elliott [Thu, 22 Dec 2016 20:50:46 +0000 (14:50 -0600)]
gclient: Delete unused code

Delete two #if 0 blocks from pre-2012 that have old calls to
num2str without the current number of arguments and hardcode
"KB/s" units.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoLine up colons across read, write, and trim thread stats
Robert Elliott [Thu, 22 Dec 2016 20:50:45 +0000 (14:50 -0600)]
Line up colons across read, write, and trim thread stats

Line up the colons across the read, write, and trim
thread stats rows.

Example:
drive_0: (groupid=0, jobs=576): err= 0: pid=42198: Wed Dec 21 19:51:35 2016
  read : io=420642MB, bw=33953MB/s, iops=8289.3K, runt= 12389msec
  write: io=325276MB, bw=26177MB/s, iops=6390.9K, runt= 12426msec
  trim: io=0MB, bw=0MB/s, iops=0, runt= 12426msec

becomes:
drive_0: (groupid=0, jobs=576): err= 0: pid=42198: Wed Dec 21 19:51:35 2016
   read: io=420642MB, bw=33953MB/s, iops=8289.3K, runt= 12389msec
  write: io=325276MB, bw=26177MB/s, iops=6390.9K, runt= 12426msec
   trim: io=0MB, bw=0MB/s, iops=0, runt= 12426msec

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix unit_base kb_base mixup in thread option conversion functions
Robert Elliott [Thu, 22 Dec 2016 20:50:44 +0000 (14:50 -0600)]
Fix unit_base kb_base mixup in thread option conversion functions

Don't set unit_base, which is 1 or 8, to kb_base, which is 1000
or 1024, during the convert functions used by the initial cconv test.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agotests, profiles: Use IEC prefixes for binary multiples
Robert Elliott [Thu, 22 Dec 2016 20:50:43 +0000 (14:50 -0600)]
tests, profiles: Use IEC prefixes for binary multiples

Use IEC 80000-13 prefixes for binary multiples of bytes
(KiB, MiB, etc.) rather than SI prefixes for decimal multiples
of bytes (KB, MB, etc.) in places in which the code has fixed
divide by 1024, 1024*1024, >>20, etc.  and is clearly always
using binary multiples.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoDocument trim workload choices and other nits
Robert Elliott [Thu, 22 Dec 2016 20:50:42 +0000 (14:50 -0600)]
Document trim workload choices and other nits

Add some missing trim workloads (rw=trim, randtrim) to HOWTO,
improve some wording about trims in HOWTO, fio.1 (the manpage),
and the help text in options.c.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogfio: Improve IOPS textbox labels
Robert Elliott [Thu, 22 Dec 2016 20:50:41 +0000 (14:50 -0600)]
gfio: Improve IOPS textbox labels

Change the three textboxes labeled IOPS to Read IOPS, Write IOPS,
and Trim IOPS, add Trim to the Blocksize label, and fix Steve's
last name in the Credits.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAvoid using units in option defaults
Robert Elliott [Thu, 22 Dec 2016 20:50:40 +0000 (14:50 -0600)]
Avoid using units in option defaults

Change option defaults and some generated option strings to specify
the number of bytes without units, so they are immune to the
interpretation of the units (e.g. is 1M interpreted as
1048576 or 1000000).

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUpdate Windows maintainer contact details
Rebecca Cran [Tue, 3 Jan 2017 00:20:06 +0000 (17:20 -0700)]
Update Windows maintainer contact details

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooptions: mark steadystate option parents
Jens Axboe [Thu, 29 Dec 2016 16:07:29 +0000 (09:07 -0700)]
options: mark steadystate option parents

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agorbd: style fixups
Jens Axboe [Sat, 24 Dec 2016 02:54:47 +0000 (19:54 -0700)]
rbd: style fixups

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agorbd: remove duplicate _fio_rbd_connect
Pan Liu [Fri, 23 Dec 2016 16:35:55 +0000 (00:35 +0800)]
rbd: remove duplicate _fio_rbd_connect

Signed-off-by: Pan Liu <pan.liu@istuary.com>
4 years agofio: Add support for auto detect dev-dax and libpmemblk engines
Dave Jiang [Tue, 20 Dec 2016 23:31:45 +0000 (16:31 -0700)]
fio: Add support for auto detect dev-dax and libpmemblk engines

Added support code in configure to detect whether libpmem and libpmemblk
are installed. dev-dax engine is enabled if libpmem is installed.
pmemblk engine is installed if libpmem and libpmemblk are installed.
A --disable-pmem flag is added to explicitly disable the libpmem based
engines even if the libs are detected.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix compile on OSX
Jens Axboe [Tue, 20 Dec 2016 21:58:06 +0000 (14:58 -0700)]
Fix compile on OSX

And probably other platforms, that don't have the posix fadvise
hints.

Fixes: ecb2083d3955 ("options: full control of fadvise hinting")
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFio 2.16 fio-2.16
Jens Axboe [Tue, 20 Dec 2016 06:12:56 +0000 (23:12 -0700)]
Fio 2.16

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooptions: full control of fadvise hinting
Jens Axboe [Mon, 19 Dec 2016 20:58:42 +0000 (13:58 -0700)]
options: full control of fadvise hinting

Previously fio supported only 0/1 for this hint. 0 would not do any
hints, but 1 (the default) would hint based on what the fio workload
was - fio would tell the kernel it was a random workload, if it was
random, or a sequential for a sequential workload.

Now fio also supports setting what the hint type should be explicitly,
by allowing a setting of 'random' or 'sequential' for full control
of the hint.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofio: add additional support for dev-dax ioengine
Dave Jiang [Fri, 16 Dec 2016 22:59:06 +0000 (15:59 -0700)]
fio: add additional support for dev-dax ioengine

Fixup options.c to add the dev-dax to ioengine list. Update fio.1 and
HOWTO as well. Also adding example fio job file.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge https://bitbucket.org/vincentfu/fio-steadystate into steady-state-2
Jens Axboe [Fri, 16 Dec 2016 17:58:55 +0000 (10:58 -0700)]
Merge https://bitbucket.org/vincentfu/fio-steadystate into steady-state-2

4 years agouse type double in creating floating point JSON values
Vincent Fu [Tue, 13 Dec 2016 20:40:46 +0000 (15:40 -0500)]
use type double in creating floating point JSON values

struct json_value uses a double to store floating point values. Have json_create_value_float also take a double (instead of a float) as an argument.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofio: move _FORTIFY_SOURCE to only when optimization is turned on
Dave Jiang [Thu, 15 Dec 2016 21:51:28 +0000 (14:51 -0700)]
fio: move _FORTIFY_SOURCE to only when optimization is turned on

When compile with debug, without the proper -O flag the compiler complains:
/usr/include/features.h:331:4: warning: #warning _FORTIFY_SOURCE requires
compiling with optimization (-O) [-Wcpp]
#  warning _FORTIFY_SOURCE requires compiling with optimization (-O)
Moving _FORTIFY_SOURCE to optimized compile option.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofio: add device dax engine
Dave Jiang [Wed, 7 Dec 2016 22:54:30 +0000 (15:54 -0700)]
fio: add device dax engine

Adding an ioengine that access DAX device directly. Very similar to the
mmap engine. Uses the libpmem's pmem_memcpy_persist() call for writes.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoChange misleading error message for invalid size= value
Jakub Sitnicki [Tue, 13 Dec 2016 16:27:24 +0000 (17:27 +0100)]
Change misleading error message for invalid size= value

The error message refers to the IO size, yet another parameter, but we
actually compare the size parameter value against the block size.

Change the message to reflect the code, and make it easier for the user
to see what values are we comparing.

Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge git://git.kernel.dk/fio into steady-state
Vincent Fu [Tue, 13 Dec 2016 14:20:52 +0000 (09:20 -0500)]
Merge git://git.kernel.dk/fio into steady-state

4 years agoiolog: add support for replay_scale option
Sitsofe Wheeler [Sat, 3 Dec 2016 09:24:30 +0000 (09:24 +0000)]
iolog: add support for replay_scale option

Enable replay_scale (and replay_align) when playing back fio's log
format by making ipo_bytes_align() available from a header.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
4 years agoblktrace: Fix replay_align 32 bit truncation
Sitsofe Wheeler [Sat, 3 Dec 2016 09:31:34 +0000 (09:31 +0000)]
blktrace: Fix replay_align 32 bit truncation

When using replay_align the aligned offset was truncated when the
original offset was greater than 2^32. Fix this by forcing the
calculation to happen with 64 bits.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
4 years agoiolog: Ignore re-add/re-open with replay_redirect
Sitsofe Wheeler [Mon, 12 Dec 2016 12:11:39 +0000 (12:11 +0000)]
iolog: Ignore re-add/re-open with replay_redirect

The HOWTO entry for replay_redirect says:
"Replay_redirect causes all IOPS to be replayed onto
the single specified device regardless of the device it was
recorded from. i.e. replay_redirect=/dev/sdc would cause all
IO in the blktrace or iolog to be replayed onto /dev/sdc.
This means multiple devices will be replayed onto a single
device, if the trace contains multiple devices."

Unfortunately trying to replay the following iolog (saved as
redirtest.fiolog):
fio version 2 iolog
/dev/sdb add
/dev/sdc add
/dev/sdb open
/dev/sdc open
/dev/sdb read 0 512
/dev/sdc read 512 512
/dev/sdb close
/dev/sdc close

with an fio line like
fio --name=redirtest --replay_redirect=/dev/nullb0 --read_iolog=redirtest.fiolog
winds up hitting this assertion:
fio: ioengines.c:413: td_io_open_file: Assertion `!fio_file_open(f)' failed.
which occurs because after redirection is applied /dev/nullb0 is added and
opened twice.

Fix this by ignoring subsequent adds or opens for the same file during
iolog replay setup when replay_redirect is in effect.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
4 years agosteadystate: make test script work better under Windows
Vincent Fu [Fri, 9 Dec 2016 20:54:32 +0000 (15:54 -0500)]
steadystate: make test script work better under Windows

4 years agosteadystate: make file for read testing optional on posix systems
Vincent Fu [Fri, 9 Dec 2016 18:36:25 +0000 (13:36 -0500)]
steadystate: make file for read testing optional on posix systems

4 years agosteadystate: improve output of test script
Vincent Fu [Thu, 8 Dec 2016 20:21:51 +0000 (15:21 -0500)]
steadystate: improve output of test script

4 years agosteadystate: rename options->ss to options->ss_state since ss is used elsewhere to...
Vincent Fu [Thu, 8 Dec 2016 20:00:02 +0000 (15:00 -0500)]
steadystate: rename options->ss to options->ss_state since ss is used elsewhere to refer to struct steadystate_data

4 years agosteadystate: clean up checks for when steadystate termination is not engaged
Vincent Fu [Thu, 8 Dec 2016 19:49:57 +0000 (14:49 -0500)]
steadystate: clean up checks for when steadystate termination is not engaged

4 years agosteadystate: use uint64_t for storing bw and iops calculations and related values...
Vincent Fu [Wed, 7 Dec 2016 19:31:17 +0000 (14:31 -0500)]
steadystate: use uint64_t for storing bw and iops calculations and related values. Also fix an overflow with group_bw on 32-bit platforms.

4 years agosteadystate: fix alignment in stat.h
Vincent Fu [Tue, 6 Dec 2016 18:11:09 +0000 (13:11 -0500)]
steadystate: fix alignment in stat.h

This was not problematic in my Linux and OSX testing, but Windows clients did not successfully receive thread_stat without this padding.

4 years agosteadystate: ensure that pointers in thread_stat occupy the same amount of storage...
Vincent Fu [Tue, 6 Dec 2016 17:22:52 +0000 (12:22 -0500)]
steadystate: ensure that pointers in thread_stat occupy the same amount of storage in 32- and 64-bit platforms

4 years agosteadystate: rename TODO to STEADYSTATE-TODO
Vincent Fu [Fri, 2 Dec 2016 20:03:13 +0000 (15:03 -0500)]
steadystate: rename TODO to STEADYSTATE-TODO

4 years agosteadystate: instead of including ss_sum_y in thread_stat record whether ss_sum_y...
Vincent Fu [Fri, 2 Dec 2016 18:25:07 +0000 (13:25 -0500)]
steadystate: instead of including ss_sum_y in thread_stat record whether ss_sum_y is nonzero in ss_state via __FIO_SS_BUFFER_FULL

4 years agosteadystate: implement transmission of steadystate data over the wire in client/serve...
Vincent Fu [Fri, 2 Dec 2016 17:51:36 +0000 (12:51 -0500)]
steadystate: implement transmission of steadystate data over the wire in client/server mode

1) Pare down thread_stat steadystate members to only those that need to go over the wire
2) Plumb up sending/receiving of steadystate data including ring buffers

4 years agosteadystate: add line for output-format=normal
Vincent Fu [Wed, 23 Nov 2016 16:06:00 +0000 (11:06 -0500)]
steadystate: add line for output-format=normal

Output contains whether steady state was attained, mean bw & iops, and value of steady state criterion

4 years agosteadystate: get rid of ->ss_pct and encode this information in ->state via __FIO_SS_PCT
Vincent Fu [Tue, 22 Nov 2016 19:06:32 +0000 (14:06 -0500)]
steadystate: get rid of ->ss_pct and encode this information in ->state via __FIO_SS_PCT

This leaves a 32-bit filler in thread_options_pack that can be removed when it is possible to maintain required byte alignment

4 years agosteadystate: rename __FIO_SS_LAST to __FIO_SS_DATA
Vincent Fu [Tue, 22 Nov 2016 18:49:18 +0000 (13:49 -0500)]
steadystate: rename __FIO_SS_LAST to __FIO_SS_DATA

Use __FIO_SS_DATA to indicate whether a job's td contains pointers to the steadystate ring buffers. When group reporting is enabled this will be the last job in the reporting group. When group reporting is not enabled this flag will be set for every job.

4 years agosteadystate: update man page
Vincent Fu [Wed, 2 Nov 2016 19:34:22 +0000 (15:34 -0400)]
steadystate: update man page

4 years agosteadystate: add example job file
Vincent Fu [Fri, 28 Oct 2016 20:05:47 +0000 (16:05 -0400)]
steadystate: add example job file

4 years agommap engine: remove unused variable mmap_map_mask
Ross Zwisler [Mon, 5 Dec 2016 16:45:06 +0000 (09:45 -0700)]
mmap engine: remove unused variable mmap_map_mask

The global variable 'mmap_map_mask' was introduced in this commit:

commit ac8931124126 ("mmap engine: allow large files on 32-bit archs")

and was used in fio_mmapio_prep() to calculate f->mmap_off.  This usage was
removed in this commit:

commit ff455a04f4e7 ("mmap engine: fix for large files on 32-bit archs")

but the definition and setup code for 'mmap_map_mask' were left intact.

Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agolog: fix for crash with rate IO and logging
Jens Axboe [Fri, 2 Dec 2016 04:23:47 +0000 (21:23 -0700)]
log: fix for crash with rate IO and logging

If we run with an IO rate, and it happens to be low enough that
we never need to wait for the queue full condition, we will never
regrow the log and hence crash in stat.c on this line:

assert(iolog->pending->nr_samples < iolog->pending->max_samples);

Fix this by shuffling the code around a bit, so io_u_quiesce()
regrows the log, if needed, after having reaped completions.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix conversion in fio_server_send_ts
Vincent Fu [Thu, 1 Dec 2016 17:20:08 +0000 (12:20 -0500)]
Fix conversion in fio_server_send_ts

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoserver: bump protocol version
Sitsofe Wheeler [Sun, 27 Nov 2016 20:58:55 +0000 (20:58 +0000)]
server: bump protocol version

Previous commits change the size of various variables so signal
incompatibility.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
4 years agostat: Change access to io_sample union
Sitsofe Wheeler [Sun, 27 Nov 2016 17:07:10 +0000 (17:07 +0000)]
stat: Change access to io_sample union

Compiling on certain platforms produces warnings like

stat.c: In function ‘add_clat_sample’:
stat.c:2241:28: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]

because the size of a pointer is different to that of a long. Work
around this by explicitly naming the union used in io_sample, forcing
the "val" in the union to be a uint64_t, changing function parameters to
accept a union (rather than an unsigned long) and adding helper
functions to wrap values into a union.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
4 years agoSilence compiler warnings
Sitsofe Wheeer [Sun, 27 Nov 2016 07:53:29 +0000 (07:53 +0000)]
Silence compiler warnings

- Mark __parse_jobs_ini and fmt_desc[] as static as they aren't used
  outside their modules.
- Cast the shifted trim state enum to an unsigned int to silence
  spurious Wshift-sign-overflow warning.

Signed-off-by: Sitsofe Wheeer <sitsofe@yahoo.com>
4 years agoeta: Fix ETA oddness at crossover points
Sitsofe Wheeler [Sun, 27 Nov 2016 20:44:30 +0000 (20:44 +0000)]
eta: Fix ETA oddness at crossover points

- Avoid generating a NaN (because casting NaN to an integer is
  undefined) due to no work having been done by clamping ETA to the
  timeout in that scenario.
- Fix the ETA spiking just as ramp time finishes by not adding it to the
  ETA calculation when it's over.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
4 years agoeta: Fix ramp time ETA
Sitsofe Wheeler [Sun, 27 Nov 2016 17:13:27 +0000 (17:13 +0000)]
eta: Fix ramp time ETA

Ramp time ETA in thread_eta() was was assuming all rate directions were
being used and that bytes_total was in kilobytes.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
4 years agoioengines: Fix td->io_issues[ddir] over decrement
Sitsofe Wheeer [Sun, 27 Nov 2016 10:34:53 +0000 (10:34 +0000)]
ioengines: Fix td->io_issues[ddir] over decrement

If an ioengine returned FIO_Q_BUSY, td->io_issues[ddir] would be
over decremented: once in td_io_queue() and then again in
requeue_io_u(). This was because td_io_queue() reverted various values
upon finding the queue busy but requeue_io_u() found the I/O was still
in flight so decremented again. This then showed up as unsigned
underflow on td->io_issues[ddir].

Fix this by clearing IO_U_F_FLIGHT on the I/O when we find the queue was
busy in td_io_queue().

Signed-off-by: Sitsofe Wheeer <sitsofe@yahoo.com>
4 years agofio: Fix (unsigned) integer overflow issues
Sitsofe Wheeler [Sun, 20 Nov 2016 20:55:06 +0000 (20:55 +0000)]
fio: Fix (unsigned) integer overflow issues

This patch tries to address issues found when compiling fio with clang's
undefined behaviour sanitizer on 32 bit Linux. While this generated many
false positives it did identify a number of subtle issues when
rates/amounts/time calculations went above 32 bit values.

The following job was used:
[global]
time_based
runtime=180m
bs=256k
rw=randrw
norandommap
rate=512M
ioengine=null
direct=1
iodepth=8
thread
numjobs=10
ramp_time=20m

[null]
size=1P

It should fix https://github.com/axboe/fio/issues/270 because various 32
bit ETA variables were overflowing due to the microsecond arithmetic
being done on them (even on 64 bit systems).

This patch should also fix https://github.com/axboe/fio/issues/194 and
https://github.com/axboe/fio/issues/266 because 64 bit Windows is a
LLP64 platform so ints and longs still default to 32 bits.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
4 years agosteadystate: reject job if steadystate options are not consistent within reporting...
Vincent Fu [Fri, 28 Oct 2016 19:19:57 +0000 (15:19 -0400)]
steadystate: reject job if steadystate options are not consistent within reporting group

4 years agosteadystate: bug fixes
Vincent Fu [Thu, 27 Oct 2016 20:36:43 +0000 (16:36 -0400)]
steadystate: bug fixes

1) Fix steadystate option parsing callback bug
2) Re-insert missing multiplier in slope calculation
3) Revert 'attained' in the JSON output to be either 0 or 1

4 years agoImprove informativeness about directIO support or rather lackthereof on Solaris when...
Sam Zaydel [Wed, 16 Nov 2016 18:16:55 +0000 (10:16 -0800)]
Improve informativeness about directIO support or rather lackthereof on Solaris when errno is set to ENOTTY

4 years agoposixaio: fix bad type passed to memset
Jens Axboe [Tue, 15 Nov 2016 17:42:58 +0000 (10:42 -0700)]
posixaio: fix bad type passed to memset

We want the full array cleared, not just the first element.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix memory leak on tmp_buf
Colin Ian King [Mon, 14 Nov 2016 00:05:37 +0000 (00:05 +0000)]
Fix memory leak on tmp_buf

tmp_buf is allocated but not free'd, causing a minor memory leak

Signed-off-by: Colin Ian King <colin.king@canonical.com>
4 years agorbd: fix crash with zero sized image
Pan Liu [Sat, 12 Nov 2016 15:26:57 +0000 (23:26 +0800)]
rbd: fix crash with zero sized image

When the user create a rbd image with size zero, and specify the
"size" larger than zero in fio option, rbd will crash.

Signed-off-by: Pan Liu <pan.liu@istuary.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix duplicated typos from 42d97b5c in fio(1)
Tomohiro Kusumi [Wed, 2 Nov 2016 11:51:30 +0000 (20:51 +0900)]
Fix duplicated typos from 42d97b5c in fio(1)

Some typos are duplicated since fio(1) is written based on HOWTO,
(but not the same since commits that update HOWTO aren't always
in sync with fio(1)).

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoRemove extra space in tausworthe32 warning message
Bruce Cran [Sat, 29 Oct 2016 05:12:15 +0000 (23:12 -0600)]
Remove extra space in tausworthe32 warning message

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'HOWTO-cleanup' of https://github.com/szaydel/fio
Jens Axboe [Tue, 1 Nov 2016 15:10:44 +0000 (09:10 -0600)]
Merge branch 'HOWTO-cleanup' of https://github.com/szaydel/fio

4 years agoFix minor spelling mistakes in HOWTO document
Sam Zaydel [Tue, 1 Nov 2016 14:42:55 +0000 (07:42 -0700)]
Fix minor spelling mistakes in HOWTO document

4 years agobackend: cleanup check for completion/issue byte checking
Jens Axboe [Fri, 28 Oct 2016 18:34:02 +0000 (12:34 -0600)]
backend: cleanup check for completion/issue byte checking

Same code, just different array. Put it in a shared helper.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd blockdev_size() support for OpenBSD
Tomohiro Kusumi [Fri, 21 Oct 2016 17:07:33 +0000 (02:07 +0900)]
Add blockdev_size() support for OpenBSD

This commit is OpenBSD version of 1116dc04 in 2015.
This is exactly the same as the previous commit for NetBSD.

OpenBSD still has block device as a file type, thus it differs
from FreeBSD/DragonFlyBSD regarding FIO_HAVE_CHARDEV_SIZE.

Note that fio doesn't compile on OpenBSD 5.9 (hits lex error),
but this this commit should still be good since this function
alone compiles on OpenBSD.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd blockdev_size() support for NetBSD
Tomohiro Kusumi [Fri, 21 Oct 2016 17:07:32 +0000 (02:07 +0900)]
Add blockdev_size() support for NetBSD

This commit is NetBSD version of 1116dc04 in 2015.

NetBSD still has block device as a file type, thus it differs
from FreeBSD/DragonFlyBSD regarding FIO_HAVE_CHARDEV_SIZE.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFio 2.15 fio-2.15
Jens Axboe [Tue, 25 Oct 2016 18:38:13 +0000 (12:38 -0600)]
Fio 2.15

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUse the POSIX `EDEADLK` instead of the Linux `EDEADLOCK`
Bruce Cran [Mon, 24 Oct 2016 18:08:40 +0000 (12:08 -0600)]
Use the POSIX `EDEADLK` instead of the Linux `EDEADLOCK`

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: end IO loop early, if the job is marked as terminated
Jens Axboe [Mon, 24 Oct 2016 16:51:04 +0000 (10:51 -0600)]
backend: end IO loop early, if the job is marked as terminated

If we terminate a job, we don't want that job to attempt to grab
resources that might already have been freed.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofio: make job reap timeout 5 minutes
Jens Axboe [Mon, 24 Oct 2016 16:50:24 +0000 (10:50 -0600)]
fio: make job reap timeout 5 minutes

1 minute is a bit too slow, if we're waiting on a lot of IO syncing.
Let's make it 5 minutes. This will still catch someone hung on a
kernel bug, but hopefully should not trigger for normal waits.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agomutex: clear mutex when removed
Jens Axboe [Mon, 24 Oct 2016 16:49:48 +0000 (10:49 -0600)]
mutex: clear mutex when removed

This ensures that any attempt to grab it after it has been removed
will trigger an assert, instead of just silently hanging.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: if we can't grab stat_mutex, report a deadlock error and exit
Theodore Ts'o [Sun, 23 Oct 2016 14:35:14 +0000 (08:35 -0600)]
backend: if we can't grab stat_mutex, report a deadlock error and exit

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUse fmt -w WIDTH option instead of -WIDTH
Tomohiro Kusumi [Thu, 20 Oct 2016 14:03:18 +0000 (23:03 +0900)]
Use fmt -w WIDTH option instead of -WIDTH

Most systems support -WIDTH option which is an abbreviated form
of -w WIDTH or --width=WIDTH option, but some OS don't support
-WIDTH format.

As far as I've checked (on actual systems or online manpages),
Linux/FreeBSD/DragonFlyBSD/OpenBSD/Solaris/AIX/Cygwin support
-WIDTH format, but NetBSD doesn't support this (see below).
IRIX and HPUX may not support either according to online manpages.

-w WIDTH format is basically supported by all systems that have
fmt command, so this is probably better in terms of portability.

--
    CC crc/crc64.o
fmt: unknown option -- 1
Usage: fmt [-Cr] [-g <goal>] [-m|w <max>] [<files>..]
         fmt [-Cr] [<goal>] [<max>] [<files>]
    CC crc/md5.o
fmt: unknown option -- 1
Usage: fmt [-Cr] [-g <goal>] [-m|w <max>] [<files>..]
         fmt [-Cr] [<goal>] [<max>] [<files>]
...

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoRemove getopt_long_only macro from NetBSD header
Tomohiro Kusumi [Thu, 20 Oct 2016 14:03:17 +0000 (23:03 +0900)]
Remove getopt_long_only macro from NetBSD header

If CONFIG_GETOPT_LONG_ONLY isn't defined, fio is compiled with
its own getopt_long_only(), so this macro is no longer needed.
(os/os-netbsd.h was added before getopt_long_only() support)

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'rbd-poll'
Jens Axboe [Wed, 19 Oct 2016 15:24:04 +0000 (09:24 -0600)]
Merge branch 'rbd-poll'

4 years agorbd: poll cleanups
Jens Axboe [Wed, 19 Oct 2016 15:23:22 +0000 (09:23 -0600)]
rbd: poll cleanups

Would be nice to cleanup the ifdef in rbd_iter_events() as well.

Signed-off-by: Jens Axboe <axboe@fb.com>