fio.git
2 years agoadd ifdef __sun__ for Solaris specific code
Tomohiro Kusumi [Tue, 29 Aug 2017 21:12:13 +0000 (00:12 +0300)]
add ifdef __sun__ for Solaris specific code

ENOTTY on direct I/O setup failure via directio(3C) is Solaris (ZFS)
specific (as also mentioned in ZoL commit[1]).

Linux and many others which support O_DIRECT never come here, but
others supporting direct I/O via OS specific way don't necessarily
have the same ENOTTY behavior or ZFS itself.

This commit brings back the original log before 42f1ee68 for others.

[1] https://github.com/zfsonlinux/zfs/commit/a08c76a8ad63c28384ead72b53a3d7ef73f39357

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: add OpenBSD to direct I/O unsupported platform
Tomohiro Kusumi [Tue, 29 Aug 2017 21:12:12 +0000 (00:12 +0300)]
HOWTO: add OpenBSD to direct I/O unsupported platform

OpenBSD has neither open(O_DIRECT) nor its kernel specific way.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agocleanup NetBSD/OpenBSD header
Tomohiro Kusumi [Tue, 29 Aug 2017 21:12:11 +0000 (00:12 +0300)]
cleanup NetBSD/OpenBSD header

* NetBSD's <sys/rb.h> was renamed to <sys/rbtree.h> 7 years ago.
* Drop unneeded undefs.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'direct_layout_fix' of https://github.com/sitsofe/fio
Jens Axboe [Mon, 28 Aug 2017 15:57:30 +0000 (09:57 -0600)]
Merge branch 'direct_layout_fix' of https://github.com/sitsofe/fio

2 years agofilesetup: add direct=1 failure warning to layout
Sitsofe Wheeler [Mon, 21 Aug 2017 21:15:00 +0000 (22:15 +0100)]
filesetup: add direct=1 failure warning to layout

Add warning about the filesystem not liking direct=1/buffered=0 to
initial layout code.

v2:
- Update warning to include /buffered=0 so it matches other EINVAL
  warnings
- Minor commit message update

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agofilesetup: align layout buffer
Sitsofe Wheeler [Mon, 21 Aug 2017 21:15:00 +0000 (22:15 +0100)]
filesetup: align layout buffer

Since commit 6e344dc3445bfec0583072e82bea728ab8d54d58 ("filesetup: keep
OS_O_DIRECT flag when pre-allocating file") when direct=1 fio will set
the OS_O_DIRECT flag during file layout. Unfortunately this makes the
following job fail with due to an "Invalid argument" on Linux:

$ rm -f fiodirect.tmp; ./fio --direct=1 --bs=4k --filename=fiodirect.tmp \
 --size=16k --name=directtest
directtest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.0-9-gb427
Starting 1 process
directtest: Laying out IO file (1 file / 0MiB)
fio: pid=35410, err=22/file:filesetup.c:227, func=write, error=Invalid argument

(where fiodirect.tmp is on a filesystem that supports O_DIRECT such as
ext4 and the job used to succeed)

The open(2) man page for Linux says:

"Under Linux 2.4, transfer sizes, and the alignment of the user buffer
and the file offset must all be multiples of the logical block size of
the filesystem.  Since Linux 2.6.0, alignment to the logical block size
of the underlying storage (typically 512 bytes) suffices."

The above is cryptic but suggests that on Linux 2.6 and above kernels
the transfer size, the *memory alignment of the user buffer* and the
file offset must all be multiples of the underlying logical block size
when using O_DIRECT.

This commit forces the memory of fio's layout I/O buffer to be aligned
to the system's page size in an attempt to meet the memory alignment
requirement. Altough approximate, this should handle all but the most
unusual setups for the near future and matches what init_io_u() does in
backend.c.

v2:
- Align buffer pointer to page size rather than maximum block size
- Update commit message, include example

Fixes https://github.com/axboe/fio/issues/429 .

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoengines/windowsaio: kill useless forward declarations
Jens Axboe [Sun, 27 Aug 2017 21:21:26 +0000 (15:21 -0600)]
engines/windowsaio: kill useless forward declarations

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'doc_runtime' of https://github.com/sitsofe/fio
Jens Axboe [Sun, 27 Aug 2017 16:04:28 +0000 (10:04 -0600)]
Merge branch 'doc_runtime' of https://github.com/sitsofe/fio

2 years agoMerge branch 'fixes' of https://github.com/sitsofe/fio
Jens Axboe [Sun, 27 Aug 2017 16:03:38 +0000 (10:03 -0600)]
Merge branch 'fixes' of https://github.com/sitsofe/fio

2 years agoMerge branch 'stat_base_overflow' of https://github.com/football1222/fio
Jens Axboe [Wed, 23 Aug 2017 14:34:37 +0000 (08:34 -0600)]
Merge branch 'stat_base_overflow' of https://github.com/football1222/fio

2 years agostat: increase the size of base to avoid overflow
Richard Liu [Wed, 23 Aug 2017 10:49:58 +0000 (18:49 +0800)]
stat: increase the size of base to avoid overflow

plat_idx_to_val() in stat.c has a variable base that is defined as an
unsigned int but it was designed to fit old settings where
FIO_IO_U_PLAT_GROUP_NR = 19.
With the alteration from usec to nsec in commit
d6bb626ef37d3905221ade2887b422717a07af09(nanosecond: update completion
latency recording and normal, json output to use nanoseconds)
FIO_IO_U_PLAT_GROUP_NR is now 29 and the range of latency changed from
uint32_t to uint64_t. This means in plat_idx_to_val(), variable base
can easily exceed a 32-bit range and overflow after error_bits reaches 26.

Eg.
Group MSB #discarded range of #buckets
error_bits value
27 32 26 [42949672968589934591]     64
base = 1 << (error_bits + FIO_IO_U_PLAT_BITS);
(1 << 32) will overflow
if change to:
base = ((unsigned long long) 1) << (error_bits + FIO_IO_U_PLAT_BITS);
base itself will still overflow
so base should be unsigned long long

2 years agoMerge branch 'timespec_add_msec_overflow' of https://github.com/sitsofe/fio
Jens Axboe [Tue, 22 Aug 2017 16:32:18 +0000 (10:32 -0600)]
Merge branch 'timespec_add_msec_overflow' of https://github.com/sitsofe/fio

2 years agotime: fix overflow in timespec_add_msec
Sitsofe Wheeler [Tue, 22 Aug 2017 05:19:29 +0000 (06:19 +0100)]
time: fix overflow in timespec_add_msec

Commit 8b6a404cdd2c40715885e562416c3db039912773 ("nanosecond: initial
commit changing timeval to timespec") introduced a change to
timespec_add_msec that decreased the time to (unsigned) overflow of
adj_nsec leading to strange values in the write_*_log files when using
log_avg_msec. The following fio job reproduces the problem:

./fio --ioengine=null --size=2G --time_based --runtime=11s \
 --ramp_time=5s --write_iops_log=overflow --log_avg_msec=5000 \
 --name=overflow

cat overflow_iops.1.log shows the following:
5000, 997853, 0, 0
10640, 141323, 0, 0

Compiling fio with clang's undefined behaviour
sanitizer with unsigned wraparound detection turned on by using

CC=clang ./configure --extra-cflags="-fsanitize=undefined,integer"

and running the previous job produces a lot of false positives but also
flags up this:

time.c:11:35: runtime error: unsigned integer overflow: 1000000 * 5000 cannot be represented in type 'unsigned int'

Fix this by forcing the type of the integer literal to ensure that a 64
bit multiplication takes place. Further, force the result variables to
be uint64_t allowing us to remove UL from other literals. Finally adjust
some spacing while we're here.

Fixes https://github.com/axboe/fio/issues/426 (which was also reported
on the mailing list - http://www.spinics.net/lists/fio/msg06190.html ).

Reported-by: Joseph R Gruher <joseph.r.gruher@intel.com>
Suggested-by: Vincent Fu <vincent.fu@wdc.com>
Suggested-by: Jeff Furlong <jeff.furlong@wdc.com>
Reviewed-by: Vincent Fu <vincent.fu@wdc.com>
Tested-by: Joseph R Gruher <joseph.r.gruher@intel.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoREADME: update/add mintty issue links
Sitsofe Wheeler [Fri, 18 Aug 2017 21:07:15 +0000 (22:07 +0100)]
README: update/add mintty issue links

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoconfigure: clean up libverbs configure test
Sitsofe Wheeler [Thu, 17 Aug 2017 21:45:03 +0000 (22:45 +0100)]
configure: clean up libverbs configure test

Compiling the libverbs configure test generated warnings which would
cause libverbs to be detected as unavailable when using -Werror. Fix
this by changing an include to infiniband/verbs.h and eliminate the
unused stdio.h include while we're here.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agofio: implement 64 bit network/big endian byte swapping macros
Sitsofe Wheeler [Thu, 17 Aug 2017 06:37:01 +0000 (07:37 +0100)]
fio: implement 64 bit network/big endian byte swapping macros

Add cpu_to_be64/be64_to_cpu and make the RDMA ioengine use them so it no
longer needs to include infiniband/arch.h (which generates obsolete
warnings at compile time) or ntohll (which generates deprecation
warnings at compile time).

Tested-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agodoc: remove '--runtime' command line option
Sitsofe Wheeler [Sun, 13 Aug 2017 07:06:15 +0000 (08:06 +0100)]
doc: remove '--runtime' command line option

Commit 5bba8b3acee1bf1359470a82cac030942018405a ("Remove '--runtime'
command line option") removed the --runtime option from the fio tool and
its internal help. This commit removes it from the documentation files
too.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agodoc: latency log unit is nsec
Vincent Fu [Thu, 17 Aug 2017 18:05:33 +0000 (14:05 -0400)]
doc: latency log unit is nsec

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agotravis: install additional development libraries
Sitsofe Wheeler [Wed, 16 Aug 2017 06:34:01 +0000 (07:34 +0100)]
travis: install additional development libraries

Change the travis configuration to install development libraries and
headers for librbd (Ceph/rados), glusterfs, libibverbs and librdmacm on
Linux CI builds so we can check the related engines continue to build.

This reverses commit ff56a4e2e0a87e4e3b1cc1e74547d55b295967a6 ("travis:
don't enable rbd") because the previous commit now prevents the rbd
engine being built when the librbd found is too old.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoconfigure: fail rbd configure check on wrong rados_create2 signature
Sitsofe Wheeler [Wed, 16 Aug 2017 20:45:35 +0000 (21:45 +0100)]
configure: fail rbd configure check on wrong rados_create2 signature

Alter the rbd/rados/ceph configure check so it will fail if the
rados_create2() signature is incorrect or if the function is
unavailable. Completes the suggestion in commit
ff56a4e2e0a87e4e3b1cc1e74547d55b295967a6 ("travis: don't enable rbd").

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agofio2gnuplot: minor man page heading fix
Sitsofe Wheeler [Wed, 16 Aug 2017 06:29:50 +0000 (07:29 +0100)]
fio2gnuplot: minor man page heading fix

Fix the man page heading to specify the section and change the date to
be "month year" in English.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agorbd: fixup format specifier
Sitsofe Wheeler [Wed, 16 Aug 2017 06:14:02 +0000 (07:14 +0100)]
rbd: fixup format specifier

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoFio 3.0 fio-3.0
Jens Axboe [Wed, 16 Aug 2017 20:12:33 +0000 (14:12 -0600)]
Fio 3.0

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: fix wrong kb_base= description
Tomohiro Kusumi [Wed, 16 Aug 2017 20:09:30 +0000 (23:09 +0300)]
HOWTO: fix wrong kb_base= description

Definition of [KMGTP] and [KMGTP]i for kb_base=1000 are opposite.
It instead describes how it works when kb_base=1024 which is default.

This was originally spotted by Mikhail Terekhov in the mailing list
(in a related thread about a week ago, and the patch hasn't been sent).
http://www.spinics.net/lists/fio/msg06151.html

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agogfio: update copyright
Jens Axboe [Wed, 16 Aug 2017 15:10:12 +0000 (09:10 -0600)]
gfio: update copyright

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoprinting: use bigger on-stack buffer
Jens Axboe [Wed, 16 Aug 2017 15:08:27 +0000 (09:08 -0600)]
printing: use bigger on-stack buffer

We might be writing more than 20 bytes, bump up the size to
be safe.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: update description of normal output latencies
Vincent Fu [Mon, 14 Aug 2017 18:59:44 +0000 (14:59 -0400)]
man: update description of normal output latencies

1) Describe lat (min, max, avg, stddev) line
2) Use IO latencies description for lat (nsec/usec/msec) lines
3) Fix IO latency description

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: minor fixes and backports from man page
Tomohiro Kusumi [Mon, 14 Aug 2017 15:30:26 +0000 (18:30 +0300)]
HOWTO: minor fixes and backports from man page

These are minor fixes not worth separating into different commits,
which are partly backports from the previous 3 commits, and partly
its own minor fixes.

 * Use proper argument name (option -> command).
 * Use proper quotation (``).
 * Add missing :option:`` directives.
 * Add missing :file:`` directive (stat.h).
 * Add indentation.
 * Add missing alias (mem_align=).
 * Add missing period.
 * Add $ for fio command line examples
 * Cleanup ordering of some ioengine specific options.
 * Change inconsistent quotations for "1..N" and "N" to always use ``.
 * Avoid mixing of whitespace and tab indentation within the same section/block.
 * Correct fio commands' name (fio-genzipf, fio_generate_plots)
 * Correct typos ("used,where", "See disable_slat" in disable_slat)
 * Change "IO" to "I/O" which has been corrected in the past commits as well,
   except for "IO" in the output of fio itself.
 * Mention verification trigger (using magic sysrq) is Linux specific.
 * Mention "Disk stats" output (using sysfs) is Linux specific.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: minor fixes for sections before "JOB PARAMETERS" for consistency
Tomohiro Kusumi [Mon, 14 Aug 2017 15:30:25 +0000 (18:30 +0300)]
man: minor fixes for sections before "JOB PARAMETERS" for consistency

This is the continuation of below series of patches as well as the
previous 2 commits to sync the man page with HOWTO.

[PATCH 0/7] Sync fio(1) man page with HOWTO
http://www.spinics.net/lists/fio/msg06008.html

This commit consists of minor fixes to sync formats in the earlier
sections (made by above series of pathches) with the previous 2 commits.
The contents remain the same.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: sync "OUTPUT" section and after with HOWTO
Tomohiro Kusumi [Mon, 14 Aug 2017 15:30:24 +0000 (18:30 +0300)]
man: sync "OUTPUT" section and after with HOWTO

This is the continuation of below series of patches as well as the
previous commit to sync the man page with HOWTO. This commit covers the
rest of the sections.

[PATCH 0/7] Sync fio(1) man page with HOWTO
http://www.spinics.net/lists/fio/msg06008.html

This commit syncs the man page with HOWTO. It does not create new contents
that didn't exist in HOWTO, however formats (indentations, directives,
quotations, etc) are modified for some sections so that the entire man
page has consistent style.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: sync "JOB PARAMETERS" section with HOWTO
Tomohiro Kusumi [Mon, 14 Aug 2017 15:30:23 +0000 (18:30 +0300)]
man: sync "JOB PARAMETERS" section with HOWTO

This is the continuation of below series of patches to sync the man page
with HOWTO. This commit covers the entire job parameters section.

[PATCH 0/7] Sync fio(1) man page with HOWTO
http://www.spinics.net/lists/fio/msg06008.html

This commit syncs the man page with HOWTO. It does not create new contents
that didn't exist in HOWTO, however formats (indentations, directives,
quotations, etc) are modified for some sections so that the entire man
page has consistent style.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: update and clarify description of latencies in normal output
Vincent Fu [Mon, 14 Aug 2017 18:59:43 +0000 (14:59 -0400)]
HOWTO: update and clarify description of latencies in normal output

1) Add a description for the lat (min, max, avg, stddev) line
2) Use the old 'IO latencies' description for the lat (nsec/usec/msec) section
3) Fix the description of the 'IO latency' section

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agostat: change indentation of the lat (nsec/usec/msec) section in the normal output
Vincent Fu [Mon, 14 Aug 2017 18:59:42 +0000 (14:59 -0400)]
stat: change indentation of the lat (nsec/usec/msec) section in the normal output

With mixed IO the normal output looks like:

$ ./fio --name=test --ioengine=null --time_based --runtime=5s --size=1G --rw=randrw

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-2.21-14-ge22fd-dirty
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=5749MiB/s,w=5748MiB/s][r=1472k,w=1472k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=16555: Tue Jun 20 16:08:00 2017
   read: IOPS=1488k, BW=5814MiB/s (6096MB/s)(28.4GiB/5001msec)
...
  write: IOPS=1488k, BW=5812MiB/s (6094MB/s)(28.4GiB/5001msec)
...
    lat (nsec) : 20=15.47%, 50=83.90%, 100=0.49%, 250=0.11%, 500=0.02%
    lat (nsec) : 750=0.01%, 1000=0.01%
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=99.84%, sys=0.00%, ctx=181, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=7443094,7440426,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
...

The 'lat (nsec) : 20=15.47%...' section is indented inside the 'write' block,
giving the impression that these describe only write latencies when they are
actually latencies for all of the IOs in the job.

With this patch the normal output looks like:

$ ./fio --name=test --ioengine=null --time_based --runtime=5s --size=1G --rw=randrw

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-2.21-14-g4ed22a-dirty
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=5879MiB/s,w=5880MiB/s][r=1505k,w=1505k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=16686: Tue Jun 20 16:12:43 2017
   read: IOPS=1523k, BW=5950MiB/s (6239MB/s)(29.1GiB/5001msec)
    clat (nsec): min=15, max=114599, avg=25.38, stdev=139.48
     lat (nsec): min=47, max=257952, avg=58.18, stdev=269.48
    clat percentiles (nsec):
     |  1.00th=[   18],  5.00th=[   19], 10.00th=[   20], 20.00th=[   21],
     | 30.00th=[   22], 40.00th=[   23], 50.00th=[   23], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   27], 90.00th=[   27], 95.00th=[   28],
     | 99.00th=[   42], 99.50th=[   45], 99.90th=[  101], 99.95th=[  165],
     | 99.99th=[ 6368]
   bw (  MiB/s): min= 5722, max= 6081, per=0.10%, avg=5950.14, stdev=126.61
  write: IOPS=1523k, BW=5948MiB/s (6236MB/s)(29.0GiB/5001msec)
    clat (nsec): min=15, max=127752, avg=25.56, stdev=158.32
     lat (nsec): min=63, max=230121, avg=79.84, stdev=317.00
    clat percentiles (nsec):
     |  1.00th=[   18],  5.00th=[   19], 10.00th=[   20], 20.00th=[   21],
     | 30.00th=[   22], 40.00th=[   23], 50.00th=[   23], 60.00th=[   25],
     | 70.00th=[   26], 80.00th=[   27], 90.00th=[   27], 95.00th=[   28],
     | 99.00th=[   42], 99.50th=[   46], 99.90th=[  105], 99.95th=[  175],
     | 99.99th=[ 6624]
   bw (  MiB/s): min= 5727, max= 6079, per=0.10%, avg=5947.07, stdev=125.05
  lat (nsec)   : 20=9.40%, 50=90.27%, 100=0.22%, 250=0.08%, 500=0.01%
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=99.84%, sys=0.00%, ctx=150, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=7617184,7614394,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=5950MiB/s (6239MB/s), 5950MiB/s-5950MiB/s (6239MB/s-6239MB/s), io=29.1GiB (31.2GB), run=5001-5001msec
  WRITE: bw=5948MiB/s (6236MB/s), 5948MiB/s-5948MiB/s (6236MB/s-6236MB/s), io=29.0GiB (31.2GB), run=5001-5001msec

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'ci' of https://github.com/sitsofe/fio
Jens Axboe [Mon, 14 Aug 2017 14:32:56 +0000 (08:32 -0600)]
Merge branch 'ci' of https://github.com/sitsofe/fio

2 years agobackend: cleanup overlap submission logic
Jens Axboe [Mon, 14 Aug 2017 14:28:54 +0000 (08:28 -0600)]
backend: cleanup overlap submission logic

Move it into a helper, since we're calling it from two places.
Only check if depth is > 1, since the current io_u is already
accounted as one.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'serialize_overlap' of https://github.com/sitsofe/fio
Jens Axboe [Mon, 14 Aug 2017 14:24:12 +0000 (08:24 -0600)]
Merge branch 'serialize_overlap' of https://github.com/sitsofe/fio

2 years agoiolog: tidy up log_io_piece() conditional
Sitsofe Wheeler [Wed, 10 May 2017 05:33:31 +0000 (06:33 +0100)]
iolog: tidy up log_io_piece() conditional

Code inspection shows there are currently guards in the call sites of
log_io_piece() that prevent log_io_piece() being called when
td->o.verify == VERIFY_NONE so skip checking it within log_io_piece().

Tested-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoiolog: remove random layout verification optimisation
Sitsofe Wheeler [Thu, 27 Apr 2017 06:22:26 +0000 (07:22 +0100)]
iolog: remove random layout verification optimisation

Running the following fio jobs unexpectedly reports a verification
failure:
rm /tmp/tmp.fio; ./fio --iodepth=1 \
 --verify=pattern --verify_fatal=1 --size=100M --bsrange=512-128k \
 --rw=randwrite --verify_backlog=128 --filename=/tmp/tmp.fio \
 --verify_pattern="%o" --name=spuriousmismatch1

rm /tmp/tmp.fio; ./fio --iodepth=1 \
 --verify=crc32c --verify_fatal=1 --size=100M --bs=4k \
 --rw=randwrite --verify_backlog=20 --filename=/tmp/tmp.fio \
 --percentage_random=50 --randseed=86 --name=spuriousmismatch2

In the case of the first job, using a bsrange where the start and end
are different can cause random write I/O to overlap an already written
region making the original data unverifiable.

For the second job, when percentage_random is between 1 and 99 the same
offset can be generated multiple times but only the last write to that
offset should be verified.

Rather than special casing the growing number of random jobs that might
generate overlaps while still having a randommap, and given preallocation
during layout is the default where possible, just remove the overwrite=0
optimisation thus forcing all random jobs to be checked for overlaps. It
is still possible to force the old behaviour by setting verifysort=0.

Fixes https://github.com/axboe/fio/issues/335 and
https://github.com/axboe/fio/issues/344 .

Tested-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoiolog: fix double free when verified I/O overlaps
Sitsofe Wheeler [Thu, 27 Apr 2017 06:29:56 +0000 (07:29 +0100)]
iolog: fix double free when verified I/O overlaps

When running
valgrind ./fio --randseed=1 --ioengine=libaio --thread --rw=randrw \
 --random_distribution=zipf:1.4 --filename=/tmp/fiofile --io_limit=50M \
 --verify=crc32c --name=verifyfree --iodepth=32 --bsrange=512-1M --size=100M

valgrind reports:
==29301== Invalid read of size 4
==29301==    at 0x44ADFC: io_completed (io_u.c:1835)
==29301==    by 0x44B215: ios_completed (io_u.c:1924)
==29301==    by 0x44B683: io_u_queued_complete (io_u.c:1983)
==29301==    by 0x46FA6F: wait_for_completions (backend.c:455)
==29301==    by 0x471568: do_io (backend.c:1046)
==29301==    by 0x474405: thread_main (backend.c:1746)
==29301==    by 0x576E6B9: start_thread (pthread_create.c:333)
==29301==    by 0x5C8E82C: clone (clone.S:109)
==29301==  Address 0x62cf988 is 72 bytes inside a block of size 88 free'd
==29301==    at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)

When the __ipo is still in-flight we shouldn't free it when it overlaps
because it will be used at I/O completion time. Fixes
https://github.com/axboe/fio/issues/336 .

Tested-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agofio: add serialize_overlap option
Sitsofe Wheeler [Sun, 23 Apr 2017 21:54:54 +0000 (22:54 +0100)]
fio: add serialize_overlap option

If this isn't set (the default) fio can submit write I/Os that overlap
other in-flight I/Os leading to potential data races. For example the
following job frequently fails at the verification stage:

./fio --random_distribution=zipf:1.6 --direct=1 --filename \
 /tmp/fiofile --ioengine=posixaio --iodepth=32 --size=20M --bs=4k \
 --rw=randwrite --verify=crc32c --name=verifyoverlap

When serialize_overlap=1 fio avoids creating such races.

Thanks to Rachel Lunnon (StorMagic) for helping me debug the initial
version of this!

Fixes: https://github.com/axboe/fio/issues/335

v2: Fix merge conflict and add missing conversion.
v3: Add man page, fix serialize_overlap disabling, improve commit
    message.

Tested-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoci: make CI builds fail on compilation warnings
Sitsofe Wheeler [Sun, 13 Aug 2017 08:03:46 +0000 (09:03 +0100)]
ci: make CI builds fail on compilation warnings

Fio currently builds without warnings using its default options on at
least the platforms where CI is run (Linux, macOS, Windows).
Unfortunately it's easy to overlook changes that start generating
warnings on a platform you don't use regularly. Make this problem more
visible by changing appveyor/travis CI builds to fail when compilation
warnings are produced (but don't make this change for normal manual
builds).

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoMakefile: modify make test to use a filesystem file
Sitsofe Wheeler [Sun, 13 Aug 2017 07:47:40 +0000 (08:47 +0100)]
Makefile: modify make test to use a filesystem file

Change the second job done by "make test" to a use a real file on the
filesystem where fio is built to help highlight problems (like the one
introduced by commit 2c3e17be).

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoHOWTO: fix unit type suffix in "Parameter types" section to upper case
Tomohiro Kusumi [Wed, 9 Aug 2017 13:44:34 +0000 (16:44 +0300)]
HOWTO: fix unit type suffix in "Parameter types" section to upper case

Redo 9ff84231('HOWTO: fix unit type suffix in "Parameter types" section').

> These should be all in lower case or all in upper case, but not both.
> The man page already has these in lower case.

Change above to all upper case based on below comment, instead of all
in lower. Fix the man page (which originally had in lower) as well.

RE: [PATCH 2/4] HOWTO: fix unit type suffix in "Parameter types" section
http://www.spinics.net/lists/fio/msg06146.html
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoAdd ability to keep memory-mapped files
Stephen Bates [Tue, 8 Aug 2017 20:26:56 +0000 (14:26 -0600)]
Add ability to keep memory-mapped files

By default file backed memory mappings are unlink()'ed after use. This
patch keeps the files if they already existed. We don't check for
errors on access() since we will catch them on the open().

Discovered this when doing p2pmem testing and fio kept deleting the
/dev/p2pmem0 files...

Changes since v1
  Altered based on feedback from Jens to avoid using an option and
  test for file existance instead.

Signed-off-by: Stephen Bates <sbates@raithlin.com>
Changed by me to use a td->flags flag, instead of adding a new
member to thread options.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'fio-jsonplus-patches' of https://github.com/vincentkfu/fio
Jens Axboe [Mon, 7 Aug 2017 19:44:01 +0000 (13:44 -0600)]
Merge branch 'fio-jsonplus-patches' of https://github.com/vincentkfu/fio

2 years agoman: add section describing json+ output format
Vincent Fu [Mon, 7 Aug 2017 19:05:32 +0000 (15:05 -0400)]
man: add section describing json+ output format

2 years agoHOWTO: add section providing details about json+ output format
Vincent Fu [Thu, 27 Jul 2017 17:57:11 +0000 (13:57 -0400)]
HOWTO: add section providing details about json+ output format

2 years agotools: add fio_jsonplus_clat2csv
Vincent Fu [Wed, 28 Jun 2017 15:47:28 +0000 (11:47 -0400)]
tools: add fio_jsonplus_clat2csv

tools/fio_jsonplus_clat2csv converts JSON+ completion latency data to CSV
format

e6200d822f0d2b9da5600ad2429be0e9e140e125 eliminated an earlier script that
carried out a similar conversion for the original JSON+ format. I assumed
that the new JSON+ format would have no use for a similar script, but I was
wrong.

2 years agomove skip_bad= option to engines/mtd.c
Tomohiro Kusumi [Mon, 7 Aug 2017 17:37:54 +0000 (20:37 +0300)]
move skip_bad= option to engines/mtd.c

This is ioengine specific, thus should be defined in engines/mtd.c.

This option is useless unless engines/mtd.c is enabled.
It could be conditionally compiled using CONFIG_MTD, but all
other ioengines define their own option structure in their code.

The only functional difference is that now .category is changed
to FIO_OPT_C_ENGINE. This is also something done by all other
ioengine specific options, though ->category isn't really used
for anything meaningfull.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: use proper (or drop wrong usage of) option type =bool
Tomohiro Kusumi [Mon, 7 Aug 2017 17:37:53 +0000 (20:37 +0300)]
HOWTO: use proper (or drop wrong usage of) option type =bool

There are options whose .type field are FIO_OPT_BOOL, but not using
=bool for arg type. Likewise there are options whose .type field
are not FIO_OPT_BOOL, but using =bool for arg type.

The options without explicit arg types in HOWTO seem to be for
FIO_OPT_STR_SET, which is not the same as FIO_OPT_BOOL.

The man page also has these, but not the same as the ones in HOWTO.
Also fix a typo "(mtd)skipbad=bool" in the man page which should be
"(mtd)skip_bad=bool".

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: fix unit type suffix in "Parameter types" section
Tomohiro Kusumi [Mon, 7 Aug 2017 17:37:52 +0000 (20:37 +0300)]
HOWTO: fix unit type suffix in "Parameter types" section

These should be all in lower case or all in upper case, but not both.
Given the examples of these are mostly in lower case, this commit
uses lower.

The man page already has these in lower case.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: add proper indentation to "PARAMETER TYPES" section
Tomohiro Kusumi [Mon, 7 Aug 2017 17:37:51 +0000 (20:37 +0300)]
man: add proper indentation to "PARAMETER TYPES" section

This commit fixes "int" subsection of "PARAMETER TYPES" section
which had no indentation, while other sections normally do have
proper indentation.

The corresponding section in HOWTO also has indentation.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: minor fix and backport from man page
Tomohiro Kusumi [Tue, 1 Aug 2017 19:49:28 +0000 (22:49 +0300)]
HOWTO: minor fix and backport from man page

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: fix wrong "here follows the complete list of fio job parameters" position
Tomohiro Kusumi [Tue, 1 Aug 2017 19:49:27 +0000 (22:49 +0300)]
HOWTO: fix wrong "here follows the complete list of fio job parameters" position

f80dba8d('Convert documentation to reStructured text.')
made this sentence appear in the wrong place.
It should appear before "Units" job parameter section.

The man page has this right as it's not synced (will be soon).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: fix wrong description of trim_backlog=
Tomohiro Kusumi [Tue, 1 Aug 2017 19:49:26 +0000 (22:49 +0300)]
HOWTO: fix wrong description of trim_backlog=

f80dba8d('Convert documentation to reStructured text.')
mixed this up with trim_verify_zero= located before this one.

The man page has this right as it's not synced (will be soon).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'master' of https://github.com/dublio/fio
Jens Axboe [Mon, 31 Jul 2017 14:15:17 +0000 (08:15 -0600)]
Merge branch 'master' of https://github.com/dublio/fio

2 years agofilesetup: keep OS_O_DIRECT flag when pre-allocating file
weiping zhang [Sat, 29 Jul 2017 08:47:59 +0000 (16:47 +0800)]
filesetup: keep OS_O_DIRECT flag when pre-allocating file

When direct=1 the file open for pre-allocation should also set OS_O_DIRECT so
we fail earlier if the filesystem/file doesn't support direct operations.

Signed-off-by: weiping zhang <zhangweiping@didichuxing.com>
2 years agodon't print native_fallocate() error if ENOSYS
Tomohiro Kusumi [Thu, 27 Jul 2017 20:38:37 +0000 (23:38 +0300)]
don't print native_fallocate() error if ENOSYS

log_err(ENOSYS) on native_fallocate() failure which was newly added
by 2c3e17be('filesetup: add native fallocate') should be silenced
or somehow be printed only once instead of being a per file message.

This happens on a platform like FreeBSD where posix_fallocate(3)
exists, but native_fallocate() is ENOSYS.

This commit just silences it if errno is set to ENOSYS on return.
"native" is the default mode, thus not printing ENOSYS won't be any
confusing unless fallocate=native is explicitly specified, and
native_fallocate() has dprint() for ENOSYS case anyway.

--
 # uname
 FreeBSD
 # ./fio --name=xxx --ioengine=sync --rw=read --bs=4k --size=10m --nrfiles=20
 xxx: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
 fio-2.99-23-gdad0
 Starting 1 process
 xxx: Laying out IO files (20 files / total 10MiB)
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 ...

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agopvsync2: Add hipri_percentage option
Stephen Bates [Thu, 27 Jul 2017 20:02:15 +0000 (14:02 -0600)]
pvsync2: Add hipri_percentage option

Allow the user to set the probability of a pvsync2 IO being high
priority. Add a new option for the pvsync2 engine called
--hipri_percentage will takes values in the range (0, 100) and use
that as a probability for setting RWF_HIPRI. The default for this
option is 100 so exisiting users of --hirpi are unaffected.

Changes since v3:
  Remove unwanted #ifdefs

Changes since v2:
  s/_probability/_percentage/

Changes since v1:
  Use --hipri_probability rather than impacting --hipri
  Correct the random number generator call

Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'minor_fixes' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 27 Jul 2017 14:11:40 +0000 (08:11 -0600)]
Merge branch 'minor_fixes' of https://github.com/sitsofe/fio

2 years agofio: refactor fallocate defines
Sitsofe Wheeler [Thu, 27 Jul 2017 05:06:10 +0000 (06:06 +0100)]
fio: refactor fallocate defines

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoinit: force fallocate_mode to none when fallocate is unsupported
Sitsofe Wheeler [Thu, 27 Jul 2017 04:50:41 +0000 (05:50 +0100)]
init: force fallocate_mode to none when fallocate is unsupported

Commit 2c3e17be ("filesetup: add native fallocate") broke fio on Windows
because fallocate_mode wound up being set to 0 on platforms without any
type of fallocate. Work around this by forcing fallocate_mode to
FIO_FALLOCATE_NONE on such platforms.

This should address the issue mention in the mail
http://www.spinics.net/lists/fio/msg06085.html ("The commit 'filesetup:
add native fallocate' breaks on Cygwin").

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agodoc: minor grammar fixes
Sitsofe Wheeler [Thu, 20 Jul 2017 06:10:52 +0000 (07:10 +0100)]
doc: minor grammar fixes

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agodoc: add block size to log file format
Sitsofe Wheeler [Tue, 18 Jul 2017 06:46:52 +0000 (07:46 +0100)]
doc: add block size to log file format

Commit a3ae5b057588f555a1544ec749167a44c5a071aa ("Update documentation
on log file formats") added documentation for the log format but left
out the block size column so let's add it here.

While we're here clean up some formatting in log_offset, add a reference
to Log File Formats and mention that it defaults to off.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agotime: Add chosen clocksource debug
Sitsofe Wheeler [Sun, 9 Jul 2017 19:54:38 +0000 (20:54 +0100)]
time: Add chosen clocksource debug

Make it a bit easier to see which clocksource was used by printing its
enumeration value when --debug=time is used.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoexamples: add a butterfly seek job file
Sitsofe Wheeler [Sun, 9 Jul 2017 16:10:42 +0000 (17:10 +0100)]
examples: add a butterfly seek job file

Add a job file that approximates doing
read at start
read at end
read at start+1
read at end-1
[...]
read at end-1
read at start+1
read at end
read at start

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoarch: raise an error when compiling for an unknown ARM platform
Sitsofe Wheeler [Sun, 9 Jul 2017 07:32:47 +0000 (08:32 +0100)]
arch: raise an error when compiling for an unknown ARM platform

The nop, read_barrier() and write_barrier() macros have to always be
defined but due to the way os/arch-arm.h is written it is possible for
them to be undefined when the ARM platform is not recognized.

Change things so compiling for an unknown ARM platform raises an
explicit compilation error at the point of the problem.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoos: add missing include for bswap_* on BSDs
Sitsofe Wheeler [Sun, 9 Jul 2017 07:09:35 +0000 (08:09 +0100)]
os: add missing include for bswap_* on BSDs

Commit 0dcebdf4a70ef0d8144b8fcba763ae87e7fc74b5 ("configure: add endian
check") removed the sys/endian.h include from the os/*.h BSD headers but
this include is needed to access the bswap_* macros. The lack of this
include wouldn't show up unless you were compiling for a big-endian BSD
platform.

This commit adds the include back for {Free,Net,Open}BSD and adds it for
the first time to the DragonFlyBSD header.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agotravis: get rid of non-breaking space characters
Sitsofe Wheeler [Sun, 9 Jul 2017 06:48:16 +0000 (07:48 +0100)]
travis: get rid of non-breaking space characters

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoHOWTO: remove unnecessary escaping
Sitsofe Wheeler [Sun, 9 Jul 2017 06:28:56 +0000 (07:28 +0100)]
HOWTO: remove unnecessary escaping

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agodrop logging when blkdev invalidation failed on unsupported platforms
Tomohiro Kusumi [Tue, 25 Jul 2017 19:05:44 +0000 (22:05 +0300)]
drop logging when blkdev invalidation failed on unsupported platforms

22de5d77('Use ENOTSUP if OS doesn't support blkdev page cache invalidation')
changed errno values being used on supported platforms to ENOTSUP,
and also log_info'd it via strerror(), but strerror() isn't working
on some platform and instead prints irrelevant string.

Re: fio Windows Invalidate
http://www.spinics.net/lists/fio/msg06084.html

The logging of ENOTSUP could just be dropped since this isn't a real
failure, as many platforms either don't have the idea of invalidating
blkdev's page cache mapping, or use chrdev for block devices.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: add missing [netsplice] for ioengine specific options
Tomohiro Kusumi [Tue, 25 Jul 2017 19:05:43 +0000 (22:05 +0300)]
HOWTO: add missing [netsplice] for ioengine specific options

These seem to be supported whether a socket is read/written by
send/recv or splice.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: fix broken ioengine specific option format
Tomohiro Kusumi [Tue, 25 Jul 2017 19:05:42 +0000 (22:05 +0300)]
man: fix broken ioengine specific option format

The parentheses here are supposed to be for the name of ioengines,
but not for the name of options, according to other correct ones.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: fix "should be to to"
Tomohiro Kusumi [Tue, 25 Jul 2017 19:05:41 +0000 (22:05 +0300)]
HOWTO: fix "should be to to"

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoCorrectly detect whether ioengine_load can exit early
Ben Walker [Wed, 19 Jul 2017 19:27:57 +0000 (12:27 -0700)]
Correctly detect whether ioengine_load can exit early

Just the presence of io_ops isn't enough - the requested
ioengine may have changed.

Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2 years agobuild: Sort file list
Bernhard M. Wiedemann [Sun, 16 Jul 2017 19:46:31 +0000 (21:46 +0200)]
build: Sort file list

to make build reproducible.
Because filesystems/readdir returns entries in underministic order
and the linker orders functions depending on input order
the resulting fio binary differed across builds
(which we do in disposable VMs for openSUSE)

See https://reproducible-builds.org/ for why this matters.

Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: add missing \ for fcntl(2) in write_hint option
Tomohiro Kusumi [Sun, 16 Jul 2017 12:40:20 +0000 (15:40 +0300)]
man: add missing \ for fcntl(2) in write_hint option

8f4b9f24 ('Update documentation for write_hint') was missing \.
Without this '|' becomes visible.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: fix wrong info on sync_file_range=
Tomohiro Kusumi [Fri, 14 Jul 2017 07:24:03 +0000 (10:24 +0300)]
man: fix wrong info on sync_file_range=

This part isn't in sync with HOWTO, but also simply wrong.
HOWTO as well as the actual option have SYNC_FILE_RANGE_WAIT_AFTER for this.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoUpdate documentation for write_hint
Jens Axboe [Thu, 13 Jul 2017 15:39:01 +0000 (09:39 -0600)]
Update documentation for write_hint

We no longer have fadvise_stream, adapt the documentation to
describe the stable 'write_hint' option.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoparse: enable options to be marked dont-free
Jens Axboe [Wed, 12 Jul 2017 22:44:07 +0000 (16:44 -0600)]
parse: enable options to be marked dont-free

For profiles, we can't free the options prematurely.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoprofiles/act: use the right options
Jens Axboe [Wed, 12 Jul 2017 22:32:24 +0000 (16:32 -0600)]
profiles/act: use the right options

We have a double set - one we actually set, and then one we
check. Oops, fix that up.

Fixes: 7b504eddc4b0 ("parse: get rid of raw option offsets")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoreturn correct error code for unhandled addr.
Pan Liu [Mon, 10 Jul 2017 16:28:45 +0000 (00:28 +0800)]
return correct error code for unhandled addr.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
2 years agoFio 2.99 fio-2.99
Jens Axboe [Fri, 7 Jul 2017 18:45:04 +0000 (12:45 -0600)]
Fio 2.99

Test release before we wrap it up and make a 3.0.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoinit: add comment as to how we seed read/write/trim generators
Jens Axboe [Fri, 7 Jul 2017 14:37:49 +0000 (08:37 -0600)]
init: add comment as to how we seed read/write/trim generators

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: sequence random buflen generation individually
Jens Axboe [Fri, 7 Jul 2017 14:33:57 +0000 (08:33 -0600)]
io_u: sequence random buflen generation individually

For trimwrite, we rely on write+trim generating identical
buffer lengths, if they have identical settings. If we use
bssplit, then we generate them using a random generator
according to the percentages set. This means we may not get
the same offsets in sequence.

Split the seeding into separate random generators, and ensure
that we seed the write and trim ones identically, if we are
doing trim writes.

Fixes: https://github.com/axboe/fio/issues/399
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooslib/libmtd: kill dead code
Jens Axboe [Thu, 6 Jul 2017 14:09:22 +0000 (08:09 -0600)]
oslib/libmtd: kill dead code

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: fix indentation for options and job parameters
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:16 +0000 (21:08 +0300)]
HOWTO: fix indentation for options and job parameters

HOWTO mostly uses tab indentations for command line options and job
file parameters, so replace white spaces with tabs.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoHOWTO: minor backports from the man page
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:15 +0000 (21:08 +0300)]
HOWTO: minor backports from the man page

This commit mainly backports some minor fixups from the man page
sections covered in the several previous commits.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: refer to REPORTING-BUGS for bug reporting
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:14 +0000 (21:08 +0300)]
man: refer to REPORTING-BUGS for bug reporting

REPORTING-BUGS should be mentioned here rather than README, provided
REPORTING-BUGS also mentions to see README for the mailing list
in addition to details of bug reporting.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: sync "PARAMETER TYPES" section with HOWTO
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:13 +0000 (21:08 +0300)]
man: sync "PARAMETER TYPES" section with HOWTO

This commit brings in updates from HOWTO to man page. As a result it
replaces some entire paragraphs with the corresponding ones in HOWTO,
though not much difference for the replaced part as it only had small
modification basically saying the same thing.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: sync "JOB FILE PARAMETERS" section with HOWTO
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:12 +0000 (21:08 +0300)]
man: sync "JOB FILE PARAMETERS" section with HOWTO

This commit replaces the entire section with the corresponding section
in HOWTO, since the man page is only slightly different from HOWTO
regarding this section.

It also renames the following section names to sync with HOWTO.
  "JOB PARAMETERS" to "JOB FILE PARAMETERS"
  "JOB PARAMETERS"'s sub section "Types" to non-sub section "PARAMETER TYPES"
  "JOB PARAMETERS"'s sub section "Parameter List" to non-sub section "JOB DESCRIPTION"

Since section names aren't following the typical man page convention
(man-pages(7) in Linux) to begin with, changing the section names
to conform to HOWTO does no harm unless there's a reason to keep names
that were once made.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: sync "JOB FILE FORMAT" section with HOWTO
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:11 +0000 (21:08 +0300)]
man: sync "JOB FILE FORMAT" section with HOWTO

This commit brings in updates from HOWTO to man page. As a result it
replaces some entire paragraphs with the corresponding ones in HOWTO,
though not much difference for the replaced part as it only had small
modification basically saying the same thing.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman: sync OPTIONS section with HOWTO
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:10 +0000 (21:08 +0300)]
man: sync OPTIONS section with HOWTO

This commit brings in updates from HOWTO. The exception is --debug
option, which is too verbose yet not that important for all debug types
to be listed in details.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'fallocate_native' of https://github.com/sitsofe/fio
Jens Axboe [Mon, 3 Jul 2017 22:51:31 +0000 (16:51 -0600)]
Merge branch 'fallocate_native' of https://github.com/sitsofe/fio

2 years agoHOWTO: indent v3 terse output definition
Sitsofe Wheeler [Mon, 3 Jul 2017 21:16:13 +0000 (22:16 +0100)]
HOWTO: indent v3 terse output definition

Without indentation Sphinx complains:
../HOWTO:3336: WARNING: Literal block expected; none found.
../HOWTO:3336: WARNING: Literal block expected; none found.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agofilesetup: add native fallocate
Sitsofe Wheeler [Fri, 30 Jun 2017 06:32:19 +0000 (07:32 +0100)]
filesetup: add native fallocate

- Implement a native fallocate mode that only logs an error (if it's
  implemented) and falls back to the same behaviour as fallocate=none if
  it fails
- Add a native OSX and Linux fallocate
- Update man page and HOWTO with new native and defaults

v4:
- Update commit message
- Update man page and HOWTO

Fixes https://github.com/axboe/fio/issues/22 - we now print a message if
fallocate fails which gives the user a hint that behaviour might differ
from other case along with a workaround suggestion (fallocate=none).

Fixes https://github.com/axboe/fio/issues/376 - we now have
fallocate=none fallback behaviour if native fallocate fails. On at least
glibc platforms, posix_fallocate emulates fallocate if the
platform/filesystem doesn't support it (see
https://www.gnu.org/software/libc/manual/html_node/Storage-Allocation.html#index-posix_005ffallocate
for some of the caveats) resulting in strange layout performance thus
making it less attractive than fallocate=none behaviour for fio.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agogettime: reduce test CPU clock entries to 1000
Jens Axboe [Sun, 2 Jul 2017 22:21:47 +0000 (16:21 -0600)]
gettime: reduce test CPU clock entries to 1000

We don't need 10,000 and it's just wasting time.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofilesetup: abstract out fallocate helper
Jens Axboe [Sun, 2 Jul 2017 22:15:32 +0000 (16:15 -0600)]
filesetup: abstract out fallocate helper

This is in preparation for improving how we handle fallocate.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofio: make gauss a duplicate of normal for file_service_type
Sitsofe Wheeler [Wed, 28 Jun 2017 21:53:06 +0000 (22:53 +0100)]
fio: make gauss a duplicate of normal for file_service_type

To be consistent with random_distribution add a normal parameter to
file_service_type and state that gauss is an alias of it.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agodoc: fix random_distribution Gaussian parameter name
Sitsofe Wheeler [Wed, 28 Jun 2017 21:30:24 +0000 (22:30 +0100)]
doc: fix random_distribution Gaussian parameter name

random_distribution's parameter for a Gaussian distribution is called
"normal". Note this differs to file_service_type where the Gaussian
distribution really is selected with the "gauss" parameter.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>