fio.git
2 years agoFio 2.20 fio-2.20
Jens Axboe [Fri, 19 May 2017 14:25:27 +0000 (08:25 -0600)]
Fio 2.20

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agostat: Re-add output of basic bw information if bw_log is not written
Andreas Herrmann [Fri, 19 May 2017 14:09:27 +0000 (16:09 +0200)]
stat: Re-add output of basic bw information if bw_log is not written

Starting with 9e4438f ("stat: don't trust per_unit_log() if log is
NULL") fio no longer provides basic bw information (e.g. min, max,
p_agg, mean, stdev) when write_bw_log is not used. Thus you have to
write a bw_log to obtain these values. (But in this case you can
compute those values yourself.) I think this doesn't make sense.  Not
writing entire bw log shouldn't prevent general sampling and
calculation of bw stats.

This patch reactivates sampling for bw and iops to be able to
calculate basic stats for both independend of writing of entire
{bw,iops}_logs.

Also check for log that is really of interest in __add_samples() it
doesn't make sense to check for bw_log if iops information needs to be
written.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Fixes: 9e4438f ("stat: don't trust per_unit_log() if log is NULL")
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoRemove leftover warnings
Jens Axboe [Thu, 18 May 2017 18:53:38 +0000 (12:53 -0600)]
Remove leftover warnings

Previous commit that fixed up the preadv2/pwritev2 wrappers
left some debug #warning in there by accident. Kill them.

Fixes: 7b5c648f6a5c ("Fix wrap issue with 64-bit pwritev2/preadv2")
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoFix wrap issue with 64-bit pwritev2/preadv2
Jens Axboe [Thu, 18 May 2017 17:59:09 +0000 (11:59 -0600)]
Fix wrap issue with 64-bit pwritev2/preadv2

Apparently I missed that Linux does a clever trick to optimize
how we pass in lo/hi offsets for 64-bit, and it encodes the
full offset in just the low part. This caused corruption when
writing with pwritev2 for laver sizes, and bad data for preadv2
for larger sizes.

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoman page: include reference to version 4 of the terse format
Jens Axboe [Wed, 10 May 2017 03:08:12 +0000 (21:08 -0600)]
man page: include reference to version 4 of the terse format

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agooptions: force refill_buffers with pattern and any reads
Jens Axboe [Thu, 4 May 2017 14:43:27 +0000 (08:43 -0600)]
options: force refill_buffers with pattern and any reads

This will cause writes to always refill the buffer, which is needed
to avoid having the reads garble the contents.

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agogettime: make utime_since_now and mtime_since_now consistent in how they record the...
Vincent Fu [Sun, 16 Apr 2017 19:04:28 +0000 (15:04 -0400)]
gettime: make utime_since_now and mtime_since_now consistent in how they record the caller and put this all behind FIO_DEBUG_TIME

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoclient/server: make sure that all elements in io_u_lat_m[] are transferred and received
Vincent Fu [Sun, 16 Apr 2017 19:04:27 +0000 (15:04 -0400)]
client/server: make sure that all elements in io_u_lat_m[] are transferred and received

FIO_IO_U_LAT_U_NR is 10 whereas FIO_IO_U_LAT_M_NR is 12. So we must iterate
over io_u_lat_u and io_u_lat_m separately.

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agostat: reset_io_stats: fix a problem, rearrange some code
Vincent Fu [Sun, 16 Apr 2017 19:04:26 +0000 (15:04 -0400)]
stat: reset_io_stats: fix a problem, rearrange some code

1) Make sure all the io_u_lat_u/m are reset
2) Only need to zero out total_submit/complete once
3) Reset total/short/drop_io_u inside the other loop that iterates over data directions

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoRevert "tools/fio_latency2csv.py: add tool that converts json+ to CSV"
Vincent Fu [Sun, 16 Apr 2017 19:04:24 +0000 (15:04 -0400)]
Revert "tools/fio_latency2csv.py: add tool that converts json+ to CSV"

This reverts commit 6cd4d583c75042bf1edd3d2d5c6626883755bfe4.
The script is no longer needed because the json+ output format now
automatically does the transformation carried out by the script.

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agostat: change json+ output format so that instead of printing the raw clat data struct...
Vincent Fu [Sun, 16 Apr 2017 19:04:23 +0000 (15:04 -0400)]
stat: change json+ output format so that instead of printing the raw clat data structure, use actual durations instead of array indices and print only bins with nonzero counts

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMerge branch 'wip-remove-disconnect' of https://github.com/liupan1111/fio
Jens Axboe [Wed, 3 May 2017 14:14:09 +0000 (08:14 -0600)]
Merge branch 'wip-remove-disconnect' of https://github.com/liupan1111/fio

2 years agoremove redundant _fio_rbd_disconnect, which is already called in
Pan Liu [Wed, 3 May 2017 11:20:14 +0000 (19:20 +0800)]
remove redundant _fio_rbd_disconnect, which is already called in
fio_rbd_cleaup

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
2 years agoremove out-of-date comment
Pan Liu [Wed, 3 May 2017 11:18:14 +0000 (19:18 +0800)]
remove out-of-date comment

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
2 years agoconfigure: Add missing $val != "yes" test to override compile_prog() result
Tomohiro Kusumi [Mon, 1 May 2017 18:48:19 +0000 (21:48 +0300)]
configure: Add missing $val != "yes" test to override compile_prog() result

Since it's possible for some platforms to use statically configured
configurations (e.g. see around L350 for Cygwin), it should test if
the variable hasn't been set to "yes" before initializing the variable
with "no", so that compile_prog() result can be safely ignored when
the platform knows it's supported.

ca205a75 (configure: Make Cygwin take regular configure path) started
this due to a major change in the configuration path, which changed
Cygwin to take the normal path taken by all other platforms, and
these two were added after that change.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoconfigure: Add missing ##########...
Tomohiro Kusumi [Mon, 1 May 2017 18:48:18 +0000 (21:48 +0300)]
configure: Add missing ##########...

And remove unneeded blank lines from embedded test C code.
Both are just for better readability.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoconfigure: Check gfio test result via return value (not printf)
Tomohiro Kusumi [Mon, 1 May 2017 18:48:17 +0000 (21:48 +0300)]
configure: Check gfio test result via return value (not printf)

Test code compiled by compile_prog() mostly return 0 on success
whether binaries are executed or not. This one actually runs and
the result can be checked without printf.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoconfigure: Add void* cast to avoid bogus warning
Tomohiro Kusumi [Mon, 1 May 2017 18:48:16 +0000 (21:48 +0300)]
configure: Add void* cast to avoid bogus warning

This warning has nothing to do with what the code is trying to test.
This test code should keep it volatile, so cast to void* to silence
a warning on discarding volatile.

--
 # uname
 Linux
 # grep memset config.log -A2 | head -3
 /tmp/fio-conf-15226-60011-1419.c:6:10: warning: passing argument 1 of 'memset' discards 'volatile' qualifier from pointer target type [-Wdiscarded-qualifiers]
    memset(&cid, 0, sizeof(cid));
           ^

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoconfigure: Add missing <string.h> to avoid bogus warning
Tomohiro Kusumi [Mon, 1 May 2017 18:48:15 +0000 (21:48 +0300)]
configure: Add missing <string.h> to avoid bogus warning

This warning has nothing to do with what the code is trying to test.
This isn't testing if memset(3) is available, so just silence the
warning by including an appropriate header.

--
 # uname
 Linux
 # grep memset config.log -A2 | head -3
 /tmp/fio-conf-23441-58170-6422.c:5:3: warning: implicit declaration of function 'memset' [-Wimplicit-function-declaration]
    memset(&cid, 0, sizeof(cid));
    ^~~~~~

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoconfigure: output_sym CONFIG_GFIO
Tomohiro Kusumi [Mon, 1 May 2017 18:48:14 +0000 (21:48 +0300)]
configure: output_sym CONFIG_GFIO

No reason not to output_sym CONFIG_GFIO, when all others do this.
Having CONFIG_GFIO in config-host.h doesn't harm anything.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoos/os-android.h: fix shared memory support
Omri Mor [Mon, 1 May 2017 00:09:22 +0000 (17:09 -0700)]
os/os-android.h: fix shared memory support

Fixes: #352 ("Android and shared memory")
Signed-off-by: Omri Mor <omor1@asu.edu>
2 years agoDrop triple X for cpu affinity for OpenBSD
Tomohiro Kusumi [Sun, 30 Apr 2017 17:41:00 +0000 (20:41 +0300)]
Drop triple X for cpu affinity for OpenBSD

(Sorry, resending this as gmail refused an email title including triple X)

This isn't really a fio's TODO item since no such thing seems to
exist (or have existed) in OpenBSD as of 2017, according to below
reference.

NetBSD is in more or less similar situation, except that NetBSD
has pthread_[gs]etaffinity_np() for threads, though it has another
problem that they don't take pid/tid.

CPU affinity comes to OpenBSD (2009)
http://undeadly.org/cgi?action=article&sid=20090324210236

tedu old comment about cpu affinity. (2012)
http://openbsd-archive.7691.n7.nabble.com/tedu-old-comment-about-cpu-affinity-tt172330.html

how to use cpu affinity from user space (2013)
http://openbsd-archive.7691.n7.nabble.com/how-to-use-cpu-affinity-from-user-space-td222323.html

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoImplement shm_attach_to_open_removed() for OpenBSD
Tomohiro Kusumi [Sun, 30 Apr 2017 17:35:10 +0000 (20:35 +0300)]
Implement shm_attach_to_open_removed() for OpenBSD

Add implementation which had been left by
0cfe2089 (Add runtime handlers for 97900ebf for FreeBSD/DragonFlyBSD).

This implementation is based on what
97900ebf (shm: have os remove shared memory if fio dies unexpectedly)
says. There seems to be no easy way to statically test versions like
KERNEL_VERSION() in Linux kernel.

The code assumes major/minor are both < 10. The minor has never been
10 or above. The major won't be 10 anytime soon based on the past
release cycle (current version is 6.1 after more than 2 decades).
https://en.wikipedia.org/wiki/OpenBSD_version_history

--
 # uname
 OpenBSD
 # uname -r
 6.1
 # cat ./test1.c
 #include <stdio.h>
 #include "./os/os.h"
 #include "./os/os-openbsd.h"
 int main(void) {
         printf("%d\n", shm_attach_to_open_removed());
         return 0;
 }
 # gcc -Wall -g ./test1.c
 # ./a.out
 1

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoTurn off lex by default on OpenBSD
Tomohiro Kusumi [Sun, 30 Apr 2017 17:35:09 +0000 (20:35 +0300)]
Turn off lex by default on OpenBSD

Do the same as what AIX does. OpenBSD (at least recent ones) hits
below syntax error. Feel free to revert this commit and properly
fix this if possible.

CC y.tab.c
exp/expression-parser.y:68: syntax error
%parse-param { long long *result }
^
gmake: *** [Makefile:341: y.tab.c] Error 1

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoFix "'RB_ROOT' undeclared" error on OpenBSD
Tomohiro Kusumi [Sun, 30 Apr 2017 17:35:08 +0000 (20:35 +0300)]
Fix "'RB_ROOT' undeclared" error on OpenBSD

OpenBSD has below compiler error due to RB_ROOT name conflict between
lib/rbtree.h and OpenBSD header. os/os-{net,open}bsd.h actually have
had a workaround for this by undefing their RB_XXX macros (which fio
doesn't use) for possible conflicts, but then "lib/rbtree.h" needs
to be included after os/os.h which includes os/os-openbsd.h.

t/dedupe.c: In function 'main':
t/dedupe.c:583: error: 'RB_ROOT' undeclared (first use in this function)
t/dedupe.c:583: error: (Each undeclared identifier is reported only once
t/dedupe.c:583: error: for each function it appears in.)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoFix "C99 inline functions are not supported" warning on OpenBSD
Tomohiro Kusumi [Sun, 30 Apr 2017 17:35:07 +0000 (20:35 +0300)]
Fix "C99 inline functions are not supported" warning on OpenBSD

OpenBSD has below compiler warning due to hist_sum() being inlined.
It should be fine with no inline (or move it to iolog.h with static
if we somehow want to keep it inlined).

iolog.c:702: warning: C99 inline functions are not supported; using GNU89
iolog.c:702: warning: to disable this warning use -fgnu89-inline or the gnu_inline function attribute

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoFix "cast from pointer to integer of different size" warning on OpenBSD
Tomohiro Kusumi [Sun, 30 Apr 2017 17:35:06 +0000 (20:35 +0300)]
Fix "cast from pointer to integer of different size" warning on OpenBSD

OpenBSD has below compiler warning due to pthread_t being typedef'd
to struct pthread*.

 os/os-openbsd.h: In function 'gettid':
 os/os-openbsd.h:71: warning: cast from pointer to integer of different size

--
 # uname
 OpenBSD
 # grep pthread_t /usr/include -rI | grep typedef | head -1
 /usr/include/pthread.h:typedef struct   pthread                 *pthread_t;
 # cat ./p1.c
 #include <stdio.h>
 #include <pthread.h>
 int main(void) {
         pthread_t id = 1234;
         printf("%d\n", (int)id);
         return 0;
 }
 # gcc -Wall -g ./p1.c
 ./p1.c: In function 'main':
 ./p1.c:4: warning: initialization makes pointer from integer without a cast
 ./p1.c:5: warning: cast from pointer to integer of different size
 # ./a.out
 1234

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMerge branch 'zero_ioengine_flags' of https://github.com/sitsofe/fio
Jens Axboe [Sun, 30 Apr 2017 22:49:22 +0000 (16:49 -0600)]
Merge branch 'zero_ioengine_flags' of https://github.com/sitsofe/fio

2 years agoMerge branch 'fix-348' of https://github.com/omor1/fio
Jens Axboe [Sun, 30 Apr 2017 22:48:24 +0000 (16:48 -0600)]
Merge branch 'fix-348' of https://github.com/omor1/fio

2 years agoos/os-linux.h: fix broken byte swap intrinsics
Omri Mor [Sun, 30 Apr 2017 22:25:06 +0000 (15:25 -0700)]
os/os-linux.h: fix broken byte swap intrinsics

Signed-off-by: Omri Mor <omor1@asu.edu>
2 years agoos/os-android.h: use byte swap intrinsics if available
Omri Mor [Sun, 30 Apr 2017 22:23:39 +0000 (15:23 -0700)]
os/os-android.h: use byte swap intrinsics if available

Signed-off-by: Omri Mor <omor1@asu.edu>
2 years agoos/os-android.h: fix broken shmget() due to ndk-r15
Omri Mor [Sun, 30 Apr 2017 21:32:03 +0000 (14:32 -0700)]
os/os-android.h: fix broken shmget() due to ndk-r15

Fixes: #346 ("Compiling for Android using ndk-r15")
Signed-off-by: Omri Mor <omor1@asu.edu>
2 years agofio.h: zero old flag bits when setting new ioengine flags
Sitsofe Wheeler [Sat, 29 Apr 2017 08:46:43 +0000 (09:46 +0100)]
fio.h: zero old flag bits when setting new ioengine flags

Fix a small bug in td_set_ioengine_flags() where it wouldn't zero old
bits in td->flags before setting new flag bits.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoconfigure: fix broken test for cuda
Jens Axboe [Wed, 26 Apr 2017 21:24:36 +0000 (15:24 -0600)]
configure: fix broken test for cuda

Fixes: 03553853 ("GPUDirect RDMA support")
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agothread_options: kill two unused pads
Jens Axboe [Wed, 26 Apr 2017 21:22:50 +0000 (15:22 -0600)]
thread_options: kill two unused pads

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoserver: bump protocol version
Jens Axboe [Wed, 26 Apr 2017 21:06:24 +0000 (15:06 -0600)]
server: bump protocol version

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agogpu: kill a lot of useless ifdefs
Jens Axboe [Wed, 26 Apr 2017 21:05:07 +0000 (15:05 -0600)]
gpu: kill a lot of useless ifdefs

We just have to guard how we set the mem type, we don't have
to guard any check for MEM_CUDA_MALLOC inside CONFIG_CUDA
ifdefs.

Additionally, fix up some bad style.

Fixes: 03553853 ("GPUDirect RDMA support")
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agogpu: ensure that we convert gpu_dev_id options
Jens Axboe [Wed, 26 Apr 2017 21:04:21 +0000 (15:04 -0600)]
gpu: ensure that we convert gpu_dev_id options

Fixes: 03553853 ("GPUDirect RDMA support")
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMerge branch 'gpudirect' of https://github.com/yufeiren/fio
Jens Axboe [Wed, 26 Apr 2017 20:54:22 +0000 (14:54 -0600)]
Merge branch 'gpudirect' of https://github.com/yufeiren/fio

2 years agoGPUDirect RDMA support
Yufei Ren [Wed, 19 Apr 2017 19:42:13 +0000 (15:42 -0400)]
GPUDirect RDMA support

2 years agoseqlock: add simple user space code for sequence locks
Jens Axboe [Wed, 26 Apr 2017 20:20:21 +0000 (14:20 -0600)]
seqlock: add simple user space code for sequence locks

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake oslib/linux-dev-lookup.c a stand-alone library
Tomohiro Kusumi [Fri, 14 Apr 2017 21:06:26 +0000 (00:06 +0300)]
Make oslib/linux-dev-lookup.c a stand-alone library

It needs to include linux-dev-lookup.h from local directory where
linux-dev-lookup.c is also located, to be able to use this as
a stand-alone library, which is useful for debugging purpose.
In fact, most of the files under oslib/ directory do things this way.

It also doesn't need to include os/os.h as it has no dependency.

--
 # cat ./test10.c
 #include <stdio.h>
 #include "oslib/linux-dev-lookup.h"
 int main(void) {
         /* just to see if it compiles */
         blktrace_lookup_device(NULL, NULL, 0, 0);
         return 0;
 }
 # gcc -Wall -g ./test10.c oslib/linux-dev-lookup.c

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake oslib/strlcat.c a stand-alone library
Tomohiro Kusumi [Fri, 14 Apr 2017 21:06:25 +0000 (00:06 +0300)]
Make oslib/strlcat.c a stand-alone library

It needs to include strlcat.h from local directory where strlcat.c
is also located, to be able to use this as a stand-alone library,
which is useful for debugging purpose. In fact, most of the files
under oslib/ directory do things this way.

--
 # cat ./test9.c
 #include <stdio.h>
 #include "oslib/strlcat.h"
 int main(void) {
         /* just to see if it compiles */
         strlcat(NULL, NULL, 0);
         return 0;
 }
 # gcc -Wall -g ./test9.c ./oslib/strlcat.c

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake lib/mountcheck.c a stand-alone library
Tomohiro Kusumi [Fri, 14 Apr 2017 21:06:24 +0000 (00:06 +0300)]
Make lib/mountcheck.c a stand-alone library

It needs to include mountcheck.h from local directory where
mountcheck.c is also located, to be able to use this as
a stand-alone library, which is useful for debugging purpose.
In fact, most of the files under lib/ directory do things this way.

--
 # cat ./test11.c
 #include <stdio.h>
 #include "lib/mountcheck.h"
 int main(int argc, char **argv) {
         printf("%d\n", device_is_mounted(argv[1]));
         return 0;
 }

--
 # uname
 Linux
 # gcc -Wall -g -DCONFIG_GETMNTENT ./test11.c ./lib/mountcheck.c
 # ./a.out /dev/sda1
 1

--
 # uname
 DragonFly
 # gcc -Wall -g -DCONFIG_GETMNTINFO ./test11.c ./lib/mountcheck.c
 # ./a.out /dev/da0s1a
 1

--
 # uname
 NetBSD
 # gcc -Wall -g -DCONFIG_GETMNTINFO_STATVFS ./test11.c ./lib/mountcheck.c
 # ./a.out /dev/sd0a
 1

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake lib/zipf.c a stand-alone library
Tomohiro Kusumi [Fri, 14 Apr 2017 21:06:23 +0000 (00:06 +0300)]
Make lib/zipf.c a stand-alone library

lib/zipf.c not having dependency on fio header (log.h) enables it
to be a stand-alone library, which is useful for debugging purpose.
In fact, most of the files under lib/ directory do things this way.

lib/zipf.c doesn't use log.h (log(3) is apparently from -lm).

This requires the previous commit.

--
 # cat ./test8.c
 #include <stdio.h>
 #include "lib/zipf.h"
 int main(void) {
         /* just to see if it compiles */
         zipf_init(NULL, 0, 0, 0);
         return 0;
 }
 # gcc -Wall -g -lm -DBITS_PER_LONG=64 ./test8.c ./lib/zipf.c ./lib/rand.c ./lib/pattern.c ./lib/strntol.c

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake lib/rand.c a stand-alone library
Tomohiro Kusumi [Fri, 14 Apr 2017 21:06:22 +0000 (00:06 +0300)]
Make lib/rand.c a stand-alone library

It needs to include pattern.h from local directory where rand.c
is also located, to be able to use this as a stand-alone library,
which is useful for debugging purpose. In fact, most of the files
under lib/ directory do things this way.

--
 # cat ./test7.c
 #include <stdio.h>
 #include "lib/rand.h"
 int main(void) {
         /* just to see if it compiles */
         init_rand(NULL, 0);
         return 0;
 }
 # gcc -Wall -g -DBITS_PER_LONG=64 ./test7.c ./lib/rand.c ./lib/pattern.c ./lib/strntol.c

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake lib/pattern.c a stand-alone library
Tomohiro Kusumi [Fri, 14 Apr 2017 21:06:21 +0000 (00:06 +0300)]
Make lib/pattern.c a stand-alone library

lib/pattern.c not having dependency on fio.h enables it to be
a stand-alone library, which is useful for debugging purpose.
In fact, most of the files under lib/ directory do things this way.

This requires the previous commit.

--
 # cat ./test6.c
 #include <stdio.h>
 #include "lib/pattern.h"
 int main(void) {
         /* just to see if it compiles */
         paste_format_inplace(NULL, 0, NULL, 0, NULL);
         return 0;
 }
 # gcc -Wall -g ./test6.c ./lib/pattern.c ./lib/strntol.c

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake lib/strntol.c a stand-alone library
Tomohiro Kusumi [Fri, 14 Apr 2017 21:06:20 +0000 (00:06 +0300)]
Make lib/strntol.c a stand-alone library

It needs to include strntol.h from local directory where strntol.c
is also located, to be able to use this as a stand-alone library,
which is useful for debugging purpose. In fact, most of the files
under lib/ directory do things this way.

--
 # cat ./test5.c
 #include <stdio.h>
 #include "lib/strntol.h"
 int main(void) {
         /* just to see if it compiles */
         strntol(NULL, 0, NULL, 0);
         return 0;
 }
 # gcc -Wall -g ./test5.c ./lib/strntol.c

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoDrop the only local variable declaration within a for-loop (C99)
Tomohiro Kusumi [Fri, 14 Apr 2017 21:06:19 +0000 (00:06 +0300)]
Drop the only local variable declaration within a for-loop (C99)

Nothing happens at least with gcc by default, but this is the only
C99 (or after) specific declaration, so change it to normal style.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoFix num2str() output when modulo != -1U
Tomohiro Kusumi [Fri, 14 Apr 2017 21:06:18 +0000 (00:06 +0300)]
Fix num2str() output when modulo != -1U

I'm not sure what 05463816 actually fixed (as it only says "fix"),
but this isn't properly working at least for some input numbers unless
".%s" part of sprintf() is meant to be something other than decimal
part of the input number.

This commit might be breaking something that 05463816 had fixed,
which then needs to be rejected, but it at least prints decimal number
as expected. For example, when 12345 is 12.0556640625 KiB, and 23456
is 22.90625 KiB,

 # python -c "print(12345.0 / 1024)"
 12.0556640625
 # python -c "print(23456.0 / 1024)"
 22.90625

running this code as well as fio result in

 # cat ./test1.c
 #include <stdio.h>
 #include "lib/num2str.h"
 int main(void) {
         printf("%s-%s\n",
                 num2str(12345, 4, 1, 1, N2S_BYTE),
                 num2str(23456, 4, 1, 1, N2S_BYTE));
         return 0;
 }
 # gcc -Wall -g -DCONFIG_STATIC_ASSERT ./test1.c ./lib/num2str.c

below with the current implementation

 # ./a.out
 12.6KiB-22.1KiB
 # ./fio --name=xxxxx --ioengine=sync --rw=read --size=1m --unlink=1 --bsrange=12345:23456 | grep "rw=read"
 xxxxx: (g=0): rw=read, bs=(R) 12.6KiB-22.1KiB, (W) 12.6KiB-22.1KiB, (T) 12.6KiB-22.1KiB, ioengine=sync, iodepth=1

whereas below with this commit.

 # ./a.out
 12.1KiB-22.9KiB
 # ./fio --name=xxxxx --ioengine=sync --rw=read --size=1m --unlink=1 --bsrange=12345:23456 | grep "rw=read"
 xxxxx: (g=0): rw=read, bs=(R) 12.1KiB-22.9KiB, (W) 12.1KiB-22.9KiB, (T) 12.1KiB-22.9KiB, ioengine=sync, iodepth=1

-- Also see below
http://www.spinics.net/lists/fio/msg05720.html
 > > + sprintf(fmt, "%%.%df", (int)(maxlen - strlen(tmp) - 1));
 > > + sprintf(tmp, fmt, (double)modulo / (double)thousand[!!pow2]);
 >
 > I suspect one of the goals of that function was to restrict itself
 > to integer math and avoid invoking the floating point unit (which
 > might not even exist on some CPUs).

-- These are some more verification with various parameters
 # cat ./testx.c
 #include <stdio.h>
 #include "lib/num2str.h"
 static void f(uint64_t n, int maxlen, int base, int pow2, int units) {
  printf("%10jd %s\n", n, num2str(n, maxlen, base, pow2, units));
 }
 int main(void) {
  /* 1 */
  f(1, 1, 1, 1, N2S_BYTE);
  f(1, 1, 1, 0, N2S_BYTE);
  printf("====================\n");
  /* 10 */
  f(10, 2, 1, 1, N2S_BYTE);
  f(10, 2, 1, 0, N2S_BYTE);
  printf("====================\n");
  /* 1000 */
  f(1000, 5, 1, 1, N2S_BYTE);
  f(1000, 5, 1, 0, N2S_BYTE);
  f(1000, 4, 1, 1, N2S_BYTE);
  f(1000, 4, 1, 0, N2S_BYTE);
  f(1000, 3, 1, 1, N2S_BYTE);
  f(1000, 3, 1, 0, N2S_BYTE);
  f(1000, 2, 1, 1, N2S_BYTE);
  f(1000, 2, 1, 0, N2S_BYTE);
  f(1000, 1, 1, 1, N2S_BYTE);
  f(1000, 1, 1, 0, N2S_BYTE);
  printf("====================\n");
  /* 1024 */
  f(1024, 5, 1, 1, N2S_BYTE);
  f(1024, 5, 1, 0, N2S_BYTE);
  f(1024, 4, 1, 1, N2S_BYTE);
  f(1024, 4, 1, 0, N2S_BYTE);
  f(1024, 3, 1, 1, N2S_BYTE);
  f(1024, 3, 1, 0, N2S_BYTE);
  f(1024, 2, 1, 1, N2S_BYTE);
  f(1024, 2, 1, 0, N2S_BYTE);
  f(1024, 1, 1, 1, N2S_BYTE);
  f(1024, 1, 1, 0, N2S_BYTE);
  printf("====================\n");
  /* 12345 */
  f(12345, 6, 1, 1, N2S_BYTE);
  f(12345, 6, 1, 0, N2S_BYTE);
  f(12345, 5, 1, 1, N2S_BYTE);
  f(12345, 5, 1, 0, N2S_BYTE);
  f(12345, 4, 1, 1, N2S_BYTE);
  f(12345, 4, 1, 0, N2S_BYTE);
  f(12345, 3, 1, 1, N2S_BYTE);
  f(12345, 3, 1, 0, N2S_BYTE);
  f(12345, 2, 1, 1, N2S_BYTE);
  f(12345, 2, 1, 0, N2S_BYTE);
  f(12345, 1, 1, 1, N2S_BYTE);
  f(12345, 1, 1, 0, N2S_BYTE);
  printf("====================\n");
  /* 1234567 */
  f(1234567, 8, 1, 1, N2S_BYTE);
  f(1234567, 8, 1, 0, N2S_BYTE);
  f(1234567, 7, 1, 1, N2S_BYTE);
  f(1234567, 7, 1, 0, N2S_BYTE);
  f(1234567, 6, 1, 1, N2S_BYTE);
  f(1234567, 6, 1, 0, N2S_BYTE);
  f(1234567, 5, 1, 1, N2S_BYTE);
  f(1234567, 5, 1, 0, N2S_BYTE);
  f(1234567, 4, 1, 1, N2S_BYTE);
  f(1234567, 4, 1, 0, N2S_BYTE);
  f(1234567, 3, 1, 1, N2S_BYTE);
  f(1234567, 3, 1, 0, N2S_BYTE);
  f(1234567, 2, 1, 1, N2S_BYTE);
  f(1234567, 2, 1, 0, N2S_BYTE);
  f(1234567, 1, 1, 1, N2S_BYTE);
  f(1234567, 1, 1, 0, N2S_BYTE);
  printf("====================\n");
  /* 1234567 with base 1024 */
  f(1234567, 8, 1024, 1, N2S_BYTE);
  f(1234567, 8, 1024, 0, N2S_BYTE);
  f(1234567, 7, 1024, 1, N2S_BYTE);
  f(1234567, 7, 1024, 0, N2S_BYTE);
  f(1234567, 6, 1024, 1, N2S_BYTE);
  f(1234567, 6, 1024, 0, N2S_BYTE);
  f(1234567, 5, 1024, 1, N2S_BYTE);
  f(1234567, 5, 1024, 0, N2S_BYTE);
  f(1234567, 4, 1024, 1, N2S_BYTE);
  f(1234567, 4, 1024, 0, N2S_BYTE);
  f(1234567, 3, 1024, 1, N2S_BYTE);
  f(1234567, 3, 1024, 0, N2S_BYTE);
  f(1234567, 2, 1024, 1, N2S_BYTE);
  f(1234567, 2, 1024, 0, N2S_BYTE);
  f(1234567, 1, 1024, 1, N2S_BYTE);
  f(1234567, 1, 1024, 0, N2S_BYTE);
  printf("====================\n");
  /* 1234567 with base 1024 with N2S_BIT */
  f(1234567, 9, 1024, 1, N2S_BIT);
  f(1234567, 9, 1024, 0, N2S_BIT);
  f(1234567, 8, 1024, 1, N2S_BIT);
  f(1234567, 8, 1024, 0, N2S_BIT);
  f(1234567, 7, 1024, 1, N2S_BIT);
  f(1234567, 7, 1024, 0, N2S_BIT);
  f(1234567, 6, 1024, 1, N2S_BIT);
  f(1234567, 6, 1024, 0, N2S_BIT);
  f(1234567, 5, 1024, 1, N2S_BIT);
  f(1234567, 5, 1024, 0, N2S_BIT);
  f(1234567, 4, 1024, 1, N2S_BIT);
  f(1234567, 4, 1024, 0, N2S_BIT);
  f(1234567, 3, 1024, 1, N2S_BIT);
  f(1234567, 3, 1024, 0, N2S_BIT);
  f(1234567, 2, 1024, 1, N2S_BIT);
  f(1234567, 2, 1024, 0, N2S_BIT);
  f(1234567, 1, 1024, 1, N2S_BIT);
  f(1234567, 1, 1024, 0, N2S_BIT);
  printf("====================\n");
  return 0;
 }
 # gcc -Wall -g -DCONFIG_STATIC_ASSERT ./testx.c ./lib/num2str.c
 # ./a.out
          1 1B
          1 1B
 ====================
         10 10B
         10 10B
 ====================
       1000 1000B
       1000 1000B
       1000 1000B
       1000 1000B
       1000 0.0KiB
       1000 1.0kB
       1000 1KiB
       1000 1kB
       1000 1KiB
       1000 1kB
 ====================
       1024 1024B
       1024 1024B
       1024 1024B
       1024 1024B
       1024 1.0KiB
       1024 1.0kB
       1024 1KiB
       1024 1kB
       1024 1KiB
       1024 1kB
 ====================
      12345 12345B
      12345 12345B
      12345 12345B
      12345 12345B
      12345 12.1KiB
      12345 12.3kB
      12345 12KiB
      12345 12kB
      12345 12KiB
      12345 12kB
      12345 0MiB
      12345 0MB
 ====================
    1234567 1234567B
    1234567 1234567B
    1234567 1234567B
    1234567 1234567B
    1234567 1205.6KiB
    1234567 1234.6kB
    1234567 1206KiB
    1234567 1235kB
    1234567 1206KiB
    1234567 1235kB
    1234567 1.2MiB
    1234567 1.2MB
    1234567 1MiB
    1234567 1MB
    1234567 1MiB
    1234567 1MB
 ====================
    1234567 1234567KiB
    1234567 1234567kB
    1234567 1234567KiB
    1234567 1234567kB
    1234567 1205.6MiB
    1234567 1234.6MB
    1234567 1206MiB
    1234567 1235MB
    1234567 1206MiB
    1234567 1235MB
    1234567 1.2GiB
    1234567 1.2GB
    1234567 1GiB
    1234567 1GB
    1234567 1GiB
    1234567 1GB
 ====================
    1234567 9876536Kibit
    1234567 9876536kbit
    1234567 9876536Kibit
    1234567 9876536kbit
    1234567 9876536Kibit
    1234567 9876536kbit
    1234567 9645.1Mibit
    1234567 9876.5Mbit
    1234567 9645Mibit
    1234567 9877Mbit
    1234567 9645Mibit
    1234567 9877Mbit
    1234567 9.4Gibit
    1234567 9.9Gbit
    1234567 9Gibit
    1234567 10Gbit
    1234567 9Gibit
    1234567 10Gbit
 ====================

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agostat: cleanup iops/bw logging functions
Jens Axboe [Wed, 26 Apr 2017 17:13:25 +0000 (11:13 -0600)]
stat: cleanup iops/bw logging functions

They are basically identical. Abstract out into a helper, and
use it for both iops and bw logging.

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agostat: make next log time decision cleaner
Jens Axboe [Wed, 26 Apr 2017 00:14:00 +0000 (18:14 -0600)]
stat: make next log time decision cleaner

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoReturn non-negtive error in order to print right error msg
mychoxin [Wed, 19 Apr 2017 12:34:19 +0000 (20:34 +0800)]
Return non-negtive error in order to print right error msg

Signed-off-by: Pan Liu <liupan1111@gmail.com>
2 years agoMerge branch 'pull-2' of https://github.com/dmonakhov/fio
Jens Axboe [Mon, 10 Apr 2017 17:41:09 +0000 (11:41 -0600)]
Merge branch 'pull-2' of https://github.com/dmonakhov/fio

2 years agodoc: add ftruncate engine documentation and example jobfile
Sitsofe Wheeler [Mon, 10 Apr 2017 15:37:00 +0000 (18:37 +0300)]
doc: add ftruncate engine documentation and example jobfile

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoFix num2str() output when maxlen <= strlen(tmp)
Tomohiro Kusumi [Tue, 4 Apr 2017 20:22:17 +0000 (23:22 +0300)]
Fix num2str() output when maxlen <= strlen(tmp)

Since a local variable decimals is unsigned int, this conditional
 if (decimals <= 1)
needs cast as shown below.
 if ((int)decimals <= 1)

Otherwise it results in showing some garbage in case of maxlen == 0,

 # cat ./test0.c
 #include <stdio.h>
 #include "lib/num2str.h"
 int main(void) {
         printf("%s\n", num2str(123, 3, 1, 1, N2S_BYTE));
         printf("%s\n", num2str(123, 2, 1, 1, N2S_BYTE));
         printf("%s\n", num2str(123, 1, 1, 1, N2S_BYTE));
         printf("%s\n", num2str(123, 0, 1, 1, N2S_BYTE));
         return 0;
 }
 # gcc -Wall -g -DCONFIG_STATIC_ASSERT ./test0.c ./lib/num2str.c
 # ./a.out
 123B
 0KiB
 0KiB
 0.0?$-JB
 # ./a.out
 123B
 0KiB
 0KiB
 0.0#;bB
 # ./a.out
 123B
 0KiB
 0KiB
 0.0|B

whereas with this commit it prints "0B".

 # gcc -Wall -g -DCONFIG_STATIC_ASSERT ./test0.c ./lib/num2str.c
 # ./a.out
 123B
 0KiB
 0KiB
 0B

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake lib/num2str.c a stand-alone library by adding lib/num2str.h
Tomohiro Kusumi [Tue, 4 Apr 2017 20:22:16 +0000 (23:22 +0300)]
Make lib/num2str.c a stand-alone library by adding lib/num2str.h

lib/num2str.c not having dependency on fio.h by adding a new header
lib/num2str.h enables it to be a stand-alone library function, which
is useful for debugging purpose. In fact, most of the files under
lib/ directory do things this way.

--
 # cat ./test1.c
 #include <stdio.h>
 #include "lib/num2str.h"
 int main(void) {
         printf("%s\n", num2str(1024, 4, 1, 1, N2S_BYTE));
         return 0;
 }
 # gcc -Wall -g -DCONFIG_STATIC_ASSERT ./test1.c ./lib/num2str.c
 # ./a.out
 1024B

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake lib/memalign.c a stand-alone library
Tomohiro Kusumi [Tue, 4 Apr 2017 20:22:15 +0000 (23:22 +0300)]
Make lib/memalign.c a stand-alone library

lib/memalign.c not having dependency on fio.h enables it to be
a stand-alone library, which is useful for debugging purpose.
In fact, most of the files under lib/ directory do things this way.

(For lib/memalign.c, it was actually myself who introduced this
dependency as a part of changes made by 248c9436, so this commit
is partially reverting that)

--
 # cat ./test3.c
 #include <stdio.h>
 #include "lib/memalign.h"
 int main(void) {
         char *p = fio_memalign(4096, 1000000);
         printf("%p\n", p);
         fio_memfree(p, 1000000);
         return 0;
 }
 # gcc -Wall -g ./test3.c ./lib/memalign.c
 # ./a.out
 0x7fbb7bbbc000

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMake lib/prio_tree.c a stand-alone library
Tomohiro Kusumi [Tue, 4 Apr 2017 20:22:14 +0000 (23:22 +0300)]
Make lib/prio_tree.c a stand-alone library

lib/prio_tree.c not having dependency on fio.h enables it to be
a stand-alone library, which is useful for debugging purpose.
In fact, most of the files under lib/ directory do things this way.

--
 # cat ./test2.c
 #include <stdio.h>
 #include "lib/prio_tree.h"
 int main(void) {
         struct prio_tree_root tree;
         struct prio_tree_node node;
         INIT_PRIO_TREE_ROOT(&tree);
         INIT_PRIO_TREE_NODE(&node);
         prio_tree_insert(&tree, &node);
         prio_tree_remove(&tree, &node);
         printf("%d\n", prio_tree_empty(&tree));
         return 0;
 }
 # gcc -Wall -g -DBITS_PER_LONG=64 ./test2.c ./lib/prio_tree.c
 # ./a.out
 1

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agodump_td_info() doesn't really need to be a function
Tomohiro Kusumi [Tue, 4 Apr 2017 20:22:12 +0000 (23:22 +0300)]
dump_td_info() doesn't really need to be a function

since it's short enough, and too specific to be called as dump_td_info().

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoDon't silently terminate td when no I/O performed due to error
Tomohiro Kusumi [Tue, 4 Apr 2017 20:22:11 +0000 (23:22 +0300)]
Don't silently terminate td when no I/O performed due to error

Some runtime configurations can cause threads/processes to terminate
without any I/O performed, yet with no explicit error message, which
is quite confusing. In the example below, fio finishes with neither
error nor regular statistics due to file offset + bs being larger
than the file size while in get_io_u().

This commit calls log_err() when this happens. Since it's difficult
to tell the exact reason after thread has left the main I/O loop[*],
it gives advice to use a command line option --debug=io, similar to
the way td_io_init() gives advice.

[*] It can't just replace dprint(FD_IO, ...) calls with td_verror()
for e.g. while in get_io_u(), since those are also used by non error
paths too.

--
 # ./fio --name=xxxxx --ioengine=sync --rw=read --bs=2k --size=1m --nrfiles=1k --unlink=1
 xxxxx: (g=0): rw=read, bs=(R) 2048B-2048B, (W) 2048B-2048B, (T) 2048B-2048B, ioengine=sync, iodepth=1
 fio-2.18-59-g618e
 Starting 1 process
 xxxxx: Laying out IO files (1024 files / total 1MiB)

 Run status group 0 (all jobs):

 Disk stats (read/write):
     dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
   sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agobackend: include data direction in debug poisson rate print
Jens Axboe [Sat, 8 Apr 2017 16:59:27 +0000 (10:59 -0600)]
backend: include data direction in debug poisson rate print

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoSplit poisson rate control into read/write/trim
Jens Axboe [Fri, 7 Apr 2017 22:04:31 +0000 (16:04 -0600)]
Split poisson rate control into read/write/trim

We currently have just one set of controls for this, which
messes things up if we have separate iops/bandwidth settings
for reads and writes, for instance.

Split this into per-data direction controls, so that we can
control them independently.

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoengine: add ftruncate ioengine
Dmitry Monakhov [Fri, 7 Apr 2017 04:59:01 +0000 (07:59 +0300)]
engine: add ftruncate ioengine

Currently fio does many things but it still cannot substitute fsx
because it does not support ftruncate. Let's add this.

2 years agoengine: e4defrag fix error reporting
Dmitry Monakhov [Fri, 7 Apr 2017 04:56:49 +0000 (07:56 +0300)]
engine: e4defrag fix error reporting

IOC_MOVE_EXT returns 0 if the requested range is beyond i_size. In that
case we currently set io_u->resid = io_u->xfer_buflen, but
io_queue_event() interprets that as a zero read and fails with EIO. We
definitely do not want this behaviour so set io_u->error explicitly to
EINVAL so we fail with that instead when we are in this scenario.

2 years agoFio 2.19 fio-2.19
Jens Axboe [Tue, 4 Apr 2017 14:30:59 +0000 (08:30 -0600)]
Fio 2.19

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agobackend: move freeing of td->mutex to main thread
Jens Axboe [Mon, 3 Apr 2017 14:46:34 +0000 (08:46 -0600)]
backend: move freeing of td->mutex to main thread

On some platforms, we can end up freeing td->mutex while the wakee
is still inside fio_mutex_up(). This introduces a use-after-free
condition, and we crash.

Free the main thread_data mutex from the main thread instead,
when we know the process/thread is long dead and gone.

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agotime: use correct size type for usecs
Jens Axboe [Sun, 2 Apr 2017 22:01:18 +0000 (16:01 -0600)]
time: use correct size type for usecs

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agotime: fix overflow in timeval_add_msec()
Chris Taylor [Sun, 2 Apr 2017 21:54:02 +0000 (15:54 -0600)]
time: fix overflow in timeval_add_msec()

There is an issue in FIO when using the detailed bandwidth and iops
logging with averaging over a period of time > 1 second.  It seems that
usecs overflows which later causes negative time diff values resulting
in skewed toward 0 results.  I have attached a potential fix that should
prevent usecs from going beyond 1000000.

[Modified by Jens to put the adj_sec in the branch.]

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoSeparate io_u from ioengine [3/3] - rename ioengine.h to ioengines.h
Tomohiro Kusumi [Tue, 28 Mar 2017 20:03:01 +0000 (23:03 +0300)]
Separate io_u from ioengine [3/3] - rename ioengine.h to ioengines.h

This is based on the previous commits which added a new header
io_u.h and separated io_u related from ioengine.h and ioengines.c.

Since now that ioengine.h and ioengines.c are 1:1 in their contents
(i.e. prototypes and implementation), they should be based on the
same name as well.

This could possibly break outbox ioengines if they include ioengine.h,
but all the inbox ones just include fio.h (probably because that's
what engines/skeleton_external.c does), so the impact is probably
limited, and fio doesn't guarantee anything to outbox ones anyway.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoSeparate io_u from ioengine [2/3] - move io_u functions
Tomohiro Kusumi [Tue, 28 Mar 2017 20:03:00 +0000 (23:03 +0300)]
Separate io_u from ioengine [2/3] - move io_u functions

Move io_u functions from ioengines.c to io_u.c whose prototypes
are now located in io_u.h after the previous commit.

Prior to the previous commit, ioengine.h originally separated io_u
related function prototypes and ioengine related function prototypes
(by comment) based on whether they invoke struct ioengine_ops member
functions or not. Prototypes for these two functions have existed
in io_u side, thus they should be moved to io_u.c respectively.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoSeparate io_u from ioengine [1/3] - add io_u.h
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:59 +0000 (23:02 +0300)]
Separate io_u from ioengine [1/3] - add io_u.h

Extract io_u related structures and functions from ioengine.h to
a new header io_u.h which is intended to be a header for io_u.c
(similar to the way there is io_u_queue.h for io_u_queue.c),
since there isn't much reason for them to be a part of ioengine.h.

This commit does nothing other than separating them out (and make
other related files include the new header).

Confirmed this compiles on Linux, Cygwin and several BSDs.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoDrop prototype of unused function td_io_sync()
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:58 +0000 (23:02 +0300)]
Drop prototype of unused function td_io_sync()

858a3d47 in 2006 has removed (actually renamed) this function,
so this prototype hasn't been used since then.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoIgnore pre-read for character devices
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:57 +0000 (23:02 +0300)]
Ignore pre-read for character devices

pre_read_file() could ignore a chrdev due to its purpose of providing
unbuffered access to devices not limited to seekable disk devices,
while the purpose of this function (i.e. pre_read= option) is to
lseek(2) the given offset and page cache whatever read via read(2)
which is basically for filesystems and blkdevs.

(This commit directly goes on top of the one before previous one)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoDon't proceed with error set when failed to pre-read files/devices
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:56 +0000 (23:02 +0300)]
Don't proceed with error set when failed to pre-read files/devices

pre_read_files() never fails (never returns negative), and because
of this behavior, fio threads can continue regardless of td_verror()
call while prereading, and results in statistics with error in it,
which is quite confusing as to how it should be interpreted for regular
users.

This commit makes pre_read_files() return -1 if preread fails, and
eventually aborts fio, which is also the case with some other options.
It also changes non error return from 1 to 0, since all it matters
is if it's <0 or not, and fio normally uses 0 for success.

Not sure what the proper way of fixing this is if it legitimately
failed for some error on prereading though. Some options just print
a message to indicate it's unsupported and continue normally withtout
any error set.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoIgnore pre-read for FIO_NOIO td
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:55 +0000 (23:02 +0300)]
Ignore pre-read for FIO_NOIO td

pre_read_file() should ignore FIO_NOIO (i.e. cpuio) in addition to
FIO_PIPEIO. It does/should fail anyway since FIO_NOIO td is supposed
to have -1 for fd for not supporting any I/O.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoDrop redundant "ignore invalidate option" message from 21c1b29e
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:54 +0000 (23:02 +0300)]
Drop redundant "ignore invalidate option" message from 21c1b29e

The message added by 21c1b29e ("fio: ignore invalidate option...")
is correct in the sense that it resets invalidate= option, however
since invalidation is enabled by default, this message appears whenever
pre_read=1 is set (unless invalidate=0 is also explicitly set at the
same time, which is unliekly).

This is annoying and shouldn't have been added. 21c1b29e updating
HOWTO was enough, sorry.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoHOWTO: Mention niche detail of range format options
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:53 +0000 (23:02 +0300)]
HOWTO: Mention niche detail of range format options

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoDon't malloc more than necessary on extending/prereading file
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:52 +0000 (23:02 +0300)]
Don't malloc more than necessary on extending/prereading file

If td->o.max_bs[DDIR_XXX] were larger than file size to extend/preread,
it only needs to malloc the file size.

Given that td->o.max_bs[DDIR_XXX] are tunable parameters, it's better
not to malloc whatever size specified if needed size is smaller than
that. Users could specify as large as maximum of uint.

I dropped "bs = td->o.max_bs[DDIR_WRITE];" inside the first while loop
since now it works the same with or without this.

(This commit directly goes on top of the previous one)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoTest malloc result when allocation size is tunable
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:51 +0000 (23:02 +0300)]
Test malloc result when allocation size is tunable

The allocation size td->o.max_bs[DDIR_XXX] can be specified by user,
so test the result although it can be over committed on some platforms.
In theory allocation size could be as large as maximum of uint.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoFix return value of make_filename() when no filename_format
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:49 +0000 (23:02 +0300)]
Fix return value of make_filename() when no filename_format

This is unlikely to happen in the first place since ->filename_format
has a default value, and option parser can also detect an empty string,
but it should return buf which is sprintf'd right before returning.

A caller expects this function to return filename string (which is
buf arg itself in this case), and it also doesn't handle NULL return.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoconfigure: add a --disable-rdma flag to control rdma deps
Mike Frysinger [Tue, 21 Mar 2017 13:20:32 +0000 (07:20 -0600)]
configure: add a --disable-rdma flag to control rdma deps

Signed-off-by: Mike Frysinger <vapier@gentoo.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoTest fsync/fdatasync/sync_file_range for the next i/o only if should_fsync(td)
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:47 +0000 (23:28 +0200)]
Test fsync/fdatasync/sync_file_range for the next i/o only if should_fsync(td)

Checking an inline function should_fsync() first is better since
should_fsync() itself contains td_write() as a condition to return true,
and there's no need for non-write td to go through the rest of checks
when they result in false due to td->io_issues[DDIR_WRITE] being 0.

(This commit directly goes on top of the previous one)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoTest uint,int before division uint/int for the next i/o
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:46 +0000 (23:28 +0200)]
Test uint,int before division uint/int for the next i/o

When selecting ddir for the next i/o, test td->io_issues[DDIR_WRITE]
before division which is more expensive.

if (x && y && !(y % x))
rather than
if (x && !(y % x) && y)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoFix a comment after f227e2b6
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:45 +0000 (23:28 +0200)]
Fix a comment after f227e2b6

Add "for regular files" since it's true only on regfiles.
Also see below.
http://www.spinics.net/lists/fio/msg05646.html

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoHOWTO: Mention fsync=/fsyncdata= are set to 0 by default
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:44 +0000 (23:28 +0200)]
HOWTO: Mention fsync=/fsyncdata= are set to 0 by default

I think people could think sync ioengine and variants would somehow
fsync(2) for writes from what their name sound like, while they do
not unless these options are explicitly set (default 0).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoHOWTO: Mention cpuload= is mandatory for cpuio
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:43 +0000 (23:28 +0200)]
HOWTO: Mention cpuload= is mandatory for cpuio

fio jobs using cpuio fail without this option value specified,
so mention it's mandatory (or have default value for this ?).

fio: pid=9320, err=22/file:engines/cpu.c:75, func=cpuio, error=cpu thread needs rate (cpuload=)
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoDefine struct sk_out in server.h (not server.c)
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:42 +0000 (23:28 +0200)]
Define struct sk_out in server.h (not server.c)

struct sk_out isn't local to server.c, so it should be visible
in its header (instead of having struct sk_out; in fio.h).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoReplace redundant TD_F_NOIO flag with td->io_ops_init
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:41 +0000 (23:28 +0200)]
Replace redundant TD_F_NOIO flag with td->io_ops_init

The only reason TD_F_NOIO exists in addition to FIO_NOIO is because
fio_running_or_pending_io_threads() needs to know whether td has
successfully initialized ioengine, whereas FIO_NOIO is statically
set regardless of ioengines's ->init() result.

This commit adds a new td field ->io_ops_init to inidicate ->init()
result, so that td needs no extra bit field for each ioengine like
TD_F_NOIO. It was rather odd that td was unable to tell ->init()
result afterward when ->init() failure (returning non zero) doesn't
mean aborting fio itself. This commit also changes TD_F_NOIO to
TD_F_RESERVED as it's no longer used.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoMerge branch 'wip-fix-bs-title' of https://github.com/liupan1111/fio
Jens Axboe [Mon, 20 Mar 2017 01:49:04 +0000 (19:49 -0600)]
Merge branch 'wip-fix-bs-title' of https://github.com/liupan1111/fio

2 years agomake the bs info output clearer.
Pan Liu [Mon, 20 Mar 2017 01:42:07 +0000 (09:42 +0800)]
make the bs info output clearer.

2 years agois_power_of_2() should return bool
Jens Axboe [Fri, 17 Mar 2017 16:39:42 +0000 (10:39 -0600)]
is_power_of_2() should return bool

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoio_u: we don't need to set power_2 to false
Jens Axboe [Fri, 17 Mar 2017 16:33:37 +0000 (10:33 -0600)]
io_u: we don't need to set power_2 to false

We always set it before checking, no need for initializing it.

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agofixed the error=invalid argument when the lower bound of bsrange is not power of 2.
Pan Liu [Fri, 17 Mar 2017 16:24:09 +0000 (00:24 +0800)]
fixed the error=invalid argument when the lower bound of bsrange is not power of 2.

Signed-off-by: Pan Liu <liupan1111@gmail.com>
2 years agoAdd 'stats' option
Jens Axboe [Thu, 16 Mar 2017 20:43:37 +0000 (14:43 -0600)]
Add 'stats' option

This option controls whether or not we collect runtime stats
and show stat output for a given job. Defaults to on.

Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agofilesetup: remove bogus message on forcing file size
Jens Axboe [Thu, 16 Mar 2017 20:33:14 +0000 (14:33 -0600)]
filesetup: remove bogus message on forcing file size

This trigger all the time when size isn't being set and we
just query it, it's pointless and confusing to log. Kill
it with fire.

Fixes: c4aa2d08 ("Fixup for a minor 0 byte file size case")
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoconfigure: add generic pshared mutex test
Sitsofe Wheeler [Sun, 26 Feb 2017 11:47:28 +0000 (11:47 +0000)]
configure: add generic pshared mutex test

Add a feature test to configure to determine if a platform has
interprocess synchronization primitives that work with threads and
remove the hardcoded enablement in the individual os/os-* headers. This
should work better for platforms where support was overlooked or didn't
exist at some point in the past but has since been added. Also update
the README to not single out FreeBSD for being thread only as this might
change...

This is based off work by the Boost project
(https://github.com/boostorg/interprocess/blob/boost-1.63.0/include/boost/interprocess/detail/workaround.hpp#L45
) that found that OSX 10.6 and Cygwin are buggy even though
_POSIX_THREAD_PROCESS_SHARED says this feature should be work on them.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoConditionally enable FIO_HAVE_PSHARED_MUTEX on FreeBSD
Tomohiro Kusumi [Mon, 13 Mar 2017 18:42:37 +0000 (20:42 +0200)]
Conditionally enable FIO_HAVE_PSHARED_MUTEX on FreeBSD

FreeBSD ports has had an outbox patch to conditionally enable
FIO_HAVE_PSHARED_MUTEX, which can be upstream'd.

This commit brings in below diff from FreeBSD ports.
https://github.com/freebsd/freebsd-ports/commit/271a84d2df7bc9b983fe2fabe1615d01202d429b

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agomanpage: Add URL links to HOWTO/README
Tomohiro Kusumi [Mon, 13 Mar 2017 18:15:02 +0000 (20:15 +0200)]
manpage: Add URL links to HOWTO/README

Add links to the official repository since whether HOWTO/README
get installed somewhere in fs is distributions/OS dependent.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2 years agoFix errval variable to be positive errno value
Tomohiro Kusumi [Mon, 13 Mar 2017 18:15:01 +0000 (20:15 +0200)]
Fix errval variable to be positive errno value

The local variable errval is used for strerror(3) which only works
with positive values (i.e. -EXXX doesn't return an expected string).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>