fio.git
3 years agoioengines: Fix td->io_issues[ddir] over decrement
Sitsofe Wheeer [Sun, 27 Nov 2016 10:34:53 +0000 (10:34 +0000)]
ioengines: Fix td->io_issues[ddir] over decrement

If an ioengine returned FIO_Q_BUSY, td->io_issues[ddir] would be
over decremented: once in td_io_queue() and then again in
requeue_io_u(). This was because td_io_queue() reverted various values
upon finding the queue busy but requeue_io_u() found the I/O was still
in flight so decremented again. This then showed up as unsigned
underflow on td->io_issues[ddir].

Fix this by clearing IO_U_F_FLIGHT on the I/O when we find the queue was
busy in td_io_queue().

Signed-off-by: Sitsofe Wheeer <sitsofe@yahoo.com>
3 years agofio: Fix (unsigned) integer overflow issues
Sitsofe Wheeler [Sun, 20 Nov 2016 20:55:06 +0000 (20:55 +0000)]
fio: Fix (unsigned) integer overflow issues

This patch tries to address issues found when compiling fio with clang's
undefined behaviour sanitizer on 32 bit Linux. While this generated many
false positives it did identify a number of subtle issues when
rates/amounts/time calculations went above 32 bit values.

The following job was used:
[global]
time_based
runtime=180m
bs=256k
rw=randrw
norandommap
rate=512M
ioengine=null
direct=1
iodepth=8
thread
numjobs=10
ramp_time=20m

[null]
size=1P

It should fix https://github.com/axboe/fio/issues/270 because various 32
bit ETA variables were overflowing due to the microsecond arithmetic
being done on them (even on 64 bit systems).

This patch should also fix https://github.com/axboe/fio/issues/194 and
https://github.com/axboe/fio/issues/266 because 64 bit Windows is a
LLP64 platform so ints and longs still default to 32 bits.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoImprove informativeness about directIO support or rather lackthereof on Solaris when...
Sam Zaydel [Wed, 16 Nov 2016 18:16:55 +0000 (10:16 -0800)]
Improve informativeness about directIO support or rather lackthereof on Solaris when errno is set to ENOTTY

3 years agoposixaio: fix bad type passed to memset
Jens Axboe [Tue, 15 Nov 2016 17:42:58 +0000 (10:42 -0700)]
posixaio: fix bad type passed to memset

We want the full array cleared, not just the first element.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix memory leak on tmp_buf
Colin Ian King [Mon, 14 Nov 2016 00:05:37 +0000 (00:05 +0000)]
Fix memory leak on tmp_buf

tmp_buf is allocated but not free'd, causing a minor memory leak

Signed-off-by: Colin Ian King <colin.king@canonical.com>
3 years agorbd: fix crash with zero sized image
Pan Liu [Sat, 12 Nov 2016 15:26:57 +0000 (23:26 +0800)]
rbd: fix crash with zero sized image

When the user create a rbd image with size zero, and specify the
"size" larger than zero in fio option, rbd will crash.

Signed-off-by: Pan Liu <pan.liu@istuary.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix duplicated typos from 42d97b5c in fio(1)
Tomohiro Kusumi [Wed, 2 Nov 2016 11:51:30 +0000 (20:51 +0900)]
Fix duplicated typos from 42d97b5c in fio(1)

Some typos are duplicated since fio(1) is written based on HOWTO,
(but not the same since commits that update HOWTO aren't always
in sync with fio(1)).

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove extra space in tausworthe32 warning message
Bruce Cran [Sat, 29 Oct 2016 05:12:15 +0000 (23:12 -0600)]
Remove extra space in tausworthe32 warning message

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'HOWTO-cleanup' of https://github.com/szaydel/fio
Jens Axboe [Tue, 1 Nov 2016 15:10:44 +0000 (09:10 -0600)]
Merge branch 'HOWTO-cleanup' of https://github.com/szaydel/fio

3 years agoFix minor spelling mistakes in HOWTO document
Sam Zaydel [Tue, 1 Nov 2016 14:42:55 +0000 (07:42 -0700)]
Fix minor spelling mistakes in HOWTO document

3 years agobackend: cleanup check for completion/issue byte checking
Jens Axboe [Fri, 28 Oct 2016 18:34:02 +0000 (12:34 -0600)]
backend: cleanup check for completion/issue byte checking

Same code, just different array. Put it in a shared helper.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd blockdev_size() support for OpenBSD
Tomohiro Kusumi [Fri, 21 Oct 2016 17:07:33 +0000 (02:07 +0900)]
Add blockdev_size() support for OpenBSD

This commit is OpenBSD version of 1116dc04 in 2015.
This is exactly the same as the previous commit for NetBSD.

OpenBSD still has block device as a file type, thus it differs
from FreeBSD/DragonFlyBSD regarding FIO_HAVE_CHARDEV_SIZE.

Note that fio doesn't compile on OpenBSD 5.9 (hits lex error),
but this this commit should still be good since this function
alone compiles on OpenBSD.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd blockdev_size() support for NetBSD
Tomohiro Kusumi [Fri, 21 Oct 2016 17:07:32 +0000 (02:07 +0900)]
Add blockdev_size() support for NetBSD

This commit is NetBSD version of 1116dc04 in 2015.

NetBSD still has block device as a file type, thus it differs
from FreeBSD/DragonFlyBSD regarding FIO_HAVE_CHARDEV_SIZE.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFio 2.15 fio-2.15
Jens Axboe [Tue, 25 Oct 2016 18:38:13 +0000 (12:38 -0600)]
Fio 2.15

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse the POSIX `EDEADLK` instead of the Linux `EDEADLOCK`
Bruce Cran [Mon, 24 Oct 2016 18:08:40 +0000 (12:08 -0600)]
Use the POSIX `EDEADLK` instead of the Linux `EDEADLOCK`

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: end IO loop early, if the job is marked as terminated
Jens Axboe [Mon, 24 Oct 2016 16:51:04 +0000 (10:51 -0600)]
backend: end IO loop early, if the job is marked as terminated

If we terminate a job, we don't want that job to attempt to grab
resources that might already have been freed.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: make job reap timeout 5 minutes
Jens Axboe [Mon, 24 Oct 2016 16:50:24 +0000 (10:50 -0600)]
fio: make job reap timeout 5 minutes

1 minute is a bit too slow, if we're waiting on a lot of IO syncing.
Let's make it 5 minutes. This will still catch someone hung on a
kernel bug, but hopefully should not trigger for normal waits.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agomutex: clear mutex when removed
Jens Axboe [Mon, 24 Oct 2016 16:49:48 +0000 (10:49 -0600)]
mutex: clear mutex when removed

This ensures that any attempt to grab it after it has been removed
will trigger an assert, instead of just silently hanging.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: if we can't grab stat_mutex, report a deadlock error and exit
Theodore Ts'o [Sun, 23 Oct 2016 14:35:14 +0000 (08:35 -0600)]
backend: if we can't grab stat_mutex, report a deadlock error and exit

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse fmt -w WIDTH option instead of -WIDTH
Tomohiro Kusumi [Thu, 20 Oct 2016 14:03:18 +0000 (23:03 +0900)]
Use fmt -w WIDTH option instead of -WIDTH

Most systems support -WIDTH option which is an abbreviated form
of -w WIDTH or --width=WIDTH option, but some OS don't support
-WIDTH format.

As far as I've checked (on actual systems or online manpages),
Linux/FreeBSD/DragonFlyBSD/OpenBSD/Solaris/AIX/Cygwin support
-WIDTH format, but NetBSD doesn't support this (see below).
IRIX and HPUX may not support either according to online manpages.

-w WIDTH format is basically supported by all systems that have
fmt command, so this is probably better in terms of portability.

--
    CC crc/crc64.o
fmt: unknown option -- 1
Usage: fmt [-Cr] [-g <goal>] [-m|w <max>] [<files>..]
         fmt [-Cr] [<goal>] [<max>] [<files>]
    CC crc/md5.o
fmt: unknown option -- 1
Usage: fmt [-Cr] [-g <goal>] [-m|w <max>] [<files>..]
         fmt [-Cr] [<goal>] [<max>] [<files>]
...

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove getopt_long_only macro from NetBSD header
Tomohiro Kusumi [Thu, 20 Oct 2016 14:03:17 +0000 (23:03 +0900)]
Remove getopt_long_only macro from NetBSD header

If CONFIG_GETOPT_LONG_ONLY isn't defined, fio is compiled with
its own getopt_long_only(), so this macro is no longer needed.
(os/os-netbsd.h was added before getopt_long_only() support)

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'rbd-poll'
Jens Axboe [Wed, 19 Oct 2016 15:24:04 +0000 (09:24 -0600)]
Merge branch 'rbd-poll'

3 years agorbd: poll cleanups
Jens Axboe [Wed, 19 Oct 2016 15:23:22 +0000 (09:23 -0600)]
rbd: poll cleanups

Would be nice to cleanup the ifdef in rbd_iter_events() as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agouse poll() and rbd_poll_io_events to speed up io retrieval.
Pan Liu [Wed, 19 Oct 2016 14:42:21 +0000 (22:42 +0800)]
use poll() and rbd_poll_io_events to speed up io retrieval.

3 years agoHOWTO: update to include iolog replay
Jens Axboe [Tue, 18 Oct 2016 15:12:45 +0000 (09:12 -0600)]
HOWTO: update to include iolog replay

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: add support for 'replay_no_stall'
Jens Axboe [Tue, 18 Oct 2016 15:11:04 +0000 (09:11 -0600)]
iolog: add support for 'replay_no_stall'

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: enable replay_redirect on iolog replay
Jens Axboe [Mon, 17 Oct 2016 20:51:54 +0000 (14:51 -0600)]
iolog: enable replay_redirect on iolog replay

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd device_is_mounted() support for NetBSD
Tomohiro Kusumi [Mon, 17 Oct 2016 14:17:41 +0000 (23:17 +0900)]
Add device_is_mounted() support for NetBSD

This commit is NetBSD version of e7e136da in 2015. It's basically
the same as the one for FreeBSD/DragonFlyBSD, but NetBSD takes
statvfs instead of statfs, so I had to add CONFIG_GETMNTINFO_STATVFS
to differentiate from CONFIG_GETMNTINFO. It could be renamed if
there is any better name, as it sounds strange.

compile_prog() takes -Werror for local CFLAGS. Without -Werror,
the existing getmntinfo(3) test code for FreeBSD/DragonFlyBSD only
gives a warning on NetBSD and results in trying to compile wrong
device_is_mounted().

(The reason this hasn't been reported as a problem is because of
the ./configure bug fixed by the previous commit. It fails to
compile the test code unless st is a pointer.)

The remaining one, OpenBSD seems to have the same interface as
FreeBSD/DragonFlyBSD, so the existing test code will probably work,
but fio doesn't compile on OpenBSD 5.9 due to other compile time
errors (lex error) that have nothing to do with this.

--
 # uname
 NetBSD
 # cat ./test1.c
 #include <stdio.h>
 #include "./lib/mountcheck.h"
 int main(int argc, char *argv[]) {
         printf("%s %d\n", argv[1], device_is_mounted(argv[1]));
         return 0;
 }
 # gcc -Wall -g ./test1.c ./lib/mountcheck.o -o ./test1
 # ./test1 /dev/wd0a /* UFS */
 /dev/wd0a 1
 # ./test1 procfs /* procfs */
 procfs 1
 # ./test1 tmpfs /* tmpfs */
 tmpfs 1
 # ./test1 invalid /* irrelevant string */
 invalid 0

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix e7e136da (Add device_is_mounted() support for BSDs)
Tomohiro Kusumi [Mon, 17 Oct 2016 14:17:40 +0000 (23:17 +0900)]
Fix e7e136da (Add device_is_mounted() support for BSDs)

The actual C code was good, but the code to check getmntinfo(3)
in ./configure was missing * for st. getmntinfo(3) on FreeBSD/
DragonFlyBSD/etc takes statfs** as an argument.

I failed to catch this error as gcc without -Werror only gives
warning and sets $? to 0, though as a result it didn't break anything.

It did nothing to Linux since getmntinfo(3) doesn't compile on
Linux whether arg is * or **.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd FIO_HAVE_FS_STAT/get_fs_free_size() support for OpenBSD
Tomohiro Kusumi [Mon, 17 Oct 2016 14:17:39 +0000 (23:17 +0900)]
Add FIO_HAVE_FS_STAT/get_fs_free_size() support for OpenBSD

This commit is OpenBSD version of 60f840a8 in 2015.
This is the same as 60f840a8, as well as the previous commit
for NetBSD, as it uses POSIX statvfs.

Note that fio doesn't compile on OpenBSD 5.9 (hits lex error),
but this this commit should still be good since this function
alone compiles on OpenBSD.

(Note that the function was renamed with _free in c08ad04c)

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd FIO_HAVE_FS_STAT/get_fs_free_size() support for NetBSD
Tomohiro Kusumi [Mon, 17 Oct 2016 14:17:38 +0000 (23:17 +0900)]
Add FIO_HAVE_FS_STAT/get_fs_free_size() support for NetBSD

This commit is NetBSD version of 60f840a8 in 2015.
This is the same as 60f840a8, as it uses POSIX statvfs.

(Note that the function was renamed with _free in c08ad04c)

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUpdate REPORTING-BUGS
Jens Axboe [Fri, 14 Oct 2016 20:07:43 +0000 (14:07 -0600)]
Update REPORTING-BUGS

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd alias for 'disable_bw_measurement' option
Jens Axboe [Wed, 12 Oct 2016 14:59:25 +0000 (08:59 -0600)]
Add alias for 'disable_bw_measurement' option

'disable_bw', similarly to the stats options.

Reported-by: Bruce Cran <bruce.cran@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoImplement nice() for Windows
Bruce Cran [Wed, 12 Oct 2016 02:22:47 +0000 (03:22 +0100)]
Implement nice() for Windows

Map the following ranges for the "nice" option:

Less than -15:  High
-1 through -15: Above normal
0:              Normal
1 through 15:   Below normal
More than 15:   Low/Idle

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUpdate bandwidth log documentation
Omar Sandoval [Wed, 5 Oct 2016 22:15:29 +0000 (15:15 -0700)]
Update bandwidth log documentation

The behavior of --bandwidth-log and write_bw_log has changed multiple
times, but the documentation hasn't been updated accordingly.

Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: make write_*_log prefix optional
Omar Sandoval [Wed, 5 Oct 2016 22:15:28 +0000 (15:15 -0700)]
iolog: make write_*_log prefix optional

Providing write_bw_log=jobname for every job in a jobfile gets pretty
tedious. This way, we can just do, e.g.,

[global]
write_bw_log

and all jobs will be logged. The documentation seems to suggest that
this is the intended behavior, and it looks like that used to be the
case.

Fixes: 203160d52b86 ("options: get rid of more .cb option usage we don't need")
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: fix --bandwidth-log segfaults
Omar Sandoval [Wed, 5 Oct 2016 22:15:27 +0000 (15:15 -0700)]
iolog: fix --bandwidth-log segfaults

The global bandwidth log doesn't have thread data.

Fixes: 1e613c9c2393 ("This commit / feature adds completion latency histogram output to fio")
Fixes: 3aea75b14510 ("Option for changing log files to use Unix epoch")
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: fix --bandwidth-log without argument
Omar Sandoval [Wed, 5 Oct 2016 22:15:26 +0000 (15:15 -0700)]
init: fix --bandwidth-log without argument

--bandwidth-log doesn't do anything with the argument that it requires,
and the documentation doesn't indicate that it takes an argument.

Fixes: b46928282e0a ("[PATCH] Add full command line parameter support")
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agowindowsaio: fix completion thread affinitization
Bruce Cran [Thu, 6 Oct 2016 03:13:54 +0000 (03:13 +0000)]
windowsaio: fix completion thread affinitization

Affinitize the windowsaio completion threads to the same CPUs as the
submission threads.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofilehash: fix init/exit
Jens Axboe [Tue, 27 Sep 2016 16:28:29 +0000 (10:28 -0600)]
filehash: fix init/exit

We do it too later, move it earlier.

Fixes: 63a26e05622b ("filehash: move to separate allocation")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: remove unused variable
Jens Axboe [Mon, 26 Sep 2016 16:07:26 +0000 (10:07 -0600)]
init: remove unused variable

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofilehash: move to separate allocation
Jens Axboe [Mon, 26 Sep 2016 07:40:52 +0000 (01:40 -0600)]
filehash: move to separate allocation

We currently make it part of the thread structure allocation, but
there's no need to do that since we have smalloc available.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: bump max jobs to 4k
Jens Axboe [Mon, 26 Sep 2016 07:29:25 +0000 (01:29 -0600)]
fio: bump max jobs to 4k

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosmalloc: OOM fixups
Jens Axboe [Sun, 25 Sep 2016 20:22:56 +0000 (14:22 -0600)]
smalloc: OOM fixups

Warn directly in smalloc() instead of having the callers do it.
Add a hint on how to solve a situation where we run out of
shared memory.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobloom: don't enforce minimum entry count
Jens Axboe [Sun, 25 Sep 2016 20:10:24 +0000 (14:10 -0600)]
bloom: don't enforce minimum entry count

We defaulted to 1G of entries, which is crazy. Assume the caller
passes in the right amount.

Fixes: 1b2a83dcda75 ("file: add bloom filter to avoid quadratic lookup behavior")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofile: add bloom filter to avoid quadratic lookup behavior
Jens Axboe [Sun, 25 Sep 2016 19:57:32 +0000 (13:57 -0600)]
file: add bloom filter to avoid quadratic lookup behavior

If we have a lot of jobs, with a lot of subjobs, and they each
have a lot of files, we get pathetic startup performance
because each file add will check all the previously added
files.

Fixes: bcbfeefa7bce ("fio: add multi directory support")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: re-call sinit() if we change the smallc pool size
Jens Axboe [Sun, 25 Sep 2016 19:56:59 +0000 (13:56 -0600)]
init: re-call sinit() if we change the smallc pool size

Just adds more pools, but that should be fine at init time.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosmalloc: fixup --alloc-size
Jens Axboe [Sun, 25 Sep 2016 19:45:59 +0000 (13:45 -0600)]
smalloc: fixup --alloc-size

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobloom: allow to pass in whether to set bits for strings
Jens Axboe [Sun, 25 Sep 2016 19:43:51 +0000 (13:43 -0600)]
bloom: allow to pass in whether to set bits for strings

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobloom: add string version
Jens Axboe [Fri, 23 Sep 2016 17:57:00 +0000 (11:57 -0600)]
bloom: add string version

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobloom: hashes take byte lengths, not nwords
Jens Axboe [Fri, 23 Sep 2016 17:38:58 +0000 (11:38 -0600)]
bloom: hashes take byte lengths, not nwords

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobloom: use bool
Jens Axboe [Fri, 23 Sep 2016 17:33:02 +0000 (11:33 -0600)]
bloom: use bool

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agostat: check if ctime_r() ends in a newline before stripping
Jens Axboe [Wed, 21 Sep 2016 03:58:34 +0000 (21:58 -0600)]
stat: check if ctime_r() ends in a newline before stripping

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoos/windows/posix.c: ensure that ctime_r() adds a newline
Jens Axboe [Wed, 21 Sep 2016 03:57:25 +0000 (21:57 -0600)]
os/windows/posix.c: ensure that ctime_r() adds a newline

The normal POSIX versions do that. And as fio proper strips the
newline, we end up losing the last digit if the Windows version
does not add the newline.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoWindows: fix Time_tToSystemTime function to be 64-bit compliant
Josh Sinykin [Mon, 19 Sep 2016 03:17:30 +0000 (04:17 +0100)]
Windows: fix Time_tToSystemTime function to be 64-bit compliant

Use LONGLONG instead of LARGE_INTEGER to avoid problems on 64-bit
systems causing a crash.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix garbage characters in json output caused by time_buf being uninitialized
Josh Sinykin [Mon, 19 Sep 2016 03:25:02 +0000 (04:25 +0100)]
Fix garbage characters in json output caused by time_buf being uninitialized

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: kill SW_F_EXITED
Jens Axboe [Fri, 16 Sep 2016 18:48:32 +0000 (12:48 -0600)]
workqueue: kill SW_F_EXITED

We don't use the flag, it's set but never tested. On exit, we use
pthread_join() to wait for the thread to exit.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: dprint() casts for 32-bit warnings
Jens Axboe [Fri, 16 Sep 2016 17:22:09 +0000 (11:22 -0600)]
iolog: dprint() casts for 32-bit warnings

Cast size_t to unsigned long before printing.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFixup two compile warnings
Jens Axboe [Fri, 16 Sep 2016 17:15:59 +0000 (11:15 -0600)]
Fixup two compile warnings

- Don't put code before variables.

- uintptr_t cast to fix a warning on 32-bit

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFio 2.14 fio-2.14
Jens Axboe [Fri, 16 Sep 2016 17:02:09 +0000 (11:02 -0600)]
Fio 2.14

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: harden clockid_t test
Jens Axboe [Fri, 16 Sep 2016 16:08:15 +0000 (10:08 -0600)]
configure: harden clockid_t test

Don't assume we can assign '0' to a clockid_t type, let's just
memset it to be on the safe side.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agomac: fix for 10.12 having clockid_t
Jens Axboe [Fri, 16 Sep 2016 16:05:44 +0000 (10:05 -0600)]
mac: fix for 10.12 having clockid_t

Previous versions did not have this type, so we defined our own.
As of 10.12 it does seem to, so add a configure test for this.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient: coding style fixups
Jens Axboe [Fri, 16 Sep 2016 15:27:41 +0000 (09:27 -0600)]
client: coding style fixups

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'client-unique-log-names' of https://github.com/bengland2/fio into log...
Jens Axboe [Fri, 16 Sep 2016 15:26:30 +0000 (09:26 -0600)]
Merge branch 'client-unique-log-names' of https://github.com/bengland2/fio into log-unique

3 years agoio_u: fix overflow in 64-bit bssplit calculation
Jens Axboe [Fri, 16 Sep 2016 15:11:04 +0000 (09:11 -0600)]
io_u: fix overflow in 64-bit bssplit calculation

If we use a random generator that has a 64-bit output, then we'll
overflow in the calculation for what block size to select when
bssplit= is used.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'master' of https://github.com/jan--f/fio
Jens Axboe [Thu, 15 Sep 2016 13:57:38 +0000 (07:57 -0600)]
Merge branch 'master' of https://github.com/jan--f/fio

3 years agoadd simple fio.service to start fio server with systemd
Jan Fajerski [Thu, 15 Sep 2016 10:57:35 +0000 (12:57 +0200)]
add simple fio.service to start fio server with systemd

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
3 years agoclient: ignore SEND_ETA, if we can't fin a reply command
Jens Axboe [Tue, 13 Sep 2016 15:08:00 +0000 (09:08 -0600)]
client: ignore SEND_ETA, if we can't fin a reply command

This can happen if we time out a reply, but the the reply comes
in a bit later. For that case, we already decremented the ETA
count, so we should just ignore the late reply.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMakefile: fixup java path for libhdfs
Jens Axboe [Mon, 12 Sep 2016 14:10:45 +0000 (08:10 -0600)]
Makefile: fixup java path for libhdfs

Fixes: 8dc1a870973e ("The fixed CPU architecture in the Makefile will make failure on ppc64le")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: pass in right pointer to def thread options free
Jens Axboe [Mon, 12 Sep 2016 01:14:10 +0000 (19:14 -0600)]
init: pass in right pointer to def thread options free

Fixes: a609f12a1701 ("options: remove dependency of 'o' being first in 'td'")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosafety first!
Ben England [Thu, 8 Sep 2016 18:54:06 +0000 (14:54 -0400)]
safety first!

3 years agoadhere to fio coding standards
Ben England [Tue, 6 Sep 2016 17:12:07 +0000 (13:12 -0400)]
adhere to fio coding standards

3 years agogenerate unique pathname for each fio --client log file
Ben England [Tue, 6 Sep 2016 16:21:30 +0000 (12:21 -0400)]
generate unique pathname for each fio --client log file

3 years agoMakes use of configparser portable to older versions by:
Karl Cronburg [Tue, 6 Sep 2016 14:22:00 +0000 (10:22 -0400)]
Makes use of configparser portable to older versions by:
- relying on its' own NoOptionError exception
- using getter method instead of dictionary overriding
- and using readfp() as older version does not autodetect fp vs string types

Signed-off-by: Karl Cronburg <karl.cronburg@gmail.com>
3 years agoserver: bump version
Jens Axboe [Fri, 2 Sep 2016 17:24:59 +0000 (11:24 -0600)]
server: bump version

Previous histogram commit changes the client/server protocol,
bump the version.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'histogram-client-server' of https://github.com/cronburg/fio
Jens Axboe [Fri, 2 Sep 2016 17:24:15 +0000 (11:24 -0600)]
Merge branch 'histogram-client-server' of https://github.com/cronburg/fio

3 years agojesd219: fix alignment
Jeff Furlong [Mon, 29 Aug 2016 21:43:27 +0000 (15:43 -0600)]
jesd219: fix alignment

Previously we added the bssplit option to accommodate more advanced
blocksize lists.  However, blockalign appears to default to the smallest
element in the bssplit list.  If so, we must update the example JESD219
workload to explicitly specify blockalign=4k, to satisfy the JESD219
workload requirement ("data payloads greater than or equal to 4096 bytes
are aligned on 4k boundaries").  There is no explicit alignment
requirement for <4KB blocksizes, so at this time, there seems to be no
need to specify a separate alignment for some blocksizes in bssplit
list.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFIO-VERSION-GEN: fix dirty repo tracking
Jens Axboe [Mon, 29 Aug 2016 19:40:08 +0000 (13:40 -0600)]
FIO-VERSION-GEN: fix dirty repo tracking

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofilelock: bool conversion
Jens Axboe [Mon, 29 Aug 2016 17:44:32 +0000 (11:44 -0600)]
filelock: bool conversion

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agotrim: convert to bool
Jens Axboe [Mon, 29 Aug 2016 16:37:44 +0000 (10:37 -0600)]
trim: convert to bool

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: check if we need to update rusage stats, if stat_mutex is busy
Jens Axboe [Fri, 26 Aug 2016 20:39:30 +0000 (14:39 -0600)]
backend: check if we need to update rusage stats, if stat_mutex is busy

Even with the fix to check if we need to update rusage happening
right before the stat_mutex lock, we can still deadlock. This
looks something like the below:

helper_thread                   job

lock(stat_mutex);
                                lock(stat_mutex);
down(td->rusage_sem);

And now are both effectively locked in an ABBA deadlock. The helper
thread is waiting for the job to update it's rusage, but the job
is stuck waiting for the stat_mutex.

Fix this by doing a trylock on the stat_mutex, and if it fails,
ensure that we update rusage.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoClient / server code for handling histograms. The server:
Karl Cronburg [Wed, 24 Aug 2016 20:12:20 +0000 (16:12 -0400)]
Client / server code for handling histograms. The server:

- Deals with subtracting consecutive histograms so that client
  doesn't have to recreate linked list.
- Uses existing IOLOG command code to create packets, checking
  for IO_LOG_TYPE_HIST when necessary.

And the client:

- Reconstructs the pointers to the histogram bins from the packet
  format, namely (sample_0, hist_0, sample_1, hist_1, ...) in lieu
  of the current (sample_0, sample_1, ...) format used for the (void *)
  cur_log->samples.
- Flushes histograms to file with updated pointer calculation to get
  ith sample, disabling subtraction in hist_sum() using a null
  pointer.

This does not cover plain-text transmission mode (i.e. when zlib is
not present during fio compilation).

Signed-off-by: Karl Cronburg <kcronbur@redhat.com>
3 years agofile: fio_files_done() can return bool
Jens Axboe [Fri, 26 Aug 2016 14:34:41 +0000 (08:34 -0600)]
file: fio_files_done() can return bool

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofile: fix numjobs > 1 and implied jobname as filename
Jens Axboe [Fri, 26 Aug 2016 03:00:55 +0000 (21:00 -0600)]
file: fix numjobs > 1 and implied jobname as filename

If we have a jobfile that looks like:

[global]
numjobs=4

[/dev/somedevice]

Then we fail jobs 2 and on, since we don't properly add those
files. Fix this by checking if we're generating a filename
based on the jobname.

Fixes: bcbfeefa7bce ("fio: add multi directory support")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'epoch-histograms' of https://github.com/cronburg/fio
Jens Axboe [Thu, 25 Aug 2016 22:03:20 +0000 (16:03 -0600)]
Merge branch 'epoch-histograms' of https://github.com/cronburg/fio

3 years agoGive job file to fiologparser_hist.py so that it can auto detect
Karl Cronburg [Thu, 25 Aug 2016 21:37:21 +0000 (17:37 -0400)]
Give job file to fiologparser_hist.py so that it can auto detect
log_hist_msec. This commit also adds handling of unix epoch
timestamps by fiologparser_hist.py.

Signed-off-by: Karl Cronburg <kcronbur@redhat.com>
3 years agostat: don't trust per_unit_log() if log is NULL
Jens Axboe [Mon, 22 Aug 2016 19:23:29 +0000 (13:23 -0600)]
stat: don't trust per_unit_log() if log is NULL

We return 'false' for that case, but it's not really false. Fixes
a sigbus error on sparc, without logging.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: use proper include for PATH_MAX
Jens Axboe [Sat, 20 Aug 2016 16:28:57 +0000 (10:28 -0600)]
verify: use proper include for PATH_MAX

Resolves: https://github.com/axboe/fio/issues/231
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoManual page for fiologparser_hist.py and Makefile updates to install
Karl Cronburg [Thu, 18 Aug 2016 22:56:17 +0000 (18:56 -0400)]
Manual page for fiologparser_hist.py and Makefile updates to install
them.

Signed-off-by: Karl Cronburg <kcronbur@redhat.com>
3 years agoMerge branch 'epoch' of https://github.com/cronburg/fio
Jens Axboe [Thu, 18 Aug 2016 20:38:21 +0000 (14:38 -0600)]
Merge branch 'epoch' of https://github.com/cronburg/fio

3 years agoOption for changing log files to use Unix epoch instead of being
Karl Cronburg [Thu, 18 Aug 2016 16:48:47 +0000 (12:48 -0400)]
Option for changing log files to use Unix epoch instead of being
zero-based (when fio starts) epoch.

This makes it easier to analyze the data in the context of other
benchmarking tools running at the same time as fio.

Signed-off-by: Karl Cronburg <kcronbur@redhat.com>
3 years agoparse: fix void * pointer math complaint
Jens Axboe [Thu, 18 Aug 2016 16:50:33 +0000 (10:50 -0600)]
parse: fix void * pointer math complaint

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd basic write/read-and-verify example job file
Jens Axboe [Thu, 18 Aug 2016 16:03:49 +0000 (10:03 -0600)]
Add basic write/read-and-verify example job file

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: use the proper enum type for the shifted IO engine flags
Jens Axboe [Thu, 18 Aug 2016 14:17:53 +0000 (08:17 -0600)]
fio: use the proper enum type for the shifted IO engine flags

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix backwards reads with --size smaller than the file size
Jens Axboe [Tue, 16 Aug 2016 21:22:17 +0000 (15:22 -0600)]
Fix backwards reads with --size smaller than the file size

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agohistogram: style and list fixups
Jens Axboe [Tue, 16 Aug 2016 20:51:41 +0000 (14:51 -0600)]
histogram: style and list fixups

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'histogram-delta' of https://github.com/cronburg/fio into histogram
Jens Axboe [Tue, 16 Aug 2016 20:47:09 +0000 (14:47 -0600)]
Merge branch 'histogram-delta' of https://github.com/cronburg/fio into histogram

3 years agoMake histogram samples non-cumulative by tracking a linked-list
Karl Cronburg [Tue, 16 Aug 2016 19:44:09 +0000 (15:44 -0400)]
Make histogram samples non-cumulative by tracking a linked-list
of the most recent histogram and differencing it when we print
to the log file(s). Linked list of pointers used to minimize
runtime impact on recording side, instead choosing to do
subtraction on the logging (when logs get printed to file) side.

This helps cleanup fiologparser_hist.py, makes the log files easier
to understand at a glance, reduces file size(s), and averts the
possibility of integer overflow in the logs (for a long enough fio
job).

3 years agoVarious cleanups
Jens Axboe [Tue, 16 Aug 2016 05:36:11 +0000 (23:36 -0600)]
Various cleanups

- Killing unused declarations

- Bool

- Making code static

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogfio: fix auto-start of backend
Jens Axboe [Tue, 16 Aug 2016 04:05:27 +0000 (22:05 -0600)]
gfio: fix auto-start of backend

Signed-off-by: Jens Axboe <axboe@fb.com>