fio.git
4 years agoFio 2.2.4 fio-2.2.4
Jens Axboe [Sun, 4 Jan 2015 02:46:55 +0000 (19:46 -0700)]
Fio 2.2.4

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobloom: kill unused function
Jens Axboe [Sat, 3 Jan 2015 20:48:30 +0000 (13:48 -0700)]
bloom: kill unused function

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoaxmap: make axmap_first_free() static
Ming Lei [Sat, 3 Jan 2015 20:41:32 +0000 (13:41 -0700)]
axmap: make axmap_first_free() static

Pulled from an earlier patch from Ming, that we no longer need.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoRevert "axmap: ensure we lock down the maps for shared access"
Jens Axboe [Sat, 3 Jan 2015 20:40:24 +0000 (13:40 -0700)]
Revert "axmap: ensure we lock down the maps for shared access"

This reverts commit 12bde3697fc230d7af560cecc8ac469ab47643f8.

4 years agoRevert "axmap: fix deadlock"
Jens Axboe [Sat, 3 Jan 2015 20:40:16 +0000 (13:40 -0700)]
Revert "axmap: fix deadlock"

This reverts commit d555a23db84364ab942361a719a4379605bdbc41.

4 years agoGet rid of lockfile callback
Jens Axboe [Sat, 3 Jan 2015 19:15:57 +0000 (12:15 -0700)]
Get rid of lockfile callback

We don't need it, since the option priorities take care of
ordering.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoaxmap: fix deadlock
Ming Lei [Sat, 3 Jan 2015 14:21:09 +0000 (22:21 +0800)]
axmap: fix deadlock

axmap_first_free() is always called with axmap->lock held,
so needn't to acquire the lock inside the function.

The deadlock is introduced in commit 12bde3697fc230d7a(axmap:
ensure we lock down the maps for shared access).

Given axmap_first_free() is only called inside lib/axmap.c,
this patch declares the function as static. In the future,
if external users need the function, axmap lock can be
considered at that time.

Reported-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Ming Lei <ming.lei@canonical.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: add basic init cpuclock test
Jens Axboe [Tue, 23 Dec 2014 17:41:34 +0000 (10:41 -0700)]
gettime: add basic init cpuclock test

Some systems claim to have a synced and constant rate tsc, even if
they don't. So add the basic cpuclock-test test as part of fio
trusting the CPU clock, to avoid using the CPU clock on a system
where the TSC truly isn't synced between CPUs.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFio 2.2.3 fio-2.2.3
Jens Axboe [Tue, 23 Dec 2014 03:06:45 +0000 (20:06 -0700)]
Fio 2.2.3

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooptions: ensure that we check all candidates on is_set check
Jens Axboe [Tue, 23 Dec 2014 03:03:50 +0000 (20:03 -0700)]
options: ensure that we check all candidates on is_set check

We can have multiple options that touch the same variable. One
example is cpus_allowed or cpumask, both will set the same
backend variable. But if we set cpumask= and later check if
cpus_allowed is set, then we want that to report true.

This breaks certain options since b2a9e6496494.

Reported-by: Vincent Fu <Vincent.Fu@sandisk.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFio 2.2.2 fio-2.2.2
Jens Axboe [Mon, 22 Dec 2014 16:03:50 +0000 (09:03 -0700)]
Fio 2.2.2

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix default scrambling of buffers
Jens Axboe [Sat, 20 Dec 2014 21:42:56 +0000 (14:42 -0700)]
Fix default scrambling of buffers

Commit 1bf2498d3111 inadvertently made us not scramble by default.
The logic was a bit convoluted, so clear it up and scramble if:

- Asked so by adding scramble_buffers=1
- Or, if not asked to explicitly, but zero_buffers not set to 1.

Reported-by: Chris Wilson <Chris.Wilson@nutanix.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: hide fio_gtod_cpumask if not used
Jens Axboe [Fri, 19 Dec 2014 22:04:10 +0000 (15:04 -0700)]
gettime: hide fio_gtod_cpumask if not used

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: only use CPU affinity if available
Jens Axboe [Fri, 19 Dec 2014 22:03:15 +0000 (15:03 -0700)]
gettime: only use CPU affinity if available

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFio 2.2.1 fio-2.2.1
Jens Axboe [Fri, 19 Dec 2014 19:27:49 +0000 (12:27 -0700)]
Fio 2.2.1

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoaxmap: ensure we lock down the maps for shared access
Jens Axboe [Fri, 19 Dec 2014 15:15:55 +0000 (08:15 -0700)]
axmap: ensure we lock down the maps for shared access

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooptions: turn missing options into debug parse messages
Jens Axboe [Fri, 19 Dec 2014 15:04:48 +0000 (08:04 -0700)]
options: turn missing options into debug parse messages

It's not an error, we expect it to happen if we try and check
for set options that aren't available (due to being platform
specific, libraries missing, etc).

This fixes cases where fio would spew:

fio: no option found at offset 1240

and other offsets.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooptions: add debug code for failure to lookup option names
Jens Axboe [Fri, 19 Dec 2014 04:51:21 +0000 (21:51 -0700)]
options: add debug code for failure to lookup option names

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agomutex: add __fio_mutex_remove()
Jens Axboe [Fri, 19 Dec 2014 02:44:18 +0000 (19:44 -0700)]
mutex: add __fio_mutex_remove()

We have to remember to free the condvar, so add a __fio_mutex_remove()
for cases that init a mutex inside an smalloc'ed region with
__fio_mutex_init().

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Thu, 18 Dec 2014 17:20:38 +0000 (10:20 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

4 years agot/btrace2fio: add ability to add specific options through -a
Jens Axboe [Thu, 18 Dec 2014 17:18:30 +0000 (10:18 -0700)]
t/btrace2fio: add ability to add specific options through -a

Can be used multiple times to add more options.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoparse: remove the arithmetic parser checking
Jens Axboe [Wed, 17 Dec 2014 20:27:32 +0000 (13:27 -0700)]
parse: remove the arithmetic parser checking

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFio 2.2.0 fio-2.2.0
Jens Axboe [Wed, 17 Dec 2014 17:46:46 +0000 (10:46 -0700)]
Fio 2.2.0

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: fix compile warning for !ARCH_HAVE_CPU_CLOCK
Jens Axboe [Wed, 17 Dec 2014 16:08:01 +0000 (09:08 -0700)]
gettime: fix compile warning for !ARCH_HAVE_CPU_CLOCK

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: offset CPU cycle counter by initial value
Jens Axboe [Wed, 17 Dec 2014 15:51:23 +0000 (08:51 -0700)]
gettime: offset CPU cycle counter by initial value

Should then be safe for the full 2^64 cycles, and we push
the more expensive variable division to later in the run (which
is probably never).

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agostat: always show disk util in terse v3/v4
Jens Axboe [Wed, 17 Dec 2014 15:30:30 +0000 (08:30 -0700)]
stat: always show disk util in terse v3/v4

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: cleanup for FIO_DEBUG_TIME
Jens Axboe [Wed, 17 Dec 2014 06:03:54 +0000 (23:03 -0700)]
gettime: cleanup for FIO_DEBUG_TIME

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: fix overflow in cycle to usec conversion
Jens Axboe [Wed, 17 Dec 2014 05:40:37 +0000 (22:40 -0700)]
gettime: fix overflow in cycle to usec conversion

If this multiplication overflows:

usecs = (t * inv_cycles_per_usec) / 16777216UL;

then usecs is 2^64/2^24, which is 1099511627776. Divide that by
10^6 to get seconds, and that is 1099511. Since we cached the
old value previously, we'd get stuck with this amount of seconds.

To avoid turning this into an expensive division, have a check
and only divide if we have to. This avoids the overflow.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime-thread: set and allow multiple CPUs
Jens Axboe [Wed, 17 Dec 2014 03:38:53 +0000 (20:38 -0700)]
gettime-thread: set and allow multiple CPUs

Also use the new option checking framework instead of carrying
a flag for whether it's set or not.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime-thread: fix missing startup mutex
Jens Axboe [Wed, 17 Dec 2014 02:49:54 +0000 (19:49 -0700)]
gettime-thread: fix missing startup mutex

It'll segfault if gtod_cpu= is used, ensure we pass in the mutex.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: improve gettimeofday() offload support
Jens Axboe [Wed, 17 Dec 2014 02:43:55 +0000 (19:43 -0700)]
gettime: improve gettimeofday() offload support

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: limit warning on CPU clock
Jens Axboe [Tue, 16 Dec 2014 22:37:25 +0000 (15:37 -0700)]
gettime: limit warning on CPU clock

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: don't attempt to fixup what looks like a backwards clock
Jens Axboe [Tue, 16 Dec 2014 22:13:45 +0000 (15:13 -0700)]
gettime: don't attempt to fixup what looks like a backwards clock

It could just be a wrap. The code is buggy, kill it, we'll deal
with the wrap later.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/mmap: clear partial mmap flag on close
Jens Axboe [Tue, 16 Dec 2014 17:46:24 +0000 (10:46 -0700)]
engines/mmap: clear partial mmap flag on close

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/mmap: fix segfault on large devices and 32-bit archs
Jens Axboe [Tue, 16 Dec 2014 17:35:07 +0000 (10:35 -0700)]
engines/mmap: fix segfault on large devices and 32-bit archs

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: use monotonic clock for ETA, if we have it
Jens Axboe [Tue, 16 Dec 2014 02:54:57 +0000 (19:54 -0700)]
backend: use monotonic clock for ETA, if we have it

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: fix off-by-one in nsec calculation
Jens Axboe [Mon, 15 Dec 2014 20:45:53 +0000 (13:45 -0700)]
backend: fix off-by-one in nsec calculation

If it's equal to 10^9 we need to bump seconds and subtract 10^9
from nsec, not only if it's larger than.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofile: unionize lfsr/randommap
Jens Axboe [Mon, 15 Dec 2014 16:38:43 +0000 (09:38 -0700)]
file: unionize lfsr/randommap

We only use one of them, add file flags to distinguish between the
two.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd helpers for getting/setting file engine data
Jens Axboe [Mon, 15 Dec 2014 15:26:11 +0000 (08:26 -0700)]
Add helpers for getting/setting file engine data

Since it's a uint64_t, we need casting to get/set the engine data
or some platforms will complain. Encapsulate that in helper macros.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofile: move mmap related data to engines/mmap.c where it belongs
Jens Axboe [Mon, 15 Dec 2014 02:17:27 +0000 (19:17 -0700)]
file: move mmap related data to engines/mmap.c where it belongs

We should not keep this in the generic file structure, it belongs
in file engine data for the specific engine.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Mon, 15 Dec 2014 02:03:12 +0000 (19:03 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

4 years agoFix read/write mix and different levels of randomness
Jens Axboe [Mon, 15 Dec 2014 02:01:24 +0000 (19:01 -0700)]
Fix read/write mix and different levels of randomness

Stephen reports:

When using fio configuration below:

[global]
ioengine=libaio
direct=1
runtime=600
bs=32k
iodepth=8
rw=randrw
rwmixread=80
percentage_random=100,0

[drive1]
filename=/dev/sda

I am expecting to see 80% reads, 20% writes where all reads are random
and all writes are sequential. I captured a bus trace of traffic to the
disk and the bus trace reflected as much with one issue. The write
commands are essentially random. Each write begins at a new random LBA.
If 2 or more writes occur in a row, the LBA's are sequential based on
the block size BUT I feel the heart of this feature would be to emulate
a large file write during random access. With that in mind would it be
possible for sequential reads or writes within mixed sequential/random
workload to remember the last LBA accessed? In this scenario the writes
would still only take up 20% of the workload but when a write did occur
it should be the next sequential step from the last write.

-----

This is due to the fact that fio tracks last start/end on a unified
basis, not per read/write/trim direction. Fix that.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoeta.c: check malloc return code
Robert Elliott [Sat, 13 Dec 2014 04:50:30 +0000 (21:50 -0700)]
eta.c: check malloc return code

Check the malloc return code in get_jobs_eta.

Signed-off-by: Robert Elliott <elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: fix should_check_rate() typo in wait_for_completions()
Jens Axboe [Fri, 12 Dec 2014 18:34:28 +0000 (11:34 -0700)]
backend: fix should_check_rate() typo in wait_for_completions()

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: abstract out waiting for completions
Jens Axboe [Fri, 12 Dec 2014 15:49:12 +0000 (08:49 -0700)]
backend: abstract out waiting for completions

We do it both in do_verify() and in do_io(), so it's duplicated code.
And we want to potentially add a completor thread, so lets move this
to a helper.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUpdate size/io_size descriptions
Jens Axboe [Thu, 11 Dec 2014 02:21:37 +0000 (19:21 -0700)]
Update size/io_size descriptions

Hopefully this is clearer. Also add an alias for io_limit to io_size,
since that is probably more descriptive of what this option does.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobtrace2fio: add support for collapsing close entries
Jens Axboe [Wed, 10 Dec 2014 21:01:36 +0000 (22:01 +0100)]
btrace2fio: add support for collapsing close entries

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoImproved support for libhdfs
Fabrice Bacchella [Wed, 10 Dec 2014 16:39:48 +0000 (17:39 +0100)]
Improved support for libhdfs

4 years agogluster: wire up sync/datasync opcodes
Jens Axboe [Wed, 10 Dec 2014 15:12:56 +0000 (08:12 -0700)]
gluster: wire up sync/datasync opcodes

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix server/client set_options conversion
Jens Axboe [Tue, 9 Dec 2014 21:12:47 +0000 (14:12 -0700)]
Fix server/client set_options conversion

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUse fio_option_is_set() for ioprio setting
Jens Axboe [Tue, 9 Dec 2014 21:10:05 +0000 (14:10 -0700)]
Use fio_option_is_set() for ioprio setting

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAlways scramble buffers, if scramble_buffers is set
Jens Axboe [Tue, 9 Dec 2014 20:59:58 +0000 (13:59 -0700)]
Always scramble buffers, if scramble_buffers is set

This makes scramble_buffers and zero_buffers cooperate nicely.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoGet rid of _set variables
Jens Axboe [Tue, 9 Dec 2014 20:17:33 +0000 (13:17 -0700)]
Get rid of _set variables

We can just check the option set mask now.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooptions: add support for checking if an option has been set
Jens Axboe [Tue, 9 Dec 2014 19:58:15 +0000 (12:58 -0700)]
options: add support for checking if an option has been set

We often just do a non-zero check, but an option can be set
and be set to zero, so we can't currently detect that. For
complicated options, we add a second variable to mark if it
has been set or not. This allows us to directly check if a
given option was set by the user or not.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUpdate date and email in man page
Jens Axboe [Sun, 7 Dec 2014 01:47:43 +0000 (18:47 -0700)]
Update date and email in man page

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooptions: strip off =optval when matching
Jens Axboe [Sat, 6 Dec 2014 16:53:46 +0000 (09:53 -0700)]
options: strip off =optval when matching

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooptions: show closest match for unknown job option
Jens Axboe [Sat, 6 Dec 2014 16:17:52 +0000 (09:17 -0700)]
options: show closest match for unknown job option

Just like we did for the command line options.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agolib/rand: cleanup __fill_random_buffer()
Jens Axboe [Fri, 5 Dec 2014 18:04:19 +0000 (11:04 -0700)]
lib/rand: cleanup __fill_random_buffer()

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agolib/rand: fix bug with non uint64_t aligned random buffer fill
Jens Axboe [Fri, 5 Dec 2014 17:48:01 +0000 (10:48 -0700)]
lib/rand: fix bug with non uint64_t aligned random buffer fill

Now that we honor the compression percentage, we can easily
get buffer fills that are not aligned to uint64_t. Make
sure that __fill_random_buf() handles this correctly.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoparse: make suggestions for unknown options
Jens Axboe [Fri, 5 Dec 2014 16:35:40 +0000 (09:35 -0700)]
parse: make suggestions for unknown options

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agosync: use io_u_log_error() for more detailed error messages
Jens Axboe [Fri, 5 Dec 2014 16:35:19 +0000 (09:35 -0700)]
sync: use io_u_log_error() for more detailed error messages

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: cleanup
Jens Axboe [Thu, 4 Dec 2014 23:49:53 +0000 (16:49 -0700)]
verify: cleanup

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUse specified compression/pattern for verify buffers too
Jens Axboe [Thu, 4 Dec 2014 22:40:32 +0000 (15:40 -0700)]
Use specified compression/pattern for verify buffers too

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoDon't clear 'refill_buffers' unconditionally for pattern fill
Jens Axboe [Thu, 4 Dec 2014 15:27:21 +0000 (08:27 -0700)]
Don't clear 'refill_buffers' unconditionally for pattern fill

If we have asked for a specific buffer compression, then we need
the refill option (if set).

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUse specified buffer_pattern (if given) for all io_u fills
Jens Axboe [Thu, 4 Dec 2014 02:55:33 +0000 (19:55 -0700)]
Use specified buffer_pattern (if given) for all io_u fills

For compression, we use a fixed '0' pattern. But if the user
specified a pattern to use in the job file, then we should
use that instead. It could slightly skew the compression ratio
for long patterns, but that is to be expected.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoRemove 'td' parameter from fill_pattern()
Jens Axboe [Thu, 4 Dec 2014 02:25:13 +0000 (19:25 -0700)]
Remove 'td' parameter from fill_pattern()

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix a few spelling errors
Jens Axboe [Wed, 3 Dec 2014 17:10:59 +0000 (10:10 -0700)]
Fix a few spelling errors

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofio: remove deprecated --latency-log from manpage
Elliott, Robert (Server Storage) [Tue, 2 Dec 2014 20:54:32 +0000 (13:54 -0700)]
fio: remove deprecated --latency-log from manpage

The --latency-log argument is no longer accepted; fio returns
an error message saying it is deprecated.  So, remove it from
the manpage.

Signed-off-by: Robert Elliott <elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Mon, 1 Dec 2014 16:44:02 +0000 (09:44 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

4 years agoBump disk util stats from 32 to 64-bit values
Jens Axboe [Mon, 1 Dec 2014 16:42:48 +0000 (09:42 -0700)]
Bump disk util stats from 32 to 64-bit values

Robert reports:

A long run can result in wrapping in the disk stats
(ios, merge, ticks, in_queue).

In this example, there were 212,245,615,134 read IOs
(that is from uint64_t total_io_u[3]):
  read : io=101207GB, bw=491309KB/s, iops=982618, runt=216000016msec
  cpu          : usr=0.09%, sys=4.33%, ctx=152784302066, majf=0, minf=2928
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.5%, 16=0.5%, 32=1.0%, >=64=1.0%
     submit    : 0=0.0%, 4=18.0%, 8=6.6%, 16=8.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=15.0%, 32=0.1%, 64=0.1%, >=64=0.1%
     issued    : total=r=212245615134/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=96

Run status group 0 (all jobs):
   READ: io=101207GB, aggrb=491309KB/s, minb=491309KB/s, maxb=491309KB/s, mint=216000016msec, maxt=216000016msec

All the disk stats, however, wrap at ios=4,294,967,295 (32 bit).
In this case it caught sdi falling a bit behind the others,
but all of them have wrapped multiple times.

Disk stats (read/write):
  sdb: ios=796157304/0, merge=0/0, ticks=722039638/0, in_queue=735284999, util=100.00%
  sdc: ios=725235994/0, merge=0/0, ticks=725893305/0, in_queue=738790349, util=100.00%
  sdd: ios=722370867/0, merge=0/0, ticks=724540972/0, in_queue=738926102, util=100.00%
  sde: ios=722490025/0, merge=0/0, ticks=724775894/0, in_queue=745581851, util=100.00%
  sdf: ios=719665329/0, merge=0/0, ticks=724281840/0, in_queue=740201203, util=100.00%
  sdg: ios=721386377/0, merge=0/0, ticks=725529483/0, in_queue=740729310, util=100.00%
  sdh: ios=720461888/0, merge=0/0, ticks=724759222/0, in_queue=741212131, util=100.00%
  sdi: ios=3280272329/0, merge=0/0, ticks=3337080572/0, in_queue=3340290775, util=81.57%
  sdj: ios=717395167/0, merge=0/0, ticks=723762993/0, in_queue=739136654, util=100.00%
  sdk: ios=824804416/0, merge=0/0, ticks=722973383/0, in_queue=734915061, util=100.00%
  sdl: ios=787317365/0, merge=0/0, ticks=720963594/0, in_queue=736506238, util=100.00%
  sdm: ios=791112845/0, merge=0/0, ticks=723371287/0, in_queue=745590636, util=100.00%
  sdn: ios=786009870/0, merge=0/0, ticks=721111447/0, in_queue=733786474, util=100.00%
  sdo: ios=787226466/0, merge=0/0, ticks=722200707/0, in_queue=737673635, util=100.00%
  sdp: ios=789062478/0, merge=0/0, ticks=723648481/0, in_queue=737820918, util=100.00%
  sdq: ios=786121713/0, merge=0/0, ticks=722203635/0, in_queue=752578258, util=100.00%

These structures are the reason:
struct disk_util_stats {
        uint32_t ios[2];
        uint32_t merges[2];
        uint64_t sectors[2];
        uint32_t ticks[2];
        uint32_t io_ticks;
        uint32_t time_in_queue;
        uint64_t msec;
};
struct disk_util_agg {
        uint32_t ios[2];
        uint32_t merges[2];
        uint64_t sectors[2];
        uint32_t ticks[2];
        uint32_t io_ticks;
        uint32_t time_in_queue;
        uint32_t slavecount;
        fio_fp64_t max_util;
};

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMake gfio build with --build-static
Jens Axboe [Tue, 25 Nov 2014 23:02:52 +0000 (16:02 -0700)]
Make gfio build with --build-static

We can't build the gtk app statically, so filter out the static part
for just that. Fix up the Makefile so that fio doesn't include
all the gtk libs as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFixup t/ test apps for lfsr_next change()
Jens Axboe [Mon, 24 Nov 2014 15:07:03 +0000 (08:07 -0700)]
Fixup t/ test apps for lfsr_next change()

Reported-by: Bruce Cran <bruce@cran.org.uk>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agolfsr: don't pass in last value to lfsr_next()
Jens Axboe [Mon, 24 Nov 2014 01:41:11 +0000 (18:41 -0700)]
lfsr: don't pass in last value to lfsr_next()

It's cached in the 'fl' struct. This means we can move the max
block calculation outside if the lfsr part, too.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: account for wasted time in ipo stall
Jens Axboe [Fri, 21 Nov 2014 21:33:33 +0000 (14:33 -0700)]
iolog: account for wasted time in ipo stall

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoblktrace: correct setting of iodepth
Jens Axboe [Fri, 21 Nov 2014 20:49:00 +0000 (13:49 -0700)]
blktrace: correct setting of iodepth

We want to check for == 1, not != 1. Also set iodepth_low, like
we would have done for a "normal" job file.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoserver: fix missing ETA in some cases
Jens Axboe [Wed, 19 Nov 2014 16:34:01 +0000 (09:34 -0700)]
server: fix missing ETA in some cases

If we get an ETA request from the client, we pass in 'force' to
ignore any time constraints. But we fail to pass the force through,
so we can fail sending the requested ETA and the client just looks
like nothing is running even though it is.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agotrigger: always send trigger, not just when remote command is set
Jens Axboe [Wed, 19 Nov 2014 16:21:16 +0000 (09:21 -0700)]
trigger: always send trigger, not just when remote command is set

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoclient: defer local trigger execute until after state is received
Jens Axboe [Wed, 19 Nov 2014 16:09:01 +0000 (09:09 -0700)]
client: defer local trigger execute until after state is received

Don't run it immediately, we have to wait until we receive the
data from the server.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoHOWTO: add description and examples of verify triggers
Jens Axboe [Wed, 19 Nov 2014 16:06:42 +0000 (09:06 -0700)]
HOWTO: add description and examples of verify triggers

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agotrigger: enable separate remote and local trigger
Jens Axboe [Wed, 19 Nov 2014 15:36:12 +0000 (08:36 -0700)]
trigger: enable separate remote and local trigger

Change the options so that we have the trigger file and command
separate, and remote and local trigger command separate as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoKill off -Wshadow again
Jens Axboe [Wed, 19 Nov 2014 03:14:23 +0000 (20:14 -0700)]
Kill off -Wshadow again

Too many warnings, even for system headers.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: allow empty/no trigger file
Jens Axboe [Wed, 19 Nov 2014 02:16:06 +0000 (19:16 -0700)]
verify: allow empty/no trigger file

If we use a trigger timeout, we don't need the file.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix some shadow warnings for some gcc variants
Jens Axboe [Wed, 12 Nov 2014 18:11:20 +0000 (11:11 -0700)]
Fix some shadow warnings for some gcc variants

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: verify_state_gen_name() should check size
Jens Axboe [Wed, 12 Nov 2014 03:51:37 +0000 (20:51 -0700)]
verify: verify_state_gen_name() should check size

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'verify-trigger'
Jens Axboe [Tue, 11 Nov 2014 18:10:35 +0000 (11:10 -0700)]
Merge branch 'verify-trigger'

4 years agoFio 2.1.14 fio-2.1.14
Jens Axboe [Tue, 11 Nov 2014 18:09:59 +0000 (11:09 -0700)]
Fio 2.1.14

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoclient: replace duplicate code with read_data()
Jens Axboe [Tue, 11 Nov 2014 17:24:00 +0000 (10:24 -0700)]
client: replace duplicate code with read_data()

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd support for verify triggers and verify state saving
Jens Axboe [Tue, 11 Nov 2014 03:34:00 +0000 (20:34 -0700)]
Add support for verify triggers and verify state saving

This allows you to (for instance) instantly terminate a verify
write workload, and then later that everything was written
correctly up to that very point. This can be useful for testing
powercut scenarios, which is often problematic on storage
devices.

The trigger part is a file based notification scheme, similar
to what is provided for the status dumps. When triggered,
fio will exit immediately and write the verify state safely
to disk. A trigger can be accompanied by a trigger command.
Say you wanted to test powercut safety, the trigger could be
something that immediately cut power to the machine.

The verify state is either saved locally (if run locally),
or saved over the network if run in client/server mode.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/btrace2fio: don't dec inflight for requeue
Jens Axboe [Tue, 11 Nov 2014 16:39:11 +0000 (09:39 -0700)]
t/btrace2fio: don't dec inflight for requeue

We don't inc it again on issue, so we'll end up with an assert
trigger on inflight going negative.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/btrace2fio: fix divide-by-zero in rate calculation
Jens Axboe [Tue, 11 Nov 2014 15:09:04 +0000 (08:09 -0700)]
t/btrace2fio: fix divide-by-zero in rate calculation

If usec is less than 1000, it'd crash.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMakefile: add -Wshadow
Jens Axboe [Tue, 11 Nov 2014 15:07:09 +0000 (08:07 -0700)]
Makefile: add -Wshadow

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFixup some of the time (usec) based conversions
Stephen M. Cameron [Tue, 11 Nov 2014 03:31:26 +0000 (20:31 -0700)]
Fixup some of the time (usec) based conversions

Modified by Jens to fixup some of the mtime/utime confusion,
and add a specific ->is_time to options to get rid of future
issues in this area.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoGet rid if ddir_trim() macro
Jens Axboe [Mon, 10 Nov 2014 22:40:24 +0000 (15:40 -0700)]
Get rid if ddir_trim() macro

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix crash on threads being reaped before they are created
Jens Axboe [Mon, 10 Nov 2014 21:08:50 +0000 (14:08 -0700)]
Fix crash on threads being reaped before they are created

If we do that, then ->rusage_sem is never allocated.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd option for statically build fio
Jens Axboe [Mon, 10 Nov 2014 18:18:06 +0000 (11:18 -0700)]
Add option for statically build fio

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/btrace2fio: cap depth if we don't see completion traces
Jens Axboe [Mon, 10 Nov 2014 15:52:23 +0000 (08:52 -0700)]
t/btrace2fio: cap depth if we don't see completion traces

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix for a race when fio prints I/O statistics periodically
Vasily Tarasov [Mon, 10 Nov 2014 03:22:24 +0000 (20:22 -0700)]
Fix for a race when fio prints I/O statistics periodically

Below is the demonstration for the latest code in git:

Job-file:

[job]
filename=/dev/sda
rw=randread
blocksize=4096
direct=1
time_based=1
runtime=300
numjobs=16

Command line:

fio --output-format=json --status-interval=10  /tmp/job.file | tee /tmp/fio.out

After parsing json and printing per-job "read->runtime" and "read->iops" in two
columns, one can see this:

runtime iops
9854 4762
19861 4724
29867 4711
39873 4705
49880 4701
59886 4698
69894 4696
79900 4694
89907 4693
99913 4692
109919 4692
119926 4691
129933 4691
139940 4691
149946 4691
159953 4690
169959 4690
179966 4690
189972 4690
199978 4690
209984 4690
219990 4690
229996 4690
240002 4690
250009 4690
260015 4690
270021 4690
280027 4690
290033 4690
600043 2344 <--- You can see a spurious jump in runtime here
300001 4690

The problem is two-fold, IMHO:

1) td->ts.runtime[DDIR_READ/DDIR_WRITE/DDIR_TRIM] are set in microseconds
   instead of milliseconds in thread_main() function. The values are later
   converted to milliseconds (after exiting from the loop in thread_main()),
   but it is too late for the out-of-band access by the helper_thread
   (the one that prints periodic statistics).

2) The helper_thread in __show_running_run_stats() temporarily adds to
   td->ts.runtime[DDIR_READ/DDIR_WRITE/DDIR_TRIM]
   the time that has passed since the beginning of the loop (td->start).
   However, the same addition happens in thread_main() by the job
   thread and td->start is not reset atomically with this addition.
   As a result, the runtime can be added twice. (That's why in the output
   we see 600043, which is approximately twice larger than the
   previous reading).

My proposal is to make the job threads updating both td->ts.runtime and
td->start atomically by using an already existing stat_mutex.
__show_running_run_stats() already takes this mutex where needed.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoclient/server: percentile_precision wasn't net converted
Jens Axboe [Sat, 8 Nov 2014 01:47:41 +0000 (18:47 -0700)]
client/server: percentile_precision wasn't net converted

We'd get 0 precision on the percentile dumps, which was confusing
and lost info.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: fix intermittent crash on exit with verify enabled
Jens Axboe [Fri, 7 Nov 2014 22:41:24 +0000 (15:41 -0700)]
iolog: fix intermittent crash on exit with verify enabled

Commit 9342d5f807795 added flist_first_entry(), but the
conversion io iolog hist list pruning was wrong. Fix it.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd time_since_genesis()
Jens Axboe [Fri, 7 Nov 2014 17:36:33 +0000 (10:36 -0700)]
Add time_since_genesis()

Signed-off-by: Jens Axboe <axboe@fb.com>