fio.git
5 years agoAllow random overwrite workloads to write io_limit
Justin Eno [Thu, 29 Jan 2015 20:27:58 +0000 (12:27 -0800)]
Allow random overwrite workloads to write io_limit

Limiting the write phase of (random) overwrite workloads to 'size'
before the verify phase doesn't make sense, so allow them to write
up to io_limit.  This better matches verify_only behavior, and
allows the [verify-job] below to succeed:

[global]
thread=1
direct=1
ioengine=sync
verify=meta
bs=1k
offset=0
size=5M
io_limit=6M
norandommap=1
verify_fatal=1
randseed=42

[write-job]
filename=datafile.tmp
rw=randwrite
do_verify=0

[verify-job]
stonewall
create_serialize=0
filename=datafile.tmp
rw=randwrite
do_verify=1
verify_only=1

Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime.h: use time_t instead of size_t for copy of tv_sec
Castor Fu [Fri, 13 Feb 2015 14:46:04 +0000 (06:46 -0800)]
gettime.h: use time_t instead of size_t for copy of tv_sec

This addresses a small nit that our compiler was complaining about.
Thanks!

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: Fix latency log for verify commands.
Gwendal Grignou [Mon, 9 Feb 2015 03:53:20 +0000 (19:53 -0800)]
verify: Fix latency log for verify commands.

When commands when requeued for the verify operation,
their start time was not reset, resulting in bogus latency graphs.

Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix for verify_only (do_dry_run()) broken by 74d6277f
Justin Eno [Wed, 28 Jan 2015 22:13:28 +0000 (14:13 -0800)]
Fix for verify_only (do_dry_run()) broken by 74d6277f

Previous commit to backend.c tracks io_limit more closely by
counting submitted (in-flight) I/O instead of completed I/O.
do_dry_run() does not submit I/O, so its I/O is not counted,
and it loops forever.
Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoconfigure: fix libnuma_v2 probe to work with -Werror
Castor Fu [Thu, 29 Jan 2015 18:07:36 +0000 (10:07 -0800)]
configure: fix libnuma_v2 probe to work with -Werror

I noticed that libnuma_v2 wasn't getting detected in our build
environment because we would get an error that mask was unused.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMoral license update
Jens Axboe [Fri, 6 Feb 2015 16:09:29 +0000 (09:09 -0700)]
Moral license update

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoUpdate the moral license
Jens Axboe [Fri, 30 Jan 2015 17:38:20 +0000 (09:38 -0800)]
Update the moral license

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: remember to tear down clock cpumask on normal exit
Jens Axboe [Thu, 29 Jan 2015 17:23:39 +0000 (09:23 -0800)]
gettime: remember to tear down clock cpumask on normal exit

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: fix unitialized variable on Solaris
Jens Axboe [Thu, 29 Jan 2015 16:32:28 +0000 (08:32 -0800)]
gettime: fix unitialized variable on Solaris

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoidletime: maintain cpuset over lifetime of idle thread
Jens Axboe [Wed, 28 Jan 2015 21:47:48 +0000 (14:47 -0700)]
idletime: maintain cpuset over lifetime of idle thread

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: initialize cpusets properly
Jens Axboe [Wed, 28 Jan 2015 21:42:10 +0000 (14:42 -0700)]
gettime: initialize cpusets properly

We can't just use memset(), some platforms have specific init/exit
routines for cpusets/masks.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: if setaffinity fails, print the errno error
Jens Axboe [Wed, 28 Jan 2015 21:34:10 +0000 (14:34 -0700)]
gettime: if setaffinity fails, print the errno error

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFixing typo
Felipe Franciosi [Wed, 28 Jan 2015 19:06:32 +0000 (19:06 +0000)]
Fixing typo

5 years agodedupe_percentage should work even if compress_percentage is not set
Vasily Tarasov [Wed, 28 Jan 2015 16:10:30 +0000 (09:10 -0700)]
dedupe_percentage should work even if compress_percentage is not set

Currently dedupe_percentage option does not work properly if
compress_percentage is not set. This can be easily demonstrated using
the following job file:

[job-/dev/dm-0]
filename=/dev/dm-0
rw=randwrite
blocksize=4096
direct=1
ioengine=libaio
iodepth=32
dedupe_percentage=50
time_based=1
runtime=5
numjobs=1

All writes will contain zeroes for the above job file.

The problem is in fill_io_buffer() function that switches dedupe random
state only if o->compress_percentage is set.

This patch updates the condition appropriately in fill_io_buffer()
appropriately.

Signed-off-by: Vasily Tarasov <tarasov@vasily.name>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoBetter accommodate random writes larger than blockalign
Justin Eno [Tue, 27 Jan 2015 22:23:20 +0000 (14:23 -0800)]
Better accommodate random writes larger than blockalign

fill_io_u() fails prematurely if the randomly-chosen offset satisfies
blockalign but not min_bs, i.e., the offset lies too near the end of
the target region.  This change honors both parameters.

Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoparse: ensure strings are pre-terminated when using strncpy()
Jens Axboe [Thu, 22 Jan 2015 21:51:56 +0000 (14:51 -0700)]
parse: ensure strings are pre-terminated when using strncpy()

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Wed, 21 Jan 2015 23:57:00 +0000 (00:57 +0100)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

5 years agoFio 2.2.5 fio-2.2.5
Jens Axboe [Wed, 21 Jan 2015 20:07:43 +0000 (13:07 -0700)]
Fio 2.2.5

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoshm: disable iomem=shm* for shm disabled
Jens Axboe [Wed, 21 Jan 2015 01:38:11 +0000 (18:38 -0700)]
shm: disable iomem=shm* for shm disabled

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoconfigure: add --disable-shm to disable use of shm
Jens Axboe [Tue, 20 Jan 2015 23:29:33 +0000 (16:29 -0700)]
configure: add --disable-shm to disable use of shm

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agonet: don't record/verify UDP sequence numbers if buffer is too small
Steven Noonan [Sat, 17 Jan 2015 00:46:11 +0000 (16:46 -0800)]
net: don't record/verify UDP sequence numbers if buffer is too small

This causes a bunch of out-of-bounds accesses if you have really small buffer
sizes (i.e. 16 bytes will crash).

Signed-off-by: Steven Noonan <steven@uplinklabs.net>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agocrc/test: sha1 should do _final() after _update()
Jens Axboe [Fri, 16 Jan 2015 23:10:43 +0000 (16:10 -0700)]
crc/test: sha1 should do _final() after _update()

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix segfault with client/server and minimal output
Jens Axboe [Fri, 16 Jan 2015 21:25:47 +0000 (14:25 -0700)]
Fix segfault with client/server and minimal output

Fixes: bf2921f02651
Reported-by: Ben England <bengland@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agosha256: fix verify failure
Jens Axboe [Fri, 16 Jan 2015 21:25:15 +0000 (14:25 -0700)]
sha256: fix verify failure

After the checksumming update, it's required that we call the
_final() to get consistent checksums between data generation
and data verification. Seems to only affect sha256, but we should
do it for sha1 and md5 too.

Fixes: f99d67f932ab
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agosha256: fix verify failure
Jens Axboe [Fri, 16 Jan 2015 20:12:53 +0000 (21:12 +0100)]
sha256: fix verify failure

After the checksumming update, it's required that we call the
_final() to get consistent checksums between data generation
and data verification. Seems to only affect sha256, but we should
do it for sha1 and md5 too.

Fixes: f99d67f932ab
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: always check completion list for low depth verifies
Jens Axboe [Fri, 16 Jan 2015 17:25:05 +0000 (10:25 -0700)]
verify: always check completion list for low depth verifies

If we have done less IO than the depth, we need to check the
pending list.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoioengine: if we get BUSY in queuing, adjust accounting
Jens Axboe [Fri, 16 Jan 2015 17:06:02 +0000 (10:06 -0700)]
ioengine: if we get BUSY in queuing, adjust accounting

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoImprove precision of the io_limit setting
Jens Axboe [Fri, 16 Jan 2015 17:03:11 +0000 (10:03 -0700)]
Improve precision of the io_limit setting

For async engines, we look only at completions. But we could have
a bunch inflight with a high queue depth, making us go higher than
we should.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoImprove rate limiting
Jens Axboe [Thu, 15 Jan 2015 17:39:12 +0000 (10:39 -0700)]
Improve rate limiting

If we have pending IO, we should commit it before going to sleep,
not just wait for IO that has already been issued.

Also improve the delay functions to return the time spent, so
that users don't have to track tha separately if they care about
the precision.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agorbd example: 'invalidate=false' no longer necessary
Jens Axboe [Wed, 14 Jan 2015 16:31:35 +0000 (09:31 -0700)]
rbd example: 'invalidate=false' no longer necessary

As of commit 903b2812e9c7, rbd does support cache invalidation.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoclient: take better care to return failure from fio_handle_clients()
Jens Axboe [Wed, 14 Jan 2015 04:47:43 +0000 (21:47 -0700)]
client: take better care to return failure from fio_handle_clients()

If any error has been noted in the ->error field, we need to pass
back a non-zero return from fio_handle_clients().

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agosmalloc: limit to 1 pool, and bump size to 16MB
Jens Axboe [Wed, 14 Jan 2015 04:35:31 +0000 (21:35 -0700)]
smalloc: limit to 1 pool, and bump size to 16MB

We can easily add new pools, but if we do this after jobs have been
forked off, then it wont really be shared storage as the new mmap
mappings will only apply to the process that instantiated them.
Since we moved the randommap mappings to private storage, we should
not need a lot of room for shared allocations. Hence I hope the 16MB
will suffice.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoaxmap: random maps are private, don't get them from smalloc
Jens Axboe [Wed, 14 Jan 2015 04:33:53 +0000 (21:33 -0700)]
axmap: random maps are private, don't get them from smalloc

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofilelock: fix segfault on some use cases of log file locking
Jens Axboe [Wed, 14 Jan 2015 04:06:45 +0000 (21:06 -0700)]
filelock: fix segfault on some use cases of log file locking

If we end up diving into a new smalloc pool, we could add
file lock references that meant nothing to other processes.

Fixes: 243bfe190245
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agommap backend invalidate fix
Yoshinori Sato [Wed, 7 Jan 2015 03:16:36 +0000 (12:16 +0900)]
mmap backend invalidate fix

madvice before mmap. moved later.

Signed-off-by Yoshinori Sato <yo-satoh@sios.com>

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix disk utils being updated too often
Jens Axboe [Wed, 7 Jan 2015 22:04:39 +0000 (15:04 -0700)]
Fix disk utils being updated too often

Fixes: f893b76d5745
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdjust ctime_r buf down to 32
Jens Axboe [Tue, 6 Jan 2015 19:03:09 +0000 (12:03 -0700)]
Adjust ctime_r buf down to 32

We need 'at least 26 bytes', but not all 64.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd timestamp to json output
Steve ODriscoll [Tue, 6 Jan 2015 19:00:27 +0000 (12:00 -0700)]
Add timestamp to json output

Modified by Jens to add humanly readable time as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: include min/max cycle counts for CPU clock
Jens Axboe [Tue, 6 Jan 2015 16:36:25 +0000 (09:36 -0700)]
gettime: include min/max cycle counts for CPU clock

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoserver: use fio_gettime() for timeout variables
Jens Axboe [Mon, 5 Jan 2015 20:51:22 +0000 (13:51 -0700)]
server: use fio_gettime() for timeout variables

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: fix potentially unterminated name
Jens Axboe [Mon, 5 Jan 2015 15:52:37 +0000 (08:52 -0700)]
verify: fix potentially unterminated name

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agomemory: double check that mmapfd isn't -1 before closing
Jens Axboe [Mon, 5 Jan 2015 15:50:41 +0000 (08:50 -0700)]
memory: double check that mmapfd isn't -1 before closing

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogluster: io_u->error should be a positive error value
Jens Axboe [Mon, 5 Jan 2015 15:50:26 +0000 (08:50 -0700)]
gluster: io_u->error should be a positive error value

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoclient: fix potential buffer overrun in server name copy
Jens Axboe [Mon, 5 Jan 2015 15:47:01 +0000 (08:47 -0700)]
client: fix potential buffer overrun in server name copy

Not an issue right now since pdu.server is larger than the buffer,
but that could change at some point. Better be safe.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoinit: fix potential mem leak on multiple trigger options
Jens Axboe [Mon, 5 Jan 2015 15:46:35 +0000 (08:46 -0700)]
init: fix potential mem leak on multiple trigger options

Free the ones we already set, if any.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAvoid potential buffer overflow in make_filename()
Jens Axboe [Sun, 4 Jan 2015 20:33:37 +0000 (13:33 -0700)]
Avoid potential buffer overflow in make_filename()

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agocleanup: add defines for default pareto or zipf values
Jens Axboe [Sun, 4 Jan 2015 17:02:34 +0000 (10:02 -0700)]
cleanup: add defines for default pareto or zipf values

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.2.4 fio-2.2.4
Jens Axboe [Sun, 4 Jan 2015 02:46:55 +0000 (19:46 -0700)]
Fio 2.2.4

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobloom: kill unused function
Jens Axboe [Sat, 3 Jan 2015 20:48:30 +0000 (13:48 -0700)]
bloom: kill unused function

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoaxmap: make axmap_first_free() static
Ming Lei [Sat, 3 Jan 2015 20:41:32 +0000 (13:41 -0700)]
axmap: make axmap_first_free() static

Pulled from an earlier patch from Ming, that we no longer need.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoRevert "axmap: ensure we lock down the maps for shared access"
Jens Axboe [Sat, 3 Jan 2015 20:40:24 +0000 (13:40 -0700)]
Revert "axmap: ensure we lock down the maps for shared access"

This reverts commit 12bde3697fc230d7af560cecc8ac469ab47643f8.

5 years agoRevert "axmap: fix deadlock"
Jens Axboe [Sat, 3 Jan 2015 20:40:16 +0000 (13:40 -0700)]
Revert "axmap: fix deadlock"

This reverts commit d555a23db84364ab942361a719a4379605bdbc41.

5 years agoGet rid of lockfile callback
Jens Axboe [Sat, 3 Jan 2015 19:15:57 +0000 (12:15 -0700)]
Get rid of lockfile callback

We don't need it, since the option priorities take care of
ordering.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoaxmap: fix deadlock
Ming Lei [Sat, 3 Jan 2015 14:21:09 +0000 (22:21 +0800)]
axmap: fix deadlock

axmap_first_free() is always called with axmap->lock held,
so needn't to acquire the lock inside the function.

The deadlock is introduced in commit 12bde3697fc230d7a(axmap:
ensure we lock down the maps for shared access).

Given axmap_first_free() is only called inside lib/axmap.c,
this patch declares the function as static. In the future,
if external users need the function, axmap lock can be
considered at that time.

Reported-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Ming Lei <ming.lei@canonical.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: add basic init cpuclock test
Jens Axboe [Tue, 23 Dec 2014 17:41:34 +0000 (10:41 -0700)]
gettime: add basic init cpuclock test

Some systems claim to have a synced and constant rate tsc, even if
they don't. So add the basic cpuclock-test test as part of fio
trusting the CPU clock, to avoid using the CPU clock on a system
where the TSC truly isn't synced between CPUs.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.2.3 fio-2.2.3
Jens Axboe [Tue, 23 Dec 2014 03:06:45 +0000 (20:06 -0700)]
Fio 2.2.3

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agooptions: ensure that we check all candidates on is_set check
Jens Axboe [Tue, 23 Dec 2014 03:03:50 +0000 (20:03 -0700)]
options: ensure that we check all candidates on is_set check

We can have multiple options that touch the same variable. One
example is cpus_allowed or cpumask, both will set the same
backend variable. But if we set cpumask= and later check if
cpus_allowed is set, then we want that to report true.

This breaks certain options since b2a9e6496494.

Reported-by: Vincent Fu <Vincent.Fu@sandisk.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.2.2 fio-2.2.2
Jens Axboe [Mon, 22 Dec 2014 16:03:50 +0000 (09:03 -0700)]
Fio 2.2.2

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix default scrambling of buffers
Jens Axboe [Sat, 20 Dec 2014 21:42:56 +0000 (14:42 -0700)]
Fix default scrambling of buffers

Commit 1bf2498d3111 inadvertently made us not scramble by default.
The logic was a bit convoluted, so clear it up and scramble if:

- Asked so by adding scramble_buffers=1
- Or, if not asked to explicitly, but zero_buffers not set to 1.

Reported-by: Chris Wilson <Chris.Wilson@nutanix.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: hide fio_gtod_cpumask if not used
Jens Axboe [Fri, 19 Dec 2014 22:04:10 +0000 (15:04 -0700)]
gettime: hide fio_gtod_cpumask if not used

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: only use CPU affinity if available
Jens Axboe [Fri, 19 Dec 2014 22:03:15 +0000 (15:03 -0700)]
gettime: only use CPU affinity if available

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.2.1 fio-2.2.1
Jens Axboe [Fri, 19 Dec 2014 19:27:49 +0000 (12:27 -0700)]
Fio 2.2.1

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoaxmap: ensure we lock down the maps for shared access
Jens Axboe [Fri, 19 Dec 2014 15:15:55 +0000 (08:15 -0700)]
axmap: ensure we lock down the maps for shared access

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agooptions: turn missing options into debug parse messages
Jens Axboe [Fri, 19 Dec 2014 15:04:48 +0000 (08:04 -0700)]
options: turn missing options into debug parse messages

It's not an error, we expect it to happen if we try and check
for set options that aren't available (due to being platform
specific, libraries missing, etc).

This fixes cases where fio would spew:

fio: no option found at offset 1240

and other offsets.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agooptions: add debug code for failure to lookup option names
Jens Axboe [Fri, 19 Dec 2014 04:51:21 +0000 (21:51 -0700)]
options: add debug code for failure to lookup option names

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agomutex: add __fio_mutex_remove()
Jens Axboe [Fri, 19 Dec 2014 02:44:18 +0000 (19:44 -0700)]
mutex: add __fio_mutex_remove()

We have to remember to free the condvar, so add a __fio_mutex_remove()
for cases that init a mutex inside an smalloc'ed region with
__fio_mutex_init().

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Thu, 18 Dec 2014 17:20:38 +0000 (10:20 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

5 years agot/btrace2fio: add ability to add specific options through -a
Jens Axboe [Thu, 18 Dec 2014 17:18:30 +0000 (10:18 -0700)]
t/btrace2fio: add ability to add specific options through -a

Can be used multiple times to add more options.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoparse: remove the arithmetic parser checking
Jens Axboe [Wed, 17 Dec 2014 20:27:32 +0000 (13:27 -0700)]
parse: remove the arithmetic parser checking

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.2.0 fio-2.2.0
Jens Axboe [Wed, 17 Dec 2014 17:46:46 +0000 (10:46 -0700)]
Fio 2.2.0

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: fix compile warning for !ARCH_HAVE_CPU_CLOCK
Jens Axboe [Wed, 17 Dec 2014 16:08:01 +0000 (09:08 -0700)]
gettime: fix compile warning for !ARCH_HAVE_CPU_CLOCK

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: offset CPU cycle counter by initial value
Jens Axboe [Wed, 17 Dec 2014 15:51:23 +0000 (08:51 -0700)]
gettime: offset CPU cycle counter by initial value

Should then be safe for the full 2^64 cycles, and we push
the more expensive variable division to later in the run (which
is probably never).

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agostat: always show disk util in terse v3/v4
Jens Axboe [Wed, 17 Dec 2014 15:30:30 +0000 (08:30 -0700)]
stat: always show disk util in terse v3/v4

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: cleanup for FIO_DEBUG_TIME
Jens Axboe [Wed, 17 Dec 2014 06:03:54 +0000 (23:03 -0700)]
gettime: cleanup for FIO_DEBUG_TIME

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: fix overflow in cycle to usec conversion
Jens Axboe [Wed, 17 Dec 2014 05:40:37 +0000 (22:40 -0700)]
gettime: fix overflow in cycle to usec conversion

If this multiplication overflows:

usecs = (t * inv_cycles_per_usec) / 16777216UL;

then usecs is 2^64/2^24, which is 1099511627776. Divide that by
10^6 to get seconds, and that is 1099511. Since we cached the
old value previously, we'd get stuck with this amount of seconds.

To avoid turning this into an expensive division, have a check
and only divide if we have to. This avoids the overflow.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime-thread: set and allow multiple CPUs
Jens Axboe [Wed, 17 Dec 2014 03:38:53 +0000 (20:38 -0700)]
gettime-thread: set and allow multiple CPUs

Also use the new option checking framework instead of carrying
a flag for whether it's set or not.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime-thread: fix missing startup mutex
Jens Axboe [Wed, 17 Dec 2014 02:49:54 +0000 (19:49 -0700)]
gettime-thread: fix missing startup mutex

It'll segfault if gtod_cpu= is used, ensure we pass in the mutex.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: improve gettimeofday() offload support
Jens Axboe [Wed, 17 Dec 2014 02:43:55 +0000 (19:43 -0700)]
gettime: improve gettimeofday() offload support

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: limit warning on CPU clock
Jens Axboe [Tue, 16 Dec 2014 22:37:25 +0000 (15:37 -0700)]
gettime: limit warning on CPU clock

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogettime: don't attempt to fixup what looks like a backwards clock
Jens Axboe [Tue, 16 Dec 2014 22:13:45 +0000 (15:13 -0700)]
gettime: don't attempt to fixup what looks like a backwards clock

It could just be a wrap. The code is buggy, kill it, we'll deal
with the wrap later.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoengines/mmap: clear partial mmap flag on close
Jens Axboe [Tue, 16 Dec 2014 17:46:24 +0000 (10:46 -0700)]
engines/mmap: clear partial mmap flag on close

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoengines/mmap: fix segfault on large devices and 32-bit archs
Jens Axboe [Tue, 16 Dec 2014 17:35:07 +0000 (10:35 -0700)]
engines/mmap: fix segfault on large devices and 32-bit archs

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobackend: use monotonic clock for ETA, if we have it
Jens Axboe [Tue, 16 Dec 2014 02:54:57 +0000 (19:54 -0700)]
backend: use monotonic clock for ETA, if we have it

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobackend: fix off-by-one in nsec calculation
Jens Axboe [Mon, 15 Dec 2014 20:45:53 +0000 (13:45 -0700)]
backend: fix off-by-one in nsec calculation

If it's equal to 10^9 we need to bump seconds and subtract 10^9
from nsec, not only if it's larger than.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofile: unionize lfsr/randommap
Jens Axboe [Mon, 15 Dec 2014 16:38:43 +0000 (09:38 -0700)]
file: unionize lfsr/randommap

We only use one of them, add file flags to distinguish between the
two.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd helpers for getting/setting file engine data
Jens Axboe [Mon, 15 Dec 2014 15:26:11 +0000 (08:26 -0700)]
Add helpers for getting/setting file engine data

Since it's a uint64_t, we need casting to get/set the engine data
or some platforms will complain. Encapsulate that in helper macros.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofile: move mmap related data to engines/mmap.c where it belongs
Jens Axboe [Mon, 15 Dec 2014 02:17:27 +0000 (19:17 -0700)]
file: move mmap related data to engines/mmap.c where it belongs

We should not keep this in the generic file structure, it belongs
in file engine data for the specific engine.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Mon, 15 Dec 2014 02:03:12 +0000 (19:03 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

5 years agoFix read/write mix and different levels of randomness
Jens Axboe [Mon, 15 Dec 2014 02:01:24 +0000 (19:01 -0700)]
Fix read/write mix and different levels of randomness

Stephen reports:

When using fio configuration below:

[global]
ioengine=libaio
direct=1
runtime=600
bs=32k
iodepth=8
rw=randrw
rwmixread=80
percentage_random=100,0

[drive1]
filename=/dev/sda

I am expecting to see 80% reads, 20% writes where all reads are random
and all writes are sequential. I captured a bus trace of traffic to the
disk and the bus trace reflected as much with one issue. The write
commands are essentially random. Each write begins at a new random LBA.
If 2 or more writes occur in a row, the LBA's are sequential based on
the block size BUT I feel the heart of this feature would be to emulate
a large file write during random access. With that in mind would it be
possible for sequential reads or writes within mixed sequential/random
workload to remember the last LBA accessed? In this scenario the writes
would still only take up 20% of the workload but when a write did occur
it should be the next sequential step from the last write.

-----

This is due to the fact that fio tracks last start/end on a unified
basis, not per read/write/trim direction. Fix that.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoeta.c: check malloc return code
Robert Elliott [Sat, 13 Dec 2014 04:50:30 +0000 (21:50 -0700)]
eta.c: check malloc return code

Check the malloc return code in get_jobs_eta.

Signed-off-by: Robert Elliott <elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobackend: fix should_check_rate() typo in wait_for_completions()
Jens Axboe [Fri, 12 Dec 2014 18:34:28 +0000 (11:34 -0700)]
backend: fix should_check_rate() typo in wait_for_completions()

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobackend: abstract out waiting for completions
Jens Axboe [Fri, 12 Dec 2014 15:49:12 +0000 (08:49 -0700)]
backend: abstract out waiting for completions

We do it both in do_verify() and in do_io(), so it's duplicated code.
And we want to potentially add a completor thread, so lets move this
to a helper.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoUpdate size/io_size descriptions
Jens Axboe [Thu, 11 Dec 2014 02:21:37 +0000 (19:21 -0700)]
Update size/io_size descriptions

Hopefully this is clearer. Also add an alias for io_limit to io_size,
since that is probably more descriptive of what this option does.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobtrace2fio: add support for collapsing close entries
Jens Axboe [Wed, 10 Dec 2014 21:01:36 +0000 (22:01 +0100)]
btrace2fio: add support for collapsing close entries

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoImproved support for libhdfs
Fabrice Bacchella [Wed, 10 Dec 2014 16:39:48 +0000 (17:39 +0100)]
Improved support for libhdfs

5 years agogluster: wire up sync/datasync opcodes
Jens Axboe [Wed, 10 Dec 2014 15:12:56 +0000 (08:12 -0700)]
gluster: wire up sync/datasync opcodes

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix server/client set_options conversion
Jens Axboe [Tue, 9 Dec 2014 21:12:47 +0000 (14:12 -0700)]
Fix server/client set_options conversion

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoUse fio_option_is_set() for ioprio setting
Jens Axboe [Tue, 9 Dec 2014 21:10:05 +0000 (14:10 -0700)]
Use fio_option_is_set() for ioprio setting

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAlways scramble buffers, if scramble_buffers is set
Jens Axboe [Tue, 9 Dec 2014 20:59:58 +0000 (13:59 -0700)]
Always scramble buffers, if scramble_buffers is set

This makes scramble_buffers and zero_buffers cooperate nicely.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoGet rid of _set variables
Jens Axboe [Tue, 9 Dec 2014 20:17:33 +0000 (13:17 -0700)]
Get rid of _set variables

We can just check the option set mask now.

Signed-off-by: Jens Axboe <axboe@fb.com>