fio.git
5 years agofifo: use minmax.h instead of rolling its own min/max
Jens Axboe [Tue, 23 Sep 2014 16:35:49 +0000 (10:35 -0600)]
fifo: use minmax.h instead of rolling its own min/max

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agodedupe: if percentage is 100, don't go through random + math
Jens Axboe [Mon, 22 Sep 2014 20:20:05 +0000 (14:20 -0600)]
dedupe: if percentage is 100, don't go through random + math

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoBasic support for dedupe
Jens Axboe [Mon, 22 Sep 2014 16:02:07 +0000 (10:02 -0600)]
Basic support for dedupe

This adds and option, dedupe_percentage, that controls how many of
the write IO buffers are identical. For instance, if this is set:

dedupe_percentage=70

then 70% of the write IO buffers will have identical contents. The
specific contents are, as before, controlled by the various options
that set buffer contents or buffer compressibility.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobtrace2fio: expand rate options, bug fixes
Jens Axboe [Thu, 18 Sep 2014 15:23:55 +0000 (17:23 +0200)]
btrace2fio: expand rate options, bug fixes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agobtrace2fio: add rate output
Jens Axboe [Wed, 17 Sep 2014 15:58:31 +0000 (17:58 +0200)]
btrace2fio: add rate output

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobtrace2fio: move file tracking to btrace_pid
Jens Axboe [Wed, 17 Sep 2014 02:17:55 +0000 (20:17 -0600)]
btrace2fio: move file tracking to btrace_pid

We want to keep btrace_out as per-device, potentially. So move
this part since it's per process.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofio: print io_u errors on one line
Robert Elliott [Tue, 16 Sep 2014 22:09:48 +0000 (17:09 -0500)]
fio: print io_u errors on one line

Print io_u errors on one line and with one log_err command
to keep them from being interrupted by other prints

Old:
fio: io_u error on file /dev/sdt: Input/output error5.1MB/0KB/0KB /s]
[227K/0/0 iops] [eta 02d:10h:53m:36s]
     read offset=145442430976, buflen=4096
fio: io_u error on file /dev/sdt: Input/output error
     read offset=397760724992, buflen=4096

New:
fio: io_u error on file /dev/sdr: Input/output error: write offset=1532626153472, buflen=4096
fio: io_u error on file /dev/sdt: Input/output error: write offset=102684229632, buflen=4096

Signed-off-by: Robert Elliott <elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFixup data-direction to name translation
Jens Axboe [Wed, 17 Sep 2014 02:07:59 +0000 (20:07 -0600)]
Fixup data-direction to name translation

We had two of them, and they were both wrong since trim got
introduced. Replace by a single implementation in io_ddir.h,
right where it's defined.

Reported-by: Elliott, Robert <Elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobtrace2fio: cleanups and inflight hashing
Jens Axboe [Tue, 16 Sep 2014 16:28:42 +0000 (18:28 +0200)]
btrace2fio: cleanups and inflight hashing

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agobtrace2fio: use percentage_random and add filename option
Jens Axboe [Tue, 16 Sep 2014 15:40:03 +0000 (17:40 +0200)]
btrace2fio: use percentage_random and add filename option

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agobtrace2fio: set runtime
Jens Axboe [Tue, 16 Sep 2014 15:35:55 +0000 (17:35 +0200)]
btrace2fio: set runtime

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Tue, 16 Sep 2014 15:07:51 +0000 (17:07 +0200)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

5 years agoAdd basic tool to turn blktrace into fio job file
Jens Axboe [Tue, 16 Sep 2014 15:06:52 +0000 (17:06 +0200)]
Add basic tool to turn blktrace into fio job file

No documentation yet, it's just a start.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agooptions: fix single use if / for bssplit
Jens Axboe [Tue, 16 Sep 2014 00:51:32 +0000 (18:51 -0600)]
options: fix single use if / for bssplit

It was buggy in that it did not extend to 100%, if nobody else
was part of the group.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.1.12 fio-2.1.12
Jens Axboe [Fri, 12 Sep 2014 18:09:32 +0000 (12:09 -0600)]
Fio 2.1.12

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoserver: fix wrong use of exit() in server when daemonized
Jens Axboe [Thu, 11 Sep 2014 23:05:58 +0000 (17:05 -0600)]
server: fix wrong use of exit() in server when daemonized

We can't use exit(), or we'll trigger the atexit() hook and
free the shared mem used for mutexes, etc. This can cause a
hang when using a daemonized fio server.

Reported-by: Castor Fu <castor@egocast.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix hangs due to iodepth_low
Robert Elliott [Thu, 4 Sep 2014 19:51:05 +0000 (13:51 -0600)]
Fix hangs due to iodepth_low

With some combinations of iodepth, iodepth_batch, iodepth_batch_complete,
and io_depth_low, do_io hangs after reaping the first set of completions
since io_u_queued_complete is called requesting more completions than
td->cur_depth.

Example printing min_evts and td->cur_depth in the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s]
<fio never exits>

Fix this by adjusting min_evts to the current_depth if that is smaller.

Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15
time_based

Modified by Jens to fix the check, and move it into
io_u_queued_complete() instead of at the caller site.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix crash in client with NULL hostname
Castor Fu [Fri, 22 Aug 2014 22:16:44 +0000 (17:16 -0500)]
Fix crash in client with NULL hostname

Fix this by adding an empty hostname.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd string support for buffer_pattern
Jens Axboe [Fri, 22 Aug 2014 19:02:02 +0000 (14:02 -0500)]
Add string support for buffer_pattern

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agooptions: warn if we fail to make sense of buffer_pattern
Jens Axboe [Fri, 22 Aug 2014 15:22:03 +0000 (10:22 -0500)]
options: warn if we fail to make sense of buffer_pattern

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoUpdate io engine version
Jens Axboe [Tue, 19 Aug 2014 20:16:29 +0000 (15:16 -0500)]
Update io engine version

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd unlink hook to ioengine API, gluster ioengine
Castor Fu [Tue, 19 Aug 2014 16:28:53 +0000 (09:28 -0700)]
Add unlink hook to ioengine API, gluster ioengine

fio would just call unlink even with engines that are not using the
operating systems file namespace... This provides a hook to allow
overriding that, with a default handler, and implements it for the
gluster ioengine.

There are others which it'd probably make sense I'm sure.

Huamin Chen looked over my changes to the gluster code earlier...
>I like this unlink idea, it would be great if you can also make unlink optional (if my coding reading is correct). This looks a great pull request candidate to fio. Please ping Axboe after you are done. He is not actively watching pull requests.

>Please also feel free to augment gluster code and pull me for review if necessary.

-castor

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofix re-open failure issue when using gluster ioengine
rootfs [Tue, 19 Aug 2014 18:42:45 +0000 (14:42 -0400)]
fix re-open failure issue when using gluster ioengine

Signed-off-by: rootfs <hchen@redhat.com>
5 years agoUpdate libhdfs engine documention and options
Manish Mandlik [Thu, 14 Aug 2014 17:45:16 +0000 (11:45 -0600)]
Update libhdfs engine documention and options

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd support for HDFS IO engine
Manish Mandlik [Wed, 13 Aug 2014 19:36:52 +0000 (13:36 -0600)]
Add support for HDFS IO engine

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd missing IO engines to the HOWTO
Jens Axboe [Wed, 13 Aug 2014 19:35:37 +0000 (13:35 -0600)]
Add missing IO engines to the HOWTO

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: turn off numberio verification for meta and time_based
Jens Axboe [Thu, 7 Aug 2014 21:27:31 +0000 (15:27 -0600)]
verify: turn off numberio verification for meta and time_based

Will only work for the first iteration, then fail miserably.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agocrc/test: a few more cleanups and code unifications
Jens Axboe [Fri, 25 Jul 2014 13:15:46 +0000 (15:15 +0200)]
crc/test: a few more cleanups and code unifications

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agocrc/test: cleanup and better precision
Jens Axboe [Fri, 25 Jul 2014 08:12:26 +0000 (10:12 +0200)]
crc/test: cleanup and better precision

Make sure we're a bit cache warm at least, and don't allocate, free,
and randomize a buffer for each hash.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAllow reset of offset_increment counter
Jiri Horky [Fri, 25 Jul 2014 07:55:03 +0000 (09:55 +0200)]
Allow reset of offset_increment counter

Reset offset_increment only within a sub-group of jobs, eg when
numjobs=x is used to automatically group jobs together.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoDon't grab stat mutex for final stat output
Jens Axboe [Fri, 25 Jul 2014 07:51:56 +0000 (09:51 +0200)]
Don't grab stat mutex for final stat output

We should not need it, and there's a bug with getting stuck on it
that hasn't been fully debugged/understood yet. So for now, lets
just forget about the mutex for final output.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd test case for previous verify crash
Jens Axboe [Thu, 24 Jul 2014 12:50:36 +0000 (14:50 +0200)]
Add test case for previous verify crash

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agonull: add FIO_FAKEIO flag
Jens Axboe [Wed, 23 Jul 2014 14:19:48 +0000 (16:19 +0200)]
null: add FIO_FAKEIO flag

Tell the backend that null just fakes IO, and use this knowledge
to have verifies automagically work when using null. This allows
testing of a wider range of paths with ioengine=null.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: fix a bug with verify_async
Jens Axboe [Wed, 23 Jul 2014 14:11:43 +0000 (16:11 +0200)]
verify: fix a bug with verify_async

There's a race between marking an io_u as deferred, completing it,
and checking of that flag. It cannot be done reliably for async
verify threads. So change the mechanism to have the verify_async
part pull the io_u completely, so we don't have to check for a flag
in it after we have run ->end_io().

This fixes a bug with verify_async, where fio would crash with
this message:

fio: io_u.c:1315: __get_io_u: Assertion `io_u->flags & IO_U_F_FREE' failed

This race has always existed, but was made considerably worse with
this commit:

commit 2ae0b204743d6b4048c6fffd46c6280a70f2ecd1
Author: Jens Axboe <axboe@kernel.dk>
Date:   Tue May 28 14:16:55 2013 +0200

    Replace list based free/busy/requeue list with FIFO + ring

    Cache friendliness of the list is pretty low. This has
    provably lower overhead.

since we moved from a single list holding the io, to a separate
verify list and io_u queues. The above bug is happening because
the io_u ends up on both the freelist and the pending verify list,
causing mayhem.

Reported-by: scameron@beardog.cce.hp.com
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: ensure that verify interval is smaller or equal to blocksize
Jens Axboe [Wed, 23 Jul 2014 07:47:26 +0000 (09:47 +0200)]
verify: ensure that verify interval is smaller or equal to blocksize

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agostat: move stat mutex grab to thread
Jens Axboe [Mon, 21 Jul 2014 09:32:13 +0000 (11:32 +0200)]
stat: move stat mutex grab to thread

No need to grab it from the parent.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: disable numberio check for multiple block sizes
Jens Axboe [Mon, 21 Jul 2014 09:31:49 +0000 (11:31 +0200)]
verify: disable numberio check for multiple block sizes

Yet another corner case that doesn't work...

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.1.11 fio-2.1.11
Jens Axboe [Wed, 16 Jul 2014 08:22:43 +0000 (10:22 +0200)]
Fio 2.1.11

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoblktrace: various bug fixes and improvements
Jens Axboe [Mon, 14 Jul 2014 13:07:11 +0000 (15:07 +0200)]
blktrace: various bug fixes and improvements

- Use the normal debug logging for notify events, as they are expected
  to happen.

- Correct a bug where a workload is considered mixed, even if it is
  only reads (note: just cosmetical, no writes were issued by fio).

- Handle trim more appropriately - fio supports it as a separate
  'data direction', so handle it as such.

- Fix a bug in the delay accounting, causing blktrace to not replay
  correctly. This could cause either too fast replays, or replays
  that would delay way too much.

- Consider TRIMs as writes, as far as --readonly being set is
  concerned.

- Don't consider < 100 usec delays noise, fio can do this quite
  accurately.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd some basic debug output for log compression
Jens Axboe [Wed, 9 Jul 2014 21:01:03 +0000 (23:01 +0200)]
Add some basic debug output for log compression

Guarded by --debug=compress, like the other debug measures.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix two error paths
Andreas Gruenbacher [Wed, 9 Jul 2014 17:37:33 +0000 (19:37 +0200)]
Fix two error paths

In generic_open_file(), when an error is detected, return 1 to indicate
the failure.

In get_file_sizes(), when an error other than ENOENT is detected, fail
immediately and don't clear the error.

5 years agoFix cases where td->terminate is set, but terminate clock not marked
Jens Axboe [Wed, 9 Jul 2014 09:24:12 +0000 (11:24 +0200)]
Fix cases where td->terminate is set, but terminate clock not marked

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd thread number to log filename
Jens Axboe [Wed, 9 Jul 2014 08:31:34 +0000 (10:31 +0200)]
Add thread number to log filename

Since commit aee2ab6775d9, we open the log files for writing and
truncate the output. This breaks when you have multiple jobs with
the same name, as will happen with numjobs=x. Instead of reverting
this part of the commit, lets keep the truncate open and instead
store log files separately. This makes more sense anyway, since
otherwise the logs are all the entries from job X, then all the
entries from job Y, etc. It's easy enough to collate the logs,
should anyone want to do that.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: run compression work at slightly elevated nice level
Jens Axboe [Wed, 9 Jul 2014 07:52:20 +0000 (09:52 +0200)]
iolog: run compression work at slightly elevated nice level

Add support for work items to contain a priority, and use this
directly with nice. Run compression at nice 1, to make it slightly
less important than the actual IO jobs.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAllow threads 60 seconds to exit before being forceful
Jens Axboe [Wed, 9 Jul 2014 06:46:12 +0000 (08:46 +0200)]
Allow threads 60 seconds to exit before being forceful

Right now the fio status thread can sit forever waiting for jobs
to exit, when ctrl-c (or another signal) is sent. Be a bit more
brutal and force quit jobs if they haven't exited on their own
in 60 seconds. That should be long enough to ensure that they
are stuck in some way.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: fix link without zlib
Jens Axboe [Tue, 8 Jul 2014 12:24:54 +0000 (14:24 +0200)]
iolog: fix link without zlib

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: fix compile error for non-zlib
Jens Axboe [Tue, 8 Jul 2014 12:18:13 +0000 (14:18 +0200)]
iolog: fix compile error for non-zlib

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: propagate errors to caller
Jens Axboe [Tue, 8 Jul 2014 12:14:23 +0000 (14:14 +0200)]
iolog: propagate errors to caller

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agomutex: move pthread_cond_signal() outside of lock
Jens Axboe [Tue, 8 Jul 2014 07:46:37 +0000 (09:46 +0200)]
mutex: move pthread_cond_signal() outside of lock

Generally best practice to drop the lock before waking, if
possible.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agotp: move pthread_cond_signal() outside of lock
Jens Axboe [Tue, 8 Jul 2014 07:45:55 +0000 (09:45 +0200)]
tp: move pthread_cond_signal() outside of lock

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd some new code comments on the log compress/decompress
Jens Axboe [Tue, 8 Jul 2014 07:44:51 +0000 (09:44 +0200)]
Add some new code comments on the log compress/decompress

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: fix bug when decompressing chunks with different sequence numbers
Jens Axboe [Fri, 4 Jul 2014 03:57:29 +0000 (21:57 -0600)]
iolog: fix bug when decompressing chunks with different sequence numbers

We need to re-start on inflate, like we do on deflate. Otherwise
we get stream end when the first chunk is done. Keep track of
this in iolog_file_inflate(), since we know how much we need to
decompress.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoflist: add flist_first_entry()
Jens Axboe [Fri, 4 Jul 2014 03:19:57 +0000 (21:19 -0600)]
flist: add flist_first_entry()

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: use a define instead of a raw bit mask
Jens Axboe [Fri, 4 Jul 2014 00:01:11 +0000 (18:01 -0600)]
iolog: use a define instead of a raw bit mask

Add a comment too, while in there.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: get rid of ic->nofree
Jens Axboe [Thu, 3 Jul 2014 23:58:49 +0000 (17:58 -0600)]
iolog: get rid of ic->nofree

Push the free out of flush_chunk(), then the caller just does it
instead. Unifies the log write vs compress as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: get rid of one section of CONFIG_ZLIB ifdef
Jens Axboe [Thu, 3 Jul 2014 23:56:55 +0000 (17:56 -0600)]
iolog: get rid of one section of CONFIG_ZLIB ifdef

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoconfigure: add a note to install zlib-devel, if not there
Jens Axboe [Thu, 3 Jul 2014 22:20:27 +0000 (16:20 -0600)]
configure: add a note to install zlib-devel, if not there

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoinit: fix dead check for !td
Jens Axboe [Thu, 3 Jul 2014 21:33:24 +0000 (15:33 -0600)]
init: fix dead check for !td

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: do stat() after fopen()
Jens Axboe [Thu, 3 Jul 2014 21:32:07 +0000 (15:32 -0600)]
iolog: do stat() after fopen()

Avoid static checker complaining about a potential race here,
ordering doesn't matter to fio.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agorbd: fix leak in error path
Jens Axboe [Thu, 3 Jul 2014 21:31:32 +0000 (15:31 -0600)]
rbd: fix leak in error path

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoglusterfs: fix leak in error path
Jens Axboe [Thu, 3 Jul 2014 21:30:37 +0000 (15:30 -0600)]
glusterfs: fix leak in error path

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: remove dead code that stored a gzip header
Jens Axboe [Thu, 3 Jul 2014 20:17:45 +0000 (14:17 -0600)]
iolog: remove dead code that stored a gzip header

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoconfigure: remove typo for zlib
Jens Axboe [Thu, 3 Jul 2014 20:17:02 +0000 (14:17 -0600)]
configure: remove typo for zlib

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMove tp.[ch] to lib/
Jens Axboe [Thu, 3 Jul 2014 20:16:25 +0000 (14:16 -0600)]
Move tp.[ch] to lib/

It's just helper code, not fio core.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd support for storing compressed logs
Jens Axboe [Thu, 3 Jul 2014 19:42:28 +0000 (13:42 -0600)]
Add support for storing compressed logs

If log_store_compressed is set, fio won't decompress for storing
them in a file. Instead, you can use --inflate-log=logfile.fz to
output the contents to stdout.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofio.1 - escape the escape character so it shows up
Jeff Moyer [Thu, 3 Jul 2014 18:20:28 +0000 (14:20 -0400)]
fio.1 - escape the escape character so it shows up

Hi, Jens,

In the filename= text, the escape character gets eaten.  This fixes it
for me.

Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoBail out if glfs_creat failed
Tiziano Müller [Thu, 3 Jul 2014 16:22:29 +0000 (18:22 +0200)]
Bail out if glfs_creat failed

Otherwise we will get segfaults from gfapi when it tries to access the
invalid fd.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agotp: remove debug start/shutdown printfs
Jens Axboe [Thu, 3 Jul 2014 15:41:58 +0000 (09:41 -0600)]
tp: remove debug start/shutdown printfs

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoPass O_DIRECT/O_SYNC to glfs_open if requested.
Tiziano Müller [Thu, 3 Jul 2014 14:48:39 +0000 (16:48 +0200)]
Pass O_DIRECT/O_SYNC to glfs_open if requested.

GlusterFS supports O_DIRECT and O_SYNC.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd support for runtime log compression
Jens Axboe [Thu, 3 Jul 2014 15:10:39 +0000 (09:10 -0600)]
Add support for runtime log compression

If log_compression=64M is specified, fio will "bite" off chunks of
the IO logs and runtime compress them with zlib. This can greatly
reduce the amount of memory required to do iops/bw/lat logging of
a run, at the cost of using some background CPU for the compression.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agolog: fix use-after-free
Jens Axboe [Wed, 2 Jul 2014 19:36:34 +0000 (13:36 -0600)]
log: fix use-after-free

Introduced in commit cb7e0ace739c.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agocconv: convert ->log_offset on the wire
Jens Axboe [Wed, 2 Jul 2014 15:08:37 +0000 (09:08 -0600)]
cconv: convert ->log_offset on the wire

Forgotten in commit ae58885234

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years ago--latency-log is now gone
Jens Axboe [Tue, 1 Jul 2014 22:38:02 +0000 (16:38 -0600)]
--latency-log is now gone

Use per-job options for this.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoCleanup logging
Jens Axboe [Tue, 1 Jul 2014 22:29:12 +0000 (16:29 -0600)]
Cleanup logging

If the log knows the filename it is supposed to log to, then we
don't need to have part of that on the init side and the other
part on the flush side. So clean it up so that the caller of
setup_log() provides everything we need. This gets rid of
various ways of ending the log, and opens us up to doing
log flushing mid-run.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoclient: fix missing init of 'i'
Jens Axboe [Tue, 1 Jul 2014 22:07:59 +0000 (16:07 -0600)]
client: fix missing init of 'i'

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoclient: fix missing pdu->log_offset endianness conversion
Jens Axboe [Tue, 1 Jul 2014 14:47:11 +0000 (08:47 -0600)]
client: fix missing pdu->log_offset endianness conversion

Make the iolog sample number uint64_t as well, so we have
the same size across 32 and 64-bit archs.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd a typecheck for the endianness conversions
Jens Axboe [Tue, 1 Jul 2014 14:46:44 +0000 (08:46 -0600)]
Add a typecheck for the endianness conversions

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd option for including byte offset for each log entry
Jens Axboe [Tue, 1 Jul 2014 02:59:03 +0000 (20:59 -0600)]
Add option for including byte offset for each log entry

When iops/latency/bw logs are used in the job, we can note the offset
at which the specific data entry took place.

Based on an initial patch from Abutalib Aghayev <agayev@gmail.com>.
Extended by me to:

- Not do this by default, as not to add 8 bytes to each log entry
  stored.

- Add support for the client/server protocol.

- Add documentation.

We should bump the server protocol rev again, but that was just done
in the previous commit, so...

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoserver: update protocol version to 35
Jens Axboe [Mon, 30 Jun 2014 21:31:59 +0000 (15:31 -0600)]
server: update protocol version to 35

We pack structs now, so not compatible with previous version.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoserver: pack over-the-wire structs
Jens Axboe [Mon, 30 Jun 2014 14:51:33 +0000 (08:51 -0600)]
server: pack over-the-wire structs

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix typo in io_u.c::get_next_file()
Andrey Kuzmin [Sat, 28 Jun 2014 02:21:22 +0000 (20:21 -0600)]
Fix typo in io_u.c::get_next_file()

We should check ops->get_next_file() is TD_F_PROFILE_OPS
is set, not if it's unset.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoclient: fix ETA run_str
Jens Axboe [Fri, 27 Jun 2014 21:30:06 +0000 (15:30 -0600)]
client: fix ETA run_str

We never touched it before, which meant the client would just
display an empty string. Make a valiant effort to allocate and
output the run_str.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoeta: only pass real size of jobs_eta
Jens Axboe [Fri, 27 Jun 2014 21:15:23 +0000 (15:15 -0600)]
eta: only pass real size of jobs_eta

We might send this over the wire, so need sending more than we
actually used.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix struct jobs_eta packing on 32 vs 64-bit archs
Jens Axboe [Fri, 27 Jun 2014 21:01:06 +0000 (15:01 -0600)]
Fix struct jobs_eta packing on 32 vs 64-bit archs

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoVerify: Tighten header length check in verify_header()
Andreas Gruenbacher [Thu, 26 Jun 2014 17:25:30 +0000 (19:25 +0200)]
Verify: Tighten header length check in verify_header()

Fix-up to commit 5964842c: Tighten the header length check as well, don't only
change the message.

5 years agoAdd configure flags for gfapi and rbd
Tiziano Müller [Thu, 26 Jun 2014 12:20:20 +0000 (14:20 +0200)]
Add configure flags for gfapi and rbd

Add --disable-{gfapi,rbd} to disable gfapi/rbd support even if available.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoVerify: Fix and improve verify_header()
Andreas Gruenbacher [Tue, 24 Jun 2014 20:13:50 +0000 (22:13 +0200)]
Verify: Fix and improve verify_header()

Report errors directly from verify_header(); requires passing in the header
number and length.  Correctly report the file name, offset, and expected header
size in all error messages.

5 years agoupdate_condensed_str(): Fix for empty input strings
Andreas Gruenbacher [Tue, 24 Jun 2014 11:09:07 +0000 (13:09 +0200)]
update_condensed_str(): Fix for empty input strings

Empty input strings resulted in unterminated result strings.

5 years agoCast input argument for json_object_add_value_int to long long
Puthikorn Voravootivat [Tue, 24 Jun 2014 22:59:53 +0000 (15:59 -0700)]
Cast input argument for json_object_add_value_int to long long

The latency related (and some other) variables type is uint32_t
but the JSON integer type is long long. So we need to cast
the numbers to long long type to make JSON output correctly.

Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoGet the crctest results properly aligned
Jens Axboe [Tue, 24 Jun 2014 16:12:47 +0000 (10:12 -0600)]
Get the crctest results properly aligned

Before:

md5: 534.03 MB/sec
crc64: 392.31 MB/sec
crc32: 352.36 MB/sec
crc32c: 8225.43 MB/sec
crc16: 384.46 MB/sec
crc7: 386.79 MB/sec
sha1: 381.15 MB/sec
sha256: 186.47 MB/sec
sha512: 303.18 MB/sec
xxhash: 6329.90 MB/sec

After:

md5:   548.59 MB/sec
crc64:   381.45 MB/sec
crc32:   347.89 MB/sec
crc32c:  8577.36 MB/sec
crc16:   396.56 MB/sec
crc7:   396.72 MB/sec
sha1:   385.87 MB/sec
sha256:   206.60 MB/sec
sha512:   314.62 MB/sec
xxhash:  6293.64 MB/sec

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoEnsure that job name is set before being used
Jens Axboe [Tue, 24 Jun 2014 15:19:30 +0000 (09:19 -0600)]
Ensure that job name is set before being used

Andreas reports:

When passing the --readonly option, with the following write job (and
probably with any write job), I get an error message with a NULL job
name. Job description used:

[write]
thread
bs=64k
direct=1
ioengine=sync
size=128m
filename=test.tmp
rw=write
do_verify=0

Reported-by: Andreas Grünbacher <andreas.gruenbacher@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd a get_jobs_eta() to return jobs eta information
Jens Axboe [Tue, 24 Jun 2014 14:40:21 +0000 (08:40 -0600)]
Add a get_jobs_eta() to return jobs eta information

We duplicate this code in eta.c and server.c, consolidate it.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoEnsure we have enough room for the ETA runstr
Jens Axboe [Tue, 24 Jun 2014 01:07:12 +0000 (19:07 -0600)]
Ensure we have enough room for the ETA runstr

The calculation of 1:4 is wrong, it's 1:5. And we missed the similar
change in server.c.

Reported-by: Andreas Grünbacher <andreas.gruenbacher@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix two minor typos
Andreas Gruenbacher [Mon, 23 Jun 2014 21:17:08 +0000 (23:17 +0200)]
Fix two minor typos

5 years agoREADME: update to include official mirrors for the git repo
Jens Axboe [Mon, 23 Jun 2014 13:52:48 +0000 (07:52 -0600)]
README: update to include official mirrors for the git repo

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd support for compiling for ESX
Jens Axboe [Wed, 18 Jun 2014 22:30:09 +0000 (15:30 -0700)]
Add support for compiling for ESX

With contributions from Ryan Haynes <rhaynes@fusionio.com>

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix build from cmake
George Dowding [Wed, 18 Jun 2014 16:36:53 +0000 (09:36 -0700)]
Fix build from cmake

George reports:

I spent a few hours on this. Might save someone else some time.

I'm working in a larger project that uses cmake for its builds. cmake
supports adding external projects using the ExternalProject_Add
function.

But, there is some issue in the fio Makefile that causes a problem
when run from cmake. I could never reproduce it from the command line.

I was able to get the build to happen by changing the implicit rule
.c.o to use pattern notation. Not sure if this will work in all cases
or has other side effects.

Without this change, I was getting this error.

make[4]: *** No rule to make target `gettime.o', needed by `fio'.  Stop.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFixup run_str[] condensing with client/server
Jens Axboe [Tue, 17 Jun 2014 15:22:34 +0000 (09:22 -0600)]
Fixup run_str[] condensing with client/server

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd string condensing for the ETA output
Jens Axboe [Mon, 16 Jun 2014 20:42:05 +0000 (14:42 -0600)]
Add string condensing for the ETA output

If you run a lot of jobs, the ETA output string will become way too
long. Add some condensing so that instead of displaying:

Jobs: 80 (f=80): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [10.2% done] [2110MB/0KB/0KB /s] [540K/0/0 iops] [eta 53m:55s]

we just display

Jobs: 80 (f=80): [r(80)] [10.2% done] [2110MB/0KB/0KB /s] [540K/0/0 iops] [eta 53m:55s]

instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofio: Eliminate compilation warning in ppc
Sumanth K [Mon, 16 Jun 2014 05:38:16 +0000 (11:08 +0530)]
fio: Eliminate compilation warning in ppc

Eliminate compilation warning : 'atb_clocktest' defined but
not used. Function is declared as static in header file. So compiler will
know if they used or not. Just eliminate the warning by #if 0 /unless we do
use it.

Signed-off-by: Sumanth K <sumantk2@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>