fio.git
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Sun, 6 Apr 2014 15:57:59 +0000 (09:57 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agoFix crash in setup_files() if no files are defined
Jens Axboe [Sun, 6 Apr 2014 15:56:10 +0000 (09:56 -0600)]
Fix crash in setup_files() if no files are defined

Commit 002fe73409d1e moved the division by o->nr_files outside
of the file loop, so for error cases we could end up with
doing the division without having files defined. Crash ensues.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMakefile: add doc target to generate PDFs from the man pages
Jens Axboe [Fri, 4 Apr 2014 14:28:17 +0000 (08:28 -0600)]
Makefile: add doc target to generate PDFs from the man pages

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix JSON_INTEGER overflow on Windows by changing datatype to 'long long'
Bruce Cran [Thu, 3 Apr 2014 18:17:00 +0000 (12:17 -0600)]
Fix JSON_INTEGER overflow on Windows by changing datatype to 'long long'

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoparse: fix crash with empty FIO_OPT_STR_STORE variables
Jens Axboe [Thu, 3 Apr 2014 14:18:07 +0000 (08:18 -0600)]
parse: fix crash with empty FIO_OPT_STR_STORE variables

Sébastien reports:

A crash on solaris & windows (at least) when using fio-2.1.6.1 :

A simple profile that use a env variable : directory=${DIRECTORY}

I'm starting fio without defining DIRECTORY and I'm getting a crash:

The stack on solaris is :

core 'core' of 4574:    ./fio ../fio_env.win
 0000000100035348 set_name_idx (ffffffff7fffe970100231950, 0, 16c4e8, 0, 1002319b0) + cc
 0000000100021a70 add_file (ffffffff7b400000ffffffff7fffee60, 0, 0, 100071c30ffffffff7fffe970) + 38
 000000010000daf4 add_job.part.4 (7, 100236050, 0, ffffffff7ffff268, 1, 0) + 11f0
 000000010000f444 parse_jobs_ini (0, 0, 0, 1, 1002360501002276c0) + 77c
 0000000100010db0 parse_options (5118, ffffffff7ffff968, 1, 0, 0, 10021eef0) + 304
 000000010006ff68 main (2, ffffffff7ffff968ffffffff7ffff980100228338100000000, 2800) + 20
 000000010000952c _start (0, 0, 0, 0, 0, 0) + 7c

Same thing happens with filename and other store options. Fix this
by checking in the parser whether the string is empty or not.

Reported-by: Sébastien Bouchex Bellomié <sbouchex@infovista.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoUse setvbuf() for log writing
Jens Axboe [Thu, 3 Apr 2014 03:47:27 +0000 (21:47 -0600)]
Use setvbuf() for log writing

Check if we have it - if we do, add a 1MB buffer for the file
output. This further speeds up the log writing.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoiolog: invert log_mask bits
Jens Axboe [Thu, 3 Apr 2014 02:07:28 +0000 (20:07 -0600)]
iolog: invert log_mask bits

Get rid of the all-log-mask.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agocrc: add test.h header to test.c
Jens Axboe [Thu, 3 Apr 2014 02:03:32 +0000 (20:03 -0600)]
crc: add test.h header to test.c

This way the static analyzers know that there's a reason
for the exported function.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoCleanup the parallellized log writing
Jens Axboe [Thu, 3 Apr 2014 02:01:27 +0000 (20:01 -0600)]
Cleanup the parallellized log writing

And move it all to iolog.c, this means we can un-export
some of the log functions, too.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoiolog: don't serialize the writing of all logs
Jens Axboe [Wed, 2 Apr 2014 21:46:58 +0000 (15:46 -0600)]
iolog: don't serialize the writing of all logs

Serialize based on filename, so that independent logs can be
written at the same time. If the logs are big, this can take
a while to flush to disk. It's silly to have all jobs waiting
on each other, when they could write their own logs independently.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd 'f' (Finishing) flag to status output
Jens Axboe [Wed, 2 Apr 2014 21:46:22 +0000 (15:46 -0600)]
Add 'f' (Finishing) flag to status output

Use this for finalizing logs when a thread is done, since
that can take a long time.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agomutex: add open init function and down_trylock() variant
Jens Axboe [Wed, 2 Apr 2014 21:45:25 +0000 (15:45 -0600)]
mutex: add open init function and down_trylock() variant

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoDon't crash when using filehash lock before init
Jens Axboe [Wed, 2 Apr 2014 18:25:16 +0000 (12:25 -0600)]
Don't crash when using filehash lock before init

We might be parsing and adding files (which locks the hash
lock) before we have setup threads. For that case, don't
bother locking, we're serialized at that point.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd missing crc/test.h file
Jens Axboe [Wed, 2 Apr 2014 17:41:29 +0000 (11:41 -0600)]
Add missing crc/test.h file

Commit 10aa136b forgot to add this file, unfortunately.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix propagation of error value on failure to connect to fio server
Jens Axboe [Wed, 2 Apr 2014 14:28:49 +0000 (08:28 -0600)]
Fix propagation of error value on failure to connect to fio server

Fio does detect the error and exits, but it doesn't exit with
a non-zero exit value.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMerge branch 'erwan/clang' of https://github.com/enovance/fio
Jens Axboe [Wed, 2 Apr 2014 14:20:18 +0000 (08:20 -0600)]
Merge branch 'erwan/clang' of https://github.com/enovance/fio

6 years agoMerge branch 'erwan/nobasename' of https://github.com/enovance/fio
Jens Axboe [Wed, 2 Apr 2014 13:42:33 +0000 (07:42 -0600)]
Merge branch 'erwan/nobasename' of https://github.com/enovance/fio

6 years agoio_u: Using initialized local variable
Erwan Velu [Wed, 2 Apr 2014 08:51:16 +0000 (10:51 +0200)]
io_u: Using initialized local variable

ddir was feed with io_u->ddir at init time but didn't got used before
reassign it to the same value or another one.

That was making this init useless.

This patch simplify this portion of code by using the initialized
variable.

6 years agostat: fixing bw_agg reporting
Erwan Velu [Wed, 2 Apr 2014 08:36:40 +0000 (10:36 +0200)]
stat: fixing bw_agg reporting

It was wrongly set to the mean value instead of the agg one.

6 years agoiolog: Keep full path for logs files
Erwan Velu [Wed, 2 Apr 2014 08:00:56 +0000 (10:00 +0200)]
iolog: Keep full path for logs files

The default behavior of fio was to basename() the log filename
preventing relocating it into another directory.

This patch simply removes the basename() that doesn't provides any real
feature but only limit the usage.

Patch tested with genfio with success while relocating to /tmp.

6 years agoCleanup symbols that should be static
Jens Axboe [Wed, 2 Apr 2014 03:10:36 +0000 (21:10 -0600)]
Cleanup symbols that should be static

Run analysis on symbols not used outside of their current file,
turn them into statics.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFile pre-read lseek error check
Jens Axboe [Tue, 1 Apr 2014 23:01:30 +0000 (17:01 -0600)]
File pre-read lseek error check

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd alloc_new_file()
Jens Axboe [Tue, 1 Apr 2014 22:37:03 +0000 (16:37 -0600)]
Add alloc_new_file()

Hopefully avoid a bug like commit 4db205dc fixed by abstracting
out the allocate-and-initialize part of fio_file allocation.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoEnsure we inherit global latency/bw log settings
Jens Axboe [Tue, 1 Apr 2014 19:34:49 +0000 (13:34 -0600)]
Ensure we inherit global latency/bw log settings

Note that you cannot use a filename with these, it's just
an on/off switch. For tighter control of where the log
is written (and its name), use the job option instead of
the command line on/off switch.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoProperly protect already-allocated file list
Jens Axboe [Tue, 1 Apr 2014 18:28:47 +0000 (12:28 -0600)]
Properly protect already-allocated file list

We need proper locking around it for thread based setups.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoProperly initialize shadow_fd to -1
Jens Axboe [Tue, 1 Apr 2014 18:28:15 +0000 (12:28 -0600)]
Properly initialize shadow_fd to -1

We forget that in one case, causing all sorts of issues when
we end up closing this bogus (STDIN) descriptor.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix bug in freeing of duplicated files
Andrey Kuzmin [Tue, 1 Apr 2014 18:27:04 +0000 (12:27 -0600)]
Fix bug in freeing of duplicated files

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFio 2.1.7 fio-2.1.7
Jens Axboe [Mon, 31 Mar 2014 15:05:54 +0000 (09:05 -0600)]
Fio 2.1.7

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Fri, 28 Mar 2014 14:09:35 +0000 (08:09 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agoFix wrong time conversion on --status-interval= command line option
Jens Axboe [Fri, 28 Mar 2014 14:09:10 +0000 (08:09 -0600)]
Fix wrong time conversion on --status-interval= command line option

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoserver: unlink sock file if interrupted early
Jens Axboe [Tue, 25 Mar 2014 22:06:17 +0000 (15:06 -0700)]
server: unlink sock file if interrupted early

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoserver: dump where we attempted to bind in case of failure
Jens Axboe [Tue, 25 Mar 2014 17:50:55 +0000 (10:50 -0700)]
server: dump where we attempted to bind in case of failure

Helps locate what the issue is, instead of having to use
strace and similar tools.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoserver: correct comment typo on port being after ':'
Jens Axboe [Tue, 25 Mar 2014 17:37:56 +0000 (10:37 -0700)]
server: correct comment typo on port being after ':'

We use commas in fio, since that works equally well on IPv4
and IPv6.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoPass -Wstrict-prototypes -Wold-style-definition, whitespace
Castor Fu [Thu, 20 Mar 2014 17:59:29 +0000 (10:59 -0700)]
Pass -Wstrict-prototypes -Wold-style-definition, whitespace

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoDon't re-create/unlink write file for append workload
Jens Axboe [Thu, 20 Mar 2014 15:33:13 +0000 (09:33 -0600)]
Don't re-create/unlink write file for append workload

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoUnlink after file close
Jens Axboe [Thu, 20 Mar 2014 14:28:11 +0000 (08:28 -0600)]
Unlink after file close

Sébastien reports:

I'm having an issue on windows : unlink=1 is not working (temp file are
still there) and it's working fine on unix

Looking at the code in close_and_free_files function :

[...]
if (td->o.unlink && f->filetype == FIO_TYPE_FILE) {
dprint(FD_FILE, "free unlink %s\n", f->file_name);
unlink(f->file_name);
}
[...]

Unlink() fails because the file is still open : On Solaris, truss shows
the following :

[...]
write(3, "\0\0\0\0\0 ;80\0\0\0\0\0".., 32768)   = 32768
unlink("/data/fio/random_rw.0.0")               = 0
close(3)                                        = 0
[...]

So unlink is called first.

I would put this unlinking phase AFTER the remove_file_hash call.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoPrevent file_append=1 with IO engine that can't extend files
Jens Axboe [Tue, 18 Mar 2014 19:30:32 +0000 (13:30 -0600)]
Prevent file_append=1 with IO engine that can't extend files

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoUpdate file_append documentation
Jens Axboe [Mon, 17 Mar 2014 18:55:08 +0000 (12:55 -0600)]
Update file_append documentation

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd support for file appends
Jens Axboe [Mon, 17 Mar 2014 18:51:09 +0000 (12:51 -0600)]
Add support for file appends

Add option file_append (bool). If set, fio will append to a file
instead of operate within the size of it. This is similar to
setting offset= to the size of each file.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoUpdate documentation on zero_buffers / scramble_buffers
Jens Axboe [Sat, 15 Mar 2014 01:41:07 +0000 (19:41 -0600)]
Update documentation on zero_buffers / scramble_buffers

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoDon't scramble buffers if compression is enabled
Jens Axboe [Fri, 14 Mar 2014 18:24:28 +0000 (12:24 -0600)]
Don't scramble buffers if compression is enabled

For the other buffer settings (like patterns, or whether always
to refill or not), we don't do the slight header scramble if
they are enabled. Do they same for compression settings, otherwise
we can skew the compression results.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoBump length of description field to 256 chars
Jens Axboe [Fri, 14 Mar 2014 14:41:39 +0000 (08:41 -0600)]
Bump length of description field to 256 chars

Ensure that string always end up \0 terminated, too.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoDisable status file updates on error
Jens Axboe [Thu, 13 Mar 2014 14:49:28 +0000 (08:49 -0600)]
Disable status file updates on error

If fio doesn't have permission to unlink() the status file,
it'll continually dump status updates. If this happens, stop
doing status updates and log an error indicating why.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agostat: fixup fio-dump-status file location
Jens Axboe [Wed, 12 Mar 2014 19:40:09 +0000 (13:40 -0600)]
stat: fixup fio-dump-status file location

Don't use /tmp in the actual file name. And kill unused Windows
os_get_tmpdir().

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Wed, 12 Mar 2014 16:50:29 +0000 (10:50 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agoUpdate documentation for filename/directory
Jens Axboe [Wed, 12 Mar 2014 16:49:36 +0000 (10:49 -0600)]
Update documentation for filename/directory

Sync the HOWTO and man page, and add a reference from 'directory'
to 'filename' on how to escape certain characters.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd exceeds_number_ios() helper
Jens Axboe [Wed, 12 Mar 2014 02:16:38 +0000 (20:16 -0600)]
Add exceeds_number_ios() helper

Commit 26251d8d open-coded the logic in two places, collapse
that into one function.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoTry a little harder to honor number_ios more accurately
Jens Axboe [Tue, 11 Mar 2014 16:21:18 +0000 (10:21 -0600)]
Try a little harder to honor number_ios more accurately

Previously, fio checked this on completion, which means we could
almost spill over by the iodepth setting. Check this before
issue instead, taking the in flight IOs into account as well.

Reported-by: Robert Elliott <Elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix pointer alignment bug causing SIGBUS on Sparc64
Sébastien Bouchex Bellomié [Tue, 11 Mar 2014 14:54:27 +0000 (08:54 -0600)]
Fix pointer alignment bug causing SIGBUS on Sparc64

OK, the issue was related to the alignment issue between the
thread_options structure and its "packed" version : The lexxx_to_cpu
macros are incorrectly casting the pointers and that causes an bus
error.  I have fixed the issue by avoid casting the pointers and calling
directly the "swap" method.

I have also included a fixed on the thread_options structure regarding
the ddir_seq_add variable : Its length was incorrect.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoDetect Windows operating system in ./configure
Stefan Hajnoczi [Thu, 6 Mar 2014 17:41:59 +0000 (18:41 +0100)]
Detect Windows operating system in ./configure

We should not rely on `uname -s` if we ever want cross-compiling to
work, so detect _WIN32.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoHonor ./configure --cc= option
Stefan Hajnoczi [Thu, 6 Mar 2014 17:41:58 +0000 (18:41 +0100)]
Honor ./configure --cc= option

The cc variable is assigned before parsing command-line options.  This
effectively ignores the ./configure --cc= option.  Move the cc variable
assigned below option parsing.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFio 2.1.6.1 fio-2.1.6.1
Jens Axboe [Thu, 6 Mar 2014 16:18:00 +0000 (09:18 -0700)]
Fio 2.1.6.1

Lets tag a minor 2.1.6 update that has the RHEL/CentOS5
compile issue fixed.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix compile on CentOS/RHEL5
Jens Axboe [Wed, 5 Mar 2014 22:49:54 +0000 (15:49 -0700)]
Fix compile on CentOS/RHEL5

They don't have CPU_COUNT(), so add a helper for that.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix typo in fio_cpu_count()
Jens Axboe [Tue, 4 Mar 2014 16:02:10 +0000 (09:02 -0700)]
Fix typo in fio_cpu_count()

Reported-by: Paul Howarth <paul@city-fan.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFio 2.1.6 fio-2.1.6
Jens Axboe [Mon, 3 Mar 2014 23:49:00 +0000 (15:49 -0800)]
Fio 2.1.6

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix verify_only option not working properly
Puthikorn Voravootivat [Mon, 3 Mar 2014 23:33:32 +0000 (15:33 -0800)]
Fix verify_only option not working properly

In commit c4b6117, we changed the place to call log_io_piece but
the new place won't be reached when using verify_only option so
that when we run with the verify_only option, fio won't verify
anything at all.

This patch fix that by also call the log_io_piece in the do_dry_run
function for the verify_only code path.

Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Mon, 3 Mar 2014 23:44:57 +0000 (15:44 -0800)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agosolaris: fixup new cpuset functions
Jens Axboe [Mon, 3 Mar 2014 20:55:44 +0000 (13:55 -0700)]
solaris: fixup new cpuset functions

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoSilence 'io_u might be used uninitialized' warning
Jens Axboe [Mon, 3 Mar 2014 20:55:32 +0000 (13:55 -0700)]
Silence 'io_u might be used uninitialized' warning

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoserver: fix warnings on some platforms on incompatible pointer type
Jens Axboe [Mon, 3 Mar 2014 20:53:39 +0000 (13:53 -0700)]
server: fix warnings on some platforms on incompatible pointer type

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoWindows: add install directory to the Path environment variable
Bruce Cran [Sun, 2 Mar 2014 00:05:03 +0000 (17:05 -0700)]
Windows: add install directory to the Path environment variable

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd runstate swap helpers
Jens Axboe [Sat, 1 Mar 2014 15:24:03 +0000 (08:24 -0700)]
Add runstate swap helpers

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoUpdate cpus_allowed_policy=split documentation
Jens Axboe [Sat, 1 Mar 2014 00:43:57 +0000 (16:43 -0800)]
Update cpus_allowed_policy=split documentation

Instead of erroring out on fewer CPUs than threads/processes,
fio now does roundrobin over the set.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agocpus_allowed_policy fixups
Jens Axboe [Fri, 28 Feb 2014 23:08:25 +0000 (15:08 -0800)]
cpus_allowed_policy fixups

- Make 'split' roundrobin the CPUs, if we have more threads
  than CPUs in the set.

- Properly use a CPU index, don't assume a sequential set of
  CPUs.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: fix last block never being touched by random offsets
Peter Oberparleiter [Fri, 28 Feb 2014 17:09:13 +0000 (09:09 -0800)]
fio: fix last block never being touched by random offsets

Fix the available range for random offsets which never touched the last block.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix off-by-one bug in CPU split mask handling
Jens Axboe [Fri, 28 Feb 2014 16:35:40 +0000 (08:35 -0800)]
Fix off-by-one bug in CPU split mask handling

The thread number is not zero based, it starts from 1.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agowindows: fix fio_cpu_count() definition
Jens Axboe [Fri, 28 Feb 2014 00:24:15 +0000 (16:24 -0800)]
windows: fix fio_cpu_count() definition

Don't take a 'cpu' argument, that's a leftover when this was
(temporarily) fio_cpu_isset().

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agowindows: add proper header for hweight64()
Jens Axboe [Fri, 28 Feb 2014 00:18:30 +0000 (16:18 -0800)]
windows: add proper header for hweight64()

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoUpdate IO engine version
Jens Axboe [Fri, 28 Feb 2014 00:10:43 +0000 (16:10 -0800)]
Update IO engine version

Commit 225ba9e3 changed the layout of struct io_u, so
update the engine version to indicate that external
engines need to be recompiled.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd support for cpus_allowed_policy
Jens Axboe [Thu, 27 Feb 2014 23:52:02 +0000 (15:52 -0800)]
Add support for cpus_allowed_policy

Two policies are supported:

shared cpumask is shared between all threads. This is the original
behavior.

split cpumask is split between threads, each thread gets a unique
CPU.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoBranch and cache miss speedups
Jens Axboe [Wed, 26 Feb 2014 22:31:15 +0000 (14:31 -0800)]
Branch and cache miss speedups

Just some low hanging fruit.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoblktrace: load improvements
Jens Axboe [Wed, 26 Feb 2014 21:42:13 +0000 (13:42 -0800)]
blktrace: load improvements

- Set the state explicitly to SETTING_UP when loading the
  traces. This could take a while, so tell the status thread
  that we are setting up so it knows how to display us
  correctly.

- Fix a bug with multiple devices where we didn't cache the
  major/minor properly. This made blktrace spend tons of CPU
  cycles looking up /sys.

- Handle multiple devices correctly when adding files.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFixup ->open_files if not given
Jens Axboe [Tue, 25 Feb 2014 22:01:26 +0000 (14:01 -0800)]
Fixup ->open_files if not given

For engines/users that add files with add_file() after the
job has been initialized, ensure that we bump the allowed
number of open files if it wasn't explicitly given as
an option.

This unbreaks the net engine after commit 002fe73409d1e made
fio more strict wrt open files.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agonet: fix accidental overwrite of more than just the address
Jens Axboe [Tue, 25 Feb 2014 21:43:38 +0000 (13:43 -0800)]
net: fix accidental overwrite of more than just the address

Commit 0b783341 accidentally overwrote more than just the
address, causing garbage in the connection part.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix crash with --debug=file and NULL file return
Jens Axboe [Tue, 25 Feb 2014 21:43:17 +0000 (13:43 -0800)]
Fix crash with --debug=file and NULL file return

Don't attempt to dereference that, obviously.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoEnsure that fio_get_kb_base() doesn't assume 'data' is thread_options
Jens Axboe [Tue, 25 Feb 2014 19:35:27 +0000 (11:35 -0800)]
Ensure that fio_get_kb_base() doesn't assume 'data' is thread_options

Nasty hack, but at least we don't corrupt option data.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoUnify the time handling
Jens Axboe [Fri, 21 Feb 2014 23:26:01 +0000 (15:26 -0800)]
Unify the time handling

Some options are in seconds, if no units are given. These include
runtime/timeout, startdelay, and ramp_time. Handle this
appropriately. Internally it should all be microseconds now, it's
just the conversion factor.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoImprove latency_target runs
Jens Axboe [Fri, 21 Feb 2014 21:55:31 +0000 (13:55 -0800)]
Improve latency_target runs

Reset the stats when we have found our target, and then do
another latency_window run with those settings so that the
final results reflect the probed values.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFixup time multipliers
Jens Axboe [Fri, 21 Feb 2014 19:50:09 +0000 (11:50 -0800)]
Fixup time multipliers

Commit 74454ce4 assumed default time is in msec, when it's
in reality usec. Fix that up, and also add a specific 'us'
or 'usec' multiplier (of 1).

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoxxhash: dos2unix'ize
Jens Axboe [Thu, 20 Feb 2014 21:29:42 +0000 (13:29 -0800)]
xxhash: dos2unix'ize

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd option group/category to allrandrepeat
Jens Axboe [Thu, 20 Feb 2014 21:22:48 +0000 (13:22 -0800)]
Add option group/category to allrandrepeat

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd support for the Google xxhash checksumming function
Jens Axboe [Thu, 20 Feb 2014 21:21:45 +0000 (13:21 -0800)]
Add support for the Google xxhash checksumming function

It's a lot faster than the other software defined ones, so will
be a useful alternative for the cases where crc32c-intel isn't
available.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoserver: bump protocol version
Jens Axboe [Thu, 20 Feb 2014 17:48:17 +0000 (09:48 -0800)]
server: bump protocol version

We have new options added.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: allow 0 as compress percentage
Peter Oberparleiter [Thu, 20 Feb 2014 13:20:07 +0000 (14:20 +0100)]
fio: allow 0 as compress percentage

Allow 0 as value for option compress_percentage which can be useful for
certain deduplication and compression based storage back ends.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: flush log files on test end
Peter Oberparleiter [Thu, 20 Feb 2014 13:20:05 +0000 (14:20 +0100)]
fio: flush log files on test end

Ensure proper flushing of all logs at the end of tests.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: allow to combine terse output with any selected output type
Christian Ehrhardt [Thu, 20 Feb 2014 13:20:04 +0000 (14:20 +0100)]
fio: allow to combine terse output with any selected output type

This patch adds the option --append-terse to be able to request a
combination of any given selected output format AND terse output based
on the same data.

This will help all users that want to parse the terse data for further
use, but need to look into the logs every now and then which then should
be readable.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: add multi directory support
Christian Ehrhardt [Thu, 20 Feb 2014 17:13:06 +0000 (09:13 -0800)]
fio: add multi directory support

This patch adds support for ':' seperated multiple directories at the
directory config statement in order to achieve an automatic distribution
of job clones (numjob) across directories.

That way people can distribute a load across these directories (usually
mount points of disks) automatically - changing numjob will be
sufficient to get all job clones evenly (optimal if dirs % numjobs = 0,
otherwise as good as possible) distributed at all times.

To avoid confused users old config Files will behave like they always
did, old fio binaries using new config files won't abort but just use
the first specified dir. If one specifies an explcit (non generated)
filename the distribution to many directories is also deactivated.

It also fixes an issue of events seeming out of order like when running
with --debug=file seeing the "..." message meaning "I created the
clones" prior to the last clone activities.  Now the clones are called
with index N-1 .. 1, zero being the base thread as before.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: provide an option for a startdelay range
Christian Ehrhardt [Thu, 20 Feb 2014 17:07:02 +0000 (09:07 -0800)]
fio: provide an option for a startdelay range

This patch allows the specification of start delay as range.  With a
range each thread will chose an individual startdelay out of the range.

That solves an issue of startdelay being the same for each numjob clone
and that way spreads all kind of activities e.g. that all clones with
mixed r/w jobs switch r/w at the same time.  Also all kind of other
"thundering herd" issues can be softened by some time spread due to this
option.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: allow milliseconds on all time specifiers
Christian Ehrhardt [Thu, 20 Feb 2014 13:20:01 +0000 (14:20 +0100)]
fio: allow milliseconds on all time specifiers

This patch allows all time specifiers to be specified down to
milliseconds.  Default will stay seconds for compatibility with old
configs.

It also adds documentation to the existing time units day, hour and
minute.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: allow general repeatability
Christian Ehrhardt [Thu, 20 Feb 2014 17:10:17 +0000 (09:10 -0800)]
fio: allow general repeatability

This patch adds a "allrandrepeat" option similar to the existing
randrepeat.  But other than the former option it does not only affect
the random I/O pattern, but also all remaining users of randomness as
well.

By that e.g. testcases using nrfiles, filesize ranges, blocksizesplit
ranges and so on will stay repeatable across (euqally parametrized) runs
as well.  To maintain compatibility the default is off.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Update by me to fix missing thread_options.h and cconv.c change.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: fix job clone mem leak
Christian Ehrhardt [Wed, 19 Feb 2014 18:42:41 +0000 (10:42 -0800)]
fio: fix job clone mem leak

In the loop to create clones at the bottom of add_job the function
get_new_job clones the thread_data, just to occaisonally get the
allocated pointers for filename and files overwritten a few lines later.

The dup files also duplicates the name strings so the references to
these are lost by the setting to null.

This patch fixes takes care of that and frees the memory before
discarding the pointers (found via valgrind).

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoadd example job file for the RBD engine
Danny Al-Gaaf [Mon, 17 Feb 2014 12:53:07 +0000 (13:53 +0100)]
add example job file for the RBD engine

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio.1: add rbd options to man page
Danny Al-Gaaf [Mon, 17 Feb 2014 12:53:06 +0000 (13:53 +0100)]
fio.1: add rbd options to man page

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 6e44a82a724b7ad0a559c22eec694e8afa7d2a20
git@github.com:dalgaaf/fio.git)
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agolibrbd ioengine support
Daniel Gollub [Mon, 17 Feb 2014 13:35:28 +0000 (14:35 +0100)]
librbd ioengine support

Initially only DDIR_WRITE/DDIR_READ and DDIR_SYNC are supported.
All other io_u->ddir get silently ignored.

A known issue requires to call the rbd engine always with --invalidate=0

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix early termination of cpu id string
Jens Axboe [Fri, 14 Feb 2014 15:48:22 +0000 (08:48 -0700)]
Fix early termination of cpu id string

Commit 2d60ad8e terminated the cpu id string at 12 chars,
but the output from cpuid is 12 chars. So we need one more
character, if we want it null terminated and strcmp()
working.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix another typo in gtod_reduce()
Jens Axboe [Fri, 14 Feb 2014 15:46:35 +0000 (08:46 -0700)]
Fix another typo in gtod_reduce()

Forgot one of the variables. Impressive.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix inverted logic in gtod_reduce()
Jens Axboe [Fri, 14 Feb 2014 03:04:02 +0000 (20:04 -0700)]
Fix inverted logic in gtod_reduce()

Fixes a bug in the previous commit, where someone wasn't
thinking straight...

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoDon't compute iops samples for gtod_reduce=1
Jens Axboe [Fri, 14 Feb 2014 02:59:56 +0000 (19:59 -0700)]
Don't compute iops samples for gtod_reduce=1

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoio_u_qiter: Fix buffer overrun
Sitsofe Wheeler [Thu, 13 Feb 2014 07:06:40 +0000 (07:06 +0000)]
io_u_qiter: Fix buffer overrun

In io_u_queue.h the io_u_qiter macro is loops around io_u_queue
structures. The problem comes with the end of loop initialisation:
i++, io_u = (q)->io_us[i]
For example, if io_us consists of one element and i is 0 then after the
first iteration is completed i++, io_u = (q)->io_us[i] will access
beyond the end of io_us.

Fix this by moving io_u initialisation to the expression part of the for
loop (yuck).

Found by Dr Memory.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agocpuid: Null terminate x86 CPU string.
Sitsofe Wheeler [Thu, 13 Feb 2014 06:07:52 +0000 (06:07 +0000)]
cpuid: Null terminate x86 CPU string.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@fb.com>