fio.git
6 years agoImprove latency_target runs
Jens Axboe [Fri, 21 Feb 2014 21:55:31 +0000 (13:55 -0800)]
Improve latency_target runs

Reset the stats when we have found our target, and then do
another latency_window run with those settings so that the
final results reflect the probed values.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFixup time multipliers
Jens Axboe [Fri, 21 Feb 2014 19:50:09 +0000 (11:50 -0800)]
Fixup time multipliers

Commit 74454ce4 assumed default time is in msec, when it's
in reality usec. Fix that up, and also add a specific 'us'
or 'usec' multiplier (of 1).

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoxxhash: dos2unix'ize
Jens Axboe [Thu, 20 Feb 2014 21:29:42 +0000 (13:29 -0800)]
xxhash: dos2unix'ize

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd option group/category to allrandrepeat
Jens Axboe [Thu, 20 Feb 2014 21:22:48 +0000 (13:22 -0800)]
Add option group/category to allrandrepeat

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd support for the Google xxhash checksumming function
Jens Axboe [Thu, 20 Feb 2014 21:21:45 +0000 (13:21 -0800)]
Add support for the Google xxhash checksumming function

It's a lot faster than the other software defined ones, so will
be a useful alternative for the cases where crc32c-intel isn't
available.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoserver: bump protocol version
Jens Axboe [Thu, 20 Feb 2014 17:48:17 +0000 (09:48 -0800)]
server: bump protocol version

We have new options added.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: allow 0 as compress percentage
Peter Oberparleiter [Thu, 20 Feb 2014 13:20:07 +0000 (14:20 +0100)]
fio: allow 0 as compress percentage

Allow 0 as value for option compress_percentage which can be useful for
certain deduplication and compression based storage back ends.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: flush log files on test end
Peter Oberparleiter [Thu, 20 Feb 2014 13:20:05 +0000 (14:20 +0100)]
fio: flush log files on test end

Ensure proper flushing of all logs at the end of tests.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: allow to combine terse output with any selected output type
Christian Ehrhardt [Thu, 20 Feb 2014 13:20:04 +0000 (14:20 +0100)]
fio: allow to combine terse output with any selected output type

This patch adds the option --append-terse to be able to request a
combination of any given selected output format AND terse output based
on the same data.

This will help all users that want to parse the terse data for further
use, but need to look into the logs every now and then which then should
be readable.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: add multi directory support
Christian Ehrhardt [Thu, 20 Feb 2014 17:13:06 +0000 (09:13 -0800)]
fio: add multi directory support

This patch adds support for ':' seperated multiple directories at the
directory config statement in order to achieve an automatic distribution
of job clones (numjob) across directories.

That way people can distribute a load across these directories (usually
mount points of disks) automatically - changing numjob will be
sufficient to get all job clones evenly (optimal if dirs % numjobs = 0,
otherwise as good as possible) distributed at all times.

To avoid confused users old config Files will behave like they always
did, old fio binaries using new config files won't abort but just use
the first specified dir. If one specifies an explcit (non generated)
filename the distribution to many directories is also deactivated.

It also fixes an issue of events seeming out of order like when running
with --debug=file seeing the "..." message meaning "I created the
clones" prior to the last clone activities.  Now the clones are called
with index N-1 .. 1, zero being the base thread as before.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: provide an option for a startdelay range
Christian Ehrhardt [Thu, 20 Feb 2014 17:07:02 +0000 (09:07 -0800)]
fio: provide an option for a startdelay range

This patch allows the specification of start delay as range.  With a
range each thread will chose an individual startdelay out of the range.

That solves an issue of startdelay being the same for each numjob clone
and that way spreads all kind of activities e.g. that all clones with
mixed r/w jobs switch r/w at the same time.  Also all kind of other
"thundering herd" issues can be softened by some time spread due to this
option.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: allow milliseconds on all time specifiers
Christian Ehrhardt [Thu, 20 Feb 2014 13:20:01 +0000 (14:20 +0100)]
fio: allow milliseconds on all time specifiers

This patch allows all time specifiers to be specified down to
milliseconds.  Default will stay seconds for compatibility with old
configs.

It also adds documentation to the existing time units day, hour and
minute.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: allow general repeatability
Christian Ehrhardt [Thu, 20 Feb 2014 17:10:17 +0000 (09:10 -0800)]
fio: allow general repeatability

This patch adds a "allrandrepeat" option similar to the existing
randrepeat.  But other than the former option it does not only affect
the random I/O pattern, but also all remaining users of randomness as
well.

By that e.g. testcases using nrfiles, filesize ranges, blocksizesplit
ranges and so on will stay repeatable across (euqally parametrized) runs
as well.  To maintain compatibility the default is off.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Update by me to fix missing thread_options.h and cconv.c change.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: fix job clone mem leak
Christian Ehrhardt [Wed, 19 Feb 2014 18:42:41 +0000 (10:42 -0800)]
fio: fix job clone mem leak

In the loop to create clones at the bottom of add_job the function
get_new_job clones the thread_data, just to occaisonally get the
allocated pointers for filename and files overwritten a few lines later.

The dup files also duplicates the name strings so the references to
these are lost by the setting to null.

This patch fixes takes care of that and frees the memory before
discarding the pointers (found via valgrind).

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoadd example job file for the RBD engine
Danny Al-Gaaf [Mon, 17 Feb 2014 12:53:07 +0000 (13:53 +0100)]
add example job file for the RBD engine

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio.1: add rbd options to man page
Danny Al-Gaaf [Mon, 17 Feb 2014 12:53:06 +0000 (13:53 +0100)]
fio.1: add rbd options to man page

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 6e44a82a724b7ad0a559c22eec694e8afa7d2a20
git@github.com:dalgaaf/fio.git)
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agolibrbd ioengine support
Daniel Gollub [Mon, 17 Feb 2014 13:35:28 +0000 (14:35 +0100)]
librbd ioengine support

Initially only DDIR_WRITE/DDIR_READ and DDIR_SYNC are supported.
All other io_u->ddir get silently ignored.

A known issue requires to call the rbd engine always with --invalidate=0

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix early termination of cpu id string
Jens Axboe [Fri, 14 Feb 2014 15:48:22 +0000 (08:48 -0700)]
Fix early termination of cpu id string

Commit 2d60ad8e terminated the cpu id string at 12 chars,
but the output from cpuid is 12 chars. So we need one more
character, if we want it null terminated and strcmp()
working.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix another typo in gtod_reduce()
Jens Axboe [Fri, 14 Feb 2014 15:46:35 +0000 (08:46 -0700)]
Fix another typo in gtod_reduce()

Forgot one of the variables. Impressive.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix inverted logic in gtod_reduce()
Jens Axboe [Fri, 14 Feb 2014 03:04:02 +0000 (20:04 -0700)]
Fix inverted logic in gtod_reduce()

Fixes a bug in the previous commit, where someone wasn't
thinking straight...

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoDon't compute iops samples for gtod_reduce=1
Jens Axboe [Fri, 14 Feb 2014 02:59:56 +0000 (19:59 -0700)]
Don't compute iops samples for gtod_reduce=1

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoio_u_qiter: Fix buffer overrun
Sitsofe Wheeler [Thu, 13 Feb 2014 07:06:40 +0000 (07:06 +0000)]
io_u_qiter: Fix buffer overrun

In io_u_queue.h the io_u_qiter macro is loops around io_u_queue
structures. The problem comes with the end of loop initialisation:
i++, io_u = (q)->io_us[i]
For example, if io_us consists of one element and i is 0 then after the
first iteration is completed i++, io_u = (q)->io_us[i] will access
beyond the end of io_us.

Fix this by moving io_u initialisation to the expression part of the for
loop (yuck).

Found by Dr Memory.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agocpuid: Null terminate x86 CPU string.
Sitsofe Wheeler [Thu, 13 Feb 2014 06:07:52 +0000 (06:07 +0000)]
cpuid: Null terminate x86 CPU string.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix IPv6 check on FreeBSD by including netinet/in.h
Bruce Cran [Thu, 13 Feb 2014 16:06:20 +0000 (09:06 -0700)]
Fix IPv6 check on FreeBSD by including netinet/in.h

I noticed IPv6 support was being disabled on FreeBSD; it seems the
configure script needs netinet/in.h so in6_addr is known.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agonull: style fixup
Jens Axboe [Thu, 13 Feb 2014 04:25:20 +0000 (21:25 -0700)]
null: style fixup

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoengines/null: allow build as external C++ ioengine
Daniel Gollub [Wed, 12 Feb 2014 19:13:46 +0000 (20:13 +0100)]
engines/null: allow build as external C++ ioengine

In first place to have a simple test candidate for external
ioengines using get_ioengine() method for retriving the
io_engine struct.

Can be compiled with:

g++ -O2 -g -shared -rdynamic -fPIC -o null.so null.c -DFIO_EXTERNAL_ENGINE

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoIntroduce get_ioengine for external engines
Daniel Gollub [Thu, 13 Feb 2014 04:23:31 +0000 (21:23 -0700)]
Introduce get_ioengine for external engines

This makes life easier for plugins written in C++
since they do not need to deal with struct initilization issues
with the ioengine_ops symbol.

With g++ a non-static ioengine_ops in global scope like this:

struct ioengine_ops ioengine = {
        .name           = "null",
        .version        = FIO_IOOPS_VERSION,
        .queue          = fio_null_queue,
        .commit         = fio_null_commit,
};

Results in:

cpp_null2.cc: At global scope:
cpp_null2.cc:112:1: error: C99 designator ‘name’ outside aggregate initializer
cpp_null2.cc:112:1: sorry, unimplemented: non-trivial designated initializers not supported
cpp_null2.cc:112:1: sorry, unimplemented: non-trivial designated initializers not supported
cpp_null2.cc:112:1: sorry, unimplemented: non-trivial designated initializers not supported
cpp_null2.cc:112:1: sorry, unimplemented: non-trivial designated initializers not supported
$

Example get_iongine() symbol usage:

---8<---
extern "C" {
void get_ioengine(struct ioengine_ops **ioengine_ptr) {
        struct ioengine_ops *ioengine;
        *ioengine_ptr = (struct ioengine_ops *) malloc(sizeof(struct ioengine_ops));
        ioengine = *ioengine_ptr;

        strcpy(ioengine->name, "cpp_null");
        ioengine->version        = FIO_IOOPS_VERSION;
        ioengine->queue          = fio_null_queue;
        ioengine->commit         = fio_null_commit;
        ioengine->getevents      = fio_null_getevents;
        ioengine->event          = fio_null_event;
        ioengine->init           = fio_null_init;
        ioengine->cleanup        = fio_null_cleanup;
        ioengine->open_file      = fio_null_open;
        ioengine->flags          = FIO_DISKLESSIO;
}
}
--->8---

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>
Moved get_ioengine_t typedef to ioengine.h.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix g++ warning about void* used in arithmetic
Daniel Gollub [Wed, 12 Feb 2014 18:25:42 +0000 (19:25 +0100)]
Fix g++ warning about void* used in arithmetic

../parse.h: In function ‘void* td_var(thread_options*, fio_option*, unsigned int)’:
../parse.h:110:10: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
../parse.h:112:9: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoclang: fix warnings with clang build
Jens Axboe [Wed, 12 Feb 2014 16:10:10 +0000 (09:10 -0700)]
clang: fix warnings with clang build

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMake parse.h C++ safe by avoiding "or" keyword
Daniel Gollub [Wed, 12 Feb 2014 15:24:57 +0000 (08:24 -0700)]
Make parse.h C++ safe by avoiding "or" keyword

Fixes for g++ (4.7.2) following compiler errors when fio.h
gets included (e.g. in an external C++ ioengine):

--8<---
[...]
parse.h:31:6: error: expected unqualified-id before ‘or’ token
[...]
--->8---

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>
Updated by me to apply to recent version, and fix gfio usage
of ->or as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMake file.h C++ safe by casting fio_file_flags
Daniel Gollub [Wed, 12 Feb 2014 14:51:54 +0000 (15:51 +0100)]
Make file.h C++ safe by casting fio_file_flags

Fixes for g++ (4.7.2) following compiler errors when fio.h
gets included (e.g. in an external C++ ioengine):

--8<---
[...]
os/../file.h: In function ‘void fio_file_set_open(fio_file*)’:
os/../file.h:142:1: error: invalid conversion from ‘int’ to ‘fio_file_flags’ [-fpermissive]
os/../file.h: In function ‘void fio_file_clear_open(fio_file*)’:
os/../file.h:142:1: error: invalid conversion from ‘int’ to ‘fio_file_flags’ [-fpermissive]
os/../file.h: In function ‘void fio_file_set_closing(fio_file*)’:
[...]
--->8---

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMake err.h a bit more Windows friendly
Jens Axboe [Wed, 12 Feb 2014 15:22:01 +0000 (08:22 -0700)]
Make err.h a bit more Windows friendly

Using unsigned long to hold pointers is a linuxism, clean
it up.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoparse: cleanup difference between profile and normal options
Jens Axboe [Tue, 11 Feb 2014 22:44:50 +0000 (15:44 -0700)]
parse: cleanup difference between profile and normal options

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFio 2.1.5 fio-2.1.5
Jens Axboe [Tue, 11 Feb 2014 21:49:43 +0000 (14:49 -0700)]
Fio 2.1.5

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoparse: get rid of raw option offsets
Jens Axboe [Tue, 11 Feb 2014 21:19:38 +0000 (14:19 -0700)]
parse: get rid of raw option offsets

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoPlug a free basic memory leaks
Jens Axboe [Tue, 11 Feb 2014 17:33:06 +0000 (10:33 -0700)]
Plug a free basic memory leaks

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoEnfore that lockfile= must precede filename=
Jens Axboe [Tue, 11 Feb 2014 16:12:06 +0000 (09:12 -0700)]
Enfore that lockfile= must precede filename=

The way fio parses options, we need to have the file locking
defined before we start adding files. Normally we can handle
this with option priorities, but that doesn't work across
job section boundaries. So catch this case and error out,
otherwise fio will segfault when it attempts to unlock files
at the end of a job.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix issue with openfiles= and file sizing
Jens Axboe [Tue, 11 Feb 2014 15:31:13 +0000 (08:31 -0700)]
Fix issue with openfiles= and file sizing

Fill the given size as well as we can, given the size and nr
of files.

Fix an issue where we don't properly honor how many files to
keep open at any point in time.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix issue with td->mutex being used-after-free
Jens Axboe [Mon, 10 Feb 2014 20:57:09 +0000 (13:57 -0700)]
Fix issue with td->mutex being used-after-free

Depending on how the OS schedules the threads, it's not necessarily
safe to immediately free a mutex after we have been woken up.
Remove the td->mutex freeing to normal cleanup after the job is done.

Reported-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agocrc: add option to list possible crc types
Jens Axboe [Sat, 8 Feb 2014 03:54:39 +0000 (20:54 -0700)]
crc: add option to list possible crc types

=help or =list will show them.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agocrc: ensure we properly match test name
Jens Axboe [Sat, 8 Feb 2014 03:52:32 +0000 (20:52 -0700)]
crc: ensure we properly match test name

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoREADME: update for crctest
Jens Axboe [Sat, 8 Feb 2014 03:48:29 +0000 (20:48 -0700)]
README: update for crctest

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd support for testing checksumming speed
Jens Axboe [Fri, 7 Feb 2014 21:39:33 +0000 (14:39 -0700)]
Add support for testing checksumming speed

fio --crctest will test all of them, --crctest=md5,crc32c will
test md5 and crc32c, for example.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoPrioritize lockfile= option
Jens Axboe [Fri, 7 Feb 2014 17:57:13 +0000 (10:57 -0700)]
Prioritize lockfile= option

This needs to be run before adding files, or we'll potentially
make a mess of things.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix crash with file locking and dup'ed files
Jens Axboe [Fri, 7 Feb 2014 17:56:15 +0000 (10:56 -0700)]
Fix crash with file locking and dup'ed files

Ensure that we properly inherit the file locks when
duplicating files.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoverify: always log IO in the order they are issued
Jens Axboe [Thu, 6 Feb 2014 19:17:37 +0000 (12:17 -0700)]
verify: always log IO in the order they are issued

We currently log verify_backlog verifies when they complete,
which means the sequence of verify and issue might be different.

Change this to log in one spot, prior to issue, and track the
completion state of the logged unit instead. This unifies the
handling of verifies.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMore precise fix for verify_backlog verification failure
Jens Axboe [Wed, 5 Feb 2014 20:15:22 +0000 (13:15 -0700)]
More precise fix for verify_backlog verification failure

Commit c2703bf3 could be a bit more precise - we only need to
add to the byte count if verify_backlog is enabled, since that
is the case where do_io() will run both the IO and verify IO.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix verify_backlog start verification before finish writing
Puthikorn Voravootivat [Wed, 5 Feb 2014 18:28:15 +0000 (10:28 -0800)]
Fix verify_backlog start verification before finish writing

Due to recent fix for checking rand_seed in verify phase. In write phase,
the IO transaction will be logged before completing the actual data write.
If verify_backlog is enabled, verification may start before all data has
been written. This patch disable that fix if the verify_backlog is enabled.

Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix for prematurely stopping on verify
Jens Axboe [Wed, 5 Feb 2014 19:36:02 +0000 (12:36 -0700)]
Fix for prematurely stopping on verify

Commit 20876c53 added a total byte check, but forgot to
account for potential verification bytes. Fix that.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix rand_seed mismatches in verify phase
Puthikorn Voravootivat [Wed, 5 Feb 2014 17:17:11 +0000 (10:17 -0700)]
Fix rand_seed mismatches in verify phase

In verify phase, the rand_seed generated on replay does not match
the written rand_seed.

Multiple problems are causing this:
1. In verify phase fio does not set io_u->rand_seed to compare with
   hdr->rand_seed
2. In randrw scenario, fio log is stored in red-black tree in "sorted by LBA"
   order.  Thus, it is imposible to replay the written order, or rather
   generate the seeds again in the same order.
3. In write phase, the code currently will generate rand_seed, write data
   and log rand_seed. When queuedepth > 1, it's possible the writes complete
   in a different order than rand_seed was generated.  Thus when replaying
   the log, the generated rand_seed might not match what was written.
4. verify_backlog option will start verification before all the data has been
   written and it make rand_seed replay code broken with current design.

Proposed fixes:
1. Use of existing verify_state to generate verify header.
   (and assumes this was the original intention of verify_state). And also
   adds code to replay rand_seed in verify phase.
2. If verifysort option is not enabled, store the write log in a list instead
   of the red-black tree. Otherwise, don't attempt to verify the rand_seed
   in the header.
3. In write phase,  generate rand_seed, log rand_seed, write data. I.e. log
   IO transactions in the order generated, not completed.
4. Don't verify rand_seed when verify_backlog is enabled.

Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoinit: log error on missing --output-format argument
Stefan Hajnoczi [Tue, 4 Feb 2014 13:27:11 +0000 (14:27 +0100)]
init: log error on missing --output-format argument

strcmp(optarg, "foo") will crash if --output-format was given without an
argument.  Log an error and exit properly instead of crashing.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoinit: --minimal does not take an optional argument
Stefan Hajnoczi [Tue, 4 Feb 2014 13:27:10 +0000 (14:27 +0100)]
init: --minimal does not take an optional argument

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoinit: update email address
Jens Axboe [Wed, 29 Jan 2014 23:39:04 +0000 (16:39 -0700)]
init: update email address

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoEnable IPv6 in the Windows build
Bruce Cran [Wed, 29 Jan 2014 00:34:08 +0000 (17:34 -0700)]
Enable IPv6 in the Windows build

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agofio: consolidate rand_seed to uint64_t
Grant Grundler [Tue, 28 Jan 2014 22:11:23 +0000 (15:11 -0700)]
fio: consolidate rand_seed to uint64_t

csscope showed rand_seed was defined 4 times - each a different type.
While they are used differently, the places where they overlap
should reference the same type. This patch makes three of them the same
as suggested by Jens Axboe.

Signed-off-by: Grant Grundler <grundler@chromium.org>
Updated by me to change rand_seed option to unsigned long long
and changes related to that.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoUpdate Windows build for new threading library
Bruce Cran [Tue, 28 Jan 2014 19:56:53 +0000 (12:56 -0700)]
Update Windows build for new threading library

MinGW now uses libwinpthread-1.dll instead of pthreadGC2.dll.
To simplify things, create a static executable so users can run fio
from anywhere without needing to copy the dll.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix Windows headers for IPv6
Bruce Cran [Tue, 28 Jan 2014 19:54:17 +0000 (12:54 -0700)]
Fix Windows headers for IPv6

MinGW includes all the necessary IPv6 definitions, so include ws2tcpip.h
and remove the custom definitions.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoengine: protect net engine IPv6 support with configure check
Jens Axboe [Sat, 25 Jan 2014 02:59:15 +0000 (18:59 -0800)]
engine: protect net engine IPv6 support with configure check

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoclient/server: don't reset ipv6 expectations
Jens Axboe [Sat, 25 Jan 2014 02:56:56 +0000 (18:56 -0800)]
client/server: don't reset ipv6 expectations

fio_server_parse_host() no longer falls back to ipv4, so just
pass in the actual value, not a reference.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoconfigure: add test for whether required IPv6 helpers
Jens Axboe [Sat, 25 Jan 2014 02:56:34 +0000 (18:56 -0800)]
configure: add test for whether required IPv6 helpers

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoserver: fix wrong error return on host lookup failure
Jens Axboe [Sat, 25 Jan 2014 02:52:16 +0000 (18:52 -0800)]
server: fix wrong error return on host lookup failure

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd verify_only to man page
Jens Axboe [Fri, 24 Jan 2014 20:15:07 +0000 (12:15 -0800)]
Add verify_only to man page

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdds check for rand_seed during verify phase.
Juan Casse [Tue, 17 Sep 2013 21:06:14 +0000 (14:06 -0700)]
Adds check for rand_seed during verify phase.

Improve data integrity checking of header (meta) data.
verify_header() will now return an additional error:
"verify: bad header rand seed ..."
The addition of the check of rand_seed helps detect stale data from
previous fio runs.
This patch also disambiguates the different data mismatches by returning
different error codes from verify_header().

Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdds verify_only option
Juan Casse [Tue, 17 Sep 2013 21:06:13 +0000 (14:06 -0700)]
Adds verify_only option

When this option is set, a dry run (no actual io is performed) of the
workload will be done in order to compute the numberio for each block
header without overwriting the data on disk. Then, do_verify() will be
effectively verifying data that was written in a previous fio run.
In the case that "loops" is set to more than 1, do_verify() will delay
the verification of numberio to the last iteration when the same
numberio state that would have been written to disk in a previous
fio run has been reached.

Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdds check for numberio during verify phase.
Juan Casse [Tue, 17 Sep 2013 21:06:12 +0000 (14:06 -0700)]
Adds check for numberio during verify phase.

Currently, fio checks the block offset number in a block's header during
the verify phase.
We add a check for the io number (numberio) to detect stale blocks. This
check is performed only on workloads that write data, as those workloads
know what numberio was written to each block.
td->io_issues[ddir] = 0; was removed so that numberio does not get reset
at each iteration; we want numberio to keep incrementing to reflect
how many times the same data was written.

Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Fixed typo.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoengine: more checks on IPv6 and multicast
Jens Axboe [Fri, 24 Jan 2014 15:36:43 +0000 (07:36 -0800)]
engine: more checks on IPv6 and multicast

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoserver: make the connect code fully IPv6
Jens Axboe [Fri, 24 Jan 2014 04:42:06 +0000 (20:42 -0800)]
server: make the connect code fully IPv6

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoengine: pass in right protocol family on host lookup
Jens Axboe [Fri, 24 Jan 2014 04:41:33 +0000 (20:41 -0800)]
engine: pass in right protocol family on host lookup

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoengine: IPv6 net engine updates
Jens Axboe [Fri, 24 Jan 2014 04:19:17 +0000 (20:19 -0800)]
engine: IPv6 net engine updates

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoengine: basic support for IPv6 in the net IO engine
Jens Axboe [Fri, 24 Jan 2014 00:49:37 +0000 (16:49 -0800)]
engine: basic support for IPv6 in the net IO engine

Adds 'tcpv6' and 'udpv6' protocol entries.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix potential oops in file unlocking at free time
Jens Axboe [Thu, 23 Jan 2014 00:27:32 +0000 (16:27 -0800)]
Fix potential oops in file unlocking at free time

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoserver: convert buffer pattern settings across client/server
Jens Axboe [Wed, 15 Jan 2014 16:41:31 +0000 (09:41 -0700)]
server: convert buffer pattern settings across client/server

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd option for specifically setting buffer contents
Jens Axboe [Tue, 14 Jan 2014 22:35:58 +0000 (15:35 -0700)]
Add option for specifically setting buffer contents

Fio can use zeroes, slightly scrambled data, full random data,
or specifically compressible data. With this option, the user
can now fully control the pattern written, similarly to how
verify_pattern works for verify=meta.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd option to manually seed the random generators
Jens Axboe [Sat, 11 Jan 2014 03:57:01 +0000 (20:57 -0700)]
Add option to manually seed the random generators

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoUpdate README to reflect that fio now runs on OpenBSD
Jens Axboe [Mon, 30 Dec 2013 02:42:06 +0000 (19:42 -0700)]
Update README to reflect that fio now runs on OpenBSD

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agolibfio: add missing OS's to fio_os_strings[]
Jens Axboe [Mon, 30 Dec 2013 02:22:46 +0000 (19:22 -0700)]
libfio: add missing OS's to fio_os_strings[]

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd OpenBSD support
John [Mon, 30 Dec 2013 02:20:35 +0000 (19:20 -0700)]
Add OpenBSD support

I managed to make a clean compile and a few test runs using the
attached patches.

These work on OpenBSD 5.4 at least.  I used the 2.1.4 release of
fio as a basis.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMake configure tests pass pickier compiler options.
Castor Fu [Fri, 20 Dec 2013 07:00:46 +0000 (23:00 -0800)]
Make configure tests pass pickier compiler options.

Signed-off-by: Castor Fu <castor@alumni.caltech.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd options to have fio latency profile a device
Jens Axboe [Mon, 9 Dec 2013 19:38:53 +0000 (12:38 -0700)]
Add options to have fio latency profile a device

This adds three new options:

- latency_target. This defines a specific latency target, in usec.
- latency_window. This defines the period over which fio samples.
- latency_percentile. This defines the percentage of IOs that must
  meet the criteria specified by latency_target/latency_window.

With these options set, fio will run the described workload and
vary the queue depth between 1 and iodepth= to find the best
performing spot that meets the criteria specified by the three
options.

A sample job file is also added to demonstrate how to use this.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoCalloc() cleanup
Jens Axboe [Fri, 6 Dec 2013 19:02:10 +0000 (12:02 -0700)]
Calloc() cleanup

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Wed, 27 Nov 2013 03:40:00 +0000 (20:40 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agoinit: escape description option string
Jens Axboe [Wed, 27 Nov 2013 03:39:35 +0000 (20:39 -0700)]
init: escape description option string

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoinit: allow FIO_OPT_STR_SET options to have an optional argument
Jens Axboe [Wed, 27 Nov 2013 03:19:59 +0000 (20:19 -0700)]
init: allow FIO_OPT_STR_SET options to have an optional argument

Makes them work like a bool.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoWindows: add CONFIG_TLS_THREAD=y and swap pthreadGC2 to libwinpthread-1.
Bruce Cran [Thu, 7 Nov 2013 14:04:00 +0000 (14:04 +0000)]
Windows: add CONFIG_TLS_THREAD=y and swap pthreadGC2 to libwinpthread-1.

Since GCC 4.4, MinGW has supported TLS via the __thread keyword, so
generate config-host.mak with CONFIG_TLS_THREAD=y.

MinGW now uses a new threading library libwinpthread-1.dll: install it
instead of pthreadGC2.dll.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoError out gracefully if we don't find the replay device for log replay
Jens Axboe [Thu, 21 Nov 2013 18:13:12 +0000 (11:13 -0700)]
Error out gracefully if we don't find the replay device for log replay

It would segfault before because no files got added to replay
against. Handle this more gracefully and ensure that fio exits
with a non-zero exit status.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoblktrace: add support for non-native endian format
Jens Axboe [Thu, 21 Nov 2013 16:55:49 +0000 (09:55 -0700)]
blktrace: add support for non-native endian format

The blktrace format is stored in the native endianness of
the machine it is run on. So to reply traces on a machine
with a different endianness, we need to swap the trace
fields. Detect and do this automatically.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFio 2.1.4 fio-2.1.4
Jens Axboe [Sat, 16 Nov 2013 18:15:12 +0000 (11:15 -0700)]
Fio 2.1.4

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd support for O_ATOMIC
Chris Mason [Fri, 15 Nov 2013 22:52:58 +0000 (15:52 -0700)]
Add support for O_ATOMIC

O_ATOMIC makes writes atomic, meaning that they are fully stable
on media (in the event of a power cut) when acknowledged by the
device and OS.

This only truly works on Linux with the pending patches to
add O_ATOMIC.

Updated by Jens to:

- Add man page and HOWTO description of the option
- Make O_ATOMIC imply O_DIRECT, so that it actually works if you
  don't set O_DIRECT manually.
- Add the option to the conversion list so it works for
  client/server.
- Error handling so that if atomic=1 is set and the OS does not
  support it, error out instead of just pretending it works.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix access-outside-array of o->rwmix[]
Jens Axboe [Fri, 8 Nov 2013 17:57:08 +0000 (10:57 -0700)]
Fix access-outside-array of o->rwmix[]

gcc 4.9 rightfully complains about it, though in a very obtuse way:

     CC cconv.o
In file included from thread_options.h:5:0,
                  from cconv.c:3:
cconv.c: In function 'convert_thread_options_to_cpu':
os/os.h:197:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
   __le32_to_cpu(*__val);   \
                 ^
os/os.h:176:28: note: in definition of macro '__le32_to_cpu'
  #define __le32_to_cpu(x)  (x)
                             ^
cconv.c:78:17: note: in expansion of macro 'le32_to_cpu'
    o->rwmix[i] = le32_to_cpu(top->rwmix[i]);
                  ^
cconv.c:63:2: note: containing loop
   for (i = 0; i < DDIR_RWDIR_CNT; i++) {
   ^
In file included from thread_options.h:5:0,
                  from cconv.c:3:
cconv.c: In function 'convert_thread_options_to_net':
os/os.h:209:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
   __cpu_to_le32(*__val);   \
                 ^
os/os.h:179:28: note: in definition of macro '__cpu_to_le32'
  #define __cpu_to_le32(x)  (x)
                             ^
cconv.c:372:19: note: in expansion of macro 'cpu_to_le32'
    top->rwmix[i] = cpu_to_le32(o->rwmix[i]);
                    ^
cconv.c:352:2: note: containing loop
   for (i = 0; i < DDIR_RWDIR_CNT; i++) {
   ^
In file included from thread_options.h:5:0,
                  from cconv.c:3:
cconv.c: In function 'fio_test_cconv':
os/os.h:209:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
   __cpu_to_le32(*__val);   \
                 ^
os/os.h:179:28: note: in definition of macro '__cpu_to_le32'
  #define __cpu_to_le32(x)  (x)
                             ^
cconv.c:372:19: note: in expansion of macro 'cpu_to_le32'
    top->rwmix[i] = cpu_to_le32(o->rwmix[i]);
                    ^
cconv.c:352:2: note: containing loop
   for (i = 0; i < DDIR_RWDIR_CNT; i++) {
   ^
In file included from thread_options.h:5:0,
                  from cconv.c:3:
os/os.h:197:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
   __le32_to_cpu(*__val);   \
                 ^
os/os.h:176:28: note: in definition of macro '__le32_to_cpu'
  #define __le32_to_cpu(x)  (x)
                             ^
cconv.c:78:17: note: in expansion of macro 'le32_to_cpu'
    o->rwmix[i] = le32_to_cpu(top->rwmix[i]);
                  ^
cconv.c:63:2: note: containing loop
   for (i = 0; i < DDIR_RWDIR_CNT; i++) {
   ^

Fix it by extending the rwmix indexes to the full rwdir count,
even though it doesn't support trim as part of the mix (yet).

Reported-by: Bruce Cran <bruce@cran.org.uk>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoserver: make job/connection lists private
Jens Axboe [Thu, 7 Nov 2013 20:37:09 +0000 (13:37 -0700)]
server: make job/connection lists private

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agosolaris: ensure that -D_REENTRANT gets set
Jens Axboe [Wed, 6 Nov 2013 22:37:35 +0000 (15:37 -0700)]
solaris: ensure that -D_REENTRANT gets set

Apparently some Solaris' require this for threadsafe
errno.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoposixaio: fix thread problem with using errno
Jens Axboe [Wed, 6 Nov 2013 21:47:22 +0000 (14:47 -0700)]
posixaio: fix thread problem with using errno

If we fail queueing a read or a write, use aio_error() to
retrieve the right error value. This fixes an issue on
(at least) Solaris where we get EAGAIN due to system
shortage of resources, but treat that as a random type
of error due to using errno.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoserver: get rid of compile warning on platforms where pid_t isn't int
Jens Axboe [Wed, 6 Nov 2013 18:19:42 +0000 (11:19 -0700)]
server: get rid of compile warning on platforms where pid_t isn't int

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix a couple of typos in fio2gnuplot.
Bruce Cran [Mon, 4 Nov 2013 11:59:43 +0000 (11:59 +0000)]
Fix a couple of typos in fio2gnuplot.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoUse #!/usr/bin/env to allow for interpreters in different directories.
Bruce Cran [Mon, 4 Nov 2013 11:52:08 +0000 (11:52 +0000)]
Use #!/usr/bin/env to allow for interpreters in different directories.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoserver: ensure that fio_time_init() is called before option parsing
Bruce Cran [Fri, 1 Nov 2013 18:32:40 +0000 (12:32 -0600)]
server: ensure that fio_time_init() is called before option parsing

When run in server mode, jobs are processed before fio_time_init()
is called, which causes pthread_setspecific() to be called before
pthread_key_create(). Work around this by calling it in thread_main()
too.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoImplement writev on Windows
Bruce Cran [Fri, 1 Nov 2013 18:31:03 +0000 (12:31 -0600)]
Implement writev on Windows

The network code depends on writev(), so implement it for Windows.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoUpdate Windows code to fix build breakage
Bruce Cran [Tue, 8 Oct 2013 14:05:27 +0000 (15:05 +0100)]
Update Windows code to fix build breakage

MinGW has been updated: this causes problems due to changes in header
files, including COM stuff being pulled in resulting in 'interface' being
a reserved keyword, and changes to a few POSIX definitions.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoconfigure: add option to disable libnuma usage
Castor Fu [Thu, 31 Oct 2013 17:00:54 +0000 (11:00 -0600)]
configure: add option to disable libnuma usage

Signed-off-by: Castor Fu <castor@alumni.caltech.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd json output for client/server mode
Castor Fu [Thu, 31 Oct 2013 17:00:34 +0000 (11:00 -0600)]
Add json output for client/server mode

In client/server mode, this adds support for json mode.  Each job's
details are gradually accumulated and upon completion the client
dumps the full json tree.  The tree is annotated with the server
host and port.

Signed-off-by: Castor Fu <castor@alumni.caltech.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>