fio.git
6 years agoflist: add flist_first_entry()
Jens Axboe [Fri, 4 Jul 2014 03:19:57 +0000 (21:19 -0600)]
flist: add flist_first_entry()

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoiolog: use a define instead of a raw bit mask
Jens Axboe [Fri, 4 Jul 2014 00:01:11 +0000 (18:01 -0600)]
iolog: use a define instead of a raw bit mask

Add a comment too, while in there.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoiolog: get rid of ic->nofree
Jens Axboe [Thu, 3 Jul 2014 23:58:49 +0000 (17:58 -0600)]
iolog: get rid of ic->nofree

Push the free out of flush_chunk(), then the caller just does it
instead. Unifies the log write vs compress as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoiolog: get rid of one section of CONFIG_ZLIB ifdef
Jens Axboe [Thu, 3 Jul 2014 23:56:55 +0000 (17:56 -0600)]
iolog: get rid of one section of CONFIG_ZLIB ifdef

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoconfigure: add a note to install zlib-devel, if not there
Jens Axboe [Thu, 3 Jul 2014 22:20:27 +0000 (16:20 -0600)]
configure: add a note to install zlib-devel, if not there

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoinit: fix dead check for !td
Jens Axboe [Thu, 3 Jul 2014 21:33:24 +0000 (15:33 -0600)]
init: fix dead check for !td

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoiolog: do stat() after fopen()
Jens Axboe [Thu, 3 Jul 2014 21:32:07 +0000 (15:32 -0600)]
iolog: do stat() after fopen()

Avoid static checker complaining about a potential race here,
ordering doesn't matter to fio.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agorbd: fix leak in error path
Jens Axboe [Thu, 3 Jul 2014 21:31:32 +0000 (15:31 -0600)]
rbd: fix leak in error path

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoglusterfs: fix leak in error path
Jens Axboe [Thu, 3 Jul 2014 21:30:37 +0000 (15:30 -0600)]
glusterfs: fix leak in error path

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoiolog: remove dead code that stored a gzip header
Jens Axboe [Thu, 3 Jul 2014 20:17:45 +0000 (14:17 -0600)]
iolog: remove dead code that stored a gzip header

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoconfigure: remove typo for zlib
Jens Axboe [Thu, 3 Jul 2014 20:17:02 +0000 (14:17 -0600)]
configure: remove typo for zlib

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMove tp.[ch] to lib/
Jens Axboe [Thu, 3 Jul 2014 20:16:25 +0000 (14:16 -0600)]
Move tp.[ch] to lib/

It's just helper code, not fio core.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd support for storing compressed logs
Jens Axboe [Thu, 3 Jul 2014 19:42:28 +0000 (13:42 -0600)]
Add support for storing compressed logs

If log_store_compressed is set, fio won't decompress for storing
them in a file. Instead, you can use --inflate-log=logfile.fz to
output the contents to stdout.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio.1 - escape the escape character so it shows up
Jeff Moyer [Thu, 3 Jul 2014 18:20:28 +0000 (14:20 -0400)]
fio.1 - escape the escape character so it shows up

Hi, Jens,

In the filename= text, the escape character gets eaten.  This fixes it
for me.

Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoBail out if glfs_creat failed
Tiziano Müller [Thu, 3 Jul 2014 16:22:29 +0000 (18:22 +0200)]
Bail out if glfs_creat failed

Otherwise we will get segfaults from gfapi when it tries to access the
invalid fd.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agotp: remove debug start/shutdown printfs
Jens Axboe [Thu, 3 Jul 2014 15:41:58 +0000 (09:41 -0600)]
tp: remove debug start/shutdown printfs

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoPass O_DIRECT/O_SYNC to glfs_open if requested.
Tiziano Müller [Thu, 3 Jul 2014 14:48:39 +0000 (16:48 +0200)]
Pass O_DIRECT/O_SYNC to glfs_open if requested.

GlusterFS supports O_DIRECT and O_SYNC.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd support for runtime log compression
Jens Axboe [Thu, 3 Jul 2014 15:10:39 +0000 (09:10 -0600)]
Add support for runtime log compression

If log_compression=64M is specified, fio will "bite" off chunks of
the IO logs and runtime compress them with zlib. This can greatly
reduce the amount of memory required to do iops/bw/lat logging of
a run, at the cost of using some background CPU for the compression.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agolog: fix use-after-free
Jens Axboe [Wed, 2 Jul 2014 19:36:34 +0000 (13:36 -0600)]
log: fix use-after-free

Introduced in commit cb7e0ace739c.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agocconv: convert ->log_offset on the wire
Jens Axboe [Wed, 2 Jul 2014 15:08:37 +0000 (09:08 -0600)]
cconv: convert ->log_offset on the wire

Forgotten in commit ae58885234

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years ago--latency-log is now gone
Jens Axboe [Tue, 1 Jul 2014 22:38:02 +0000 (16:38 -0600)]
--latency-log is now gone

Use per-job options for this.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoCleanup logging
Jens Axboe [Tue, 1 Jul 2014 22:29:12 +0000 (16:29 -0600)]
Cleanup logging

If the log knows the filename it is supposed to log to, then we
don't need to have part of that on the init side and the other
part on the flush side. So clean it up so that the caller of
setup_log() provides everything we need. This gets rid of
various ways of ending the log, and opens us up to doing
log flushing mid-run.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoclient: fix missing init of 'i'
Jens Axboe [Tue, 1 Jul 2014 22:07:59 +0000 (16:07 -0600)]
client: fix missing init of 'i'

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoclient: fix missing pdu->log_offset endianness conversion
Jens Axboe [Tue, 1 Jul 2014 14:47:11 +0000 (08:47 -0600)]
client: fix missing pdu->log_offset endianness conversion

Make the iolog sample number uint64_t as well, so we have
the same size across 32 and 64-bit archs.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd a typecheck for the endianness conversions
Jens Axboe [Tue, 1 Jul 2014 14:46:44 +0000 (08:46 -0600)]
Add a typecheck for the endianness conversions

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd option for including byte offset for each log entry
Jens Axboe [Tue, 1 Jul 2014 02:59:03 +0000 (20:59 -0600)]
Add option for including byte offset for each log entry

When iops/latency/bw logs are used in the job, we can note the offset
at which the specific data entry took place.

Based on an initial patch from Abutalib Aghayev <agayev@gmail.com>.
Extended by me to:

- Not do this by default, as not to add 8 bytes to each log entry
  stored.

- Add support for the client/server protocol.

- Add documentation.

We should bump the server protocol rev again, but that was just done
in the previous commit, so...

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoserver: update protocol version to 35
Jens Axboe [Mon, 30 Jun 2014 21:31:59 +0000 (15:31 -0600)]
server: update protocol version to 35

We pack structs now, so not compatible with previous version.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoserver: pack over-the-wire structs
Jens Axboe [Mon, 30 Jun 2014 14:51:33 +0000 (08:51 -0600)]
server: pack over-the-wire structs

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix typo in io_u.c::get_next_file()
Andrey Kuzmin [Sat, 28 Jun 2014 02:21:22 +0000 (20:21 -0600)]
Fix typo in io_u.c::get_next_file()

We should check ops->get_next_file() is TD_F_PROFILE_OPS
is set, not if it's unset.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoclient: fix ETA run_str
Jens Axboe [Fri, 27 Jun 2014 21:30:06 +0000 (15:30 -0600)]
client: fix ETA run_str

We never touched it before, which meant the client would just
display an empty string. Make a valiant effort to allocate and
output the run_str.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoeta: only pass real size of jobs_eta
Jens Axboe [Fri, 27 Jun 2014 21:15:23 +0000 (15:15 -0600)]
eta: only pass real size of jobs_eta

We might send this over the wire, so need sending more than we
actually used.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix struct jobs_eta packing on 32 vs 64-bit archs
Jens Axboe [Fri, 27 Jun 2014 21:01:06 +0000 (15:01 -0600)]
Fix struct jobs_eta packing on 32 vs 64-bit archs

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoVerify: Tighten header length check in verify_header()
Andreas Gruenbacher [Thu, 26 Jun 2014 17:25:30 +0000 (19:25 +0200)]
Verify: Tighten header length check in verify_header()

Fix-up to commit 5964842c: Tighten the header length check as well, don't only
change the message.

6 years agoAdd configure flags for gfapi and rbd
Tiziano Müller [Thu, 26 Jun 2014 12:20:20 +0000 (14:20 +0200)]
Add configure flags for gfapi and rbd

Add --disable-{gfapi,rbd} to disable gfapi/rbd support even if available.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoVerify: Fix and improve verify_header()
Andreas Gruenbacher [Tue, 24 Jun 2014 20:13:50 +0000 (22:13 +0200)]
Verify: Fix and improve verify_header()

Report errors directly from verify_header(); requires passing in the header
number and length.  Correctly report the file name, offset, and expected header
size in all error messages.

6 years agoupdate_condensed_str(): Fix for empty input strings
Andreas Gruenbacher [Tue, 24 Jun 2014 11:09:07 +0000 (13:09 +0200)]
update_condensed_str(): Fix for empty input strings

Empty input strings resulted in unterminated result strings.

6 years agoCast input argument for json_object_add_value_int to long long
Puthikorn Voravootivat [Tue, 24 Jun 2014 22:59:53 +0000 (15:59 -0700)]
Cast input argument for json_object_add_value_int to long long

The latency related (and some other) variables type is uint32_t
but the JSON integer type is long long. So we need to cast
the numbers to long long type to make JSON output correctly.

Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoGet the crctest results properly aligned
Jens Axboe [Tue, 24 Jun 2014 16:12:47 +0000 (10:12 -0600)]
Get the crctest results properly aligned

Before:

md5: 534.03 MB/sec
crc64: 392.31 MB/sec
crc32: 352.36 MB/sec
crc32c: 8225.43 MB/sec
crc16: 384.46 MB/sec
crc7: 386.79 MB/sec
sha1: 381.15 MB/sec
sha256: 186.47 MB/sec
sha512: 303.18 MB/sec
xxhash: 6329.90 MB/sec

After:

md5:   548.59 MB/sec
crc64:   381.45 MB/sec
crc32:   347.89 MB/sec
crc32c:  8577.36 MB/sec
crc16:   396.56 MB/sec
crc7:   396.72 MB/sec
sha1:   385.87 MB/sec
sha256:   206.60 MB/sec
sha512:   314.62 MB/sec
xxhash:  6293.64 MB/sec

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoEnsure that job name is set before being used
Jens Axboe [Tue, 24 Jun 2014 15:19:30 +0000 (09:19 -0600)]
Ensure that job name is set before being used

Andreas reports:

When passing the --readonly option, with the following write job (and
probably with any write job), I get an error message with a NULL job
name. Job description used:

[write]
thread
bs=64k
direct=1
ioengine=sync
size=128m
filename=test.tmp
rw=write
do_verify=0

Reported-by: Andreas Grünbacher <andreas.gruenbacher@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd a get_jobs_eta() to return jobs eta information
Jens Axboe [Tue, 24 Jun 2014 14:40:21 +0000 (08:40 -0600)]
Add a get_jobs_eta() to return jobs eta information

We duplicate this code in eta.c and server.c, consolidate it.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoEnsure we have enough room for the ETA runstr
Jens Axboe [Tue, 24 Jun 2014 01:07:12 +0000 (19:07 -0600)]
Ensure we have enough room for the ETA runstr

The calculation of 1:4 is wrong, it's 1:5. And we missed the similar
change in server.c.

Reported-by: Andreas Grünbacher <andreas.gruenbacher@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix two minor typos
Andreas Gruenbacher [Mon, 23 Jun 2014 21:17:08 +0000 (23:17 +0200)]
Fix two minor typos

6 years agoREADME: update to include official mirrors for the git repo
Jens Axboe [Mon, 23 Jun 2014 13:52:48 +0000 (07:52 -0600)]
README: update to include official mirrors for the git repo

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd support for compiling for ESX
Jens Axboe [Wed, 18 Jun 2014 22:30:09 +0000 (15:30 -0700)]
Add support for compiling for ESX

With contributions from Ryan Haynes <rhaynes@fusionio.com>

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix build from cmake
George Dowding [Wed, 18 Jun 2014 16:36:53 +0000 (09:36 -0700)]
Fix build from cmake

George reports:

I spent a few hours on this. Might save someone else some time.

I'm working in a larger project that uses cmake for its builds. cmake
supports adding external projects using the ExternalProject_Add
function.

But, there is some issue in the fio Makefile that causes a problem
when run from cmake. I could never reproduce it from the command line.

I was able to get the build to happen by changing the implicit rule
.c.o to use pattern notation. Not sure if this will work in all cases
or has other side effects.

Without this change, I was getting this error.

make[4]: *** No rule to make target `gettime.o', needed by `fio'.  Stop.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFixup run_str[] condensing with client/server
Jens Axboe [Tue, 17 Jun 2014 15:22:34 +0000 (09:22 -0600)]
Fixup run_str[] condensing with client/server

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd string condensing for the ETA output
Jens Axboe [Mon, 16 Jun 2014 20:42:05 +0000 (14:42 -0600)]
Add string condensing for the ETA output

If you run a lot of jobs, the ETA output string will become way too
long. Add some condensing so that instead of displaying:

Jobs: 80 (f=80): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [10.2% done] [2110MB/0KB/0KB /s] [540K/0/0 iops] [eta 53m:55s]

we just display

Jobs: 80 (f=80): [r(80)] [10.2% done] [2110MB/0KB/0KB /s] [540K/0/0 iops] [eta 53m:55s]

instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agofio: Eliminate compilation warning in ppc
Sumanth K [Mon, 16 Jun 2014 05:38:16 +0000 (11:08 +0530)]
fio: Eliminate compilation warning in ppc

Eliminate compilation warning : 'atb_clocktest' defined but
not used. Function is declared as static in header file. So compiler will
know if they used or not. Just eliminate the warning by #if 0 /unless we do
use it.

Signed-off-by: Sumanth K <sumantk2@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFio 2.1.10 fio-2.1.10
Jens Axboe [Tue, 10 Jun 2014 19:18:38 +0000 (13:18 -0600)]
Fio 2.1.10

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoiolog: remember to add bytes to total IO size
Jens Axboe [Tue, 10 Jun 2014 01:56:51 +0000 (19:56 -0600)]
iolog: remember to add bytes to total IO size

This fixes an issue with replay of iologs.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoReset file count when freeing files
Jens Axboe [Tue, 10 Jun 2014 01:52:05 +0000 (19:52 -0600)]
Reset file count when freeing files

This fixes and oops with iolog replay, and a manually specified
filename=

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix crash on loading unknown IO engine with --ioengine
Jens Axboe [Mon, 9 Jun 2014 19:54:22 +0000 (13:54 -0600)]
Fix crash on loading unknown IO engine with --ioengine

We don't clean up properly for a command line job when this happens.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix regression introduced by commit 334185e
Jens Axboe [Mon, 9 Jun 2014 19:25:13 +0000 (13:25 -0600)]
Fix regression introduced by commit 334185e

We need to run fio_time_init() before the threads are invoked,
but we should do it in global context so that all threads share
the same clock and init settings.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd aarch64 support.
Doug Kwan [Fri, 6 Jun 2014 22:29:11 +0000 (15:29 -0700)]
Add aarch64 support.

This patch adds support for aarch64 for fio.  This is tested by building
FIO on an aarch64 Ubuntu system and running some samepls.

Signed-off-by: Doug Kwan <dougkwan@google.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoglusterfs: remove glfs-handles.h include
Jens Axboe [Tue, 3 Jun 2014 14:36:20 +0000 (08:36 -0600)]
glusterfs: remove glfs-handles.h include

We don't seem to need it, and newer packages don't include
it.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMake rbd engine use aio flush
Haomai Wang [Wed, 21 May 2014 08:30:55 +0000 (16:30 +0800)]
Make rbd engine use aio flush

The fio's option "fsync" can make rbd engine issue sync request. In practice,
we would like to use aio_rbd_flush to make it non-block. So it will be more
reliable to use fio to evaluate actual io scene like qemu.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoETA no longer works when -o specified
Rik Faith [Thu, 29 May 2014 20:13:39 +0000 (14:13 -0600)]
ETA no longer works when -o specified

Regarding fio version fio-2.1.9-17-gd9b1 (after April 11, 2014):

A recent change (5e1d8745471d1f315113c5c3fd8b93809d9ddff9) caused stdout
to be closed when -o was specified, which prevents ETA output from showing
up on stdout even when, e.g., json is being saved in the file.

This patch continues to close f_out so that there is no leak if multiple -o
options are specified, but doesn't close stdout.  Hence, ETA works again
when it should.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd ->invalidate() IO engine ops
Jens Axboe [Tue, 20 May 2014 01:57:05 +0000 (19:57 -0600)]
Add ->invalidate() IO engine ops

Allow IO engines to plug in their own invalidate cache handler.
This fixes an issue on rbd, where we attempt to invalidate the
cache, but fail because it's not a valid file descriptor.

Reported-by: xan.peng@gmail.com
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agogfapi: indentify
Jens Axboe [Tue, 20 May 2014 01:53:03 +0000 (19:53 -0600)]
gfapi: indentify

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoRun the gluster engines through indent
Jens Axboe [Mon, 19 May 2014 16:12:56 +0000 (10:12 -0600)]
Run the gluster engines through indent

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMerge branch 'master' of git://github.com/cvubrugier/fio
Jens Axboe [Mon, 19 May 2014 15:50:54 +0000 (09:50 -0600)]
Merge branch 'master' of git://github.com/cvubrugier/fio

6 years agoMerge branch 'master' of git://github.com/rootfs/fio
Jens Axboe [Mon, 19 May 2014 15:49:56 +0000 (09:49 -0600)]
Merge branch 'master' of git://github.com/rootfs/fio

Signed-off-by: Jens Axboe <axboe@fb.com>
Conflicts:
configure

6 years agoFix issue with dangling pointer in the rbd engine
Xan Peng [Mon, 19 May 2014 15:39:00 +0000 (09:39 -0600)]
Fix issue with dangling pointer in the rbd engine

Without this patch, after _fio_rbd_connect() failed, fio will hang for a few seconds, display the following logs and then quit:

====console output start====
rbd_iodepth32: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=32
fio-2.1.9
Starting 1 process
rbd engine: RBD version: 0.1.8
rbd engine: RBD version: 0.1.8
rbd_open failed.
fio_rbd_connect failed.
common/Mutex.cc: In function 'void Mutex::Lock(bool)' thread 7f6e337aa780 time 2014-05-17 19:23:23.417285
common/Mutex.cc: 93: FAILED assert(r == 0)
ceph version 0.80 (b78644e)
1: (()+0x254093) [0x7f6e32502093]
2: (librados::RadosClient::shutdown()+0x1f) [0x7f6e324741df]
3: (rados_shutdown()+0x9) [0x7f6e32459209]
4: ./fio() [0x456a46]
5: (setup_files()+0xc2) [0x420872]
6: ./fio() [0x449c15]
7: (fio_backend()+0x115) [0x44a515]
8: (__libc_start_main()+0xf5) [0x7f6e311f3455]
9: ./fio() [0x40ad3c]
NOTE: a copy of the executable, or objdump -rdS is needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'
Aborted
====console output end====

With this patch, after _fio_rbd_connect() failed, fio will quit at once:

====console output start====
rbd_iodepth32: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=32
fio-2.1.9
Starting 1 process
rbd engine: RBD version: 0.1.8
rbd_open failed.
fio_rbd_connect failed.

Run status group 0 (all jobs):
====console output end====

Signed-off-by: Xan Peng <xanpeng@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFio 2.1.9 fio-2.1.9
Jens Axboe [Mon, 12 May 2014 23:18:13 +0000 (17:18 -0600)]
Fio 2.1.9

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoverify: fix verification of uncompleted or errored ios
Jens Axboe [Wed, 7 May 2014 01:06:51 +0000 (19:06 -0600)]
verify: fix verification of uncompleted or errored ios

Stoo Davies reports:

------

I'm doing some powerfail recovery testing on a storage array over iSCSI.
Host is RHEL 6.4 kernel 2.6.32-358.el6.x86_64.

With fio 2.1.2 -> 2.1.4 the job file below rides through the disks going
away, and continues I/O after they come back, without reporting any
errors.  With fio 2.1.5 -> 2.1.8 when the disks come back fio
immediately reports a meta verification error.

I captured a trace with an finisar analyzer, and can see that after the
disks come back and the host logs back in, a read is issued for an lba
which was never written to.  Since I don't see verification errors
outside of the powerfail testing, I suspect fio isn't correctly handling
failed writes during the time the disks are unavailable.

------

The issue is caused by a change between 2.1.4 and 2.1.5 that started
logging IO for verification at issue time, instead of at completion
time. If the IO only partially completes or completes in error, then
we need to adjust the verification unit so that we verify only what
fio actually wrote.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAvoid buildenv conditional in thread_option struct
Daniel Gollub [Thu, 1 May 2014 15:07:49 +0000 (17:07 +0200)]
Avoid buildenv conditional in thread_option struct

Managed to run into issues with an external ioengine
which got build with CONFIG_LIBNUMA not defined. Fio
itself got build with CONFIG_LIBNUMA this resulted
in different struct members offsets in the two different
ELF objects. Causing crashes due to invalidate offsets
inside the thread_data structure (e.g. td->io_ops->data).

Ideally all structs which might be used by external
ioengines should be independent of buildenv conditionals
like CONFIG_LIBNUMA or others.

Removed the CONFIG_LIBNUMA in thread_options.h and replaced
the libnuma specific "struct bitmask" members with strings
which hold the option's input value. This should also make
the marshaling/demarshaling in cconv.c easier.
(Note: the NUMA bits are not handled in cconv.c at the
moment. And not part of the thread_options_packed struct)

Signed-off-by: Daniel Gollub <daniel.gollub@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAvoid double declaration of leXX_to_cpu
Daniel Gollub [Thu, 1 May 2014 12:07:04 +0000 (14:07 +0200)]
Avoid double declaration of leXX_to_cpu

for external ioengines. leXX_to_cpu is a very common
declaration - e.g. Ceph is using this as well.

Signed-off-by: Daniel Gollub <daniel.gollub@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAvoid double-declaration of ARRAY_SIZE
Daniel Gollub [Thu, 1 May 2014 12:07:03 +0000 (14:07 +0200)]
Avoid double-declaration of ARRAY_SIZE

for external ioengines. ARRAY_SIZE is a very common
declaration - e.g. Ceph is using this as well.

Signed-off-by: Daniel Gollub <daniel.gollub@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agodistinguish internal/ext. builds with FIO_INTERNAL
Daniel Gollub [Thu, 1 May 2014 12:07:02 +0000 (14:07 +0200)]
distinguish internal/ext. builds with FIO_INTERNAL

To distinguish between internal and external includes of
fio.h and other headers FIO_INTERNAL get applied only for
the fio (and gfio) internal build itself.

This helps to prevent double-declaration issues by making
ambigous declaration with common names coditional based on
the FIO_INTERNAL define.

Signed-off-by: Daniel Gollub <daniel.gollub@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoReplace CONFIG_CPU_COUNT with plain CPU_COUNT
Daniel Gollub [Thu, 1 May 2014 12:07:01 +0000 (14:07 +0200)]
Replace CONFIG_CPU_COUNT with plain CPU_COUNT

Fixes double-declartion issue of CPU_COUNT when fio.h is included by
third-party projects.

Checking for a define of CPU_COUNT seems to be a common approach
to deal with different versions of sched.h

Signed-off-by: Daniel Gollub <daniel.gollub@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agocrc/test.c: fix include of time.h
Danny Al-Gaaf [Wed, 30 Apr 2014 18:48:16 +0000 (20:48 +0200)]
crc/test.c: fix include of time.h

Replace include of time.h with renamed fio_time.h to fix build.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMakefile: clean all *.d dependency files on make clean
Jens Axboe [Wed, 30 Apr 2014 21:01:00 +0000 (15:01 -0600)]
Makefile: clean all *.d dependency files on make clean

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoRename time.h for third-party include of fio.h
Daniel Gollub [Wed, 30 Apr 2014 09:25:04 +0000 (11:25 +0200)]
Rename time.h for third-party include of fio.h

External project might build external ioengines and need to
include there for fio.h. If a project set the include path
to the fio root source directory and the third-party source
holds an "time.h" as well things get complicated.

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>
Cc: Daniel Gollub <daniel.gollub@t-online.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix gfio build due to disk_stat struct seperation
Daniel Gollub [Wed, 30 Apr 2014 06:27:53 +0000 (08:27 +0200)]
Fix gfio build due to disk_stat struct seperation

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>
Cc: Daniel Gollub <daniel.gollub@t-online.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoWindows: always open files in binary mode
Bruce Cran [Mon, 28 Apr 2014 21:54:43 +0000 (15:54 -0600)]
Windows: always open files in binary mode

On Windows, open() defaults to text mode, causing control characters
to be interpreted. Avoid this by passing the _O_BINARY flag.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAdd option for io_limit
Jens Axboe [Mon, 28 Apr 2014 18:08:47 +0000 (12:08 -0600)]
Add option for io_limit

'size' denotes both the size of the region for IO, as well as the
amount of IO to transfer. Sometimes it's useful to be able to set
them separately instead. With this patch, you can do:

size=100G
io_limit=10G

and have fio do 10G of IO between 0..100G of the address space.

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Wed, 23 Apr 2014 15:59:05 +0000 (09:59 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agoinit: don't set did_arg just for an engine option
Jens Axboe [Wed, 23 Apr 2014 15:58:39 +0000 (09:58 -0600)]
init: don't set did_arg just for an engine option

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoError out if the set blocksize is larger than the file size
Jens Axboe [Wed, 16 Apr 2014 21:51:27 +0000 (23:51 +0200)]
Error out if the set blocksize is larger than the file size

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agosg/binject: only restore file flags if they have been set
Jens Axboe [Tue, 15 Apr 2014 15:07:44 +0000 (09:07 -0600)]
sg/binject: only restore file flags if they have been set

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agosg: check for error in fcntl() restore of flags
Jens Axboe [Tue, 15 Apr 2014 14:30:06 +0000 (08:30 -0600)]
sg: check for error in fcntl() restore of flags

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agobinject: check for error in fcntl() restore of flags
Jens Axboe [Tue, 15 Apr 2014 14:28:51 +0000 (08:28 -0600)]
binject: check for error in fcntl() restore of flags

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agobackend: fix potential division-by-zero
Jens Axboe [Tue, 15 Apr 2014 14:26:57 +0000 (08:26 -0600)]
backend: fix potential division-by-zero

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoidletime: fix another missing unlock on error
Jens Axboe [Tue, 15 Apr 2014 14:25:16 +0000 (08:25 -0600)]
idletime: fix another missing unlock on error

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Tue, 15 Apr 2014 14:18:36 +0000 (08:18 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agostrcpy: kill last of the suspect ones
Jens Axboe [Tue, 15 Apr 2014 14:18:08 +0000 (08:18 -0600)]
strcpy: kill last of the suspect ones

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoWindows: set sizeof(sun_path) to 260 (MAX_PATH).
Bruce Cran [Tue, 15 Apr 2014 00:33:22 +0000 (18:33 -0600)]
Windows: set sizeof(sun_path) to 260 (MAX_PATH).

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoinit: improve command line check for when to print usage/help
Jens Axboe [Mon, 14 Apr 2014 19:37:46 +0000 (13:37 -0600)]
init: improve command line check for when to print usage/help

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoinit: fix leak bug fix from commit 60c5ec73
Jens Axboe [Mon, 14 Apr 2014 19:29:52 +0000 (13:29 -0600)]
init: fix leak bug fix from commit 60c5ec73

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agodisk util: fix bug in strncpy() conversion
Jens Axboe [Mon, 14 Apr 2014 19:25:12 +0000 (13:25 -0600)]
disk util: fix bug in strncpy() conversion

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix potential NULL dereference in alloc_mem_mmap()
Jens Axboe [Mon, 14 Apr 2014 19:20:38 +0000 (13:20 -0600)]
Fix potential NULL dereference in alloc_mem_mmap()

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agodiskutil: make it clear that parent != NULL for json output
Jens Axboe [Mon, 14 Apr 2014 19:19:13 +0000 (13:19 -0600)]
diskutil: make it clear that parent != NULL for json output

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoparse: make it clear that 'cp' can't be NULL for o->off1
Jens Axboe [Mon, 14 Apr 2014 19:16:10 +0000 (13:16 -0600)]
parse: make it clear that 'cp' can't be NULL for o->off1

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoFix is_power_of_2() for longs
Jens Axboe [Mon, 14 Apr 2014 19:13:21 +0000 (13:13 -0600)]
Fix is_power_of_2() for longs

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoidletime: unlock ->start_lock in error cases
Jens Axboe [Mon, 14 Apr 2014 19:11:19 +0000 (13:11 -0600)]
idletime: unlock ->start_lock in error cases

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoAvoid overwriting allocated pid_file
Jens Axboe [Mon, 14 Apr 2014 19:09:04 +0000 (13:09 -0600)]
Avoid overwriting allocated pid_file

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agogettime: init 'failed' before use
Jens Axboe [Mon, 14 Apr 2014 18:20:04 +0000 (12:20 -0600)]
gettime: init 'failed' before use

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agostat: warn on failure to detach DU thread
Jens Axboe [Mon, 14 Apr 2014 18:15:35 +0000 (12:15 -0600)]
stat: warn on failure to detach DU thread

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agogettime: handle pthread_create() failure
Jens Axboe [Mon, 14 Apr 2014 18:14:17 +0000 (12:14 -0600)]
gettime: handle pthread_create() failure

Signed-off-by: Jens Axboe <axboe@fb.com>
6 years agoinit: silence bogus td == NULL warning
Jens Axboe [Mon, 14 Apr 2014 18:07:36 +0000 (12:07 -0600)]
init: silence bogus td == NULL warning

Signed-off-by: Jens Axboe <axboe@fb.com>