fio.git
4 years agoFio 2.1.13 fio-2.1.13
Jens Axboe [Fri, 3 Oct 2014 19:19:04 +0000 (13:19 -0600)]
Fio 2.1.13

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/libaio: modify getevents and commit error handling
Jens Axboe [Fri, 3 Oct 2014 19:10:23 +0000 (13:10 -0600)]
engines/libaio: modify getevents and commit error handling

If we get some error other than -EAGAIN in getevents(), treat
it as fatal and break.

If we get -ENOMEM on submission, reap events if we can. If we
have no pending events, treat it as a fatal error and break.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/libaio: reset 'wait_start' on non-EAGAIN
Jens Axboe [Wed, 1 Oct 2014 14:48:54 +0000 (08:48 -0600)]
engines/libaio: reset 'wait_start' on non-EAGAIN

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoBump IO engine version number
Jens Axboe [Wed, 1 Oct 2014 14:43:58 +0000 (08:43 -0600)]
Bump IO engine version number

Commit 1f440ece6127 modified the ->getevents() hook to have
a const timespec, so we should increment the engine version
number to catch that in external engines.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/libaio: fix submit loop to use 'ld->queued' for exit
Jens Axboe [Wed, 1 Oct 2014 14:42:27 +0000 (08:42 -0600)]
engines/libaio: fix submit loop to use 'ld->queued' for exit

We could have head == tail for a full ring, so use the
amount queued as the gauge instead. This would be very
hard to trigger, as it would require a full ring on entry
and the first submit would have to fail to submit anything
before we loop around.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/libaio: don't reap on EAGAIN and no pending events
Jens Axboe [Wed, 1 Oct 2014 02:28:45 +0000 (20:28 -0600)]
engines/libaio: don't reap on EAGAIN and no pending events

Instead just loop on submit, since there are no events for us
to reap. This is usually a kernel bug, violating the principle
of forward progress guarantee.

If we can't submit anything in 30 seconds, error out.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoConstify a few more hot paths
Jens Axboe [Tue, 30 Sep 2014 22:04:12 +0000 (16:04 -0600)]
Constify a few more hot paths

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/libaio: fix issue with EAGAIN
Jens Axboe [Tue, 30 Sep 2014 19:33:23 +0000 (13:33 -0600)]
engines/libaio: fix issue with EAGAIN

If we get -EAGAIN on IO submission, we should back off and reap
some events. Currently we just loop trying to resubmit, and this
has been reported to cause hangs in some virtualized environments.
There's no record of this ever being hit on raw metal.

So implement this backoff. Unfortunately it requires rewriting
the io_u tracking in libaio, since otherwise we depend on being
able to submit all the io_u's we are handed. So change the
tracking to be a ring buffer, with submission bumping the head
and completion bumping the tail. With this in place, we can't
stop issuing IO at anytime, since we'll just continue where
we left off.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoConstify 'td' in a few functions
Jens Axboe [Tue, 30 Sep 2014 19:29:57 +0000 (13:29 -0600)]
Constify 'td' in a few functions

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUse calloc() instead of malloc + memset
Jens Axboe [Tue, 30 Sep 2014 16:08:08 +0000 (10:08 -0600)]
Use calloc() instead of malloc + memset

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoHOWTO: fix typo
ordahan [Tue, 30 Sep 2014 14:18:34 +0000 (08:18 -0600)]
HOWTO: fix typo

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofilesetup: fix a bug where we overwrite the set size
Jens Axboe [Tue, 30 Sep 2014 03:23:51 +0000 (21:23 -0600)]
filesetup: fix a bug where we overwrite the set size

For the null io engine, we don't set file sizes. And we should
not overwrite a potentially legitimate o->size, if total_size
is zero.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix compile of t/btrace2fio on RHEL5/CentOS5
Jens Axboe [Mon, 29 Sep 2014 22:07:54 +0000 (16:07 -0600)]
Fix compile of t/btrace2fio on RHEL5/CentOS5

Get these errors:

t/btrace2fio.c:395: undefined reference to `__bswap_32'
t/btrace2fio.c:396: undefined reference to `__bswap_32'
t/btrace2fio.c:397: undefined reference to `__bswap_64'
t/btrace2fio.c:398: undefined reference to `__bswap_64'
t/btrace2fio.c:399: undefined reference to `__bswap_32'
t/btrace2fio.c:400: undefined reference to `__bswap_32'
t/btrace2fio.c:401: undefined reference to `__bswap_32'
t/btrace2fio.c:402: undefined reference to `__bswap_32'
t/btrace2fio.c:403: undefined reference to `__bswap_32'
t/btrace2fio.c:404: undefined reference to `__bswap_16'
t/btrace2fio.c:405: undefined reference to `__bswap_16'

Issue is that os/os-linux.h needs to include byteswap.h.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/dedupe: silence 'some_done' unused warning on some compilers
Jens Axboe [Mon, 29 Sep 2014 01:10:24 +0000 (19:10 -0600)]
t/dedupe: silence 'some_done' unused warning on some compilers

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix typo in bs_is_seq_ran option help
Jens Axboe [Sun, 28 Sep 2014 22:25:49 +0000 (16:25 -0600)]
Fix typo in bs_is_seq_ran option help

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoClarify what number_ios does in the HOWTO/man page/help
Jens Axboe [Sun, 28 Sep 2014 22:24:23 +0000 (16:24 -0600)]
Clarify what number_ios does in the HOWTO/man page/help

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix bug with zone and zone skipping and io_limit
Jens Axboe [Sun, 28 Sep 2014 22:20:58 +0000 (16:20 -0600)]
Fix bug with zone and zone skipping and io_limit

If you do:

dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write --filename=/dev/shm/1M --name=go \
--zoneskip=4k --zonesize=4k --io_limit=2M

then fio will exit after having performed 512KB of IO, instead
of the specified IO limit. This is a similar issue as was fixed
by commit ac002339c382 with rw sequencing - if we reach the
end of the file with a zone skip, wrap to the beginning.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix bug with rw sequence offset and io_limit
Jens Axboe [Sun, 28 Sep 2014 22:18:43 +0000 (16:18 -0600)]
Fix bug with rw sequence offset and io_limit

If you do:

dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --name=go --io_limit=2M

then fio will exit prematurely after having done 128 4KB ios,
so only 512KB are read. This is because after 128 ios, we have
reache the end of the file. Handle this case and wrap around
properly.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/dedupe: fixup bloom entry calculation
Jens Axboe [Sun, 28 Sep 2014 03:29:03 +0000 (21:29 -0600)]
t/dedupe: fixup bloom entry calculation

With m/n at 8x and with 5 default hashes for blooming, we should
be around ~2% false positive rate max.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobloom: up hashes to 5 by default
Jens Axboe [Sun, 28 Sep 2014 03:28:47 +0000 (21:28 -0600)]
bloom: up hashes to 5 by default

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agocrc/test: add fnv
Jens Axboe [Sun, 28 Sep 2014 03:27:48 +0000 (21:27 -0600)]
crc/test: add fnv

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd fnv hash
Jens Axboe [Sun, 28 Sep 2014 03:26:58 +0000 (21:26 -0600)]
Add fnv hash

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agocrc/test: if the checksum needs a _final(), do it in the loop
Jens Axboe [Sat, 27 Sep 2014 20:13:58 +0000 (14:13 -0600)]
crc/test: if the checksum needs a _final(), do it in the loop

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/dedupe: print dedupe ratio instead of made-up factor
Jens Axboe [Sat, 27 Sep 2014 20:12:02 +0000 (14:12 -0600)]
t/dedupe: print dedupe ratio instead of made-up factor

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMove murmur3 hash to crc/
Jens Axboe [Sat, 27 Sep 2014 15:30:56 +0000 (09:30 -0600)]
Move murmur3 hash to crc/

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agocrc/test: add jhash
Jens Axboe [Sat, 27 Sep 2014 15:29:42 +0000 (09:29 -0600)]
crc/test: add jhash

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agocrc/test: fix alignment
Jens Axboe [Sat, 27 Sep 2014 14:43:27 +0000 (08:43 -0600)]
crc/test: fix alignment

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd murmurhash3
Jens Axboe [Sat, 27 Sep 2014 14:38:42 +0000 (08:38 -0600)]
Add murmurhash3

And use it in bloom, add it to the crc tester as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: exit gracefully if device/file open fails
Jens Axboe [Sat, 27 Sep 2014 14:08:24 +0000 (08:08 -0600)]
dedupe: exit gracefully if device/file open fails

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoparse: fix issue with not filling leftover percentages correctly
Jens Axboe [Sat, 27 Sep 2014 14:08:00 +0000 (08:08 -0600)]
parse: fix issue with not filling leftover percentages correctly

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoImprove dedupe/compression buffer filling for mixed block sizes
Jens Axboe [Fri, 26 Sep 2014 21:04:58 +0000 (15:04 -0600)]
Improve dedupe/compression buffer filling for mixed block sizes

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobloom: use independent hashes
Jens Axboe [Fri, 26 Sep 2014 20:57:41 +0000 (14:57 -0600)]
bloom: use independent hashes

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoClarify that include files may not contain job sections
Jens Axboe [Fri, 26 Sep 2014 19:45:56 +0000 (13:45 -0600)]
Clarify that include files may not contain job sections

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobloom: always use a larger minimum size for bloom filter
Jens Axboe [Fri, 26 Sep 2014 19:30:00 +0000 (13:30 -0600)]
bloom: always use a larger minimum size for bloom filter

Increases accuracy at the lower end.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd HOWTO section on include files
Andrey Kuzmin [Fri, 26 Sep 2014 19:29:15 +0000 (13:29 -0600)]
Add HOWTO section on include files

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd ability to use an include file in a fio job file
Andrey Kuzmin [Fri, 26 Sep 2014 18:48:40 +0000 (12:48 -0600)]
Add ability to use an include file in a fio job file

With this patch, we can use the 'include' directive to tell
fio to include the contents of a file inside a job file.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: read in larger chunks at the time
Jens Axboe [Fri, 26 Sep 2014 18:47:14 +0000 (12:47 -0600)]
dedupe: read in larger chunks at the time

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: fix warning and segfault on -B0
Jens Axboe [Fri, 26 Sep 2014 16:16:53 +0000 (10:16 -0600)]
dedupe: fix warning and segfault on -B0

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/lfsr-test: fixup time
Jens Axboe [Fri, 26 Sep 2014 16:01:50 +0000 (10:01 -0600)]
t/lfsr-test: fixup time

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd debug helper stub for t/ programs
Jens Axboe [Fri, 26 Sep 2014 15:58:29 +0000 (09:58 -0600)]
Add debug helper stub for t/ programs

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: default to using a bloom filter to save memory
Jens Axboe [Fri, 26 Sep 2014 15:51:33 +0000 (09:51 -0600)]
dedupe: default to using a bloom filter to save memory

With the bloom filter, we can comfortably check devices up to
the petabyte range, which isn't feasible if we track each
extent.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd bloom filter
Jens Axboe [Fri, 26 Sep 2014 15:51:16 +0000 (09:51 -0600)]
Add bloom filter

Very basic, but seems to do the job.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix compile for FIO_INC_DEBUG not set
Jens Axboe [Wed, 24 Sep 2014 15:54:24 +0000 (09:54 -0600)]
Fix compile for FIO_INC_DEBUG not set

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: print threads used up front
Jens Axboe [Wed, 24 Sep 2014 15:34:47 +0000 (09:34 -0600)]
dedupe: print threads used up front

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: remove rb_lock when done
Jens Axboe [Wed, 24 Sep 2014 15:33:29 +0000 (09:33 -0600)]
dedupe: remove rb_lock when done

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: cleanups and rate estimation
Jens Axboe [Wed, 24 Sep 2014 15:30:52 +0000 (09:30 -0600)]
dedupe: cleanups and rate estimation

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Wed, 24 Sep 2014 14:21:47 +0000 (08:21 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

4 years agodedupe: don't add extent items unless we use them
Jens Axboe [Wed, 24 Sep 2014 14:21:15 +0000 (08:21 -0600)]
dedupe: don't add extent items unless we use them

Only used for collision_check and debug output, so save memory
if those aren't enabled.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/dedupe: Linux only for now
Jens Axboe [Wed, 24 Sep 2014 00:36:52 +0000 (18:36 -0600)]
t/dedupe: Linux only for now

Just need to fix up the file sizing, that's the only Linux
dependency.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix 32-bit compile warnings
Jens Axboe [Tue, 23 Sep 2014 22:06:04 +0000 (16:06 -0600)]
Fix 32-bit compile warnings

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoBuild t/ tools by default
Jens Axboe [Tue, 23 Sep 2014 21:53:32 +0000 (15:53 -0600)]
Build t/ tools by default

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: print progress indicator
Jens Axboe [Tue, 23 Sep 2014 20:40:48 +0000 (14:40 -0600)]
dedupe: print progress indicator

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: improve wording in output
Jens Axboe [Tue, 23 Sep 2014 20:29:21 +0000 (14:29 -0600)]
dedupe: improve wording in output

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd small tool to check for dedupable contents in a file/device
Jens Axboe [Tue, 23 Sep 2014 20:10:35 +0000 (14:10 -0600)]
Add small tool to check for dedupable contents in a file/device

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoChecksumming updates
Jens Axboe [Tue, 23 Sep 2014 20:08:21 +0000 (14:08 -0600)]
Checksumming updates

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agomemalign: fix off-by-one bug in alignment
Jens Axboe [Tue, 23 Sep 2014 16:59:50 +0000 (10:59 -0600)]
memalign: fix off-by-one bug in alignment

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix min/max typeof warnings
Jens Axboe [Tue, 23 Sep 2014 16:38:54 +0000 (10:38 -0600)]
Fix min/max typeof warnings

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofifo: use minmax.h instead of rolling its own min/max
Jens Axboe [Tue, 23 Sep 2014 16:35:49 +0000 (10:35 -0600)]
fifo: use minmax.h instead of rolling its own min/max

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodedupe: if percentage is 100, don't go through random + math
Jens Axboe [Mon, 22 Sep 2014 20:20:05 +0000 (14:20 -0600)]
dedupe: if percentage is 100, don't go through random + math

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoBasic support for dedupe
Jens Axboe [Mon, 22 Sep 2014 16:02:07 +0000 (10:02 -0600)]
Basic support for dedupe

This adds and option, dedupe_percentage, that controls how many of
the write IO buffers are identical. For instance, if this is set:

dedupe_percentage=70

then 70% of the write IO buffers will have identical contents. The
specific contents are, as before, controlled by the various options
that set buffer contents or buffer compressibility.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobtrace2fio: expand rate options, bug fixes
Jens Axboe [Thu, 18 Sep 2014 15:23:55 +0000 (17:23 +0200)]
btrace2fio: expand rate options, bug fixes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 years agobtrace2fio: add rate output
Jens Axboe [Wed, 17 Sep 2014 15:58:31 +0000 (17:58 +0200)]
btrace2fio: add rate output

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobtrace2fio: move file tracking to btrace_pid
Jens Axboe [Wed, 17 Sep 2014 02:17:55 +0000 (20:17 -0600)]
btrace2fio: move file tracking to btrace_pid

We want to keep btrace_out as per-device, potentially. So move
this part since it's per process.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofio: print io_u errors on one line
Robert Elliott [Tue, 16 Sep 2014 22:09:48 +0000 (17:09 -0500)]
fio: print io_u errors on one line

Print io_u errors on one line and with one log_err command
to keep them from being interrupted by other prints

Old:
fio: io_u error on file /dev/sdt: Input/output error5.1MB/0KB/0KB /s]
[227K/0/0 iops] [eta 02d:10h:53m:36s]
     read offset=145442430976, buflen=4096
fio: io_u error on file /dev/sdt: Input/output error
     read offset=397760724992, buflen=4096

New:
fio: io_u error on file /dev/sdr: Input/output error: write offset=1532626153472, buflen=4096
fio: io_u error on file /dev/sdt: Input/output error: write offset=102684229632, buflen=4096

Signed-off-by: Robert Elliott <elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFixup data-direction to name translation
Jens Axboe [Wed, 17 Sep 2014 02:07:59 +0000 (20:07 -0600)]
Fixup data-direction to name translation

We had two of them, and they were both wrong since trim got
introduced. Replace by a single implementation in io_ddir.h,
right where it's defined.

Reported-by: Elliott, Robert <Elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobtrace2fio: cleanups and inflight hashing
Jens Axboe [Tue, 16 Sep 2014 16:28:42 +0000 (18:28 +0200)]
btrace2fio: cleanups and inflight hashing

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 years agobtrace2fio: use percentage_random and add filename option
Jens Axboe [Tue, 16 Sep 2014 15:40:03 +0000 (17:40 +0200)]
btrace2fio: use percentage_random and add filename option

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 years agobtrace2fio: set runtime
Jens Axboe [Tue, 16 Sep 2014 15:35:55 +0000 (17:35 +0200)]
btrace2fio: set runtime

Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Tue, 16 Sep 2014 15:07:51 +0000 (17:07 +0200)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

4 years agoAdd basic tool to turn blktrace into fio job file
Jens Axboe [Tue, 16 Sep 2014 15:06:52 +0000 (17:06 +0200)]
Add basic tool to turn blktrace into fio job file

No documentation yet, it's just a start.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agooptions: fix single use if / for bssplit
Jens Axboe [Tue, 16 Sep 2014 00:51:32 +0000 (18:51 -0600)]
options: fix single use if / for bssplit

It was buggy in that it did not extend to 100%, if nobody else
was part of the group.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.1.12 fio-2.1.12
Jens Axboe [Fri, 12 Sep 2014 18:09:32 +0000 (12:09 -0600)]
Fio 2.1.12

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoserver: fix wrong use of exit() in server when daemonized
Jens Axboe [Thu, 11 Sep 2014 23:05:58 +0000 (17:05 -0600)]
server: fix wrong use of exit() in server when daemonized

We can't use exit(), or we'll trigger the atexit() hook and
free the shared mem used for mutexes, etc. This can cause a
hang when using a daemonized fio server.

Reported-by: Castor Fu <castor@egocast.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix hangs due to iodepth_low
Robert Elliott [Thu, 4 Sep 2014 19:51:05 +0000 (13:51 -0600)]
Fix hangs due to iodepth_low

With some combinations of iodepth, iodepth_batch, iodepth_batch_complete,
and io_depth_low, do_io hangs after reaping the first set of completions
since io_u_queued_complete is called requesting more completions than
td->cur_depth.

Example printing min_evts and td->cur_depth in the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s]
<fio never exits>

Fix this by adjusting min_evts to the current_depth if that is smaller.

Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15
time_based

Modified by Jens to fix the check, and move it into
io_u_queued_complete() instead of at the caller site.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix crash in client with NULL hostname
Castor Fu [Fri, 22 Aug 2014 22:16:44 +0000 (17:16 -0500)]
Fix crash in client with NULL hostname

Fix this by adding an empty hostname.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd string support for buffer_pattern
Jens Axboe [Fri, 22 Aug 2014 19:02:02 +0000 (14:02 -0500)]
Add string support for buffer_pattern

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agooptions: warn if we fail to make sense of buffer_pattern
Jens Axboe [Fri, 22 Aug 2014 15:22:03 +0000 (10:22 -0500)]
options: warn if we fail to make sense of buffer_pattern

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoUpdate io engine version
Jens Axboe [Tue, 19 Aug 2014 20:16:29 +0000 (15:16 -0500)]
Update io engine version

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd unlink hook to ioengine API, gluster ioengine
Castor Fu [Tue, 19 Aug 2014 16:28:53 +0000 (09:28 -0700)]
Add unlink hook to ioengine API, gluster ioengine

fio would just call unlink even with engines that are not using the
operating systems file namespace... This provides a hook to allow
overriding that, with a default handler, and implements it for the
gluster ioengine.

There are others which it'd probably make sense I'm sure.

Huamin Chen looked over my changes to the gluster code earlier...
>I like this unlink idea, it would be great if you can also make unlink optional (if my coding reading is correct). This looks a great pull request candidate to fio. Please ping Axboe after you are done. He is not actively watching pull requests.

>Please also feel free to augment gluster code and pull me for review if necessary.

-castor

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofix re-open failure issue when using gluster ioengine
rootfs [Tue, 19 Aug 2014 18:42:45 +0000 (14:42 -0400)]
fix re-open failure issue when using gluster ioengine

Signed-off-by: rootfs <hchen@redhat.com>
5 years agoUpdate libhdfs engine documention and options
Manish Mandlik [Thu, 14 Aug 2014 17:45:16 +0000 (11:45 -0600)]
Update libhdfs engine documention and options

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd support for HDFS IO engine
Manish Mandlik [Wed, 13 Aug 2014 19:36:52 +0000 (13:36 -0600)]
Add support for HDFS IO engine

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd missing IO engines to the HOWTO
Jens Axboe [Wed, 13 Aug 2014 19:35:37 +0000 (13:35 -0600)]
Add missing IO engines to the HOWTO

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: turn off numberio verification for meta and time_based
Jens Axboe [Thu, 7 Aug 2014 21:27:31 +0000 (15:27 -0600)]
verify: turn off numberio verification for meta and time_based

Will only work for the first iteration, then fail miserably.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agocrc/test: a few more cleanups and code unifications
Jens Axboe [Fri, 25 Jul 2014 13:15:46 +0000 (15:15 +0200)]
crc/test: a few more cleanups and code unifications

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agocrc/test: cleanup and better precision
Jens Axboe [Fri, 25 Jul 2014 08:12:26 +0000 (10:12 +0200)]
crc/test: cleanup and better precision

Make sure we're a bit cache warm at least, and don't allocate, free,
and randomize a buffer for each hash.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAllow reset of offset_increment counter
Jiri Horky [Fri, 25 Jul 2014 07:55:03 +0000 (09:55 +0200)]
Allow reset of offset_increment counter

Reset offset_increment only within a sub-group of jobs, eg when
numjobs=x is used to automatically group jobs together.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoDon't grab stat mutex for final stat output
Jens Axboe [Fri, 25 Jul 2014 07:51:56 +0000 (09:51 +0200)]
Don't grab stat mutex for final stat output

We should not need it, and there's a bug with getting stuck on it
that hasn't been fully debugged/understood yet. So for now, lets
just forget about the mutex for final output.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd test case for previous verify crash
Jens Axboe [Thu, 24 Jul 2014 12:50:36 +0000 (14:50 +0200)]
Add test case for previous verify crash

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agonull: add FIO_FAKEIO flag
Jens Axboe [Wed, 23 Jul 2014 14:19:48 +0000 (16:19 +0200)]
null: add FIO_FAKEIO flag

Tell the backend that null just fakes IO, and use this knowledge
to have verifies automagically work when using null. This allows
testing of a wider range of paths with ioengine=null.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: fix a bug with verify_async
Jens Axboe [Wed, 23 Jul 2014 14:11:43 +0000 (16:11 +0200)]
verify: fix a bug with verify_async

There's a race between marking an io_u as deferred, completing it,
and checking of that flag. It cannot be done reliably for async
verify threads. So change the mechanism to have the verify_async
part pull the io_u completely, so we don't have to check for a flag
in it after we have run ->end_io().

This fixes a bug with verify_async, where fio would crash with
this message:

fio: io_u.c:1315: __get_io_u: Assertion `io_u->flags & IO_U_F_FREE' failed

This race has always existed, but was made considerably worse with
this commit:

commit 2ae0b204743d6b4048c6fffd46c6280a70f2ecd1
Author: Jens Axboe <axboe@kernel.dk>
Date:   Tue May 28 14:16:55 2013 +0200

    Replace list based free/busy/requeue list with FIFO + ring

    Cache friendliness of the list is pretty low. This has
    provably lower overhead.

since we moved from a single list holding the io, to a separate
verify list and io_u queues. The above bug is happening because
the io_u ends up on both the freelist and the pending verify list,
causing mayhem.

Reported-by: scameron@beardog.cce.hp.com
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: ensure that verify interval is smaller or equal to blocksize
Jens Axboe [Wed, 23 Jul 2014 07:47:26 +0000 (09:47 +0200)]
verify: ensure that verify interval is smaller or equal to blocksize

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agostat: move stat mutex grab to thread
Jens Axboe [Mon, 21 Jul 2014 09:32:13 +0000 (11:32 +0200)]
stat: move stat mutex grab to thread

No need to grab it from the parent.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: disable numberio check for multiple block sizes
Jens Axboe [Mon, 21 Jul 2014 09:31:49 +0000 (11:31 +0200)]
verify: disable numberio check for multiple block sizes

Yet another corner case that doesn't work...

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.1.11 fio-2.1.11
Jens Axboe [Wed, 16 Jul 2014 08:22:43 +0000 (10:22 +0200)]
Fio 2.1.11

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoblktrace: various bug fixes and improvements
Jens Axboe [Mon, 14 Jul 2014 13:07:11 +0000 (15:07 +0200)]
blktrace: various bug fixes and improvements

- Use the normal debug logging for notify events, as they are expected
  to happen.

- Correct a bug where a workload is considered mixed, even if it is
  only reads (note: just cosmetical, no writes were issued by fio).

- Handle trim more appropriately - fio supports it as a separate
  'data direction', so handle it as such.

- Fix a bug in the delay accounting, causing blktrace to not replay
  correctly. This could cause either too fast replays, or replays
  that would delay way too much.

- Consider TRIMs as writes, as far as --readonly being set is
  concerned.

- Don't consider < 100 usec delays noise, fio can do this quite
  accurately.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd some basic debug output for log compression
Jens Axboe [Wed, 9 Jul 2014 21:01:03 +0000 (23:01 +0200)]
Add some basic debug output for log compression

Guarded by --debug=compress, like the other debug measures.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix two error paths
Andreas Gruenbacher [Wed, 9 Jul 2014 17:37:33 +0000 (19:37 +0200)]
Fix two error paths

In generic_open_file(), when an error is detected, return 1 to indicate
the failure.

In get_file_sizes(), when an error other than ENOENT is detected, fail
immediately and don't clear the error.

5 years agoFix cases where td->terminate is set, but terminate clock not marked
Jens Axboe [Wed, 9 Jul 2014 09:24:12 +0000 (11:24 +0200)]
Fix cases where td->terminate is set, but terminate clock not marked

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd thread number to log filename
Jens Axboe [Wed, 9 Jul 2014 08:31:34 +0000 (10:31 +0200)]
Add thread number to log filename

Since commit aee2ab6775d9, we open the log files for writing and
truncate the output. This breaks when you have multiple jobs with
the same name, as will happen with numjobs=x. Instead of reverting
this part of the commit, lets keep the truncate open and instead
store log files separately. This makes more sense anyway, since
otherwise the logs are all the entries from job X, then all the
entries from job Y, etc. It's easy enough to collate the logs,
should anyone want to do that.

Signed-off-by: Jens Axboe <axboe@fb.com>