fio.git
8 years agolibmtd: ->name and ->type_str can't be constant
Jens Axboe [Tue, 5 May 2015 18:11:55 +0000 (12:11 -0600)]
libmtd: ->name and ->type_str can't be constant

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoARM: Use generic assembly nop and barrier code for armv8-a
Milton Chiang [Tue, 5 May 2015 09:18:17 +0000 (17:18 +0800)]
ARM: Use generic assembly nop and barrier code for armv8-a

Signed-off-by: Milton Chiang <milton.chiang@mediatek.com>
8 years agoFix Runtime, IOPS, bandwidth recorded incorrectly
Brian Fulton [Mon, 4 May 2015 22:07:10 +0000 (16:07 -0600)]
Fix Runtime, IOPS, bandwidth recorded incorrectly

This happens with small sized timed_based runs. Change runtime
tabulation back to microseconds to handle sub millisecond loops.

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agolfsr-test: print total elapsed time correctly
Steven Noonan [Fri, 1 May 2015 19:49:34 +0000 (19:49 +0000)]
lfsr-test: print total elapsed time correctly

The conversion from microseconds to seconds is 1e6 not 1e9.

Signed-off-by: Steven Noonan <steven@uplinklabs.net>
Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agogettime: add support for CLOCK_MONOTONIC_RAW
Steven Noonan [Fri, 1 May 2015 19:31:44 +0000 (12:31 -0700)]
gettime: add support for CLOCK_MONOTONIC_RAW

The clock CLOCK_MONOTONIC_RAW (introduced in Linux 2.6.28) is similar to
CLOCK_MONOTONIC except that it is not subject to NTP adjustments or incremental
adjustments performed by adjtime().

Signed-off-by: Steven Noonan <steven@uplinklabs.net>
Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoblktrace: support for non-512b sector sizes
Jens Axboe [Fri, 1 May 2015 23:02:24 +0000 (17:02 -0600)]
blktrace: support for non-512b sector sizes

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoconfigure: add support for --prefix
Jens Axboe [Wed, 29 Apr 2015 16:29:16 +0000 (10:29 -0600)]
configure: add support for --prefix

Allows other installation paths that /usr/local

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoblktrace: only set O_DIRECT if the min bs is a multiple of 4k
Jens Axboe [Tue, 28 Apr 2015 21:30:02 +0000 (15:30 -0600)]
blktrace: only set O_DIRECT if the min bs is a multiple of 4k

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoAdd recommendation to set direct=0 if first O_DIRECT fails
Jens Axboe [Tue, 28 Apr 2015 21:23:21 +0000 (15:23 -0600)]
Add recommendation to set direct=0 if first O_DIRECT fails

Fio currently tells you:

"fio: first direct IO errored. File system may not support direct IO, or
 iomem_align= is bad."

but it doesn't offer a remedy. Add a suggestion to set direct=0 if this
happens. It can happen because of a badly configured job, or from
blktrace replay because the latter adds direct=1 automatically. But that
may fail, if the trace contains IO that isn't sector aligned.

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agodedupe: improve 100% dedupe case
Jens Axboe [Mon, 27 Apr 2015 15:37:11 +0000 (09:37 -0600)]
dedupe: improve 100% dedupe case

We don't need an on-stack state, just juggle the buf_state and
buf_state_prev appropriately.

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agodedupe: fix dedupe_percentage=100
Jens Axboe [Mon, 27 Apr 2015 14:38:33 +0000 (08:38 -0600)]
dedupe: fix dedupe_percentage=100

The corner case of 100% dedupable was buggy, save and restore random
state appropriate for that too.

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoio_u: fix bug in rounding of generated buffer length
Jens Axboe [Fri, 24 Apr 2015 16:41:58 +0000 (10:41 -0600)]
io_u: fix bug in rounding of generated buffer length

If the maximum blocksize isn't a multiple of the minimum blocksize,
then fio has a bug where it will round up the block size and align
it to a size larger than the IO buffer we have. This causes random
memory corruption and crashes.

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoblktrace: fix bugs in accounting
Jens Axboe [Fri, 24 Apr 2015 15:03:05 +0000 (09:03 -0600)]
blktrace: fix bugs in accounting

We need to clear the trim part of the accounting arrays, otherwise
we can use bogus values leading to attempting to allocate huge
amounts of memory. This causes fio to exit with:

fio: pid=21428, err=12/file:memory.c:242, func=iomem allocation,
error=Cannot allocate memory

Also improve the depth detection, by making it per data direction
instead of a global state.

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoMerge branch 'io-threads'
Jens Axboe [Thu, 23 Apr 2015 17:13:27 +0000 (11:13 -0600)]
Merge branch 'io-threads'

8 years agoblktrace: only probe and set depth if option isn't set
Jens Axboe [Thu, 23 Apr 2015 01:54:54 +0000 (19:54 -0600)]
blktrace: only probe and set depth if option isn't set

The comment predates fio's support for checking whether a specific
option was set, we have the power to check this properly now.

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoconfigure: check for more of the used MTD features
Jens Axboe [Wed, 22 Apr 2015 19:46:45 +0000 (13:46 -0600)]
configure: check for more of the used MTD features

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoFix off-by-one in cpus_allowed
Jens Axboe [Tue, 21 Apr 2015 15:39:32 +0000 (09:39 -0600)]
Fix off-by-one in cpus_allowed

Similar to commit b84113993b3a, fix it for cpus_allowed as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoFix off-by-one in cpu mask index handling
Jens Axboe [Tue, 21 Apr 2015 14:25:58 +0000 (08:25 -0600)]
Fix off-by-one in cpu mask index handling

This just affects how errors are printed. Kleber Sacilotto de Souza
reports:

The error message displayed by the option parser for the 'cpus_allowed'
parameter is misleading.

My system has 4 processors:

$ grep -c processor /proc/cpuinfo
4

If I provide a high number for cpus_allowed, I get the error message:

$ fio --filename=/tmp/foo.fio --cpus_allowed=5 --name=job1
fio: CPU 5 too large (max=4)
fio: failed parsing cpus_allowed=5

If it says "(max=4)", I would expect it to accept the value "4" if I
want to use the last CPU of the system (even though we know that the
CPUs are generally numbered starting with 0), but that's not what
happens:

$ fio --name=global --filename=/tmp/foo.fio --cpus_allowed=4 --name=job1
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.1.3
Starting 1 process
fio: pid=24942, err=22/file:backend.c:1192, func=cpu_set_affinity, error=Invalid argument

So for setting the affinity the CPU number really starts from 0, so 3
would be the right value in this case and the following command works as
expected:

Signed-off-by: Jens Axboe <axboe@fb.com>
8 years agoDon't export state variable
Dan Ehrenberg [Mon, 20 Apr 2015 23:20:30 +0000 (16:20 -0700)]
Don't export state variable

There was no reason to export this extra variable. It was just a typo
and it is never used.

Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFix mtd valgrind warning
Dan Ehrenberg [Thu, 16 Apr 2015 21:50:39 +0000 (14:50 -0700)]
Fix mtd valgrind warning

strlen seems to operate in 32-bit chunks, but libmtd sometimes allocates
strings without that much padding. This patch adds something extra to
a string allocation so that the mtd ioengine is clean with respect
to valgrind memory access checks.

Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFix wrong index bug
Jens Axboe [Thu, 16 Apr 2015 03:47:47 +0000 (21:47 -0600)]
Fix wrong index bug

Commit 0e4dd95c548cc re-uses 'i' as an iterator, causing us to
overrun the nr_ts allocated. This subsequently causes fio to
segfault.

Fixes: 0e4dd95c548cc
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAdd sample job file for fixed submission rate
Jens Axboe [Wed, 15 Apr 2015 16:01:56 +0000 (10:01 -0600)]
Add sample job file for fixed submission rate

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAdd man page and HOWTO for io_submit_mode option
Jens Axboe [Wed, 15 Apr 2015 15:58:58 +0000 (09:58 -0600)]
Add man page and HOWTO for io_submit_mode option

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agolibaio: don't call io_destroy(), let exit_aio() take care of it
Jens Axboe [Thu, 19 Mar 2015 20:52:54 +0000 (14:52 -0600)]
libaio: don't call io_destroy(), let exit_aio() take care of it

From the exit_aio() path, we can parallellize the freeing, hence
not taking an RCU grace period hit for each.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFirst cut at supporting IO offload
Jens Axboe [Fri, 20 Mar 2015 04:50:06 +0000 (22:50 -0600)]
First cut at supporting IO offload

rate_iops=x
io_submit_mode=offload

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agobackend: split queue io_u event handling into helper
Jens Axboe [Fri, 20 Mar 2015 04:44:05 +0000 (22:44 -0600)]
backend: split queue io_u event handling into helper

io_queue_event(), can now be used from both verify and IO path.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAdd ->bytes_done[] to struct thread_data
Jens Axboe [Fri, 20 Mar 2015 04:32:00 +0000 (22:32 -0600)]
Add ->bytes_done[] to struct thread_data

We can't keep it on the stack for async IO offload.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoChange writetrim to trimwrite
Jens Axboe [Wed, 15 Apr 2015 15:49:20 +0000 (09:49 -0600)]
Change writetrim to trimwrite

This better explains what it does. Also add HOWTO and man page entries
for trimwrite.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agomtd: example job file
Dan Ehrenberg [Tue, 14 Apr 2015 22:58:20 +0000 (15:58 -0700)]
mtd: example job file

This job file shows how one can write sequentially to an MTD device,
erase (with trim), and do a looped write/trim workload. Ignoring EIO
errors allows operations to continue as bad blocks are skipped.
The block histogram counts P/E cycles and can be used to assess
NAND flash lifetime and failure characteristics. The job file is
written to work with the nandsim fake NAND device.

Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agomtd: ioengine
Dan Ehrenberg [Tue, 14 Apr 2015 22:58:19 +0000 (15:58 -0700)]
mtd: ioengine

This patch adds an MTD ioengine. Trims are interpreted as MTD erases.
A skip_bad option calls out to the kernel to ask if the block being
operated on is bad first; if it is bad, the operation is skipped
and -EIO is returned.

Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agomtd: Import libmtd with modifications to make it compile
Dan Ehrenberg [Tue, 14 Apr 2015 22:58:18 +0000 (15:58 -0700)]
mtd: Import libmtd with modifications to make it compile

This patch imports files from mtdutils to access MTD devices. mtdutils
is licensed as GPLv2, like fio, so it can be included here. Files are
changed slightly in how they import headers so as to be compatible
with the fio directory structure.

Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agomtd: Add CONFIG_MTD to ./configure script
Dan Ehrenberg [Tue, 14 Apr 2015 22:58:17 +0000 (15:58 -0700)]
mtd: Add CONFIG_MTD to ./configure script

The CONFIG_MTD option will allow the mtd ioengine to be built. It
only works when the Linux MTD header is present.

Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoCollect a block-wise histogram of trim and write errors
Dan Ehrenberg [Tue, 14 Apr 2015 22:58:16 +0000 (15:58 -0700)]
Collect a block-wise histogram of trim and write errors

This patch collects information on how many trims or writes each trim
block sized unit can undergo before its first error. It also records
the way in which an error occurred, if one did.

Output is provided with a histogram which can be output either in
normal or json output. Terse output is omitted, since it seemed like
that would require a version rev bump.

Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAdd new writetrim rw= mode for trims preceding writes
Dan Ehrenberg [Tue, 14 Apr 2015 22:58:15 +0000 (15:58 -0700)]
Add new writetrim rw= mode for trims preceding writes

In this new mode, sequential trims and writes are interspersed by first
doing a trim at a particular offset, then doing writes starting from
that offset until the start of the next trim block, then another trim,
etc. This workload is designed to match the requirements of NAND flash,
if trims are implemented as erases.

Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAllow trim on any file type
Dan Ehrenberg [Tue, 14 Apr 2015 22:58:14 +0000 (15:58 -0700)]
Allow trim on any file type

The mtd fio engine will support trim (interpreted as erase). There is
no particular reason to tie trims to block devices, as it is only a
particular ioengine which calls the trim ioctl, not generic code.
This patch simply removes the test which checks that trim is only
run on block devices.

Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFio 2.2.7 fio-2.2.7
Jens Axboe [Wed, 15 Apr 2015 00:08:59 +0000 (18:08 -0600)]
Fio 2.2.7

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoconfigure: update to changes in fusion-aw packaging
Akshita Rajendra Jain [Thu, 9 Apr 2015 14:24:38 +0000 (08:24 -0600)]
configure: update to changes in fusion-aw packaging

Remove a dependency that is no longer used.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agogenzip: switch to jenkins hash
Jens Axboe [Wed, 8 Apr 2015 22:25:49 +0000 (16:25 -0600)]
genzip: switch to jenkins hash

Normal distribution causes a lot of collisions with the normal hash,
for some reason. This speeds it up a lot.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoUnify gauss and zipf/pareto input values
Jens Axboe [Wed, 8 Apr 2015 21:49:18 +0000 (15:49 -0600)]
Unify gauss and zipf/pareto input values

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agogenzip: cleanups
Jens Axboe [Wed, 8 Apr 2015 21:42:20 +0000 (15:42 -0600)]
genzip: cleanups

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agogenzip: add summed hit percentages
Jens Axboe [Wed, 8 Apr 2015 21:26:29 +0000 (15:26 -0600)]
genzip: add summed hit percentages

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agogenzipf: add support for normal distribution
Jens Axboe [Wed, 8 Apr 2015 17:25:27 +0000 (11:25 -0600)]
genzipf: add support for normal distribution

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoMake normal distribution takes deviations as a percentage
Jens Axboe [Wed, 8 Apr 2015 17:24:58 +0000 (11:24 -0600)]
Make normal distribution takes deviations as a percentage

This makes it independent of the address range generated.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Wed, 8 Apr 2015 17:10:59 +0000 (11:10 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

9 years agoAdd support for normal/gaussian random distributions
Jens Axboe [Wed, 8 Apr 2015 17:02:13 +0000 (11:02 -0600)]
Add support for normal/gaussian random distributions

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agofio-genzipf results output are slightly wrong, it doesn't help to understand how...
Fabrice Bacchella [Wed, 8 Apr 2015 14:49:42 +0000 (16:49 +0200)]
fio-genzipf results output are slightly wrong, it doesn't help to understand how to use it:
 * The last row was not displayed
 * some useless variables were used
 * the calculation for the output table missed a few events, so the sum of 'No Hits' column was wrong
 * Wrong sample in genzipf.c header, used old style arguments
 * the help message for -o options was misleading.
 * useless define of DEF_NR.
 * default of default row number to a more common value of 20,
   instead of 23.

9 years agosg: don't return success if non bdev/char is used
Jens Axboe [Tue, 31 Mar 2015 19:40:25 +0000 (13:40 -0600)]
sg: don't return success if non bdev/char is used

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAdd support for POSIX_FADV_STREAMID
Jens Axboe [Tue, 24 Mar 2015 16:20:22 +0000 (10:20 -0600)]
Add support for POSIX_FADV_STREAMID

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAlways update rusage before grabbing stat_mutex
Jens Axboe [Mon, 23 Mar 2015 16:19:44 +0000 (10:19 -0600)]
Always update rusage before grabbing stat_mutex

zxh reports running into a rusage_sem vs stat_mutex deadlock:

"I am running into a deadlock with fio for the following lines:

thread_main

fio_mutex_down(stat_mutex) at backend.c:1529

helper_thread_main

fio_mutex_down(td->rusage_sem) at stat.c:1467

thread_main is waiting for stat_mutex, which is already locked by
helper_thread_main in function __show_running_run_stats() at
stat.c:1441. However, the helper_thread_main is waiting for
td->rusage_sem, which is supposed to be unlocked by
check_update_rusage() in do_io() at backend.c:1525 in thread_main.

The issue is not reproducible every time, and I was using a customized
ioengine derived from rbd.c.

Is there any chance that this issue is caused by the customized io
engine? Or is there a way to get around this?"

Fix this by always updating the rusage stats before grabbing
stat_mutex, so we avoid this ABBA deadlock.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoCpu clock does not wrap if there is no CPU clock
Akash Verma [Thu, 19 Mar 2015 23:19:25 +0000 (16:19 -0700)]
Cpu clock does not wrap if there is no CPU clock

This causes compiler warnings because of unused variables cycles_start
and cycles_wrap.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoio_u_queue: add debug assert check on adding too many elements
Jens Axboe [Fri, 20 Mar 2015 04:15:02 +0000 (22:15 -0600)]
io_u_queue: add debug assert check on adding too many elements

Caught a bug in the io-threads branch, lets merge it into master.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoPack io_u/fio_file
Jens Axboe [Wed, 18 Mar 2015 22:17:41 +0000 (16:17 -0600)]
Pack io_u/fio_file

Remove two unneeded holes in the struct.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAddedd -z option to specify file size for filepaths fio files
Luca Maranzano [Wed, 18 Mar 2015 17:00:39 +0000 (18:00 +0100)]
Addedd -z option to specify file size for filepaths fio files

9 years agoRevert "Scramble IO buffers unless explicitly told not"
Jens Axboe [Wed, 18 Mar 2015 16:31:49 +0000 (10:31 -0600)]
Revert "Scramble IO buffers unless explicitly told not"

This reverts commit 7d1bb70f3d85d041b4b7dd926e98054fea980bdc.

9 years agoUpdate documentation for zero_buffers
Jens Axboe [Wed, 18 Mar 2015 16:13:33 +0000 (10:13 -0600)]
Update documentation for zero_buffers

If it is explicitly enabled, then we don't scramble the buffers as
well, unless the user has asked for that by setting
scramble_buffers=1.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoScramble IO buffers unless explicitly told not
Jens Axboe [Wed, 18 Mar 2015 00:11:07 +0000 (18:11 -0600)]
Scramble IO buffers unless explicitly told not

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoio_u: fix refill, should be random content
Jens Axboe [Tue, 17 Mar 2015 21:57:13 +0000 (15:57 -0600)]
io_u: fix refill, should be random content

Unless we asked for zeroed buffers, we fill them randomly.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoinit: fix exit status with bogus arguments
Zev Weiss [Thu, 12 Mar 2015 20:28:17 +0000 (15:28 -0500)]
init: fix exit status with bogus arguments

Had been exiting "successfully" on invalid command-line arguments.

Signed-off-by: Zev Weiss <zev@bewilderbeest.net>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agosmalloc: don't crash on being passed a bad pointer
Jens Axboe [Tue, 17 Mar 2015 16:45:18 +0000 (10:45 -0600)]
smalloc: don't crash on being passed a bad pointer

This can happen if we pass sfree() memory not allocated by smalloc.
Ignore it and log an error instead of triggering an assert().

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agorate: remove redundant code in rate_ddir()
Jens Axboe [Thu, 12 Mar 2015 16:05:25 +0000 (10:05 -0600)]
rate: remove redundant code in rate_ddir()

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agosmalloc: bump initial size to 8 pools
Jens Axboe [Tue, 3 Mar 2015 22:13:17 +0000 (15:13 -0700)]
smalloc: bump initial size to 8 pools

Commit 23bd40f944b7 shrank our pool size from 128 pools at 8MB max,
to one static 16MB pool. Christian reports:

"For our tests with about 250k files we found the smalloc pool being
 depleted.  Now for us values of 3-4 would be enough, but since it is a
 compile time switch I'd like to make it safe for everybody and set 8."

Bump the pool alloc to 8 pools again, retaining the 16MB size. That's
still substantially less than before, but should be enough for most
cases. Allocate the pools at init time, to avoid the issue that the
original commit fixed.

Also fix a bug where we failed to iterate some pools when restarting
the allocation scan, resulting in less-than-optimal exhaustion of all
pools before smalloc() gave up and returned NULL.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agofio: fix smalloc strdop allocation failure
Christian Ehrhardt [Tue, 3 Mar 2015 11:44:46 +0000 (12:44 +0100)]
fio: fix smalloc strdop allocation failure

smalloc_strdup didn't check for allocation success and thereby ran into
segfaults if the single pool went out of memory.
Now with this patch applied it is still failing, but in a more consistent
way than segfaulting.

You still get a bad allocation, but it looks like this now:
  fio: smalloc OOM
  fio: filesetup.c:1495: dup_files: Assertion `0' failed.
  Aborted

In fact the upper layers expected smalloc_strdup to retrun NULL on failure.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFio 2.2.6 fio-2.2.6
Jens Axboe [Fri, 27 Feb 2015 15:30:04 +0000 (08:30 -0700)]
Fio 2.2.6

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agofio: fix cycles_start build issue
Christian Ehrhardt [Fri, 27 Feb 2015 15:20:46 +0000 (08:20 -0700)]
fio: fix cycles_start build issue

There is an issue introduced with "commit 73df3e07 gettime: offset CPU cycle
counter by initial value". For architectures which define
ARCH_CPU_CLOCK_CYCLES_PER_USEC (currently only s390) this causes a build
error

gettime.c:174:11: error: â€˜cycles_start’ undeclared (first use in this
function)
    if (t < cycles_start && !cycles_wrap)

To make sure variables and code are only compiled in the same cases I
added a
clock definition called ARCH_CPU_CLOCK_WRAPS. We could merge
ARCH_CPU_CLOCK_WRAPS and the existing ARCH_CPU_CLOCK_CYCLES_PER_USEC
into one
if you prefer, so far nobody else uses it.

To avoid cluttering all architecture headers I enabled it by default in
arch-generic.h, so any arch not needing the wrap handling can undef
ARCH_CPU_CLOCK_WRAPS later in their headers.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoPatch to make per-thread IOPS more accurate
Ben England [Fri, 27 Feb 2015 15:14:49 +0000 (08:14 -0700)]
Patch to make per-thread IOPS more accurate

The following small patch to stat.c (using fio 2.2.4 from github)
outputs IOPS field in JSON format as a floating point number instead of
an integer.  This improves accuracy in case where fio --client runs use
lots of threads with single-digit IOPS per thread.  It seems to work,
here's a snippet of output from a short fio run with rate_iops=10 in the
job file.

     "write" : {
        "io_bytes" : 6464,
        "bw" : 646,
        "iops" : 10.10,
        "runtime" : 10000,

Why the patch: IOPS number is rounded to integer in stats.c calls to
num2str().  This doesn't sound like much of a problem because in many
use cases, with large IOPS number the error is negligible.  But in this
use case where we have many threads (we would like to get into the
thousands eventually), the IOPS/thread may be quite low and integer
rounding can introduce significant error.  For example, if we are doing
5,000 IOPS over 1,000 threads, average throughput is 5 IOPS and
potential error is ~20%, but some threads could have much higher error
in IOPS because of integer format.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFix segfault due to bad munmap()
Jens Axboe [Thu, 26 Feb 2015 22:38:42 +0000 (15:38 -0700)]
Fix segfault due to bad munmap()

Bruce reports:

The latest code from git (built using clang) causes a segfault after printing the usage text when "./fio" is run:

[New LWP 100111]
No jobs(s) defined

fio-2.2.5-28-g93eeb
[usage text]
[New Thread 801c06400 (LWP 100111/fio)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 801c06400 (LWP 100111/fio)]
flist_empty (head=0x802000040) at flist.h:119
119             return head->next == head;
Current language:  auto; currently minimal
(gdb) p head
$1 = (const struct flist_head *) 0x802000040

which is due to a bug in the filelock code, that uses
fio_mutex_remove() to remove the mutex. But that mutex is embedded
inside another mmap'ed region, hence we then segfault on later
deferencing pointers.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Thu, 26 Feb 2015 15:42:31 +0000 (08:42 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

9 years agoTypo fix in man page.
Lars Marowsky-Bree [Thu, 26 Feb 2015 09:48:31 +0000 (10:48 +0100)]
Typo fix in man page.

josn should be json, of course.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFix implicit function declaration on Android
enh [Wed, 25 Feb 2015 21:00:01 +0000 (14:00 -0700)]
Fix implicit function declaration on Android

there's no fio_set_sched_idle in os/os-android.h. copy the one from
os/os-linux.h.

current Android also has posix_madvise.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAdd missing pre-defined ARM arch compiler macro
Raphael Silva [Wed, 25 Feb 2015 13:57:40 +0000 (10:57 -0300)]
Add missing pre-defined ARM arch compiler macro

Signed-off-by: Raphael Silva <raphael.philipe@ossystems.com.br>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAdd --output-format option to manpage.
Lars Marowsky-Bree [Wed, 25 Feb 2015 15:26:19 +0000 (16:26 +0100)]
Add --output-format option to manpage.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoMake 'loops' honor full write+verify loops
Jens Axboe [Tue, 24 Feb 2015 22:50:57 +0000 (14:50 -0800)]
Make 'loops' honor full write+verify loops

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFixup whitespace damage in the two previous commits
Jens Axboe [Thu, 19 Feb 2015 16:47:34 +0000 (08:47 -0800)]
Fixup whitespace damage in the two previous commits

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoaxmap: Avoid memory leak if axmap_new() failed
Erwan Velu [Thu, 19 Feb 2015 12:09:09 +0000 (13:09 +0100)]
axmap: Avoid memory leak if axmap_new() failed

The axmap_new() function implements an exit path using the label "err"
to free the alloced memory.

That cleanup code was cleaning all the sub malloc structures but not the
main structure (axmap) itself resulting in a memory leak of
sizeof(struct axmap) bytes.

This simple patch just free axmap at the end of the cleanup code.

9 years agoiolog: Don't leak memory if fread fails in iolog_file_inflate
Erwan Velu [Thu, 19 Feb 2015 12:04:08 +0000 (13:04 +0100)]
iolog: Don't leak memory if fread fails in iolog_file_inflate

If fread fails in iolog_file_inflate(), the current code was closing the
file descriptor and return 1. But it was missing freeing the previously
malloced buffer (buf).

This patch does add a memory free of 'buf' before returning to avoid the
memory leak.

9 years agoAllow verify w/norandommap and bsrange
Justin Eno [Thu, 29 Jan 2015 22:28:38 +0000 (14:28 -0800)]
Allow verify w/norandommap and bsrange

Only intact blocks are verified; partially-overwritten blocks
are ignored
Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAllow verification of random overwrites w/ba < bs
Justin Eno [Thu, 29 Jan 2015 20:59:22 +0000 (12:59 -0800)]
Allow verification of random overwrites w/ba < bs

When blockalign is less than blocksize, random overwrite
workloads may partially overwrite blocks.  This change
evicts partially-overwritten blocks from the iolog tree
so subsequent verification targets only intact blocks.

This change also allows verification with norandommap
and bsrange, though that is left disabled.
Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoexamples/ssd-test.fio: bump size to 10G
Jens Axboe [Mon, 16 Feb 2015 22:04:11 +0000 (15:04 -0700)]
examples/ssd-test.fio: bump size to 10G

1G is a bit too tiny for most devices today, bump it to 10G.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoAllow random overwrite workloads to write io_limit
Justin Eno [Thu, 29 Jan 2015 20:27:58 +0000 (12:27 -0800)]
Allow random overwrite workloads to write io_limit

Limiting the write phase of (random) overwrite workloads to 'size'
before the verify phase doesn't make sense, so allow them to write
up to io_limit.  This better matches verify_only behavior, and
allows the [verify-job] below to succeed:

[global]
thread=1
direct=1
ioengine=sync
verify=meta
bs=1k
offset=0
size=5M
io_limit=6M
norandommap=1
verify_fatal=1
randseed=42

[write-job]
filename=datafile.tmp
rw=randwrite
do_verify=0

[verify-job]
stonewall
create_serialize=0
filename=datafile.tmp
rw=randwrite
do_verify=1
verify_only=1

Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agogettime.h: use time_t instead of size_t for copy of tv_sec
Castor Fu [Fri, 13 Feb 2015 14:46:04 +0000 (06:46 -0800)]
gettime.h: use time_t instead of size_t for copy of tv_sec

This addresses a small nit that our compiler was complaining about.
Thanks!

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoverify: Fix latency log for verify commands.
Gwendal Grignou [Mon, 9 Feb 2015 03:53:20 +0000 (19:53 -0800)]
verify: Fix latency log for verify commands.

When commands when requeued for the verify operation,
their start time was not reset, resulting in bogus latency graphs.

Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFix for verify_only (do_dry_run()) broken by 74d6277f
Justin Eno [Wed, 28 Jan 2015 22:13:28 +0000 (14:13 -0800)]
Fix for verify_only (do_dry_run()) broken by 74d6277f

Previous commit to backend.c tracks io_limit more closely by
counting submitted (in-flight) I/O instead of completed I/O.
do_dry_run() does not submit I/O, so its I/O is not counted,
and it loops forever.
Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoconfigure: fix libnuma_v2 probe to work with -Werror
Castor Fu [Thu, 29 Jan 2015 18:07:36 +0000 (10:07 -0800)]
configure: fix libnuma_v2 probe to work with -Werror

I noticed that libnuma_v2 wasn't getting detected in our build
environment because we would get an error that mask was unused.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoMoral license update
Jens Axboe [Fri, 6 Feb 2015 16:09:29 +0000 (09:09 -0700)]
Moral license update

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoUpdate the moral license
Jens Axboe [Fri, 30 Jan 2015 17:38:20 +0000 (09:38 -0800)]
Update the moral license

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agogettime: remember to tear down clock cpumask on normal exit
Jens Axboe [Thu, 29 Jan 2015 17:23:39 +0000 (09:23 -0800)]
gettime: remember to tear down clock cpumask on normal exit

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agogettime: fix unitialized variable on Solaris
Jens Axboe [Thu, 29 Jan 2015 16:32:28 +0000 (08:32 -0800)]
gettime: fix unitialized variable on Solaris

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoidletime: maintain cpuset over lifetime of idle thread
Jens Axboe [Wed, 28 Jan 2015 21:47:48 +0000 (14:47 -0700)]
idletime: maintain cpuset over lifetime of idle thread

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agogettime: initialize cpusets properly
Jens Axboe [Wed, 28 Jan 2015 21:42:10 +0000 (14:42 -0700)]
gettime: initialize cpusets properly

We can't just use memset(), some platforms have specific init/exit
routines for cpusets/masks.

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agogettime: if setaffinity fails, print the errno error
Jens Axboe [Wed, 28 Jan 2015 21:34:10 +0000 (14:34 -0700)]
gettime: if setaffinity fails, print the errno error

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoFixing typo
Felipe Franciosi [Wed, 28 Jan 2015 19:06:32 +0000 (19:06 +0000)]
Fixing typo

9 years agodedupe_percentage should work even if compress_percentage is not set
Vasily Tarasov [Wed, 28 Jan 2015 16:10:30 +0000 (09:10 -0700)]
dedupe_percentage should work even if compress_percentage is not set

Currently dedupe_percentage option does not work properly if
compress_percentage is not set. This can be easily demonstrated using
the following job file:

[job-/dev/dm-0]
filename=/dev/dm-0
rw=randwrite
blocksize=4096
direct=1
ioengine=libaio
iodepth=32
dedupe_percentage=50
time_based=1
runtime=5
numjobs=1

All writes will contain zeroes for the above job file.

The problem is in fill_io_buffer() function that switches dedupe random
state only if o->compress_percentage is set.

This patch updates the condition appropriately in fill_io_buffer()
appropriately.

Signed-off-by: Vasily Tarasov <tarasov@vasily.name>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoBetter accommodate random writes larger than blockalign
Justin Eno [Tue, 27 Jan 2015 22:23:20 +0000 (14:23 -0800)]
Better accommodate random writes larger than blockalign

fill_io_u() fails prematurely if the randomly-chosen offset satisfies
blockalign but not min_bs, i.e., the offset lies too near the end of
the target region.  This change honors both parameters.

Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoparse: ensure strings are pre-terminated when using strncpy()
Jens Axboe [Thu, 22 Jan 2015 21:51:56 +0000 (14:51 -0700)]
parse: ensure strings are pre-terminated when using strncpy()

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Wed, 21 Jan 2015 23:57:00 +0000 (00:57 +0100)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

9 years agoFio 2.2.5 fio-2.2.5
Jens Axboe [Wed, 21 Jan 2015 20:07:43 +0000 (13:07 -0700)]
Fio 2.2.5

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoshm: disable iomem=shm* for shm disabled
Jens Axboe [Wed, 21 Jan 2015 01:38:11 +0000 (18:38 -0700)]
shm: disable iomem=shm* for shm disabled

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agoconfigure: add --disable-shm to disable use of shm
Jens Axboe [Tue, 20 Jan 2015 23:29:33 +0000 (16:29 -0700)]
configure: add --disable-shm to disable use of shm

Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agonet: don't record/verify UDP sequence numbers if buffer is too small
Steven Noonan [Sat, 17 Jan 2015 00:46:11 +0000 (16:46 -0800)]
net: don't record/verify UDP sequence numbers if buffer is too small

This causes a bunch of out-of-bounds accesses if you have really small buffer
sizes (i.e. 16 bytes will crash).

Signed-off-by: Steven Noonan <steven@uplinklabs.net>
Signed-off-by: Jens Axboe <axboe@fb.com>
9 years agocrc/test: sha1 should do _final() after _update()
Jens Axboe [Fri, 16 Jan 2015 23:10:43 +0000 (16:10 -0700)]
crc/test: sha1 should do _final() after _update()

Signed-off-by: Jens Axboe <axboe@fb.com>