fio.git
12 years agoFix parser bug with ranges
Jens Axboe [Wed, 7 Sep 2011 19:21:07 +0000 (21:21 +0200)]
Fix parser bug with ranges

The change to allow -/+ for values broke ranges.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoUse r/R/w/W instead of m/M for mixed workloads with 100% reads or writes
Jens Axboe [Thu, 1 Sep 2011 15:58:11 +0000 (09:58 -0600)]
Use r/R/w/W instead of m/M for mixed workloads with 100% reads or writes

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoOnly clear string pointer in parser for sub-value options
Jens Axboe [Wed, 31 Aug 2011 22:45:03 +0000 (16:45 -0600)]
Only clear string pointer in parser for sub-value options

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoEnsure that buffer contents are random across jobs as well
Jens Axboe [Wed, 31 Aug 2011 21:20:15 +0000 (15:20 -0600)]
Ensure that buffer contents are random across jobs as well

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix parser using uninitialized memory
Jens Axboe [Wed, 31 Aug 2011 20:29:22 +0000 (14:29 -0600)]
Fix parser using uninitialized memory

Introduced by e2979754b554d6a48cffa98f41fb59bc045922ac.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd support for backwards holes
Jens Axboe [Wed, 31 Aug 2011 19:14:12 +0000 (13:14 -0600)]
Add support for backwards holes

If you did:

bs=4k
rw=read:-8k

you would essentially rewind by 8k after reading 4k, causing
the read to be sequentially backwards.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRemove duplicate code in parser
Jens Axboe [Wed, 31 Aug 2011 18:55:27 +0000 (12:55 -0600)]
Remove duplicate code in parser

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix FIO_OPT_STR_STORE without posval
Jens Axboe [Wed, 31 Aug 2011 18:49:49 +0000 (12:49 -0600)]
Fix FIO_OPT_STR_STORE without posval

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd sub-option support (sort-of) and convert libaio_userspace_reap
Jens Axboe [Wed, 31 Aug 2011 02:43:53 +0000 (20:43 -0600)]
Add sub-option support (sort-of) and convert libaio_userspace_reap

You should now use ioengine=libaio:userspace_reap instead of
the separate option.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdding userspace_libaio_reap option
Dan Ehrenberg [Wed, 31 Aug 2011 02:36:15 +0000 (20:36 -0600)]
Adding userspace_libaio_reap option

When a single thread is reading from a libaio io_context_t object
in a non-blocking polling manner (that is, with the minimum number
of events to return being 0), then it is possible to safely read
events directly from user-space, taking advantage of the fact that
the io_context_t object is a pointer to memory with a certain layout.
This patch adds an option, userspace_libaio_reap, which allows
reading events in this manner when the libaio engine is used.

You can observe its effect by setting iodepth_batch_complete=0
and seeing the change in distribution of system/user time based on
whether this new flag is set. If userspace_libaio_reap=1, then
busy polling takes place in userspace, and there is a larger amount of
usr CPU. If userspace_libaio_reap=0 (the default), then there is a
larger amount of sys CPU from the polling in the kernel.

Polling from a queue in this manner is several times faster. In my
testing, it took less than an eighth as much time to execute a
polling operation in user-space than with the io_getevents syscall.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix bad latency reporting for rated IO jobs
Jens Axboe [Tue, 30 Aug 2011 21:34:14 +0000 (15:34 -0600)]
Fix bad latency reporting for rated IO jobs

If we are going to sleep to satisfy a given rate limit, then
ensure that we flush pending IO first. Otherwise they end up
including the sleep time in the calculated completion latencies.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix compile on environment of SuperH
Nobuhiro Iwamatsu [Sat, 27 Aug 2011 21:34:37 +0000 (06:34 +0900)]
Fix compile on environment of SuperH

Signed-off-by: Nobuhiro Iwamatsu <iwamatsu@nigauri.org>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd 'null' perf test
Jens Axboe [Thu, 25 Aug 2011 18:20:14 +0000 (20:20 +0200)]
Add 'null' perf test

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoMake SH port work for packagers that don't differentiate between SH4 and SH4A
Jens Axboe [Thu, 25 Aug 2011 12:24:03 +0000 (14:24 +0200)]
Make SH port work for packagers that don't differentiate between SH4 and SH4A

Generic bits done by me, SH specific bits implemented by
Nobuhiro Iwamatsu <iwamatsu@nigauri.org>.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd possibility to make sequential IO "holed"
Jens Axboe [Thu, 25 Aug 2011 07:09:37 +0000 (09:09 +0200)]
Add possibility to make sequential IO "holed"

For sequential IO, it is now possible to add a number of bytes to
be skipped for every block read or written. Using:

bs=8k
rw=read:8k

will first read 0k->8k, then 16k->24k, and so on. This skips 8k
for every 'bs' sized block read. Similar for writes, doing

bs=4k
rw=write:4k

will write 0k->4k, then 8k->12k, etc. End result being that every
other block is written, in a sequential fashion.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoImprove Makefile for t/ test code
Jens Axboe [Tue, 16 Aug 2011 08:01:03 +0000 (10:01 +0200)]
Improve Makefile for t/ test code

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd simple stest smalloc tester
Jens Axboe [Tue, 16 Aug 2011 07:56:27 +0000 (09:56 +0200)]
Add simple stest smalloc tester

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFixup bad style in plat code
Jens Axboe [Tue, 16 Aug 2011 06:43:45 +0000 (08:43 +0200)]
Fixup bad style in plat code

- Comments
- Var declarations in code sections
- Misc style

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agostats: fix io_u_plat out-of-bound accesses (round 2)
Eric Gouriou [Tue, 16 Aug 2011 06:35:43 +0000 (08:35 +0200)]
stats: fix io_u_plat out-of-bound accesses (round 2)

Commit 833491908a1afd67 introduced the ability to report completion
latency percentiles. It also caused a memory corruption when running
with multiple threads due to out of bound accesses in show_run_stats().
The major index of the io_u_plat two-dimensional array is meant
to be DDIR_ value in {DDIR_READ, DDIR_WRITE} (i.e., {0, 1}). The
code in show_run_stats() incorrectly wrote into the array using
a major index with values {0, 1, 2}. Commit 0a0b49007cbce8d1 fixed
the out of bound accesses by increasing the size of the major
dimension of the io_u_plat array from 2 to 3.

This patch reverts the size change from 0a0b49007cbce8d1 in favor
of avoiding the out-of-bound accesses in show_run_stats().

Signed-off-by: Eric Gouriou <egouriou@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix bug in smalloc size calculation
Jiri Horky [Mon, 15 Aug 2011 13:18:03 +0000 (15:18 +0200)]
Fix bug in smalloc size calculation

Jiri reports:

I think I found a bug. If the device (file) used in fio has "wrong" size, and is used with "wrong" block size, the fio fails to allocate memory for randommap.
Following conditions must be true, considering just one thread for simplicity:

 1) number of blocks on the device are too big to fill in the default's pool
 2) number of blocks on the device must be "right", so the size of newly allocated pool by smalloc.c:add_pool is exactly as big as requested size (there are some roundings in place, so not every size triggers this error)

When this is true, function smalloc.c:__smalloc_pool fails to use the newly created (exactly big enough) pool, because of this line:

                idx = find_next_zero(pool->bitmap[i], last_idx);

which always returns non-zero value (the minimum returned value is last_idx +1), and so in our case the idx is 1 even though, the block number 0 is free. As the direct consequence, the function doesn't find enough free space in the given pool and returns NULL pointer. This leads to another try to allocate enough memory in smalloc.c:smalloc_real function, and so on...

The behavior could be reprocuded using both stable 1.57 and git version of the tool. The values that triggers the error are:
init_random_map - real_file_size: 21999995584512, o.rw_min_bs: 4096, blocks: 5371092672, num_maps: 83923323, alloc size: 671386584

whereas this block device is fine:
init_random_map - real_file_size: 19999995985920, o.rw_min_bs: 4096, blocks: 4882811520, num_maps: 76293930, alloc size: 610351440

the configuration file used:

[global]
description=CESNET_test
[cesnet]
filename=/dev/sdd
rw=randread
size=10000g
ioengine=libaio
iodepth=4
bs=4k
runtime=10m
time_based
numjobs=1
group_reporting

Get rid of the +1, it looks like a remnant from when the pools
had a header associated with them.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRDMA engine: Change wording of HOWTO section
Bart Van Assche [Mon, 15 Aug 2011 07:01:05 +0000 (09:01 +0200)]
RDMA engine: Change wording of HOWTO section

Change "iWarp" into "iWARP" ("iWarp" is the name of a supercomputer) and
"engines" into "protocols".

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Ren Yufei <renyufei83@yahoo.com.cn>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix off-by-one in io_u_plat[] array sizing
Jens Axboe [Mon, 15 Aug 2011 07:00:28 +0000 (09:00 +0200)]
Fix off-by-one in io_u_plat[] array sizing

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agostats: Add a function to report completion latency percentiles
Yu-ju Hong [Fri, 12 Aug 2011 22:53:44 +0000 (00:53 +0200)]
stats: Add a function to report completion latency percentiles

    This patch introduces two new options:
    1) Specifying --clat_percentiles enables the reporting of
       completion latency percentiles.
    2) --percentile_list allows the user to customize the
       list of percentiles to report.

Signed-off-by: Yu-ju Hong <yjhong@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoman page typo
Jens Axboe [Mon, 8 Aug 2011 07:47:13 +0000 (09:47 +0200)]
man page typo

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoUpdate softrandommap in HOWTO
Jens Axboe [Mon, 8 Aug 2011 07:07:28 +0000 (09:07 +0200)]
Update softrandommap in HOWTO

Previous commit fixed the typo in the man page, update the HOWTO
as well.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoMerge branch 'master' of ssh://router/data/git/fio
Jens Axboe [Mon, 8 Aug 2011 07:06:45 +0000 (09:06 +0200)]
Merge branch 'master' of ssh://router/data/git/fio

12 years agoFix a typo in fio.1
Zhu Yanhai [Mon, 8 Aug 2011 07:06:27 +0000 (09:06 +0200)]
Fix a typo in fio.1

softrandommap requires an bool argument actually.

Signed-off-by: Zhu Yanhai <gaoyang.zyh@taobao.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRDMA I/O engine: Fix compiler warnings
Bart Van Assche [Sun, 7 Aug 2011 19:50:52 +0000 (21:50 +0200)]
RDMA I/O engine: Fix compiler warnings

Fix 32-bit mode compiler warnings and a warning about an unused variable.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Ren Yufei <renyufei83@yahoo.com.cn>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRDMA engine: rewrite header comment block and man page section
Bart Van Assche [Sun, 7 Aug 2011 19:50:51 +0000 (21:50 +0200)]
RDMA engine: rewrite header comment block and man page section

Spelling and grammar fixes for the header comment. Also make it clear that
users can choose between the RDMA software provided by their Linux distributor
and OFED. Update a related paragraph in the man page too.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Ren Yufei <renyufei83@yahoo.com.cn>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRDMA engine spelling fix: change FIO_RDMA_MAX_IO_DPETH into FIO_RDMA_MAX_IO_DEPTH
Bart Van Assche [Sun, 7 Aug 2011 19:50:51 +0000 (21:50 +0200)]
RDMA engine spelling fix: change FIO_RDMA_MAX_IO_DPETH into FIO_RDMA_MAX_IO_DEPTH

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Ren Yufei <renyufei83@yahoo.com.cn>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agofio_generate_plots with setable resolution
Martin Steigerwald [Fri, 5 Aug 2011 17:47:51 +0000 (19:47 +0200)]
fio_generate_plots with setable resolution

Specify xres and yres as 2nd and 3rd argument, otherwise
defaults to 1024x768.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd 'wait_for_previous' as an alias for the 'stonewall' option
Jens Axboe [Wed, 3 Aug 2011 10:38:39 +0000 (12:38 +0200)]
Add 'wait_for_previous' as an alias for the 'stonewall' option

Also update man page and HOWTO, plus add missing alias for
iomem_align while in there.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRDMA update
Yufei Ren [Tue, 2 Aug 2011 10:15:30 +0000 (12:15 +0200)]
RDMA update

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRDMA IO engine
ren yufei [Mon, 1 Aug 2011 08:01:57 +0000 (10:01 +0200)]
RDMA IO engine

I have hacked an rdma ioengine based on OFED for fio which could test
both rdma memory semantic (rdma_write/rdma_read) and channel semantic
(send/recv). Would you like to merge this engine into fio?

notes
1) RDMA engine works in IB, iWarp and RoCE.
2) RDMA engine is disable by default. To enable it, execute the following
   before compile:

$ export EXTFLAGS="-DFIO_HAVE_RDMA"
$ export EXTLIBS="-libverbs -lrdmacm"

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd proper include to silence build warning on NetBSD/FreeBSD
Jens Axboe [Sat, 30 Jul 2011 11:29:58 +0000 (13:29 +0200)]
Add proper include to silence build warning on NetBSD/FreeBSD

Reported-by: Bruce Cran <bruce@cran.org.uk>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd version number to --help output
Bruce Cran [Sat, 30 Jul 2011 10:55:52 +0000 (12:55 +0200)]
Add version number to --help output

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agostats: Fix computation of summed standard deviation
Yu-ju Hong [Sat, 30 Jul 2011 07:18:13 +0000 (09:18 +0200)]
stats: Fix computation of summed standard deviation

Fix the computation of standard deviation for a group
of jobs. Please see the below link for the
approximation formula used.
<http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

The formula was originally presented here:
<ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/79/773/
CS-TR-79-773.pdf>

Signed-off-by: Yu-ju Hong <yjhong@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix bsrange read,write value option pairs
Dave Engberg [Sat, 23 Jul 2011 19:07:13 +0000 (21:07 +0200)]
Fix bsrange read,write value option pairs

The 'bs' and 'bssplit' configuration variables accept separate numbers
for read and write operations if you separate them with a comma. For
example: bssplit=1k/39:8k/50:64k/11,1k/29:8k/50:64k/21

The documentation for 'bsrange' says it works the same way, but if I try
to specify two ranges with a comma, I get an error when I run the app.
E.g. if I put this line in the configuration file: bsrange=1k-16k,2k-32k

I get this error output:

fio: pid=26861, err=22/file:engines/sync.c:62, func=xfer, error=Invalid argument

This patch fixes it.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix --timeout as global parameter
Jens Axboe [Sat, 23 Jul 2011 11:43:13 +0000 (13:43 +0200)]
Fix --timeout as global parameter

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoLibaio engine support for iodepth_batch_complete=0
Dan Ehrenberg [Sat, 23 Jul 2011 11:36:46 +0000 (13:36 +0200)]
Libaio engine support for iodepth_batch_complete=0

Previously, even if iodepth_batch_complete=0, the libaio engine
passed in non-zero values for the min_nr value for io_getevents.
This patch makes min_nr always 0 if iodepth_batch_complete == 0,
and if multiple events are required at a higher level, then
we poll io_getevents multiple times for the events.

Signed-off-by: Dan Ehrenberg <dehrenberg@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRespect iodepth_batch_complete=0 in main loop
Dan Ehrenberg [Sat, 23 Jul 2011 11:36:44 +0000 (13:36 +0200)]
Respect iodepth_batch_complete=0 in main loop

If the iodepth_batch_complete is 0, then even if the queue is full,
we still don't make min_events be 1 in the main loop. Instead, poll
in a non-blocking way until a result is found.

Signed-off-by: Dan Ehrenberg <dehrenberg@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAddes Trace file format description
Paul Dubs [Thu, 21 Jul 2011 15:26:02 +0000 (17:26 +0200)]
Addes Trace file format description

12 years agoAdded description for iops output
Paul Dubs [Thu, 21 Jul 2011 14:04:52 +0000 (16:04 +0200)]
Added description for iops output

12 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Tue, 19 Jul 2011 14:28:33 +0000 (16:28 +0200)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

12 years agoFix Solaris compile
Jens Axboe [Tue, 19 Jul 2011 14:27:43 +0000 (16:27 +0200)]
Fix Solaris compile

In file included from fio.h:24,
                  from gettime.c:10:
os/os.h:192: error: redefinition of 'gettid'
os/os-solaris.h:123: error: previous definition of 'gettid' was here
make: *** [gettime.o] Error 1

Forgot to define FIO_HAVE_GETTID

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoMake string-set options behave more like bool options
Jens Axboe [Fri, 15 Jul 2011 07:09:15 +0000 (09:09 +0200)]
Make string-set options behave more like bool options

Before this change, string-set options (like time_based) would not
complain if passed an argument, they would just always evaluate
to being set. This is very confusing if someone uses them negated,
ala:

time_based=0

since fio would interpret that as the option being set. Now we'll
do the right thing, time_based=0 will be identical to not having
the option set. And time_based=1 or just time_based will equate
to the option being set.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFio 1.57 fio-1.57
Jens Axboe [Thu, 14 Jul 2011 18:20:25 +0000 (20:20 +0200)]
Fio 1.57

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoWindows affinity fix for thread based jobs
Bruce Cran [Thu, 14 Jul 2011 14:34:02 +0000 (16:34 +0200)]
Windows affinity fix for thread based jobs

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix infinite loop on platforms with severely limited aio resources
Jens Axboe [Thu, 14 Jul 2011 10:45:13 +0000 (12:45 +0200)]
Fix infinite loop on platforms with severely limited aio resources

Problem hit on OSX, where with 4 jobs each with a queue depth larger
than what the system supports, you can get into a situation where
any given process can get EAGAIN on queueing AIO even if it has
nothing queued already. Check for this condition to avoid fio's
IO loop going into a death spiral.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoMake raw disk size detection work on OSX
Jens Axboe [Thu, 14 Jul 2011 07:12:50 +0000 (09:12 +0200)]
Make raw disk size detection work on OSX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix compile warning on OSX
Jens Axboe [Thu, 14 Jul 2011 06:39:19 +0000 (08:39 +0200)]
Fix compile warning on OSX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix killing of threads that haven't started
Michael Callahan [Thu, 14 Jul 2011 05:56:37 +0000 (07:56 +0200)]
Fix killing of threads that haven't started

If there is a thread that has not started yet (for instance with a long
startdelay or a stonewall) then a SIGINT will fire off terminate_threads
which runs through all of the threads and calls kill(pid, SIGTERM).
However the threads that have not started yet have a pid of 0.  When you
call kill(0, SIGTERM) it kills the entire thread group which can extend
out of fio.  I have included a short patch which fixes this for me.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd CPU affinity support to Windows
Bruce Cran [Thu, 14 Jul 2011 05:50:43 +0000 (07:50 +0200)]
Add CPU affinity support to Windows

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix AIX Makefile
Michael Perzl [Wed, 13 Jul 2011 18:28:07 +0000 (20:28 +0200)]
Fix AIX Makefile

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix AIX compile
Michael Perzl [Wed, 13 Jul 2011 16:43:57 +0000 (18:43 +0200)]
Fix AIX compile

When adding HPUX support the compile for AIX was broken in "lib/getopt.h".

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFio 1.56 fio-1.56
Jens Axboe [Wed, 13 Jul 2011 10:42:26 +0000 (12:42 +0200)]
Fio 1.56

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agogettid() for NetBSD
Jens Axboe [Tue, 12 Jul 2011 21:49:16 +0000 (23:49 +0200)]
gettid() for NetBSD

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd gettid() for FreeBSD/OSX/Solaris
Jens Axboe [Tue, 12 Jul 2011 20:33:53 +0000 (22:33 +0200)]
Add gettid() for FreeBSD/OSX/Solaris

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd gettid() for proper thread affinity on Linux
Jens Axboe [Tue, 12 Jul 2011 19:17:49 +0000 (21:17 +0200)]
Add gettid() for proper thread affinity on Linux

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix compile on older systems that don't have fallocate() on Linux
Jens Axboe [Tue, 12 Jul 2011 18:49:46 +0000 (20:49 +0200)]
Fix compile on older systems that don't have fallocate() on Linux

Commit a596f047 broke build en eg CentOS 5:

filesetup.c:17:26: error: linux/falloc.h: No such file or directory
filesetup.c: In function 'extend_file':
filesetup.c:95: warning: implicit declaration of function 'fallocate'
filesetup.c:95: error: 'FALLOC_FL_KEEP_SIZE' undeclared (first use in this function)
filesetup.c:95: error: (Each undeclared identifier is reported only once
filesetup.c:95: error: for each function it appears in.)
make: *** [filesetup.o] Error 1

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAllow percentage setting for size=
Jens Axboe [Tue, 12 Jul 2011 17:47:03 +0000 (19:47 +0200)]
Allow percentage setting for size=

Sometimes it's useful to set this to a particular size of a device.
Allowing percentages makes this more easy, as one does not have to
do the math outside of fio and pass in as environment variables
or custom job files.

To use, simply add a

size=15%

or whatever is needed, then fio will use 15% of the total size of
the device(s) or file(s) given. Percentage can be from 1 to 100.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoHP-UX: add fdatasync()
Jens Axboe [Mon, 11 Jul 2011 12:43:26 +0000 (14:43 +0200)]
HP-UX: add fdatasync()

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoHP-UX: has aio_fsync() and fallocate()
Jens Axboe [Mon, 11 Jul 2011 08:56:32 +0000 (10:56 +0200)]
HP-UX: has aio_fsync() and fallocate()

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoHP-UX disk query size
Jens Axboe [Mon, 11 Jul 2011 08:48:51 +0000 (10:48 +0200)]
HP-UX disk query size

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRemove old debug printf()
Jens Axboe [Mon, 11 Jul 2011 08:26:00 +0000 (10:26 +0200)]
Remove old debug printf()

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFill in os_phys_mem() stub for HP-UX
Jens Axboe [Mon, 11 Jul 2011 08:25:35 +0000 (10:25 +0200)]
Fill in os_phys_mem() stub for HP-UX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix bug in $mb_memory keyword
Jens Axboe [Mon, 11 Jul 2011 08:24:20 +0000 (10:24 +0200)]
Fix bug in $mb_memory keyword

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoTake __weak out of the compiler namespace
Bruce Cran [Sun, 10 Jul 2011 18:56:23 +0000 (20:56 +0200)]
Take __weak out of the compiler namespace

With Xcode 4 on OS X shipping LLVM by default, I tried building fio with
clang and found that the definition of __weak conflicts with something
clang defines.  I've attached a patch which replaces the double
underscores with a single underscore, removing it from the
compiler-dependant namespace.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoposixaio: fix for HP-UX
Jens Axboe [Sat, 9 Jul 2011 11:33:38 +0000 (13:33 +0200)]
posixaio: fix for HP-UX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoHP-UX typo
Jens Axboe [Sat, 9 Jul 2011 11:28:19 +0000 (13:28 +0200)]
HP-UX typo

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd note saying that HP-UX should work
Jens Axboe [Sat, 9 Jul 2011 06:58:44 +0000 (08:58 +0200)]
Add note saying that HP-UX should work

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoA few HP-UX fixes
Jens Axboe [Sat, 9 Jul 2011 06:54:26 +0000 (08:54 +0200)]
A few HP-UX fixes

Enable posixaio and net IO engine

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoMake processes work on HP-UX
Jens Axboe [Sat, 9 Jul 2011 06:31:53 +0000 (08:31 +0200)]
Make processes work on HP-UX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoImprove Windows windowsaio engine performance
Bruce Cran [Sat, 9 Jul 2011 06:20:47 +0000 (08:20 +0200)]
Improve Windows windowsaio engine performance

Always use IO completion ports on Windows since some drives benefit from
using it even with direct or sync io.
Remove PrintError function since logging of errors is done in the core.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix integer overflow in calculating large IOPS on 32-bit platforms
Bruce Cran [Sat, 9 Jul 2011 06:19:53 +0000 (08:19 +0200)]
Fix integer overflow in calculating large IOPS on 32-bit platforms

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoGet rid of warning on platforms for casting char -> int
Jens Axboe [Fri, 8 Jul 2011 19:14:57 +0000 (21:14 +0200)]
Get rid of warning on platforms for casting char -> int

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd block device size detection for HP-UX
Jens Axboe [Fri, 8 Jul 2011 19:10:30 +0000 (21:10 +0200)]
Add block device size detection for HP-UX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoInitial HP-UX port
Jens Axboe [Fri, 8 Jul 2011 18:56:06 +0000 (20:56 +0200)]
Initial HP-UX port

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix bad start_delay type
Jens Axboe [Fri, 8 Jul 2011 18:47:03 +0000 (20:47 +0200)]
Fix bad start_delay type

Should be unsigned long long, causes a bus error on HP-UX.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoSet affinity before doing buffer and io_u allocs
Jens Axboe [Fri, 8 Jul 2011 17:25:04 +0000 (19:25 +0200)]
Set affinity before doing buffer and io_u allocs

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoRewrite posixaio back off comment to be closer to actual English
Jens Axboe [Fri, 8 Jul 2011 06:33:37 +0000 (08:33 +0200)]
Rewrite posixaio back off comment to be closer to actual English

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoEnsure that we honor aio_read/write() returning EAGAIN
Jens Axboe [Thu, 7 Jul 2011 19:06:40 +0000 (21:06 +0200)]
Ensure that we honor aio_read/write() returning EAGAIN

This fixes a problem on OSX, where the number of resources for
aio is woefully small.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix ramp time random screwup
Jens Axboe [Thu, 7 Jul 2011 13:35:27 +0000 (15:35 +0200)]
Fix ramp time random screwup

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix ramp time bug
Jens Axboe [Thu, 7 Jul 2011 10:36:59 +0000 (12:36 +0200)]
Fix ramp time bug

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Wed, 6 Jul 2011 18:13:27 +0000 (20:13 +0200)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

12 years agoAdd --max-jobs/-j command line option
Jens Axboe [Wed, 6 Jul 2011 18:12:54 +0000 (20:12 +0200)]
Add --max-jobs/-j command line option

And shrink OSX max default number of jobs supported. OSX has,
by default, a very small shared memory segment available.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix for terminating threads in ramp time
Jens Axboe [Wed, 6 Jul 2011 07:05:31 +0000 (09:05 +0200)]
Fix for terminating threads in ramp time

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoMake smalloc use anonymous memory mmaps
Jens Axboe [Wed, 29 Jun 2011 07:50:08 +0000 (09:50 +0200)]
Make smalloc use anonymous memory mmaps

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agopatch from debian fio package to fix syntax errors in manpage.
Martin Steigerwald [Mon, 27 Jun 2011 14:07:24 +0000 (16:07 +0200)]
patch from debian fio package to fix syntax errors in manpage.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoman page: fix incorrect description for disable_clat option
Steven Noonan [Wed, 22 Jun 2011 07:47:09 +0000 (09:47 +0200)]
man page: fix incorrect description for disable_clat option

Signed-off-by: Steven Noonan <snoonan@amazon.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoAdd ability to invoke fallocate() FALLOC_FL_KEEP_SIZE.
Eric Gouriou [Fri, 17 Jun 2011 07:11:45 +0000 (09:11 +0200)]
Add ability to invoke fallocate() FALLOC_FL_KEEP_SIZE.

Linux offers fallocate() and the FALLOC_FL_KEEP_SIZE option as
an alternative to posix_fallocate(). When FALLOC_FL_KEEP_SIZE is
specified for an falloc request going beyond the end of the file,
the requested blocks get preallocated without changing the apparent
size of the file. This is is a commonly recommended use of fallocate()
for workloads performing append writes.

This patch modifies the fallocate option from a boolean option
to a string option accepting none/posix/keep/0/1. 'keep' is only
made available on systems where FALLOC_FL_KEEP_SIZE is available
(i.e., Linux at this time). If specified, fallocate() is used
with FALLOC_FL_KEEP_SIZE set. 'none' disables pre-allocation while
'posix' uses posix_fallocate(). The default behavior remains unchaned,
i.e., invoking posix_fallocate. The settings '0'/'1' are there to
provide backward compatibility for users who had explicitly set the
boolean option.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix fill_device with queue depth higher than 1
Steven Lang [Mon, 13 Jun 2011 07:47:36 +0000 (09:47 +0200)]
Fix fill_device with queue depth higher than 1

I found an interesting combination of using fill_device when the
iodepth is greater than 1.  All outstanding IO causes an ENOSPC error,
but only the first is treated specially.  The remainder come through
and cause an error exit condition when the IO queue is being cleared.
So it is impossible to use deep IO with fill device.

This patch should fix the issue.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFixup wrong enum used for __check_min_rate()
Steven Lang [Fri, 3 Jun 2011 19:15:50 +0000 (21:15 +0200)]
Fixup wrong enum used for __check_min_rate()

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix Makefile depend issue
Michael Zappe [Wed, 1 Jun 2011 18:15:18 +0000 (20:15 +0200)]
Fix Makefile depend issue

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFio 1.55 fio-1.55
Jens Axboe [Thu, 26 May 2011 18:57:30 +0000 (20:57 +0200)]
Fio 1.55

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Tue, 24 May 2011 19:36:35 +0000 (21:36 +0200)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

12 years agoSilence gcc 4.6 warnings
Jens Axboe [Tue, 24 May 2011 19:36:24 +0000 (21:36 +0200)]
Silence gcc 4.6 warnings

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFix overlap io_u into outside regions
Jens Axboe [Tue, 24 May 2011 09:26:16 +0000 (11:26 +0200)]
Fix overlap io_u into outside regions

Test case is a job file ala this:

[global]
ioengine=posixaio
direct=0
filename=testfile
iodepth=128
size=50M
loops=10000000000
group_reporting=1
readwrite=randwrite
do_verify=1
verify=md5
verify_fatal=1
numjobs=1
thread
bsrange=4k:1M

[thread0]
offset=0G

[thread1]
offset=50M

[thread2]
offset=100M

[thread3]
offset=150M

[thread4]
offset=200M

[thread5]
offset=250M

[thread6]
offset=300M

[thread7]
offset=350M

where io_u units from threadN ends up crossing into the zone of threadN+1.
This is due to using file->real_file_size as the boundary check for the
buffer length, where the offset itself is correctly calculated. When
using mixed block sizes, this can cause a blocksize larger than the
minimum size to span into the next region.

Fix this by using file->io_size + offset as the end delimeter. Also
improve the logic of dropping to a smaller allowed block size if we do
end up being close to spanning into the next region.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agosplice: use ->o.memalign to indicate we want aligned buffers
Jens Axboe [Fri, 13 May 2011 19:19:25 +0000 (21:19 +0200)]
splice: use ->o.memalign to indicate we want aligned buffers

Vasilis Liaskovitis reports:

---

I am trying to use the splice ioengine to read/write from/to tmpfs (/dev/shm).

Writing works fine e.g. with the following fio config:
[sw]
rw=write
size=1g
directory=/dev/shm
iodepth=32
direct=0
blocksize=512k
numjobs=1
nrfiles=1
ioengine=splice
loops=1
group_reporting
create_on_open=1
create_serialize=0

But reading the same file afterwards with the splice ioengine fails:
[sw]
rw=read
size=1g
directory=/dev/shm
iodepth=32
direct=0
blocksize=512k
numjobs=1
nrfiles=1
ioengine=splice
loops=1
group_reporting

root@server1:~/fio# ./fio  /root/fior_splice.fio
sw: (g=0): rw=read, bs=512K-512K/512K-512K, ioengine=splice, iodepth=32
fio 1.53
Starting 1 process
fio: pid=17504, err=22/file:filesetup.c:502,
func=open(/dev/shm/sw.1.0), error=Invalid argument
Run status group 0 (all jobs):

Is something misconfigured in the fio read config?
The --debug=all output is:

Starting 1 process
file     17517 setup files
file     17517 get file size for 0x7f2f87451028/0/0x7f2f87451108
diskutil 17517 /dev/shm/sw.1.0 belongs to maj/min 0/17
process  17517 pid=0: runstate 0 -> 1
process  17517 will fork
mutex    17517 wait on startup_mutex
process  17519 jobs pid=17519 started
process  17519 pid=17519: runstate 1 -> 2
mutex    17519 up startup_mutex
mutex    17519 wait on td->mutex
mutex    17517 done waiting on startup_mutex
process  17517 pid=17519: runstate 2 -> 4
mutex    17519 done waiting on td->mutex
mem      17519 Alloc 16777216 for buffers
[...]
file     17519 trying file /dev/shm/sw.1.0 10
file     17519 fd open /dev/shm/sw.1.0
file     17519 file not found in hash /dev/shm/sw.1.0
file     17519 get file /dev/shm/sw.1.0, ref=0
io       17519 invalidate cache /dev/shm/sw.1.0: 0/1073741824
file     17519 fd close /dev/shm/sw.1.0
file     17519 error 1 on open of /dev/shm/sw.1.0
file     17519 get_next_file_rr: (nil)
fio: pid=17519, got signal=11
process  17517 pid=17519: runstate 4 -> 9
process  17517 terminate group_id=-1
process  17517 setting terminate on sw/17519
diskutil 17517 update io ticks

Other fio ioengines (sync, vsync) work fine for both reading/writing
on tmpfs. Also, other splice tests can do both reads and writes to
tmpfs (e.g. splice-cp).

This happens with current fio-master or fio-1.38-1 (debian package)
thanks for any comments,

---

The reason is that splice uses ->o.odirect to set that we want aligned
buffers, but we actually have a flag for that now. So just use that.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
12 years agoFio 1.54 fio-1.54
Jens Axboe [Thu, 12 May 2011 18:06:57 +0000 (20:06 +0200)]
Fio 1.54

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>