fio.git
7 years agoFio 1.60.2 fio-1.60.2
Jens Axboe [Wed, 26 Oct 2011 07:50:25 +0000 (09:50 +0200)]
Fio 1.60.2

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix segfault with verify_async
Steven Lang [Wed, 26 Oct 2011 07:46:50 +0000 (09:46 +0200)]
Fix segfault with verify_async

At some point the filename was added to the report on verify failures,
however this broke verify_async, as the file pointer on the io_u is
set to NULL before the verify thread sees the io_u. ┬áThe result is a
segfault when there is a verify mismatch.

This patch changes the semantics of a deferred free (IO_U_F_FREE_DEF)
to have already called put_file, but not set the file pointer to NULL.
┬áThis is safe to do as the file list is only freed after all the
verify threads have been terminated.

FYI - this is the minimal config to cause the error this patch fixes,
and is what I used to test the fix.  (Not using thread still causes a
segfault, but it is reported more subtly since just a sub-process hits
it.)

[segv]
ioengine=null
size=4k
verify=meta
rw=read
thread
verify_async=1

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMake verify_dump off by default
Jens Axboe [Tue, 25 Oct 2011 20:43:36 +0000 (22:43 +0200)]
Make verify_dump off by default

It's a change of behaviour, so it should be off by default.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd core for verify_dump option
Steven Lang [Tue, 25 Oct 2011 20:41:05 +0000 (22:41 +0200)]
Add core for verify_dump option

The verify_dump option is defined and documented, but does nothing.
This one-liner enables the option.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 1.60.1 fio-1.60.1
Jens Axboe [Sat, 22 Oct 2011 16:49:46 +0000 (18:49 +0200)]
Fio 1.60.1

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix io_u->buf calculation overflow
Jens Axboe [Sat, 22 Oct 2011 16:47:21 +0000 (18:47 +0200)]
Fix io_u->buf calculation overflow

Jagadish reports:

----

following are the details of the bug in fio.

This bug in fio can show up as corruption of data when performing verify.

Description:
----------------

if  the product of block size and queudepth is greater than 4GB, io_u
buffer will not
be assigned properly due to overflow.

fio --bsrange=256k-4m --ioengine=libaio --iodepth=2064 --direct=1
--name=job3 --offset=2GB --size=14GB --rw=write
--verify_pattern=0xdeadbeef --filename=/dev/sdb

can show false corruption.

Version:
-----------
1.58

Explanation:
-----------------

in a loop fio tries to assign the data buffer to each i/o request.

static int init_io_u(struct thread_data *td)
{
        struct io_u *io_u;
        unsigned int max_bs;
        int cl_align, i, max_units;
        char *p;
...
         p = td->orig_buffer;
...
         for (i = 0; i < max_units; i++) {
...
                    io_u->buf = p + max_bs * i;
         }
}

at max_bs=4M i=1024, the integer overflows and the addresses are being
used again.
i,e i/o request 1024 will have the same data buffer as that of i/o request 0.

This is seen from fio debug log.

mem 11164 io_u alloc 0x219f530, index 0
mem 11164 io_u 0x219f530, mem 0x7f09bb62d000
mem 11164 io_u alloc 0x219f820, index 1
mem 11164 io_u 0x219f820, mem 0x7f09bba2d000

mem 11164 io_u alloc 0x225b530, index 1024
mem 11164 io_u 0x225b530, mem 0x7f09bb62d000
mem 11164 io_u alloc 0x225b820, index 1025
mem 11164 io_u 0x225b820, mem 0x7f09bba2d000

the fix is as follows:

                        io_u->buf = p + (unsigned long long)max_bs * i;

----

Fix this by continually incrementing 'p' instead, avoiding
the overflow as well.

Reported-by: Jagadish Kumar <jagadish.mukku@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 1.60 fio-1.60
Jens Axboe [Thu, 13 Oct 2011 14:30:22 +0000 (16:30 +0200)]
Fio 1.60

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix bad random offset generation for file map
Jens Axboe [Wed, 12 Oct 2011 07:42:33 +0000 (09:42 +0200)]
Fix bad random offset generation for file map

If OS_RAND_MAX and FRAND_MAX are different, then ensure that
we use the right one. If not, our lookup cache could be outside
the allowed range, thus causing a segfault by indexing
f->file_map beyond end of the array.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoEscape minus signs in manpage to fix lintian warning:
Martin Steigerwald [Wed, 5 Oct 2011 09:41:54 +0000 (11:41 +0200)]
Escape minus signs in manpage to fix lintian warning:

I: fio: hyphen-used-as-minus-sign usr/share/man/man1/fio.1.gz:934

see:
http://lintian.debian.org/tags/hyphen-used-as-minus-sign.html

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix for hosed end-of-run log numbers
Jens Axboe [Wed, 5 Oct 2011 12:07:07 +0000 (14:07 +0200)]
Fix for hosed end-of-run log numbers

For queued IO workloads, we complete the pending count in one
go at the end of the run. But the completion time isn't updated,
so we end up getting a lot of tail log entries that are identical
if the timing is against us.

Fix this by only accounting the first of the last batch of ios.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 1.59 fio-1.59
Jens Axboe [Wed, 5 Oct 2011 08:28:11 +0000 (10:28 +0200)]
Fio 1.59

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoDon't overwrite -t timeout setting in fill_def_thread()
Jens Axboe [Wed, 5 Oct 2011 07:54:30 +0000 (09:54 +0200)]
Don't overwrite -t timeout setting in fill_def_thread()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoInclude http:// git url location
Jens Axboe [Thu, 29 Sep 2011 15:21:49 +0000 (09:21 -0600)]
Include http:// git url location

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoScramble corruption fix
Jens Axboe [Tue, 27 Sep 2011 20:27:48 +0000 (14:27 -0600)]
Scramble corruption fix

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix aiocb compile warnings on HPUX
Jens Axboe [Wed, 21 Sep 2011 07:38:01 +0000 (09:38 +0200)]
Fix aiocb compile warnings on HPUX

Unlike other platforms, it seems to require aiocb64 explicitly
for long offsets. Add an os independent typedef for this.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoSilence build warning on HPUX
Jens Axboe [Wed, 21 Sep 2011 07:27:24 +0000 (09:27 +0200)]
Silence build warning on HPUX

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 years agoScramble fix
Jens Axboe [Mon, 19 Sep 2011 09:33:30 +0000 (11:33 +0200)]
Scramble fix

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 years agoAdd option for controlling buffer scrambling
Jens Axboe [Mon, 19 Sep 2011 07:24:44 +0000 (09:24 +0200)]
Add option for controlling buffer scrambling

scramble_buffers bool option, defaults to on.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 years agoScramble on a 512b boundary
Jens Axboe [Fri, 16 Sep 2011 20:45:27 +0000 (22:45 +0200)]
Scramble on a 512b boundary

Instead of just scrambling the full block at the beginning at the
end, make it a bit more clever by:

- Doing it at 512b intervals to ensure full 512b de-dupe attemts, and
- Do it at a "random" offset into that block.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 years agoScramble default write buffer contents slightly
Jens Axboe [Fri, 16 Sep 2011 20:11:23 +0000 (22:11 +0200)]
Scramble default write buffer contents slightly

If we are worried about de-dupe making the output data too easy
to compress, but don't want the full incompressible guarantee
provided by refill_buffers=1 (since it's too costly), then lets
make their life a little harder by filling the offset of the IO
into the beginning of the buffer and the start time at the end
of the buffer. This will effectively default simple de-dupe
or compression attempts, while not doing the costly full buffer
refill.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 years agoFio 1.58 fio-1.58
Jens Axboe [Fri, 16 Sep 2011 06:44:10 +0000 (08:44 +0200)]
Fio 1.58

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 years agoFix sum_stat() bug in group_reporting
Zheng Liu [Fri, 16 Sep 2011 06:20:12 +0000 (08:20 +0200)]
Fix sum_stat() bug in group_reporting

Write samples in io_stat does not be counted with rw=randread, vice versa.
Then, in sum_stat() function, it will cause 0/0 in calculating mean and S,
and 'nan' is printed. So it should return immediately when src->samples is
equal to 0.

the error result is as follows:
  write: io=8328.0KB, bw=1582.7KB/s, iops=395 , runt=  5262msec
    clat (usec): min=58 , max=293353 , avg= -nan, stdev= -nan
     lat (usec): min=59 , max=293353 , avg= -nan, stdev= -nan
    bw (KB/s) : min=   92, max=  804, per=-nan%, avg= -nan, stdev= -nan

the configuration file used:

[global]
direct=1
ioengine=psync
bs=4k
filename=/dev/sdb1
runtime=5
group_reporting
loops=50

[read]
rw=randread
numjobs=8

[write]
rw=randwrite
numjobs=8

CC: Jens Axboe <jaxboe@fusionio.com>
CC: Yu-ju Hong <yjhong@google.com>
Signed-off-by: Zheng Liu <wenqing.lz@taobao.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 years agoFix inconsistencies in with man page <param>=<value>
Steven Noonan [Wed, 14 Sep 2011 07:21:33 +0000 (09:21 +0200)]
Fix inconsistencies in with man page <param>=<value>

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8 years agoImprove wording on terse Total IO (KB) field
Jens Axboe [Tue, 13 Sep 2011 08:12:12 +0000 (10:12 +0200)]
Improve wording on terse Total IO (KB) field

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoMerge branch 'master' of ssh://router/data/git/fio
Jens Axboe [Mon, 12 Sep 2011 09:30:01 +0000 (11:30 +0200)]
Merge branch 'master' of ssh://router/data/git/fio

8 years agoAdd terse version output format command line parameter
Jens Axboe [Fri, 9 Sep 2011 19:01:37 +0000 (21:01 +0200)]
Add terse version output format command line parameter

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix disk utilization for delayed jobs
Jens Axboe [Thu, 8 Sep 2011 12:30:24 +0000 (14:30 +0200)]
Fix disk utilization for delayed jobs

We initialized it all before starting the jobs, so we would
easily see disk stats that were too low for jobs that ended
up being started later.

Fix this by moving the disk util init right before the jobs
are spawned.

Reported-by: Shrirang Phansalkar <sphansalkar@stec-inc.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoStyle fixup
Jens Axboe [Thu, 8 Sep 2011 11:08:05 +0000 (13:08 +0200)]
Style fixup

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoBetter parser fix
Jens Axboe [Wed, 7 Sep 2011 20:11:00 +0000 (22:11 +0200)]
Better parser fix

The previous one broke all postfixes for ranges, that wasn't
very nice. This one allows the proper postfix and +/- as well,
as long as the latter is seen before a digit.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRevert "Fix parser bug with ranges"
Jens Axboe [Wed, 7 Sep 2011 20:10:11 +0000 (22:10 +0200)]
Revert "Fix parser bug with ranges"

This reverts commit f20485ad77872d6c5084dead547420386a6ecd5e.

8 years agoFix parser bug with ranges
Jens Axboe [Wed, 7 Sep 2011 19:21:07 +0000 (21:21 +0200)]
Fix parser bug with ranges

The change to allow -/+ for values broke ranges.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoUse r/R/w/W instead of m/M for mixed workloads with 100% reads or writes
Jens Axboe [Thu, 1 Sep 2011 15:58:11 +0000 (09:58 -0600)]
Use r/R/w/W instead of m/M for mixed workloads with 100% reads or writes

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoOnly clear string pointer in parser for sub-value options
Jens Axboe [Wed, 31 Aug 2011 22:45:03 +0000 (16:45 -0600)]
Only clear string pointer in parser for sub-value options

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoEnsure that buffer contents are random across jobs as well
Jens Axboe [Wed, 31 Aug 2011 21:20:15 +0000 (15:20 -0600)]
Ensure that buffer contents are random across jobs as well

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix parser using uninitialized memory
Jens Axboe [Wed, 31 Aug 2011 20:29:22 +0000 (14:29 -0600)]
Fix parser using uninitialized memory

Introduced by e2979754b554d6a48cffa98f41fb59bc045922ac.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd support for backwards holes
Jens Axboe [Wed, 31 Aug 2011 19:14:12 +0000 (13:14 -0600)]
Add support for backwards holes

If you did:

bs=4k
rw=read:-8k

you would essentially rewind by 8k after reading 4k, causing
the read to be sequentially backwards.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRemove duplicate code in parser
Jens Axboe [Wed, 31 Aug 2011 18:55:27 +0000 (12:55 -0600)]
Remove duplicate code in parser

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix FIO_OPT_STR_STORE without posval
Jens Axboe [Wed, 31 Aug 2011 18:49:49 +0000 (12:49 -0600)]
Fix FIO_OPT_STR_STORE without posval

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd sub-option support (sort-of) and convert libaio_userspace_reap
Jens Axboe [Wed, 31 Aug 2011 02:43:53 +0000 (20:43 -0600)]
Add sub-option support (sort-of) and convert libaio_userspace_reap

You should now use ioengine=libaio:userspace_reap instead of
the separate option.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdding userspace_libaio_reap option
Dan Ehrenberg [Wed, 31 Aug 2011 02:36:15 +0000 (20:36 -0600)]
Adding userspace_libaio_reap option

When a single thread is reading from a libaio io_context_t object
in a non-blocking polling manner (that is, with the minimum number
of events to return being 0), then it is possible to safely read
events directly from user-space, taking advantage of the fact that
the io_context_t object is a pointer to memory with a certain layout.
This patch adds an option, userspace_libaio_reap, which allows
reading events in this manner when the libaio engine is used.

You can observe its effect by setting iodepth_batch_complete=0
and seeing the change in distribution of system/user time based on
whether this new flag is set. If userspace_libaio_reap=1, then
busy polling takes place in userspace, and there is a larger amount of
usr CPU. If userspace_libaio_reap=0 (the default), then there is a
larger amount of sys CPU from the polling in the kernel.

Polling from a queue in this manner is several times faster. In my
testing, it took less than an eighth as much time to execute a
polling operation in user-space than with the io_getevents syscall.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix bad latency reporting for rated IO jobs
Jens Axboe [Tue, 30 Aug 2011 21:34:14 +0000 (15:34 -0600)]
Fix bad latency reporting for rated IO jobs

If we are going to sleep to satisfy a given rate limit, then
ensure that we flush pending IO first. Otherwise they end up
including the sleep time in the calculated completion latencies.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix compile on environment of SuperH
Nobuhiro Iwamatsu [Sat, 27 Aug 2011 21:34:37 +0000 (06:34 +0900)]
Fix compile on environment of SuperH

Signed-off-by: Nobuhiro Iwamatsu <iwamatsu@nigauri.org>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd 'null' perf test
Jens Axboe [Thu, 25 Aug 2011 18:20:14 +0000 (20:20 +0200)]
Add 'null' perf test

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoMake SH port work for packagers that don't differentiate between SH4 and SH4A
Jens Axboe [Thu, 25 Aug 2011 12:24:03 +0000 (14:24 +0200)]
Make SH port work for packagers that don't differentiate between SH4 and SH4A

Generic bits done by me, SH specific bits implemented by
Nobuhiro Iwamatsu <iwamatsu@nigauri.org>.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd possibility to make sequential IO "holed"
Jens Axboe [Thu, 25 Aug 2011 07:09:37 +0000 (09:09 +0200)]
Add possibility to make sequential IO "holed"

For sequential IO, it is now possible to add a number of bytes to
be skipped for every block read or written. Using:

bs=8k
rw=read:8k

will first read 0k->8k, then 16k->24k, and so on. This skips 8k
for every 'bs' sized block read. Similar for writes, doing

bs=4k
rw=write:4k

will write 0k->4k, then 8k->12k, etc. End result being that every
other block is written, in a sequential fashion.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoImprove Makefile for t/ test code
Jens Axboe [Tue, 16 Aug 2011 08:01:03 +0000 (10:01 +0200)]
Improve Makefile for t/ test code

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd simple stest smalloc tester
Jens Axboe [Tue, 16 Aug 2011 07:56:27 +0000 (09:56 +0200)]
Add simple stest smalloc tester

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFixup bad style in plat code
Jens Axboe [Tue, 16 Aug 2011 06:43:45 +0000 (08:43 +0200)]
Fixup bad style in plat code

- Comments
- Var declarations in code sections
- Misc style

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agostats: fix io_u_plat out-of-bound accesses (round 2)
Eric Gouriou [Tue, 16 Aug 2011 06:35:43 +0000 (08:35 +0200)]
stats: fix io_u_plat out-of-bound accesses (round 2)

Commit 833491908a1afd67 introduced the ability to report completion
latency percentiles. It also caused a memory corruption when running
with multiple threads due to out of bound accesses in show_run_stats().
The major index of the io_u_plat two-dimensional array is meant
to be DDIR_ value in {DDIR_READ, DDIR_WRITE} (i.e., {0, 1}). The
code in show_run_stats() incorrectly wrote into the array using
a major index with values {0, 1, 2}. Commit 0a0b49007cbce8d1 fixed
the out of bound accesses by increasing the size of the major
dimension of the io_u_plat array from 2 to 3.

This patch reverts the size change from 0a0b49007cbce8d1 in favor
of avoiding the out-of-bound accesses in show_run_stats().

Signed-off-by: Eric Gouriou <egouriou@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix bug in smalloc size calculation
Jiri Horky [Mon, 15 Aug 2011 13:18:03 +0000 (15:18 +0200)]
Fix bug in smalloc size calculation

Jiri reports:

I think I found a bug. If the device (file) used in fio has "wrong" size, and is used with "wrong" block size, the fio fails to allocate memory for randommap.
Following conditions must be true, considering just one thread for simplicity:

 1) number of blocks on the device are too big to fill in the default's pool
 2) number of blocks on the device must be "right", so the size of newly allocated pool by smalloc.c:add_pool is exactly as big as requested size (there are some roundings in place, so not every size triggers this error)

When this is true, function smalloc.c:__smalloc_pool fails to use the newly created (exactly big enough) pool, because of this line:

                idx = find_next_zero(pool->bitmap[i], last_idx);

which always returns non-zero value (the minimum returned value is last_idx +1), and so in our case the idx is 1 even though, the block number 0 is free. As the direct consequence, the function doesn't find enough free space in the given pool and returns NULL pointer. This leads to another try to allocate enough memory in smalloc.c:smalloc_real function, and so on...

The behavior could be reprocuded using both stable 1.57 and git version of the tool. The values that triggers the error are:
init_random_map - real_file_size: 21999995584512, o.rw_min_bs: 4096, blocks: 5371092672, num_maps: 83923323, alloc size: 671386584

whereas this block device is fine:
init_random_map - real_file_size: 19999995985920, o.rw_min_bs: 4096, blocks: 4882811520, num_maps: 76293930, alloc size: 610351440

the configuration file used:

[global]
description=CESNET_test
[cesnet]
filename=/dev/sdd
rw=randread
size=10000g
ioengine=libaio
iodepth=4
bs=4k
runtime=10m
time_based
numjobs=1
group_reporting

Get rid of the +1, it looks like a remnant from when the pools
had a header associated with them.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRDMA engine: Change wording of HOWTO section
Bart Van Assche [Mon, 15 Aug 2011 07:01:05 +0000 (09:01 +0200)]
RDMA engine: Change wording of HOWTO section

Change "iWarp" into "iWARP" ("iWarp" is the name of a supercomputer) and
"engines" into "protocols".

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Ren Yufei <renyufei83@yahoo.com.cn>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix off-by-one in io_u_plat[] array sizing
Jens Axboe [Mon, 15 Aug 2011 07:00:28 +0000 (09:00 +0200)]
Fix off-by-one in io_u_plat[] array sizing

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agostats: Add a function to report completion latency percentiles
Yu-ju Hong [Fri, 12 Aug 2011 22:53:44 +0000 (00:53 +0200)]
stats: Add a function to report completion latency percentiles

    This patch introduces two new options:
    1) Specifying --clat_percentiles enables the reporting of
       completion latency percentiles.
    2) --percentile_list allows the user to customize the
       list of percentiles to report.

Signed-off-by: Yu-ju Hong <yjhong@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoman page typo
Jens Axboe [Mon, 8 Aug 2011 07:47:13 +0000 (09:47 +0200)]
man page typo

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoUpdate softrandommap in HOWTO
Jens Axboe [Mon, 8 Aug 2011 07:07:28 +0000 (09:07 +0200)]
Update softrandommap in HOWTO

Previous commit fixed the typo in the man page, update the HOWTO
as well.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoMerge branch 'master' of ssh://router/data/git/fio
Jens Axboe [Mon, 8 Aug 2011 07:06:45 +0000 (09:06 +0200)]
Merge branch 'master' of ssh://router/data/git/fio

8 years agoFix a typo in fio.1
Zhu Yanhai [Mon, 8 Aug 2011 07:06:27 +0000 (09:06 +0200)]
Fix a typo in fio.1

softrandommap requires an bool argument actually.

Signed-off-by: Zhu Yanhai <gaoyang.zyh@taobao.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRDMA I/O engine: Fix compiler warnings
Bart Van Assche [Sun, 7 Aug 2011 19:50:52 +0000 (21:50 +0200)]
RDMA I/O engine: Fix compiler warnings

Fix 32-bit mode compiler warnings and a warning about an unused variable.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Ren Yufei <renyufei83@yahoo.com.cn>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRDMA engine: rewrite header comment block and man page section
Bart Van Assche [Sun, 7 Aug 2011 19:50:51 +0000 (21:50 +0200)]
RDMA engine: rewrite header comment block and man page section

Spelling and grammar fixes for the header comment. Also make it clear that
users can choose between the RDMA software provided by their Linux distributor
and OFED. Update a related paragraph in the man page too.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Ren Yufei <renyufei83@yahoo.com.cn>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRDMA engine spelling fix: change FIO_RDMA_MAX_IO_DPETH into FIO_RDMA_MAX_IO_DEPTH
Bart Van Assche [Sun, 7 Aug 2011 19:50:51 +0000 (21:50 +0200)]
RDMA engine spelling fix: change FIO_RDMA_MAX_IO_DPETH into FIO_RDMA_MAX_IO_DEPTH

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Cc: Ren Yufei <renyufei83@yahoo.com.cn>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agofio_generate_plots with setable resolution
Martin Steigerwald [Fri, 5 Aug 2011 17:47:51 +0000 (19:47 +0200)]
fio_generate_plots with setable resolution

Specify xres and yres as 2nd and 3rd argument, otherwise
defaults to 1024x768.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd 'wait_for_previous' as an alias for the 'stonewall' option
Jens Axboe [Wed, 3 Aug 2011 10:38:39 +0000 (12:38 +0200)]
Add 'wait_for_previous' as an alias for the 'stonewall' option

Also update man page and HOWTO, plus add missing alias for
iomem_align while in there.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRDMA update
Yufei Ren [Tue, 2 Aug 2011 10:15:30 +0000 (12:15 +0200)]
RDMA update

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRDMA IO engine
ren yufei [Mon, 1 Aug 2011 08:01:57 +0000 (10:01 +0200)]
RDMA IO engine

I have hacked an rdma ioengine based on OFED for fio which could test
both rdma memory semantic (rdma_write/rdma_read) and channel semantic
(send/recv). Would you like to merge this engine into fio?

notes
1) RDMA engine works in IB, iWarp and RoCE.
2) RDMA engine is disable by default. To enable it, execute the following
   before compile:

$ export EXTFLAGS="-DFIO_HAVE_RDMA"
$ export EXTLIBS="-libverbs -lrdmacm"

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd proper include to silence build warning on NetBSD/FreeBSD
Jens Axboe [Sat, 30 Jul 2011 11:29:58 +0000 (13:29 +0200)]
Add proper include to silence build warning on NetBSD/FreeBSD

Reported-by: Bruce Cran <bruce@cran.org.uk>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd version number to --help output
Bruce Cran [Sat, 30 Jul 2011 10:55:52 +0000 (12:55 +0200)]
Add version number to --help output

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agostats: Fix computation of summed standard deviation
Yu-ju Hong [Sat, 30 Jul 2011 07:18:13 +0000 (09:18 +0200)]
stats: Fix computation of summed standard deviation

Fix the computation of standard deviation for a group
of jobs. Please see the below link for the
approximation formula used.
<http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

The formula was originally presented here:
<ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/79/773/
CS-TR-79-773.pdf>

Signed-off-by: Yu-ju Hong <yjhong@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix bsrange read,write value option pairs
Dave Engberg [Sat, 23 Jul 2011 19:07:13 +0000 (21:07 +0200)]
Fix bsrange read,write value option pairs

The 'bs' and 'bssplit' configuration variables accept separate numbers
for read and write operations if you separate them with a comma. For
example: bssplit=1k/39:8k/50:64k/11,1k/29:8k/50:64k/21

The documentation for 'bsrange' says it works the same way, but if I try
to specify two ranges with a comma, I get an error when I run the app.
E.g. if I put this line in the configuration file: bsrange=1k-16k,2k-32k

I get this error output:

fio: pid=26861, err=22/file:engines/sync.c:62, func=xfer, error=Invalid argument

This patch fixes it.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix --timeout as global parameter
Jens Axboe [Sat, 23 Jul 2011 11:43:13 +0000 (13:43 +0200)]
Fix --timeout as global parameter

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoLibaio engine support for iodepth_batch_complete=0
Dan Ehrenberg [Sat, 23 Jul 2011 11:36:46 +0000 (13:36 +0200)]
Libaio engine support for iodepth_batch_complete=0

Previously, even if iodepth_batch_complete=0, the libaio engine
passed in non-zero values for the min_nr value for io_getevents.
This patch makes min_nr always 0 if iodepth_batch_complete == 0,
and if multiple events are required at a higher level, then
we poll io_getevents multiple times for the events.

Signed-off-by: Dan Ehrenberg <dehrenberg@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRespect iodepth_batch_complete=0 in main loop
Dan Ehrenberg [Sat, 23 Jul 2011 11:36:44 +0000 (13:36 +0200)]
Respect iodepth_batch_complete=0 in main loop

If the iodepth_batch_complete is 0, then even if the queue is full,
we still don't make min_events be 1 in the main loop. Instead, poll
in a non-blocking way until a result is found.

Signed-off-by: Dan Ehrenberg <dehrenberg@google.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAddes Trace file format description
Paul Dubs [Thu, 21 Jul 2011 15:26:02 +0000 (17:26 +0200)]
Addes Trace file format description

8 years agoAdded description for iops output
Paul Dubs [Thu, 21 Jul 2011 14:04:52 +0000 (16:04 +0200)]
Added description for iops output

8 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Tue, 19 Jul 2011 14:28:33 +0000 (16:28 +0200)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

8 years agoFix Solaris compile
Jens Axboe [Tue, 19 Jul 2011 14:27:43 +0000 (16:27 +0200)]
Fix Solaris compile

In file included from fio.h:24,
                  from gettime.c:10:
os/os.h:192: error: redefinition of 'gettid'
os/os-solaris.h:123: error: previous definition of 'gettid' was here
make: *** [gettime.o] Error 1

Forgot to define FIO_HAVE_GETTID

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoMake string-set options behave more like bool options
Jens Axboe [Fri, 15 Jul 2011 07:09:15 +0000 (09:09 +0200)]
Make string-set options behave more like bool options

Before this change, string-set options (like time_based) would not
complain if passed an argument, they would just always evaluate
to being set. This is very confusing if someone uses them negated,
ala:

time_based=0

since fio would interpret that as the option being set. Now we'll
do the right thing, time_based=0 will be identical to not having
the option set. And time_based=1 or just time_based will equate
to the option being set.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFio 1.57 fio-1.57
Jens Axboe [Thu, 14 Jul 2011 18:20:25 +0000 (20:20 +0200)]
Fio 1.57

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoWindows affinity fix for thread based jobs
Bruce Cran [Thu, 14 Jul 2011 14:34:02 +0000 (16:34 +0200)]
Windows affinity fix for thread based jobs

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix infinite loop on platforms with severely limited aio resources
Jens Axboe [Thu, 14 Jul 2011 10:45:13 +0000 (12:45 +0200)]
Fix infinite loop on platforms with severely limited aio resources

Problem hit on OSX, where with 4 jobs each with a queue depth larger
than what the system supports, you can get into a situation where
any given process can get EAGAIN on queueing AIO even if it has
nothing queued already. Check for this condition to avoid fio's
IO loop going into a death spiral.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoMake raw disk size detection work on OSX
Jens Axboe [Thu, 14 Jul 2011 07:12:50 +0000 (09:12 +0200)]
Make raw disk size detection work on OSX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix compile warning on OSX
Jens Axboe [Thu, 14 Jul 2011 06:39:19 +0000 (08:39 +0200)]
Fix compile warning on OSX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix killing of threads that haven't started
Michael Callahan [Thu, 14 Jul 2011 05:56:37 +0000 (07:56 +0200)]
Fix killing of threads that haven't started

If there is a thread that has not started yet (for instance with a long
startdelay or a stonewall) then a SIGINT will fire off terminate_threads
which runs through all of the threads and calls kill(pid, SIGTERM).
However the threads that have not started yet have a pid of 0.  When you
call kill(0, SIGTERM) it kills the entire thread group which can extend
out of fio.  I have included a short patch which fixes this for me.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd CPU affinity support to Windows
Bruce Cran [Thu, 14 Jul 2011 05:50:43 +0000 (07:50 +0200)]
Add CPU affinity support to Windows

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix AIX Makefile
Michael Perzl [Wed, 13 Jul 2011 18:28:07 +0000 (20:28 +0200)]
Fix AIX Makefile

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix AIX compile
Michael Perzl [Wed, 13 Jul 2011 16:43:57 +0000 (18:43 +0200)]
Fix AIX compile

When adding HPUX support the compile for AIX was broken in "lib/getopt.h".

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFio 1.56 fio-1.56
Jens Axboe [Wed, 13 Jul 2011 10:42:26 +0000 (12:42 +0200)]
Fio 1.56

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agogettid() for NetBSD
Jens Axboe [Tue, 12 Jul 2011 21:49:16 +0000 (23:49 +0200)]
gettid() for NetBSD

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd gettid() for FreeBSD/OSX/Solaris
Jens Axboe [Tue, 12 Jul 2011 20:33:53 +0000 (22:33 +0200)]
Add gettid() for FreeBSD/OSX/Solaris

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAdd gettid() for proper thread affinity on Linux
Jens Axboe [Tue, 12 Jul 2011 19:17:49 +0000 (21:17 +0200)]
Add gettid() for proper thread affinity on Linux

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix compile on older systems that don't have fallocate() on Linux
Jens Axboe [Tue, 12 Jul 2011 18:49:46 +0000 (20:49 +0200)]
Fix compile on older systems that don't have fallocate() on Linux

Commit a596f047 broke build en eg CentOS 5:

filesetup.c:17:26: error: linux/falloc.h: No such file or directory
filesetup.c: In function 'extend_file':
filesetup.c:95: warning: implicit declaration of function 'fallocate'
filesetup.c:95: error: 'FALLOC_FL_KEEP_SIZE' undeclared (first use in this function)
filesetup.c:95: error: (Each undeclared identifier is reported only once
filesetup.c:95: error: for each function it appears in.)
make: *** [filesetup.o] Error 1

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoAllow percentage setting for size=
Jens Axboe [Tue, 12 Jul 2011 17:47:03 +0000 (19:47 +0200)]
Allow percentage setting for size=

Sometimes it's useful to set this to a particular size of a device.
Allowing percentages makes this more easy, as one does not have to
do the math outside of fio and pass in as environment variables
or custom job files.

To use, simply add a

size=15%

or whatever is needed, then fio will use 15% of the total size of
the device(s) or file(s) given. Percentage can be from 1 to 100.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoHP-UX: add fdatasync()
Jens Axboe [Mon, 11 Jul 2011 12:43:26 +0000 (14:43 +0200)]
HP-UX: add fdatasync()

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoHP-UX: has aio_fsync() and fallocate()
Jens Axboe [Mon, 11 Jul 2011 08:56:32 +0000 (10:56 +0200)]
HP-UX: has aio_fsync() and fallocate()

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoHP-UX disk query size
Jens Axboe [Mon, 11 Jul 2011 08:48:51 +0000 (10:48 +0200)]
HP-UX disk query size

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoRemove old debug printf()
Jens Axboe [Mon, 11 Jul 2011 08:26:00 +0000 (10:26 +0200)]
Remove old debug printf()

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFill in os_phys_mem() stub for HP-UX
Jens Axboe [Mon, 11 Jul 2011 08:25:35 +0000 (10:25 +0200)]
Fill in os_phys_mem() stub for HP-UX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoFix bug in $mb_memory keyword
Jens Axboe [Mon, 11 Jul 2011 08:24:20 +0000 (10:24 +0200)]
Fix bug in $mb_memory keyword

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoTake __weak out of the compiler namespace
Bruce Cran [Sun, 10 Jul 2011 18:56:23 +0000 (20:56 +0200)]
Take __weak out of the compiler namespace

With Xcode 4 on OS X shipping LLVM by default, I tried building fio with
clang and found that the definition of __weak conflicts with something
clang defines.  I've attached a patch which replaces the double
underscores with a single underscore, removing it from the
compiler-dependant namespace.

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoposixaio: fix for HP-UX
Jens Axboe [Sat, 9 Jul 2011 11:33:38 +0000 (13:33 +0200)]
posixaio: fix for HP-UX

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
8 years agoHP-UX typo
Jens Axboe [Sat, 9 Jul 2011 11:28:19 +0000 (13:28 +0200)]
HP-UX typo

Signed-off-by: Jens Axboe <jaxboe@fusionio.com>