fio.git
7 years agoFio 2.0.8 fio-2.0.8
Jens Axboe [Tue, 29 May 2012 11:29:12 +0000 (13:29 +0200)]
Fio 2.0.8

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoeta: ETA bandwidth was off by 1.024
Jens Axboe [Fri, 11 May 2012 18:33:02 +0000 (20:33 +0200)]
eta: ETA bandwidth was off by 1.024

Same fix as 033bbb51 essentially. Since mtime is a 1000th of a second,
for the usual option of having 1024 be the KB base, we end up being
off by 1.024 if we don't multiply by 1000 before dividing by the
runtime (and then dividing by 1024 to get to next power-of-2).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix hyphen as minus sign errors by escaping minus signs.
Martin Steigerwald [Mon, 7 May 2012 15:06:54 +0000 (17:06 +0200)]
Fix hyphen as minus sign errors by escaping minus signs.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix manpage error warning: macro `hostname'' not defined.
Martin Steigerwald [Mon, 7 May 2012 15:06:26 +0000 (17:06 +0200)]
Fix manpage error warning: macro `hostname'' not defined.

Fix manpage error warning: macro `hostname'' not defined by using
different line wrapping.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix spelling error in manpage.
Martin Steigerwald [Mon, 7 May 2012 15:06:13 +0000 (17:06 +0200)]
Fix spelling error in manpage.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix terse bandwidth
Jens Axboe [Mon, 7 May 2012 07:46:40 +0000 (09:46 +0200)]
Fix terse bandwidth

Off by 1.024

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Wed, 2 May 2012 12:29:21 +0000 (14:29 +0200)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

7 years agoRemove deprecated rwmixcycle option from the man page
Jens Axboe [Wed, 2 May 2012 11:59:09 +0000 (13:59 +0200)]
Remove deprecated rwmixcycle option from the man page

It has no effect, remove it.

Suggested-by: Roger Sibert <Roger_Sibert@xyratex.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd create_only option
Jens Axboe [Wed, 2 May 2012 11:58:02 +0000 (13:58 +0200)]
Add create_only option

If this option is set, then fio will only run the preparation
phase of a job. This means it will create all the files it
needs, but exit after that.

Suggested-by: Georg Schönberger <gschoenberger@thomas-krenn.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix man page type
Jens Axboe [Thu, 26 Apr 2012 17:39:47 +0000 (19:39 +0200)]
Fix man page type

"one" -> "done".

Reported-by: Roger Sibert <Roger_Sibert@xyratex.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agofio: Add to cgroup if parameter "cgroup" is enabled.
Tao Ma [Thu, 26 Apr 2012 10:15:46 +0000 (12:15 +0200)]
fio: Add to cgroup if parameter "cgroup" is enabled.

Currently, we only add the tasks to cgroup if cgroup_weight is set by
the user. But if we want to test other blkcg parameters like block
throttle, the fio tasks isn't added to the corresponding cgroup and
it doesn't make sense.

So add the task to the corresponding cgroup if the cgroup name is given.

Signed-off-by: Tao Ma <boyu.mt@taobao.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix for crash with more than ~500 jobs
Jens Axboe [Wed, 18 Apr 2012 18:39:48 +0000 (20:39 +0200)]
Fix for crash with more than ~500 jobs

We simply overwrite our status output buffer, not a terribly
good idea.

Reported-by: Roger Sibert <Roger_Sibert@xyratex.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoWindows build fixes
Bruce Cran [Mon, 16 Apr 2012 17:33:03 +0000 (19:33 +0200)]
Windows build fixes

Bruce writes:

SIGUSR1 isn't defined on Windows so I've defined it.

Also, MSI packages need a version string of x.x.x.x where x is between 0
and 65535 so we can't include the git hash. It's easier just to go back
to hard-coding the version for now, and I'll update it (and the product
GUID) when I create a new package (it doesn't need checked in each
time).

Also, Cygwin doesn't like $(SHELL_PATH) but $(SHELL) seems to work -
does $(SHELL) cause a problem on other platforms?

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoEnlarge probe version field
Jens Axboe [Mon, 16 Apr 2012 09:44:48 +0000 (11:44 +0200)]
Enlarge probe version field

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoGet rid of fio_version.h
Jens Axboe [Mon, 16 Apr 2012 07:52:22 +0000 (09:52 +0200)]
Get rid of fio_version.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix for non-git versions
Jens Axboe [Fri, 13 Apr 2012 17:18:49 +0000 (19:18 +0200)]
Fix for non-git versions

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoEnsure that init depends on FIO-VERSION-FILE
Jens Axboe [Fri, 13 Apr 2012 12:06:08 +0000 (14:06 +0200)]
Ensure that init depends on FIO-VERSION-FILE

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Conflicts:

Makefile

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoDocument the use of the USR1 signal for dumping runtime statistics
Jens Axboe [Thu, 12 Apr 2012 07:18:38 +0000 (09:18 +0200)]
Document the use of the USR1 signal for dumping runtime statistics

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix repeated link of fio
Jens Axboe [Wed, 11 Apr 2012 20:24:19 +0000 (22:24 +0200)]
Fix repeated link of fio

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMake fio include the git version in the version output
Jens Axboe [Wed, 11 Apr 2012 20:07:07 +0000 (22:07 +0200)]
Make fio include the git version in the version output

Makes it easier in bug reporting, don't have to ask people what
they are running...

Here's to you, Kep.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Conflicts:

Makefile

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoDefine SIGUSR1 to inform fio to dump run stats while continuing to run
Jens Axboe [Fri, 30 Mar 2012 08:30:35 +0000 (10:30 +0200)]
Define SIGUSR1 to inform fio to dump run stats while continuing to run

For long running jobs, it may be interesting to see the stats without
having to terminate the job. Set up SIGUSR1 so that it does that,
gives you the stats at that point in time.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFree disk util on exit, not on stats display
Jens Axboe [Fri, 30 Mar 2012 06:33:27 +0000 (08:33 +0200)]
Free disk util on exit, not on stats display

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.7 fio-2.0.7
Jens Axboe [Wed, 11 Apr 2012 12:49:06 +0000 (14:49 +0200)]
Fio 2.0.7

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoHOWTO: typo: write_log_log -> write_lat_log + use suffix
Lucian Adrian Grijincu [Fri, 6 Apr 2012 00:18:35 +0000 (18:18 -0600)]
HOWTO: typo: write_log_log -> write_lat_log + use suffix

- I guess 'write_log_log' was a copy-pasta typo.

- You used 'suffix' everywhere except for two places where you said
  'postfix'.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogroup reporting: fix bad values of min/max
Jens Axboe [Thu, 5 Apr 2012 14:42:11 +0000 (08:42 -0600)]
group reporting: fix bad values of min/max

If you look at the fio output, the group reporting min and max
bandwidth don't match what the actual job reports. This is due
to an error in calculating the min and max values. The aggregate
value is correct, and matches what the job reports.

This did not affect the values that are usually used for reporting,
it's only affecting the group run status output.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoiolog: remove assert in io_u overlap
Jens Axboe [Wed, 4 Apr 2012 20:11:58 +0000 (14:11 -0600)]
iolog: remove assert in io_u overlap

This can happen very rarely for time based jobs. Before we
had this as an assert since it was an impossible to hit
condition, but now it can be expected very rarely with
verify and time based jobs.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoWindows fixes
Bruce Cran [Wed, 4 Apr 2012 14:35:13 +0000 (08:35 -0600)]
Windows fixes

Only return from fio_windowsaio_getevents if the minimum number of IOs has
completed.

Add posix_fallocate implementation to avoid extending the file during the test.

Move call to ResetEvent into windowsaio_getevents where other reset code is
located.

Fix tabs vs. spaces in windowsaio.c and the installer sources.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix address truncation on Windows
Bruce Cran [Wed, 4 Apr 2012 00:00:00 +0000 (18:00 -0600)]
Fix address truncation on Windows

Avoid having PAGE_ALIGN truncate addresses by using a pointer-sized
variable for the page size and mask.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoHOWTO: emphasize units in --minimal mode: latency in usec, bw in KB/s
Lucian Adrian Grijincu [Tue, 3 Apr 2012 20:42:48 +0000 (14:42 -0600)]
HOWTO: emphasize units in --minimal mode: latency in usec, bw in KB/s

It wasn't clear what unit were used in --minimal mode.

Signed-off-by: Lucian Adrian Grijincu <lucian@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd indication of whether a job got killed
Jens Axboe [Mon, 2 Apr 2012 16:47:09 +0000 (09:47 -0700)]
Add indication of whether a job got killed

We have 'X' for exited with error, add 'K' for killed by
signal as well.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoHOWTO: update to note base of latencies in minimal/terse output
Jens Axboe [Mon, 2 Apr 2012 14:03:26 +0000 (07:03 -0700)]
HOWTO: update to note base of latencies in minimal/terse output

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoEnhance the reaped status display
Jens Axboe [Fri, 30 Mar 2012 19:21:20 +0000 (21:21 +0200)]
Enhance the reaped status display

Instead of just printing a '_' for a job that has exited,
allow the user to tell if the job exited with an error or
not. A normally reaped job that ran to completion without
errors will have the '_' displayed. If the job exited with
an error, then an 'X' will be displayed instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agostat: move final \n of terse output to the end
Jens Axboe [Fri, 23 Mar 2012 18:22:21 +0000 (19:22 +0100)]
stat: move final \n of terse output to the end

Don't print a new line, description, then another new line.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd regression test for ae2fafc8
Jens Axboe [Fri, 23 Mar 2012 12:43:19 +0000 (13:43 +0100)]
Add regression test for ae2fafc8

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoverify: verify bytes should not add to this_io_bytes
webee [Fri, 23 Mar 2012 12:41:41 +0000 (13:41 +0100)]
verify: verify bytes should not add to this_io_bytes

When rw=readwrite,verify+backlog, the verify are treated as normal
read, and bytes are added to this_io_bytes, cause fio terminate early.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAllow readwrite as an alias for rw in setting sequential read/write
Jens Axboe [Fri, 23 Mar 2012 12:40:06 +0000 (13:40 +0100)]
Allow readwrite as an alias for rw in setting sequential read/write

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.6 fio-2.0.6
Jens Axboe [Fri, 23 Mar 2012 07:09:44 +0000 (08:09 +0100)]
Fio 2.0.6

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix terse output with description set
Vikram Seth [Fri, 23 Mar 2012 07:09:09 +0000 (08:09 +0100)]
Fix terse output with description set

I have seen that using 2.05 version of fio the terse output formatting
has an issue when there is a text description given in the test file.
There is no newline separating the description and the next job stats.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdded 99.95 and 99.99 default percentiles.
Steven Lang [Sat, 17 Mar 2012 12:16:28 +0000 (13:16 +0100)]
Added 99.95 and 99.99 default percentiles.

Since there are 20 slots for percentiles, but not all used by default,
I figure it doesn't hurt to add a couple extra percentiles to the
default for tail latency capture.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd regression test for 37cf9e3c
Jens Axboe [Sat, 17 Mar 2012 11:59:12 +0000 (12:59 +0100)]
Add regression test for 37cf9e3c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoEnable read/write sequential offset to be a non-multiple of the block size
Jens Axboe [Sat, 17 Mar 2012 11:54:30 +0000 (12:54 +0100)]
Enable read/write sequential offset to be a non-multiple of the block size

The way that fio does offset math internally, if a non-multiple
of the blocksize was given as a sequence offset, it would simply
be ignored due to the block offset division rounding it down.
In other words, if you did:

rw=read:512k
bs=1m

it did not read 1m and 0 offset, 1m at 512k offset, etc. Instead
the hole is ignored.

Fix this by leaving the generated sequential offset in bytes
instead of turning it into a 'fio block'.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agotime_based: Avoid restarting main I/O loop
Daniel Ehrenberg [Fri, 16 Mar 2012 17:54:15 +0000 (18:54 +0100)]
time_based: Avoid restarting main I/O loop

Previously, when fio had written a volume of I/O equal to the size
argument, it restarted the main do_io loop.

This patch allows time_based tests to be run for longer than one
cycle in the do_io main loop. This has a couple of advantages:
 * The random number generator is not reset on each iteration
   of the loop, so running longer will reach different locations.
 * There is not a throughput-reducing point where all operations
   must be reaped before new operations are submitted.

The implementation consists of two minor changes:
 * In the do_io loop, a time_based test will not exit the loop for
   reading or writing too much data.
 * When reading or writing sequentially, the operations wrap around
   to the beginning after reading the end within the
   get_next_seq_block function.

Modified by Jens to properly restart when using a randommap
as well.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix problem with iodepth_batch_complete=0 and accounting
Jens Axboe [Fri, 16 Mar 2012 12:55:50 +0000 (13:55 +0100)]
Fix problem with iodepth_batch_complete=0 and accounting

If the queue is full, we always need to reap at least one event.
Otherwise we end up failing get_io_u() and restarting the file.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix wrap bug in mtime_since()
Jens Axboe [Fri, 16 Mar 2012 12:55:27 +0000 (13:55 +0100)]
Fix wrap bug in mtime_since()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoDocument compress_buffer_percentage, compress_buffer_chunk
Jens Axboe [Thu, 15 Mar 2012 14:02:56 +0000 (15:02 +0100)]
Document compress_buffer_percentage, compress_buffer_chunk

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoNote offset_increment parent option
Jens Axboe [Thu, 15 Mar 2012 13:53:38 +0000 (14:53 +0100)]
Note offset_increment parent option

And move it up near "offset".

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix offset_increment option and add man page entry for it
Jens Axboe [Thu, 15 Mar 2012 13:50:58 +0000 (14:50 +0100)]
Fix offset_increment option and add man page entry for it

td->thread_number starts at 1, not 0. So we need to subtract
one before multiplying with the offset.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agordma engine graceful teardown
Yufei Ren [Thu, 15 Mar 2012 13:44:47 +0000 (14:44 +0100)]
rdma engine graceful teardown

For rdma ioengine's one side operation, RDMA_WRITE or RDMA_READ,
server side gets a message from client side that the task is finished.
In previous version, the server simply exit() in td_io_commit() which
causes segmentation fault in thread mode or process hanging in process
mode. This patch achieves graceful tear down by setting up `td->done`
as UDP engine did.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoNew offset_increment option
Dan Ehrenberg [Thu, 15 Mar 2012 13:44:26 +0000 (14:44 +0100)]
New offset_increment option

This patch adds a new option to fio job files. It is described
in the HOWTO as follows:

offset_increment=int If this is provided, then the real offset becomes
the offset + offset_increment * thread_number, where the
thread number is a counter that starts at 0 and is incremented
for each job. This option is useful if there are several jobs
which are intended to operate on a file in parallel in disjoint
segments, with even spacing between the starting points.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd regression test for 82af2a7c
Jens Axboe [Tue, 13 Mar 2012 12:55:38 +0000 (13:55 +0100)]
Add regression test for 82af2a7c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix failure to verify in mixed read/write workload with backlog
Jens Axboe [Tue, 13 Mar 2012 12:45:58 +0000 (13:45 +0100)]
Fix failure to verify in mixed read/write workload with backlog

If you run a workload like this:

fio --rw=randrw --bs=4k --direct=1 --ioengine=libaio --iodepth=32
--verify=meta --verify_backlog=1024 --verify_fatal=1 --name=ver-test
--filename=foo --size=1G --verify_pattern=0xaaa

Fio ends up never actually verifying the written blocks. This happens
because as we generate an entry to be verified, the backend checks
whether this is a read/write mixed workload. It then thinks that the
READ is just that, a normal READ, when in fact it could be coming
from our verify list.

Mark such a read as coming from our verify list, so that we know if
it's a "normal" read or one generated to verify previously written
data.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMark buffer_compress_chunk as child of buffer_compress_percentage
Jens Axboe [Fri, 9 Mar 2012 18:11:25 +0000 (19:11 +0100)]
Mark buffer_compress_chunk as child of buffer_compress_percentage

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd buffer_compress_chunk option
Jens Axboe [Fri, 9 Mar 2012 18:06:24 +0000 (19:06 +0100)]
Add buffer_compress_chunk option

Instead of doing bs/2 of incompressible and zeroed data for a 50%
compression setting, use the smaller chunk size specified by
this option.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoHonor refill_buffers even for verifies
Jens Axboe [Fri, 9 Mar 2012 18:02:01 +0000 (19:02 +0100)]
Honor refill_buffers even for verifies

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoCompression buffer fill fix
Jens Axboe [Fri, 9 Mar 2012 18:00:31 +0000 (19:00 +0100)]
Compression buffer fill fix

We need to advance after the zero fill as well.

Reported-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix batch mismatch for verifies
Jens Axboe [Wed, 7 Mar 2012 08:32:58 +0000 (09:32 +0100)]
Fix batch mismatch for verifies

If we set verify_backlog=32, we expect 32 writes followed by 32
reads that verify the written data. But if we fail getting a
verify io_u, then we may decrement the batch counter when we should
not. The result is that we write 32, read 32, write 1, read 1, repeat.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd regression test for f7078f7b
Jens Axboe [Wed, 7 Mar 2012 08:26:13 +0000 (09:26 +0100)]
Add regression test for f7078f7b

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix bug that causes early termination of fio with verify_backlog
Jens Axboe [Wed, 7 Mar 2012 08:24:05 +0000 (09:24 +0100)]
Fix bug that causes early termination of fio with verify_backlog

Running this job:

[global]
bs=4k
ioengine=sync
size=100m
direct=1
filename=xxx

[write]
verify=md5
verify_backlog=32
rw=write

causes fio to exit when reads + writes have exceeded 100m, where
it should only exit when writes have exceeded 100m. Fix this by
checking the appropriate counters based on what type of job it is.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoserver: fix bad win32 indentation
Jens Axboe [Tue, 6 Mar 2012 18:34:22 +0000 (19:34 +0100)]
server: fix bad win32 indentation

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd regression test for 8a99fdf6
Jens Axboe [Tue, 6 Mar 2012 18:26:42 +0000 (19:26 +0100)]
Add regression test for 8a99fdf6

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix another verify segfault
Jens Axboe [Tue, 6 Mar 2012 18:24:49 +0000 (19:24 +0100)]
Fix another verify segfault

If you run this job:

[global]
ioengine=libaio
direct=1
filename=/tmp/foo
iodepth=128
size=10M
loops=1
group_reporting=1
readwrite=write
do_verify=0
verify=md5
numjobs=1
thread
verify_dump=1

[small_writes]
offset=0G
blocksize=512
verify_interval=1M

[large_writes]
stonewall
offset=1G
blocksize=1M
verify_interval=512

fio crashes because verify_interval is larger than the block size
in the small_writes job. Check for this in get_hdr_inc().

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd regression test for verify bug
Jens Axboe [Tue, 6 Mar 2012 18:23:54 +0000 (19:23 +0100)]
Add regression test for verify bug

It was a bug in the job files, not a zone bug after all.
The two block sizes were transposed between the pre and
post job.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAvoid segfault if verify header appears longer than io_u
Jens Axboe [Tue, 6 Mar 2012 16:46:44 +0000 (17:46 +0100)]
Avoid segfault if verify header appears longer than io_u

Running this job:

[global]
ioengine=libaio
direct=1
filename=/tmp/foo
iodepth=128
size=10M
loops=1
group_reporting=1
readwrite=write
do_verify=0
verify=md5
numjobs=1
thread
verify_dump=1

[small_writes]
offset=0G
blocksize=512

[large_writes]
offset=1G
blocksize=1M

Followed by this job:

[global]
ioengine=libaio
direct=1
filename=/tmp/foo
iodepth=128
size=1M
loops=1
group_reporting=1
readwrite=read
do_verify=1
verify=md5
verify_fatal=1
numjobs=1
thread
verify_dump=1
bs=4k

[large_reads]
offset=0G
blocksize=1M

[small_reads]
offset=1G
blocksize=512

Will cause a segfault, since the bs=512 job appears to stumble
upon a 1mb header, making fio attempt to verify a much bigger
area than what we have read.

This looks like a zone bug. For now, check in a fix that makes
fio validate the verify header a bit harder.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd client references
Jens Axboe [Tue, 6 Mar 2012 14:42:31 +0000 (15:42 +0100)]
Add client references

We have a use-after-free in the fio_handle_clients() loop.
If we receive a QUIT command, we remove the client in
fio_handle_client(). But fio_handle_clients() doesn't
have a way to detect this, so it checks client->error
after it has potentially been freed.

Add a simple reference to get rid of this problem.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoserver: fixup bug in sending jobs_eta command
Jens Axboe [Tue, 6 Mar 2012 14:37:40 +0000 (15:37 +0100)]
server: fixup bug in sending jobs_eta command

We stored nr_running in elapsed_sec, causing ETA to only
be reliable on the client when the job neared completion.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoCompression fixes
Jens Axboe [Fri, 2 Mar 2012 21:23:36 +0000 (22:23 +0100)]
Compression fixes

- memset() remainder of buffer instead of copying random segment.
  Provides more reliable compression (yes yes, you told me so).

- Allow 100 as well, just means full memset. Make that just set
  zero_buffers=1 instead, since we only need to do it once
  instead of for every write.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd buffer_compress_percentage
Jens Axboe [Fri, 2 Mar 2012 20:02:12 +0000 (21:02 +0100)]
Add buffer_compress_percentage

The option is pending testing, so not documented yet.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agolinux: fallocate() fixes
Jens Axboe [Thu, 1 Mar 2012 15:07:46 +0000 (16:07 +0100)]
linux: fallocate() fixes

Enable posix_fallocate() for glibc >= 2.6 as before, but
only enable fallocate() from 2.8 and up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRename crc functions to private namespace
Jens Axboe [Wed, 29 Feb 2012 09:01:34 +0000 (10:01 +0100)]
Rename crc functions to private namespace

Apparently we overlap with some gtk (?) crc32() functions, causing
ours to be called and subsequently crashing since the arguments
aren't the same.

Move everything to a fio_ protected namespace.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.5 fio-2.0.5
Jens Axboe [Tue, 28 Feb 2012 07:23:23 +0000 (08:23 +0100)]
Fio 2.0.5

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoReally fix IOPS logging
Jens Axboe [Tue, 28 Feb 2012 07:20:26 +0000 (08:20 +0100)]
Really fix IOPS logging

Commit 9602d8df had a bug where it mistakenly stored
the bytes last seen when doing an IOPS log entry, instead
of the blocks last seen.

Reported-by: Bill Hooper <whooper@micron.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.4 fio-2.0.4
Jens Axboe [Fri, 24 Feb 2012 09:16:46 +0000 (10:16 +0100)]
Fio 2.0.4

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix verify
Jens Axboe [Fri, 24 Feb 2012 07:26:57 +0000 (08:26 +0100)]
Fix verify

The hole commit from yesterday (d3a173a9) had a bug, where
we would checksum too much.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoclient: exit if no jobs were sent and client is idle
Jens Axboe [Thu, 23 Feb 2012 19:56:12 +0000 (20:56 +0100)]
client: exit if no jobs were sent and client is idle

If we invoke the client without arguments, it gets the probe
response and then waits for further response from the server.
But if no command line or job files were sent, the server
is idle. In that case, just quit.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoverify: put all header verification in one place
Jens Axboe [Thu, 23 Feb 2012 09:31:07 +0000 (10:31 +0100)]
verify: put all header verification in one place

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix // comment
Jens Axboe [Thu, 23 Feb 2012 07:35:04 +0000 (08:35 +0100)]
Fix // comment

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRemove holes in verify_header structure
Jens Axboe [Thu, 23 Feb 2012 07:23:18 +0000 (08:23 +0100)]
Remove holes in verify_header structure

Shrink the magic and verify_type members, and reorder so that
it packs nicely. Incompatible change, write workloads must be
re-run.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoImplement sigaction for Windows.
Bruce Cran [Wed, 22 Feb 2012 19:56:22 +0000 (19:56 +0000)]
Implement sigaction for Windows.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agostat() doesn't work with devices on Windows, so move check for '\\.\' earlier.
Bruce Cran [Wed, 22 Feb 2012 17:55:16 +0000 (17:55 +0000)]
stat() doesn't work with devices on Windows, so move check for '\\.\' earlier.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUse crc32c for the verify_header checksum
Jens Axboe [Wed, 22 Feb 2012 19:29:42 +0000 (20:29 +0100)]
Use crc32c for the verify_header checksum

It's the only hardware assisted crypto we have, and it's
widely available in any recent Intel CPU.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAuto-detect whether to use hw assisted crc32c
Jens Axboe [Wed, 22 Feb 2012 19:28:17 +0000 (20:28 +0100)]
Auto-detect whether to use hw assisted crc32c

Little point in separating the two. If the hardware assisted
crypto is available, always use it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd checksum to verify_header
Jens Axboe [Wed, 22 Feb 2012 19:11:57 +0000 (20:11 +0100)]
Add checksum to verify_header

Currently we don't know if the verify_header itself is
valid when pulled off a disk. While it will fail verify
if it is, fio might then confuse the 'expected' and
'received' data since it re-generates the corrupted
crc with the seed stored. But if the seed is corrupt,
then we fail.

So checksum the verify header, so that we know for a fact
whether the header is valid or not.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoflow: Fixing uninitialized variable
Daniel Ehrenberg [Wed, 22 Feb 2012 10:48:19 +0000 (11:48 +0100)]
flow: Fixing uninitialized variable

In the flow initialization code path, there was an uninitialized
variable that led to memory corruption in certain cases. This
patch should initialize the variable properly.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix tabbing and comment characters in Windows code.
Bruce Cran [Mon, 20 Feb 2012 20:03:48 +0000 (21:03 +0100)]
Fix tabbing and comment characters in Windows code.

Update syslog.h with proper flag values.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRemove unused Windows version.h and version.rc files.
Bruce Cran [Mon, 20 Feb 2012 20:01:50 +0000 (21:01 +0100)]
Remove unused Windows version.h and version.rc files.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd missing os/windows/posix.c file.
Bruce Cran [Mon, 20 Feb 2012 19:28:39 +0000 (19:28 +0000)]
Add missing os/windows/posix.c file.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix IOPS logging for mixed read/write workloads
Jens Axboe [Mon, 20 Feb 2012 19:19:28 +0000 (20:19 +0100)]
Fix IOPS logging for mixed read/write workloads

Similar to commit 5daa4ebe, but for the IOPS logging.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMove Windows port to MinGW
Bruce Cran [Mon, 20 Feb 2012 19:18:19 +0000 (20:18 +0100)]
Move Windows port to MinGW

- Add calls to WSAStartup in the network code as required by
  Winsock.
- Add Windows-specific init_random_state function which uses the
  Crypto API.
- Move Windows port to MinGW and update build system to create a
  64-bit binary by default.
- Install text files as .rtf so they won't open in Notepad by default
  (Wordpad understands Unix line endings; Notepad doesn't).
- Simplify WiX installer code.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd FIO_PREFERRED_CLOCK_SOURCE to allow selection of clock source on a per-platform...
Bruce Cran [Mon, 20 Feb 2012 17:07:32 +0000 (17:07 +0000)]
Add FIO_PREFERRED_CLOCK_SOURCE to allow selection of clock source on a per-platform basis.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd OS_PATH_SEPARATOR for platforms which have different path separator characters...
Bruce Cran [Mon, 20 Feb 2012 17:01:46 +0000 (17:01 +0000)]
Add OS_PATH_SEPARATOR for platforms which have different path separator characters (e.g. Windows and 'classic' MacOS).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoToken-based flow control
Dan Ehrenberg [Mon, 20 Feb 2012 10:05:14 +0000 (11:05 +0100)]
Token-based flow control

This patch allows two fio jobs to be kept to a certain
proportion of each other using token-based flow control.
There are three new parameters: flow, flow_watermark, and
flow_sleep, documented in the fio options. An example of an fio
job using these parameters is below:

[global]
norandommap
thread
time_based
runtime=30
direct=1
ioengine=libaio
iodepth=256
size=100g
bs=8k
filename=/tmp/testfile
flow_watermark=100
flow_sleep=1000

[job2]
numjobs=1
rw=write
flow=-8

[job1]
numjobs=1
rw=randread
flow=1

The motivating application of this patch was to allow random reads
and sequential writes at a particular given proportion.

This initial version is only correct when run with 'thread', as shared
state is represented with a global variable. It also only allows two
jobs to be synchronized properly. A future version might do more, but
no more functionality was needed for my application.

Tested: Ran a few fio jobs with this flow control, observing
the proportion of IOPS to match what was intended by the job file.
Varied the flow_watermark and flow_sleep parameters and observed
the effect on throughput.

Signed-off-by: Dan Ehrenberg <dehrenberg@google.com>
Modified by me to support flow_id, so an arbitrary number of flows can
be used. This means it no longer relies on global context, so it can be
used from a thread or process alike. Also added man page documentation.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix bandwidth logging for mixed read/write workloads.
Josh Carter [Mon, 20 Feb 2012 09:12:22 +0000 (10:12 +0100)]
Fix bandwidth logging for mixed read/write workloads.

fio was maintaining separate read/write stats, but only one timer for
bw_avg_time. Whichever IO direction happened to cross the timer would
get its interval logged; the other data direction would not. Now both
ddir are logged each time we cross bw_avg_time.

Where intervals don't contain any activity in a given ddir, no log
entry is made.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix assumption that pointers fits in a 'long'
Bruce Cran [Mon, 20 Feb 2012 08:34:24 +0000 (09:34 +0100)]
Fix assumption that pointers fits in a 'long'

Windows uses LLP64 model so [u]intptr_t is more correct.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoNeed glibc 2.8 or bigger for fallocate
Jens Axboe [Mon, 20 Feb 2012 08:21:45 +0000 (09:21 +0100)]
Need glibc 2.8 or bigger for fallocate

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomutex: don't attempt to use CLOCK_MONOTONIC
Jens Axboe [Mon, 20 Feb 2012 08:18:43 +0000 (09:18 +0100)]
mutex: don't attempt to use CLOCK_MONOTONIC

It's a bit of a mess. Some platforms don't have
pthread_condattr_setclock(), some have it but it appears NOT to set
the block correctly.

Accept that we have to use the realtime clock for timed mutex and just
don't touch the clock settings.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agonice() error handling
Bruce Cran [Mon, 20 Feb 2012 06:59:06 +0000 (07:59 +0100)]
nice() error handling

I've attached a patch which fixes a potential issue I noticed while
reading the POSIX specs: nice() can succeed and return -1, so it's
recommended to set errno to 0 and check it afterwards:

"As -1 is a permissible return value in a successful situation, an
application wishing to check for error situations should set errno to 0,
then call nice(), and if it returns -1, check to see whether errno is
non-zero."

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomac: doesn't have pthread_condattr_setclock()
Jens Axboe [Fri, 17 Feb 2012 15:59:09 +0000 (16:59 +0100)]
mac: doesn't have pthread_condattr_setclock()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomutex: guard pthread_condattr_setclock() by defined "have ifdef"
Jens Axboe [Fri, 17 Feb 2012 12:48:08 +0000 (13:48 +0100)]
mutex: guard pthread_condattr_setclock() by defined "have ifdef"

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomutex: more clock fixes
Jens Axboe [Thu, 16 Feb 2012 21:22:46 +0000 (22:22 +0100)]
mutex: more clock fixes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMutex timeout work-around
Jens Axboe [Wed, 15 Feb 2012 21:24:19 +0000 (22:24 +0100)]
Mutex timeout work-around

Signed-off-by: Jens Axboe <axboe@kernel.dk>