fio.git
7 years agoFio 2.0.4 fio-2.0.4
Jens Axboe [Fri, 24 Feb 2012 09:16:46 +0000 (10:16 +0100)]
Fio 2.0.4

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix verify
Jens Axboe [Fri, 24 Feb 2012 07:26:57 +0000 (08:26 +0100)]
Fix verify

The hole commit from yesterday (d3a173a9) had a bug, where
we would checksum too much.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoclient: exit if no jobs were sent and client is idle
Jens Axboe [Thu, 23 Feb 2012 19:56:12 +0000 (20:56 +0100)]
client: exit if no jobs were sent and client is idle

If we invoke the client without arguments, it gets the probe
response and then waits for further response from the server.
But if no command line or job files were sent, the server
is idle. In that case, just quit.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoverify: put all header verification in one place
Jens Axboe [Thu, 23 Feb 2012 09:31:07 +0000 (10:31 +0100)]
verify: put all header verification in one place

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix // comment
Jens Axboe [Thu, 23 Feb 2012 07:35:04 +0000 (08:35 +0100)]
Fix // comment

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRemove holes in verify_header structure
Jens Axboe [Thu, 23 Feb 2012 07:23:18 +0000 (08:23 +0100)]
Remove holes in verify_header structure

Shrink the magic and verify_type members, and reorder so that
it packs nicely. Incompatible change, write workloads must be
re-run.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoImplement sigaction for Windows.
Bruce Cran [Wed, 22 Feb 2012 19:56:22 +0000 (19:56 +0000)]
Implement sigaction for Windows.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agostat() doesn't work with devices on Windows, so move check for '\\.\' earlier.
Bruce Cran [Wed, 22 Feb 2012 17:55:16 +0000 (17:55 +0000)]
stat() doesn't work with devices on Windows, so move check for '\\.\' earlier.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUse crc32c for the verify_header checksum
Jens Axboe [Wed, 22 Feb 2012 19:29:42 +0000 (20:29 +0100)]
Use crc32c for the verify_header checksum

It's the only hardware assisted crypto we have, and it's
widely available in any recent Intel CPU.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAuto-detect whether to use hw assisted crc32c
Jens Axboe [Wed, 22 Feb 2012 19:28:17 +0000 (20:28 +0100)]
Auto-detect whether to use hw assisted crc32c

Little point in separating the two. If the hardware assisted
crypto is available, always use it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd checksum to verify_header
Jens Axboe [Wed, 22 Feb 2012 19:11:57 +0000 (20:11 +0100)]
Add checksum to verify_header

Currently we don't know if the verify_header itself is
valid when pulled off a disk. While it will fail verify
if it is, fio might then confuse the 'expected' and
'received' data since it re-generates the corrupted
crc with the seed stored. But if the seed is corrupt,
then we fail.

So checksum the verify header, so that we know for a fact
whether the header is valid or not.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoflow: Fixing uninitialized variable
Daniel Ehrenberg [Wed, 22 Feb 2012 10:48:19 +0000 (11:48 +0100)]
flow: Fixing uninitialized variable

In the flow initialization code path, there was an uninitialized
variable that led to memory corruption in certain cases. This
patch should initialize the variable properly.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix tabbing and comment characters in Windows code.
Bruce Cran [Mon, 20 Feb 2012 20:03:48 +0000 (21:03 +0100)]
Fix tabbing and comment characters in Windows code.

Update syslog.h with proper flag values.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRemove unused Windows version.h and version.rc files.
Bruce Cran [Mon, 20 Feb 2012 20:01:50 +0000 (21:01 +0100)]
Remove unused Windows version.h and version.rc files.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd missing os/windows/posix.c file.
Bruce Cran [Mon, 20 Feb 2012 19:28:39 +0000 (19:28 +0000)]
Add missing os/windows/posix.c file.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix IOPS logging for mixed read/write workloads
Jens Axboe [Mon, 20 Feb 2012 19:19:28 +0000 (20:19 +0100)]
Fix IOPS logging for mixed read/write workloads

Similar to commit 5daa4ebe, but for the IOPS logging.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMove Windows port to MinGW
Bruce Cran [Mon, 20 Feb 2012 19:18:19 +0000 (20:18 +0100)]
Move Windows port to MinGW

- Add calls to WSAStartup in the network code as required by
  Winsock.
- Add Windows-specific init_random_state function which uses the
  Crypto API.
- Move Windows port to MinGW and update build system to create a
  64-bit binary by default.
- Install text files as .rtf so they won't open in Notepad by default
  (Wordpad understands Unix line endings; Notepad doesn't).
- Simplify WiX installer code.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd FIO_PREFERRED_CLOCK_SOURCE to allow selection of clock source on a per-platform...
Bruce Cran [Mon, 20 Feb 2012 17:07:32 +0000 (17:07 +0000)]
Add FIO_PREFERRED_CLOCK_SOURCE to allow selection of clock source on a per-platform basis.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd OS_PATH_SEPARATOR for platforms which have different path separator characters...
Bruce Cran [Mon, 20 Feb 2012 17:01:46 +0000 (17:01 +0000)]
Add OS_PATH_SEPARATOR for platforms which have different path separator characters (e.g. Windows and 'classic' MacOS).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoToken-based flow control
Dan Ehrenberg [Mon, 20 Feb 2012 10:05:14 +0000 (11:05 +0100)]
Token-based flow control

This patch allows two fio jobs to be kept to a certain
proportion of each other using token-based flow control.
There are three new parameters: flow, flow_watermark, and
flow_sleep, documented in the fio options. An example of an fio
job using these parameters is below:

[global]
norandommap
thread
time_based
runtime=30
direct=1
ioengine=libaio
iodepth=256
size=100g
bs=8k
filename=/tmp/testfile
flow_watermark=100
flow_sleep=1000

[job2]
numjobs=1
rw=write
flow=-8

[job1]
numjobs=1
rw=randread
flow=1

The motivating application of this patch was to allow random reads
and sequential writes at a particular given proportion.

This initial version is only correct when run with 'thread', as shared
state is represented with a global variable. It also only allows two
jobs to be synchronized properly. A future version might do more, but
no more functionality was needed for my application.

Tested: Ran a few fio jobs with this flow control, observing
the proportion of IOPS to match what was intended by the job file.
Varied the flow_watermark and flow_sleep parameters and observed
the effect on throughput.

Signed-off-by: Dan Ehrenberg <dehrenberg@google.com>
Modified by me to support flow_id, so an arbitrary number of flows can
be used. This means it no longer relies on global context, so it can be
used from a thread or process alike. Also added man page documentation.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix bandwidth logging for mixed read/write workloads.
Josh Carter [Mon, 20 Feb 2012 09:12:22 +0000 (10:12 +0100)]
Fix bandwidth logging for mixed read/write workloads.

fio was maintaining separate read/write stats, but only one timer for
bw_avg_time. Whichever IO direction happened to cross the timer would
get its interval logged; the other data direction would not. Now both
ddir are logged each time we cross bw_avg_time.

Where intervals don't contain any activity in a given ddir, no log
entry is made.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix assumption that pointers fits in a 'long'
Bruce Cran [Mon, 20 Feb 2012 08:34:24 +0000 (09:34 +0100)]
Fix assumption that pointers fits in a 'long'

Windows uses LLP64 model so [u]intptr_t is more correct.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoNeed glibc 2.8 or bigger for fallocate
Jens Axboe [Mon, 20 Feb 2012 08:21:45 +0000 (09:21 +0100)]
Need glibc 2.8 or bigger for fallocate

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomutex: don't attempt to use CLOCK_MONOTONIC
Jens Axboe [Mon, 20 Feb 2012 08:18:43 +0000 (09:18 +0100)]
mutex: don't attempt to use CLOCK_MONOTONIC

It's a bit of a mess. Some platforms don't have
pthread_condattr_setclock(), some have it but it appears NOT to set
the block correctly.

Accept that we have to use the realtime clock for timed mutex and just
don't touch the clock settings.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agonice() error handling
Bruce Cran [Mon, 20 Feb 2012 06:59:06 +0000 (07:59 +0100)]
nice() error handling

I've attached a patch which fixes a potential issue I noticed while
reading the POSIX specs: nice() can succeed and return -1, so it's
recommended to set errno to 0 and check it afterwards:

"As -1 is a permissible return value in a successful situation, an
application wishing to check for error situations should set errno to 0,
then call nice(), and if it returns -1, check to see whether errno is
non-zero."

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomac: doesn't have pthread_condattr_setclock()
Jens Axboe [Fri, 17 Feb 2012 15:59:09 +0000 (16:59 +0100)]
mac: doesn't have pthread_condattr_setclock()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomutex: guard pthread_condattr_setclock() by defined "have ifdef"
Jens Axboe [Fri, 17 Feb 2012 12:48:08 +0000 (13:48 +0100)]
mutex: guard pthread_condattr_setclock() by defined "have ifdef"

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomutex: more clock fixes
Jens Axboe [Thu, 16 Feb 2012 21:22:46 +0000 (22:22 +0100)]
mutex: more clock fixes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMutex timeout work-around
Jens Axboe [Wed, 15 Feb 2012 21:24:19 +0000 (22:24 +0100)]
Mutex timeout work-around

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMutex fixes
Jens Axboe [Wed, 15 Feb 2012 21:20:26 +0000 (22:20 +0100)]
Mutex fixes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFixup includes
Jens Axboe [Wed, 15 Feb 2012 20:52:39 +0000 (21:52 +0100)]
Fixup includes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix typo
Jens Axboe [Wed, 15 Feb 2012 18:40:57 +0000 (19:40 +0100)]
Fix typo

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix build issue for non-cgroup support
Jens Axboe [Wed, 15 Feb 2012 16:30:35 +0000 (17:30 +0100)]
Fix build issue for non-cgroup support

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix grammatical error
Jens Axboe [Wed, 15 Feb 2012 13:27:08 +0000 (14:27 +0100)]
Fix grammatical error

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomutex: set and use the proper clock source
Jens Axboe [Wed, 15 Feb 2012 08:56:51 +0000 (09:56 +0100)]
mutex: set and use the proper clock source

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRevert "Change iolog overlap assert to a debug dump"
Jens Axboe [Sat, 11 Feb 2012 08:06:58 +0000 (09:06 +0100)]
Revert "Change iolog overlap assert to a debug dump"

Issued has been fixed, restore assert.

This reverts commit 86f40280e27474c666fea75cc705caeaf94f6346.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd job files for zone bug
Jens Axboe [Sat, 11 Feb 2012 08:06:36 +0000 (09:06 +0100)]
Add job files for zone bug

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoOnly attempt zone switch if we have zones enabled
Jens Axboe [Sat, 11 Feb 2012 08:04:02 +0000 (09:04 +0100)]
Only attempt zone switch if we have zones enabled

Just an offset isn't enough.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoChange iolog overlap assert to a debug dump
Jens Axboe [Fri, 10 Feb 2012 20:58:46 +0000 (21:58 +0100)]
Change iolog overlap assert to a debug dump

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoSlave aggregate output format fix
Jens Axboe [Fri, 10 Feb 2012 18:30:23 +0000 (19:30 +0100)]
Slave aggregate output format fix

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix missing ';' before slave disk utilization in terse output
Jens Axboe [Fri, 10 Feb 2012 18:21:50 +0000 (19:21 +0100)]
Fix missing ';' before slave disk utilization in terse output

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix a few other static code checker spotted "issues"
Jens Axboe [Thu, 9 Feb 2012 20:17:06 +0000 (21:17 +0100)]
Fix a few other static code checker spotted "issues"

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFixup a few items spotted by a static code checker
Jens Axboe [Thu, 9 Feb 2012 19:55:29 +0000 (20:55 +0100)]
Fixup a few items spotted by a static code checker

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd gettime.h
Jens Axboe [Thu, 9 Feb 2012 14:02:39 +0000 (15:02 +0100)]
Add gettime.h

time.h is a mix of gettime.c and time.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMove variables to backend, fix up line spaces
Jens Axboe [Thu, 9 Feb 2012 10:25:24 +0000 (11:25 +0100)]
Move variables to backend, fix up line spaces

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMove some code around to better separate front/backend
Jens Axboe [Thu, 9 Feb 2012 10:15:02 +0000 (11:15 +0100)]
Move some code around to better separate front/backend

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMove gtod thread to gettime.c
Jens Axboe [Thu, 9 Feb 2012 09:27:10 +0000 (10:27 +0100)]
Move gtod thread to gettime.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoInstall man page in /usr/share/man by default on OSX
Jens Axboe [Wed, 8 Feb 2012 13:19:27 +0000 (14:19 +0100)]
Install man page in /usr/share/man by default on OSX

Other packages apparently do this, and /usr/local/man isn't
in the default OSX man path.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agostat: fix typo
Jens Axboe [Tue, 7 Feb 2012 15:42:43 +0000 (16:42 +0100)]
stat: fix typo

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agostat: NaN fixes
Jens Axboe [Tue, 7 Feb 2012 11:27:28 +0000 (12:27 +0100)]
stat: NaN fixes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoEnable completion latency stat collection on verify
Steven Lang [Tue, 7 Feb 2012 08:42:59 +0000 (09:42 +0100)]
Enable completion latency stat collection on verify

A patch last year
(http://git.kernel.dk/?p=fio.git;a=commit;f=io_u.c;h=c8eeb9df1f52f28567a5937e141decc6a26ec30b)
changed the behavior of verify loads so they stopped printing
completion latency stats.  (A verify load in this case includes both
rw=*write with verify and do_verify=1 and rw=*read with verify.)

Is there any particular reason that an io_u which is going to be
verified shouldn't have its latency statistics recorded?

This patch re-enables the clat recording.

An example demonstrating the issue here:

$ ./fio --name=test --size=1m --bs=4k --verify=meta --rw=write
test: (g=0): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.2
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 1MB)

test: (groupid=0, jobs=1): err= 0: pid=27535
  read : io=1024.0KB, bw=1000.0MB/s, iops=256000 , runt=     1msec
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    0], 50.00th=[    0], 60.00th=[    0],
     | 70.00th=[    0], 80.00th=[    0], 90.00th=[    0], 95.00th=[    0],
     | 99.00th=[    0], 99.50th=[    0], 99.90th=[    0]
  write: io=1024.0KB, bw=341333KB/s, iops=85333 , runt=     3msec
    clat (usec): min=2 , max=13 , avg= 3.30, stdev= 0.78
     lat (usec): min=2 , max=14 , avg= 3.52, stdev= 0.84
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    4], 99.50th=[    4], 99.90th=[   13]
    lat (usec) : 4=35.55%, 10=14.26%, 20=0.20%
  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=46
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=256/w=256/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1024KB, aggrb=1000.0MB/s, minb=1024.0MB/s,
maxb=1024.0MB/s, mint=1msec, maxt=1msec
  WRITE: io=1024KB, aggrb=341333KB/s, minb=349525KB/s,
maxb=349525KB/s, mint=3msec, maxt=3msec

Note that no clat line is reported for reads and no clat percentiles
are listed.  If a read-only load is then executed, it has the same
thing.

$ ./fio --name=test --size=1m --bs=4k --verify=meta --rw=read
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.2
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=31694
  read : io=1024.0KB, bw=13653KB/s, iops=3413 , runt=    75msec
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    0], 50.00th=[    0], 60.00th=[    0],
     | 70.00th=[    0], 80.00th=[    0], 90.00th=[    0], 95.00th=[    0],
     | 99.00th=[    0], 99.50th=[    0], 99.90th=[    0]
  cpu          : usr=0.00%, sys=0.00%, ctx=15, majf=0, minf=36
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=256/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1024KB, aggrb=13653KB/s, minb=13981KB/s, maxb=13981KB/s,
mint=75msec, maxt=75msec

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoOptimize pattern verify
Steven Lang [Tue, 7 Feb 2012 08:42:59 +0000 (09:42 +0100)]
Optimize pattern verify

Similar to the patch last week, this optimizes the pattern verify
operation to use optimized library calls like memcmp(), and only fall
back to byte-by-byte if there is a miscompare to locate it.

This uses the same premise that the pattern is repeated as many times
as possible to do large compares in a single call.  For single byte
pattern, the setup fills the pattern space, and verify assumes it is
full.

Tested by running a script which created an 8k file with 4k bs and,
one byte at a time, tried corrupting the pattern and running a read
pass and verified it still found miscompares across the whole range of
the pattern.  This was done with a pattern length of 1 and 3 bytes.

In performance tests, this was about 8 times more efficient on
verifies than without this patch.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.3 fio-2.0.3
Jens Axboe [Tue, 7 Feb 2012 06:46:09 +0000 (07:46 +0100)]
Fio 2.0.3

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd regression test job file for commit 52c58027
Jens Axboe [Tue, 7 Feb 2012 06:43:59 +0000 (07:43 +0100)]
Add regression test job file for commit 52c58027

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Tue, 7 Feb 2012 06:42:18 +0000 (07:42 +0100)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

7 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Mon, 6 Feb 2012 21:00:06 +0000 (22:00 +0100)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

7 years agoFail if we can't satisfy the minimum block size
Jens Axboe [Mon, 6 Feb 2012 20:58:56 +0000 (21:58 +0100)]
Fail if we can't satisfy the minimum block size

Commit 799441286648bdced4f42d3040f37fd2e35eaf1d introduced a bug
where if we cannot satisfy the minimum block size from a given
offset, then we will loop forever trying to do that. Fix that bug
checking early if it is indeed a possible thing to satisfy, if
not give up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agomac: remove unused timer_create()
Jens Axboe [Fri, 3 Feb 2012 13:45:38 +0000 (14:45 +0100)]
mac: remove unused timer_create()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUse CLOCK_MONOTONIC if we can
Jens Axboe [Fri, 3 Feb 2012 11:48:16 +0000 (12:48 +0100)]
Use CLOCK_MONOTONIC if we can

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAvoid divide-by-zero for disk util on 0 msec runtime
Jens Axboe [Fri, 3 Feb 2012 11:06:19 +0000 (12:06 +0100)]
Avoid divide-by-zero for disk util on 0 msec runtime

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoOptimize pattern filling by limiting small calls to memcpy
Steven Lang [Thu, 2 Feb 2012 19:22:04 +0000 (20:22 +0100)]
Optimize pattern filling by limiting small calls to memcpy

In looking at profiling the speed of fill_pattern(), it calls memcpy()
for the fill pattern repeatedly for multibyte patterns.  So for a 4
byte pattern with 8k IO, it calls memcpy() 2048 times.

Since there is already 512 bytes reserved for the pattern, I figured a
simple solution was to use it.  This patch replicates short patterns
so they can be more efficiently copied.  (Single byte patterns are
left alone since they can make use of the much more efficient memset()
call.)

The result is a 10x performance improvement on pattern filling. (With
this patch, it's still 3x slower than when it re-uses the already
filled pattern.)

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.2 fio-2.0.2
Jens Axboe [Thu, 2 Feb 2012 18:56:03 +0000 (19:56 +0100)]
Fio 2.0.2

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoReenable io_u->buf_filled_len optimization
Jens Axboe [Thu, 2 Feb 2012 08:20:09 +0000 (09:20 +0100)]
Reenable io_u->buf_filled_len optimization

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMake random map allocation more resilient
Jens Axboe [Thu, 2 Feb 2012 07:41:28 +0000 (08:41 +0100)]
Make random map allocation more resilient

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agowindowsaio: style fixes
Jens Axboe [Wed, 1 Feb 2012 22:03:44 +0000 (23:03 +0100)]
windowsaio: style fixes

Some weird 4-space tabs in there, clean it up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agowindowsaio: fix file offset bug
Bruce Cran [Wed, 1 Feb 2012 20:46:55 +0000 (21:46 +0100)]
windowsaio: fix file offset bug

I was monitoring file IO while running fio today and noticed that the file
offset when using windowsaio was always 0.
Because the OVERLAPPED structure contains a union, by initializing Pointer
to NULL the code was also overwriting the Offset field.
I've attached a patch that fixes it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoKeep reducing jobs on ENOMEM for shmget()
Jens Axboe [Wed, 1 Feb 2012 19:11:01 +0000 (20:11 +0100)]
Keep reducing jobs on ENOMEM for shmget()

Some platforms return ENOMEM, not just EINVAL, for too large
a size.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix bad types for mac blockdev_size()
Jens Axboe [Wed, 1 Feb 2012 19:10:12 +0000 (20:10 +0100)]
Fix bad types for mac blockdev_size()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoGrow file num_maps to unsigned long
Jens Axboe [Wed, 1 Feb 2012 19:05:34 +0000 (20:05 +0100)]
Grow file num_maps to unsigned long

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Tue, 31 Jan 2012 13:07:04 +0000 (14:07 +0100)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

7 years agoFix thread hang when using async engines (libaio,etc.) when too low of a iops rate...
Ryan Marchand [Tue, 31 Jan 2012 13:05:32 +0000 (14:05 +0100)]
Fix thread hang when using async engines (libaio,etc.) when too low of a iops rate is specified.

Rate limiting logic was using thread_data->cur_depth to decide the
min_evts number to ask for during its "flush" prior to sleeping.
td->cur_depth, however, does not properly track in-flight IOs submitted
to the async engines.  Added field to thread_data structure and use
that, instead, to track IOs currently in flight.

Signed-off-by: Ryan Marchand <rmarchan@amazon.com>
Signed-off-by: Steven Noonan <snoonan@amazon.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoimplement zoned random I/O testing support
Steven Noonan [Tue, 31 Jan 2012 12:58:00 +0000 (13:58 +0100)]
implement zoned random I/O testing support

This fixes the limitation that prevents fio from doing random I/O with zones
enabled.

This also adds a 'zonerange' configuration option which may be too ambiguous
and is subject to potential renaming in the future.

When doing random I/O, it is beneficial to be able to specify how large the
addressible space is in the zone, while specifying a different metric for
how much data to read from that zone (i.e., how many samples to take from each
zone). When 'zonerange' is not specified, it defaults to be equal to the
'zonesize' option. When both are specified, 'zonerange' indicates the
size of the zone while 'zonesize' indicates the quantity of data to read from
each zone.

Signed-off-by: Steven Noonan <snoonan@amazon.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agofix early termination of runs caused by ramp_time > runtime
Steven Noonan [Mon, 30 Jan 2012 21:34:04 +0000 (13:34 -0800)]
fix early termination of runs caused by ramp_time > runtime

If ramp_time was longer than the runtime, then the run terminated immediately
after the ramp_time was exceeded. In order to avoid this, teach
runtime_exceeded about ramp_time.

Signed-off-by: Steven Noonan <snoonan@amazon.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.1 fio-2.0.1
Jens Axboe [Wed, 18 Jan 2012 08:00:20 +0000 (09:00 +0100)]
Fio 2.0.1

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix a man page typo
Zhu Yanhai [Mon, 2 Jan 2012 13:32:43 +0000 (14:32 +0100)]
Fix a man page typo

The man page line of 'fadvise_hit' is exactly opposite to its true meaning.

Signed-off-by: Zhu Yanhai <gaoyang.zyh@taobao.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agowindows: use SYSTEMDRIVE instead of hard-coding C:
Bruce Cran [Thu, 22 Dec 2011 19:17:25 +0000 (20:17 +0100)]
windows: use SYSTEMDRIVE instead of hard-coding C:

This allows FIO.bat to work on a system with multiple copies of Windows
installed.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix segfault with net io engine and no file/hostname given
Jens Axboe [Mon, 19 Dec 2011 07:57:18 +0000 (08:57 +0100)]
Fix segfault with net io engine and no file/hostname given

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agofix net engine client read server write bug
Yufei Ren [Mon, 19 Dec 2011 07:56:29 +0000 (08:56 +0100)]
fix net engine client read server write bug

The net engine was updated with specific 'listen' parameter which used
for identifying whether it is a tcp server, instead of previous
'rw=read' side must be the server. This let fio support bi-direction
net io test. fio_netio_open_file() has to follow this
principle. Otherwise, the following tcp client 'rw=read' and tcp
server 'rw=write' case doesn't work.

[global]
ioengine=net
port=8888
protocol=tcp
bs=4k
size=10g

[server]
listen
rw=write

[client]
hostname=localhost
startdelay=1
rw=read

BTW, examples/netio needs 'listen' and 'hostname' adjusted.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0 fio-2.0
Jens Axboe [Thu, 8 Dec 2011 08:52:39 +0000 (09:52 +0100)]
Fio 2.0

1.9x seems to have settled, so lets tag 2.0. This also just happens
to be on the 6th year anniversary of Fio 1.1, the first officially
tagged release!

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRemove debug/broken assert() in the IOPS logging code
Jens Axboe [Fri, 2 Dec 2011 07:45:44 +0000 (08:45 +0100)]
Remove debug/broken assert() in the IOPS logging code

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 1.99.13 fio-1.99.13
Jens Axboe [Thu, 1 Dec 2011 14:18:47 +0000 (15:18 +0100)]
Fio 1.99.13

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoOnly log the period mean if we have samples
Jens Axboe [Thu, 1 Dec 2011 14:17:24 +0000 (15:17 +0100)]
Only log the period mean if we have samples

We can't use the mean value to determine that, it could in theory
be 0.0 and still have valid samples.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd log_avg_msec option
Jens Axboe [Thu, 1 Dec 2011 08:04:31 +0000 (09:04 +0100)]
Add log_avg_msec option

By default, fio will note an entry in the bw/iops/lat logs for every
IO that is completed. This quickly eats up a lot of disk space
for long running jobs. By setting this option to eg 1000, fio will
average the results logged over 1 second instead. This reduces the
resolution of the log, but makes it more manageable.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUse the reentrant getmntent_r instead of getmntent
Zhu Yanhai [Tue, 22 Nov 2011 08:35:29 +0000 (09:35 +0100)]
Use the reentrant getmntent_r instead of getmntent

The function find_cgroup_mnt() could be called under multithread scenario, so
we should use the reentrant edition getmntent_r() instead of getmntent().

E.g, without this fix the output for below job file is wrong.

[root@localhost blkio-test]# cat small-read-file
[global]
ioengine=sync
direct=1
rw=read
bs=4k
filename=/dev/sdb
time_based
runtime=60
cgroup=small
cgroup_weight=500
numjobs=32
group_reporting
thread

[file1]

Then fio randomly says:
[cut here]
Starting 32 threads
fio: cgroup blkio does not appear to be mounted
fio: cgroup blkio does not appear to be mounted
fio: cgroup blkio does not appear to be mounted

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix initialization of lastrate when using ratemin
Josh Carter [Tue, 22 Nov 2011 08:35:29 +0000 (09:35 +0100)]
Fix initialization of lastrate when using ratemin

fio was copying td->bw_sample_time (single timeval) into td->lastrate
(array of two timevals) using the latter's size. Thus is would get one
timeval worth of good data plus who-knows-what for the second timeval.
I'm pretty sure the intent of this code was to copy bw_sample_time
into both members of lastrate.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoExpand continue_on_error to select which type of error to allow
Steven Lang [Thu, 17 Nov 2011 08:45:17 +0000 (09:45 +0100)]
Expand continue_on_error to select which type of error to allow

This expands the continue_on_error option to take a string specifying
what type of error to continue on, breaking out errors into read,
write, and verify.  (Sync, trim, and anything else not specifically a
read are considered write operations for the sake of error
continuation.)

Backwards compatibility is retained by allowing =0 and =1 values to
specify none and all, respectively.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix parsing of ioengine strings
Steven Lang [Wed, 16 Nov 2011 09:33:51 +0000 (10:33 +0100)]
Fix parsing of ioengine strings

This cleans up parsing of FIO_OPT_STR_STORE options which have defined
values (IE ioengine).  I made a few assumptions here...

1. If FIO_OPT_STR_STORE was used, that means a copy of the string is
desired, so it is always copied if the offset is non-zero.
2. If the values were the only allowed values, then FIO_OPT_STR would
have been used; therefore it is not an error to give a value not in
the list.
3. If an option callback is defined, then any values are ignored and
the callback is called to parse the string.  (Don't think this
currently applies, anywhere, but there is precendent in other options
skipping normal parsing if a callback is used.)
4. If no offsets are defined, and no callback is given, the behavior
is undefined - though this patch handles it cleanly and still calls
value callbacks; though I believe fio will still complain about a bad
options structure.

This fixes two cases (Restores it to previous behavior without
breaking new functionality).
1. External ioengines (The patch to a .so is never in the list of
possible values, so was never matching)
2. IO engines compiled in but not in the list in options.c

This patch should be applied to the 1.5x stable branch as well, though
it doesn't apply cleanly due to one line difference from my last
options patch.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 1.99.12 fio-1.99.12
Jens Axboe [Wed, 9 Nov 2011 19:26:58 +0000 (20:26 +0100)]
Fio 1.99.12

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix crash when attempting to dupe options
Jens Axboe [Wed, 9 Nov 2011 19:25:21 +0000 (20:25 +0100)]
Fix crash when attempting to dupe options

If the engine has extended options but isn't loaded, boom.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUpdate examples/netio to new option format
Jens Axboe [Wed, 9 Nov 2011 13:31:47 +0000 (14:31 +0100)]
Update examples/netio to new option format

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 1.99.11 fio-1.99.11
Jens Axboe [Wed, 9 Nov 2011 13:04:33 +0000 (14:04 +0100)]
Fio 1.99.11

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoPrivate parameters for ioengines
Steven Lang [Wed, 9 Nov 2011 13:03:34 +0000 (14:03 +0100)]
Private parameters for ioengines

Here is the polished version of the engine private options patch.  As
discussed, the global section only ever tracks the private options for
the globally defined ioengine.  For command line parameters, the
ioengine must be selected before any private options are used.  (IE
--ioengine=libaio --userspace_reap will work, but --userspace_reap
--ioengine=libaio will not.)

The userspace_reap option from libaio has been moved over to this new
option method, usage should be identical to before.
The net ioengine has been modified to use parameters, with hostname,
port, protocol and listen defined as ioengine private parameters.  The
old style of hostname=host,port,protocol no longer works, so usage
will need to be updated.  (It will spit out an error that should be
clear enough that it changed if this is tried.)  Also, with the new
way for specifying parameters, the net IO engine now allows data to
flow in either direction on TCP connections, regardless of which end
initiates the connection.

There's also a new command line argument --enghelp which can be used
to get help on ioengine private parameters, similar to --cmdhelp.
With no argument, it lists all built-in ioengine.  The argument is an
ioengine name (Or path to .so) and optionally a comma followed by a
command name, which behaves identically to --cmdhelp.

For ioengine authorship, if options are supplied, both the options
structure and the size of the storage needed must be supplied, and the
storage must be large enough to hold a pointer to struct thread_data;
that is because the options callback doesn't explicitly have a pointer
to the thread data (Normally it relies on the fact that the options
struct is the start of the thread data), so the offset 0 of the struct
must point to the thread data, and is filled in automatically.  (This
also neatly provides a guarantee that offset 0 is reserved in the
options data, so it can be safely used as a test of undefined.)

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoImprove accuracy of rate= option
Steven Lang [Wed, 9 Nov 2011 12:48:01 +0000 (13:48 +0100)]
Improve accuracy of rate= option

I noticed that the rate= option is not terribly precise in some cases.
 It gets worse the higher the selected rate.  For example:
$ fio -name=load -size=100g -ioengine=null -runtime=10 -rate=30m
One would expect that to read 300MB (307200KB) at close to 30MB/s
(30720KB/s).  However it writes 315024KB at 31499KB/s.  Further
experimentation shows that even higher rates can show bigger
discrepancies.  At the extreme end...
$ fio -name=load -size=100g -ioengine=null -runtime=10 -rate=500m
One would expect this to write 5000MB at a rate of 500MB/s
(512000KB/s).  However it writes close to double that (9536.8MB) at a
rate of over 953MB/s.  At a rate of 1GB/s and higher, the rate
limiting is effectively ignored.

This patch improves the accuracy of the rate= option across the whole
range of rates, at the cost of being very slightly more
computationally expensive.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUsage typo
Jens Axboe [Mon, 7 Nov 2011 21:07:36 +0000 (22:07 +0100)]
Usage typo

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix big endian build
Anton Blanchard [Mon, 7 Nov 2011 13:16:26 +0000 (14:16 +0100)]
Fix big endian build

I get the following error when building fio on PowerPC:

client.c: In function ‘convert_agg’:
client.c:641:22: error: invalid operands to binary & (have ‘long double’ and ‘long long unsigned int’)

It looks like we have things backwards, we should byteswap the
integer value before we convert it to a double.

Signed-off-by: Anton Blanchard <anton@samba.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoCleanup option keyword/environment substitution
Steven Lang [Fri, 28 Oct 2011 06:37:13 +0000 (08:37 +0200)]
Cleanup option keyword/environment substitution

Right now the substitution for options seems quite fragile.  Among the
issues...

- If bc had an error and returned no output, it caused a NULL reference
- Multiple variable substitutions (For example $ncpus * $pagesize)
caused an error as it tried to run bc after the first, with the second
still text
- Memory leak for every keyword substituted
- Multiplication caused shell wildcard expansion (*) of the current
directory when passing the input to bc
- Shell escape sequences would be parsed on the command line when bc is called
- Potential buffer overrun due to unchecked lengths on the input line

So I did a little cleanup to get rid of the issues.  This patch also
moves the environment variable substitution to run before the keyword
substitution, so an environment variable can now indirectly perform a
keyword substitution.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 1.99.10 fio-1.99.10
Jens Axboe [Wed, 26 Oct 2011 07:51:03 +0000 (09:51 +0200)]
Fio 1.99.10

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix segfault with verify_async
Steven Lang [Wed, 26 Oct 2011 07:46:50 +0000 (09:46 +0200)]
Fix segfault with verify_async

At some point the filename was added to the report on verify failures,
however this broke verify_async, as the file pointer on the io_u is
set to NULL before the verify thread sees the io_u.  The result is a
segfault when there is a verify mismatch.

This patch changes the semantics of a deferred free (IO_U_F_FREE_DEF)
to have already called put_file, but not set the file pointer to NULL.
 This is safe to do as the file list is only freed after all the
verify threads have been terminated.

FYI - this is the minimal config to cause the error this patch fixes,
and is what I used to test the fix.  (Not using thread still causes a
segfault, but it is reported more subtly since just a sub-process hits
it.)

[segv]
ioengine=null
size=4k
verify=meta
rw=read
thread
verify_async=1

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMake verify_dump off by default
Jens Axboe [Tue, 25 Oct 2011 20:43:36 +0000 (22:43 +0200)]
Make verify_dump off by default

It's a change of behaviour, so it should be off by default.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd core for verify_dump option
Steven Lang [Tue, 25 Oct 2011 20:41:05 +0000 (22:41 +0200)]
Add core for verify_dump option

The verify_dump option is defined and documented, but does nothing.
This one-liner enables the option.

Signed-off-by: Jens Axboe <axboe@kernel.dk>