fio.git
6 years agoMerge branch 'master' into gfio
Jens Axboe [Fri, 5 Apr 2013 10:46:54 +0000 (12:46 +0200)]
Merge branch 'master' into gfio

Conflicts:
Makefile

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agogitignore: ignore configure script outputs
Steven Noonan [Thu, 4 Apr 2013 07:05:56 +0000 (09:05 +0200)]
gitignore: ignore configure script outputs

Signed-off-by: Steven Noonan <steven@uplinklabs.net>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFixup json bandwidth output
Shaohua Li [Wed, 3 Apr 2013 06:40:17 +0000 (08:40 +0200)]
Fixup json bandwidth output

The final version sent in for inclusion had the check mixed
up, so it doesn't output bandwidth unless the calculation
failed.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoparse: fix misparse of bs=64k-128k
Jens Axboe [Fri, 29 Mar 2013 18:20:51 +0000 (12:20 -0600)]
parse: fix misparse of bs=64k-128k

We failed parsing the postfix for this case, resulting in
64 and 128k being the result (instead of 64k and 128k).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoaxmap: get rid of old debug ->fail_ok checking
Jens Axboe [Fri, 29 Mar 2013 18:08:40 +0000 (12:08 -0600)]
axmap: get rid of old debug ->fail_ok checking

Don't assert() on not being able to set a bit. We are returning
the number of bits we set anyway, so just have it return 0.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoWorkaround pthreads-win32 pthread_rwlock_init limitation.
Bruce Cran [Fri, 29 Mar 2013 16:17:25 +0000 (16:17 +0000)]
Workaround pthreads-win32 pthread_rwlock_init limitation.

On Windows, pthreads-win32 returns EINVAL if you specify an attribute when
calling pthread_rwlock_init. Work around this by only specifying the attribute
when FIO_HAVE_PSHARED_MUTEX is defined.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix rwlock error messages to specify the correct function names.
Bruce Cran [Fri, 29 Mar 2013 16:02:53 +0000 (16:02 +0000)]
Fix rwlock error messages to specify the correct function names.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agot/axmap: add test for multi bit sets
Jens Axboe [Fri, 29 Mar 2013 15:14:23 +0000 (09:14 -0600)]
t/axmap: add test for multi bit sets

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoaxmap: fix bug with max_bs/min_bs ratio being > 64
Jens Axboe [Fri, 29 Mar 2013 14:56:09 +0000 (08:56 -0600)]
axmap: fix bug with max_bs/min_bs ratio being > 64

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoconfigure: fixup build so that it's more resilient
Jens Axboe [Fri, 29 Mar 2013 14:55:32 +0000 (08:55 -0600)]
configure: fixup build so that it's more resilient

Allow people to clear/set EXTFLAGS/OPTFLAGS at will.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix usr/sys/ctx/majf/minf for -USR1 usage
Jens Axboe [Thu, 28 Mar 2013 21:08:49 +0000 (15:08 -0600)]
Fix usr/sys/ctx/majf/minf for -USR1 usage

Let the threads themselves update their stats.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix build if just one of libverbs or librdma is installed
Jens Axboe [Thu, 28 Mar 2013 18:52:17 +0000 (12:52 -0600)]
Fix build if just one of libverbs or librdma is installed

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agomutex: set pshared attributes on rw semaphore
Shaohua Li [Thu, 28 Mar 2013 14:28:51 +0000 (08:28 -0600)]
mutex: set pshared attributes on rw semaphore

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoconfigure: check for v2 of libnuma
Jens Axboe [Mon, 25 Mar 2013 19:20:08 +0000 (13:20 -0600)]
configure: check for v2 of libnuma

We use options that require 2.x.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoConsider the maximum block size difference the minimum for loop exit
Jens Axboe [Sat, 23 Mar 2013 04:56:55 +0000 (22:56 -0600)]
Consider the maximum block size difference the minimum for loop exit

For mixed block sizes, we can hit the condition where we decided
to stop even if we could have done a small block size. Don't do
another loop for those cases.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoposixaio: restart suspend list after we have used aio_suspend()
Jens Axboe [Fri, 22 Mar 2013 21:23:20 +0000 (15:23 -0600)]
posixaio: restart suspend list after we have used aio_suspend()

Otherwise we could both end up putting entries in there twice,
if there are still in flight, or having done entries in there too.
So restart the suspend list after aio_suspend(), so we only put
truly still-pending ios in there.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agomutex: fix the expression for checking the mutext magic
Jianpeng Ma [Fri, 22 Mar 2013 13:16:32 +0000 (07:16 -0600)]
mutex: fix the expression for checking the mutext magic

Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoOnly attempt file unlock if we use locking
Jens Axboe [Thu, 21 Mar 2013 16:06:58 +0000 (10:06 -0600)]
Only attempt file unlock if we use locking

Fixes a segfault on exit, if file locking isn't used.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agomutex: add magic checks
Jens Axboe [Thu, 21 Mar 2013 16:05:07 +0000 (10:05 -0600)]
mutex: add magic checks

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMerge branch 'master' into gfio
Jens Axboe [Thu, 21 Mar 2013 12:45:27 +0000 (06:45 -0600)]
Merge branch 'master' into gfio

Conflicts:
Makefile
fio.h
options.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoKill now unused lockfile_batch variable
Jens Axboe [Thu, 21 Mar 2013 12:23:14 +0000 (06:23 -0600)]
Kill now unused lockfile_batch variable

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoRework lockfile= file lock handling
Jens Axboe [Thu, 21 Mar 2013 01:57:01 +0000 (19:57 -0600)]
Rework lockfile= file lock handling

Get rid of the hand rolled rw semaphores, just use pthread
rwlocks instead. Kill the batching too, it was broken by
default, so nobody could have been using it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agolib/getopt_long: clear 'optarg' every time
Jens Axboe [Tue, 19 Mar 2013 16:47:15 +0000 (10:47 -0600)]
lib/getopt_long: clear 'optarg' every time

We don't want a stale optarg argument to be available.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix re-run issue when job size is set to "unknown"
Jens Axboe [Tue, 19 Mar 2013 13:49:35 +0000 (07:49 -0600)]
Fix re-run issue when job size is set to "unknown"

Assert that td->o.size is valid before comparing the IO amount
we did to it. This prevents re-running a job because we think
there's more work to do. But if we haven't set ->o.size to a
valid value, then that must not determine whether to re-run
the loop or not.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoA few ULL -> uint64_t conversions
Jens Axboe [Tue, 19 Mar 2013 13:48:46 +0000 (07:48 -0600)]
A few ULL -> uint64_t conversions

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFixup commit error in 56c796
Jens Axboe [Mon, 18 Mar 2013 21:34:41 +0000 (15:34 -0600)]
Fixup commit error in 56c796

I hand-fondled it, forgot to revert.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAndroid: remove static Android configuration
Aaron Carroll [Thu, 14 Mar 2013 05:57:41 +0000 (16:57 +1100)]
Android: remove static Android configuration

Android can now be configured dynamically, so remove the static config.
For this we need to modify the socklen_t check #includes which appear to
be wrong anyway.

Signed-off-by: Aaron Carroll <aaronc@cse.unsw.edu.au>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoconfigure: endianness check for cross compile
Aaron Carroll [Thu, 14 Mar 2013 05:57:40 +0000 (16:57 +1100)]
configure: endianness check for cross compile

 - Add a cross compile check: assume yes if we can't run a test binary
 - If cross compiling, revert to a compile-time endianess check. This tries
   a few possible ways to detect big endian, but otherwise assumes little. We
   rely on the run-time check to save us if the build-time check is wrong.

Signed-off-by: Aaron Carroll <aaronc@cse.unsw.edu.au>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoconfigure: compile-time word size detection
Aaron Carroll [Thu, 14 Mar 2013 05:57:39 +0000 (16:57 +1100)]
configure: compile-time word size detection

Signed-off-by: Aaron Carroll <aaronc@cse.unsw.edu.au>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoPropagate target OS from configure to Makefile
Aaron Carroll [Thu, 14 Mar 2013 05:57:38 +0000 (16:57 +1100)]
Propagate target OS from configure to Makefile

Propagate target OS from configure to Makefile (CONFIG_TARGET_OS) to avoid
duplicating the check in Makefile.  Also allow CROSS_COMPILE for the toolchain
prefix, which is common (e.g. Linux).

Signed-off-by: Aaron Carroll <aaronc@cse.unsw.edu.au>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoandroid: fix compile by not defining FIO_NO_HAVE_SHM_H twice
Aaron Carroll [Wed, 13 Mar 2013 09:09:10 +0000 (10:09 +0100)]
android: fix compile by not defining FIO_NO_HAVE_SHM_H twice

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMake test for gnuplot work with empty strings.
Martin Steigerwald [Mon, 11 Mar 2013 15:22:56 +0000 (16:22 +0100)]
Make test for gnuplot work with empty strings.

When I launch fio_generate_plots on a system where gnuplot is not installed I
get this error :

$ fio_generate_plots test
Making bw logs
/usr/bin/fio_generate_plots: 42: /usr/bin/fio_generate_plots: -: not found

That's because the test checking whether gnuplot is installed is failing
because of an empty variable :
GNUPLOT=$(which gnuplot)
if [ ! -x $GNUPLOT ]; then
        echo You need gnuplot installed to generate graphs
        exit 1
fi

Indeed the command "test -x" is exiting with code 0.

To correct this we should enclose the variable with quotes :
if [ ! -x "$GNUPLOT" ]; then

Then the script is going into the test properly :
$ fio_generate_plots test
You need gnuplot installed to generate graphs

I copied problem description by Hervé from the Debian bug report below.

Fixes:
Bug#700580: /usr/bin/fio_generate_plots: -: not found
http://bugs.debian.org/700580

Reported-By: Hervé Werner <hwerner@score-md.com>
Tested-By: Hervé Werner <hwerner@score-md.com>
Tested-By: Martin Steigerwald <ms@teamix.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAndroid: add ioprio support
Aaron Carroll [Tue, 12 Mar 2013 04:46:10 +0000 (15:46 +1100)]
Android: add ioprio support

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agolfsr: fix verification and spin bugs
Alex Pyrgiotis [Tue, 12 Mar 2013 08:28:35 +0000 (10:28 +0200)]
lfsr: fix verification and spin bugs

Changes:

1. Verification now works properly and reports for which value it fails
2. Fix a mishandling of spin incrementation which led to multiple
calculation of the same values

Signed-off-by: Alex Pyrgiotis <apyrgio@grnet.gr>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoConditionally include <sys/shm.h> based on OS support. Define missing MAP_HUGETLB...
Oleg [Tue, 12 Mar 2013 07:06:53 +0000 (00:06 -0700)]
Conditionally include <sys/shm.h> based on OS support. Define missing MAP_HUGETLB for Android.

Signed-off-by: Oleg Matcovschi <olegmatcovsky@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoRemove config-host.h on clean.
Oleg Matcovschi [Tue, 12 Mar 2013 07:04:16 +0000 (00:04 -0700)]
Remove config-host.h on clean.

Signed-off-by: Oleg Matcovschi <olegmatcovsky@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agolfsr: Fix spin related bug
Alex Pyrgiotis [Sun, 10 Mar 2013 09:58:05 +0000 (10:58 +0100)]
lfsr: Fix spin related bug

On the previous patch on cycle detection, we would incremet spin value
when:

    num_vals % cycle_length == 0

which would evaluate always to true for the first iteration since we
start with num_vals = 0. Only one corner-case is affected by this bug
and this patch should fix it.

Signed-off-by: Alex Pyrgiotis <apyrgio@grnet.gr>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agolfsr: indent table
Jens Axboe [Fri, 8 Mar 2013 18:12:15 +0000 (19:12 +0100)]
lfsr: indent table

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd a simple test for LFSR generator
Alex Pyrgiotis [Fri, 8 Mar 2013 12:37:04 +0000 (14:37 +0200)]
Add a simple test for LFSR generator

Adds a simple test suite to check the speed of the LFSR
generator and verify its results. Just run:

    make t/lfsr-test
    ./t/lfsr-test

to compile the test suite and print its usage

Signed-off-by: Alex Pyrgiotis <apyrgio@grnet.gr>
 create mode 100644 t/lfsr-test.c
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoImprove LFSR implementation
Alex Pyrgiotis [Fri, 8 Mar 2013 12:37:03 +0000 (14:37 +0200)]
Improve LFSR implementation

Changes:

1. Use Galois LFSR instead of Fibonacci LFSR.
2. Use XNOR gates instead of XOR gates.
3. Add tap sizes for LFSRs ranging from 3-bits to 15-bits.
4. Add spin parameter.

Rationale:

1. Fibonacci LFSRs have the following drawbacks:

   a. Their next state can not be computed in one cycle, since the input
      bit must be propagated serially through the XOR gates. Galois
  LFSRs however, can be computed instantly with XOR-masks.

   b. Their state changes cannot be considered "irregular", even by I/O
      standards. Actually, if the current state of an n-bit LFSR is x,
  then the next will either be (x >> 1) or (2^n + (x >> 1)). Galois
  LFSRs have instead their XOR gates interleaved with their bits,
  which means that the inner bits are changed as well, besides of
  the shifting. If the number of taps is z, this means that the
  different outcomes are 2^(z + 1).

2. An LFSR with XOR gates has the all-zeroes state as illegal. Since
   zero is valid for most I/O operations, it would be more intuitive to
   use XNOR gates, that have as the all-ones state as illegal.

3. Allow smaller I/O benchmarks to use LFSRs.

4. The spin parameter follows the same rationale as in 1b. To make the
   LFSR outcome "appear" less predictable, we can spin internally the
   LFSR state and produce the i-th number. To understand the spin
   parameter, consider the following state sequence of a 3-bit LFSR:

                             0, 2, 3, 5, 6, 1, 4
   Same LFSR, spin value 2:  0, 3, 6, 4, 2, 5, 1
   But what is the benefit from using spin? Well, the benefits are
   two-fold:

   a. For the same I/O size, we can create a different I/O sequences.

   b. Following the rationale of 1b, we can have more variable outcomes.
      If the spin value is "i" and the taps are "z", the number of
  different outcomes become i * 2^(z + 1).

Signed-off-by: Alex Pyrgiotis <apyrgio@grnet.gr>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix skewed latencies for rate IOPS
SEOKYOUNG KO [Thu, 7 Mar 2013 11:34:31 +0000 (12:34 +0100)]
Fix skewed latencies for rate IOPS

- when -rate_iops is specified, FIO periodically calls usleep() to limit IOPS

- Before usleep(), FIO always wait until the completion of all pending I/O

- For all I/O completions, FIO shows erroneous behavior of logging their
  latency, with that of the longest latency IO sample.

- w/ QD=8/ mixed R:W=33:66, up to 7 samples may get false latency log when
  waiting all I/O completions => False report of latency distribution

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix rate iops with ramp time
SEOKYOUNG KO [Thu, 7 Mar 2013 11:31:04 +0000 (12:31 +0100)]
Fix rate iops with ramp time

When the ramp time is over, fio resets the ->io_issues[]
counters, but not the rwmix counters. This causes a problem
for mixed read/write workloads that are rate limited and
include a ramp time.

Fix it by resetting rwmix_issues with the regular issues.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd --parse-only command line option
Jens Axboe [Thu, 7 Mar 2013 10:31:20 +0000 (11:31 +0100)]
Add --parse-only command line option

Sometimes it's useful just being able to verify whether a job
config is valid or not, without actually starting any jobs.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMerge branch 'master' into gfio
Jens Axboe [Mon, 4 Mar 2013 11:27:07 +0000 (12:27 +0100)]
Merge branch 'master' into gfio

Conflicts:
init.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoconfigure: fixup __sync_fetch_and_add()
Jens Axboe [Thu, 28 Feb 2013 21:13:29 +0000 (22:13 +0100)]
configure: fixup __sync_fetch_and_add()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoconfigure: add general libs for Solaris
Jens Axboe [Thu, 28 Feb 2013 21:10:22 +0000 (22:10 +0100)]
configure: add general libs for Solaris

Assume we just need them, this makes us find inet_aton()
on Solaris too.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agosolaris: ensure that TCP_NODELAY gets picked up
Jens Axboe [Thu, 28 Feb 2013 19:34:31 +0000 (20:34 +0100)]
solaris: ensure that TCP_NODELAY gets picked up

Use proper libs when linking, then we can also remove
those from the Makefile.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Thu, 28 Feb 2013 19:08:12 +0000 (20:08 +0100)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

6 years agopowerpc: use mfspr for Time Base Register reads
Steven Noonan [Thu, 28 Feb 2013 19:08:06 +0000 (20:08 +0100)]
powerpc: use mfspr for Time Base Register reads

The 'mftb' and 'mftbu' instructions are deprecated in Power ISA v2.06 in favor
of mfspr. Since CPUs supporting mftb/mftbu also support the TBR SPRs, there's
no known reason to keep mftb/mftbu.

Signed-off-by: Steven Noonan <steven@uplinklabs.net>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix compile of RDMA engine for SunOS 5.x
Jens Axboe [Thu, 28 Feb 2013 07:28:05 +0000 (08:28 +0100)]
Fix compile of RDMA engine for SunOS 5.x

- byteswap.h include that doesn't exist there. Kill it, we
  have no use for it.

- No RLIMIT_MEMLOCK on SunOS. The use in the engine is a bit
  suspect, so not a problem if we don't have it. Add configure
  check for that.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agonet: fix warning for systems without TCP_NODELAY
Jens Axboe [Thu, 28 Feb 2013 07:24:23 +0000 (08:24 +0100)]
net: fix warning for systems without TCP_NODELAY

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRevert "debug: only do getpid() if we have to"
Jens Axboe [Wed, 27 Feb 2013 11:32:43 +0000 (12:32 +0100)]
Revert "debug: only do getpid() if we have to"

This reverts commit 8037423f08649a7c3378dadf95327d6553105264.

7 years agoFIO-VERSION-GEN: use `foo` instead of $(foo)
Jens Axboe [Tue, 26 Feb 2013 20:59:33 +0000 (21:59 +0100)]
FIO-VERSION-GEN: use `foo` instead of $(foo)

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Tue, 26 Feb 2013 20:55:43 +0000 (21:55 +0100)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

7 years agoconfigure: use `foo` instead of $(foo)
Jens Axboe [Tue, 26 Feb 2013 20:55:08 +0000 (21:55 +0100)]
configure: use `foo` instead of $(foo)

Seems to work everywhere, whereas $(foo) does not work on
some variants of FreeBSD.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agopowerpc: harden the clock ATB probe a bit
Jens Axboe [Tue, 26 Feb 2013 15:11:49 +0000 (16:11 +0100)]
powerpc: harden the clock ATB probe a bit

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agopowerpc: use ATB clock, if it's available
Jens Axboe [Tue, 26 Feb 2013 13:43:02 +0000 (14:43 +0100)]
powerpc: use ATB clock, if it's available

The ATB is more precise, but we don't know if it's available.
Check by forking off a process that attempts to use it, if
we die with SIGILL, then we know it's not there.

Based on code from Steven Noonan.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: print 64-bit variable with ULL
Jens Axboe [Tue, 26 Feb 2013 12:54:20 +0000 (13:54 +0100)]
gettime: print 64-bit variable with ULL

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agodebug: make __dprint() have printf() like argument type checking
Jens Axboe [Tue, 26 Feb 2013 07:16:43 +0000 (08:16 +0100)]
debug: make __dprint() have printf() like argument type checking

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agodebug: only do getpid() if we have to
Jens Axboe [Mon, 25 Feb 2013 13:02:42 +0000 (14:02 +0100)]
debug: only do getpid() if we have to

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: use 32-bit atomic sequences
Jens Axboe [Mon, 25 Feb 2013 09:23:58 +0000 (10:23 +0100)]
gettime: use 32-bit atomic sequences

Not all platforms have 64-bit wide atomic sync_and_fetch(). If
we just check for overflow, it should be OK to use a 32-bit
sequence number.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFixup wrong types for dprint()
Jens Axboe [Mon, 25 Feb 2013 09:18:33 +0000 (10:18 +0100)]
Fixup wrong types for dprint()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: add some sanity checks to platform clock
Jens Axboe [Sun, 24 Feb 2013 20:29:35 +0000 (21:29 +0100)]
gettime: add some sanity checks to platform clock

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Sun, 24 Feb 2013 13:24:16 +0000 (14:24 +0100)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

7 years agoAdd LICENSE to Windows installer, fix filename extensions and update URL.
Bruce Cran [Sun, 24 Feb 2013 11:15:26 +0000 (11:15 +0000)]
Add LICENSE to Windows installer, fix filename extensions and update URL.

Add the LICENSE file to the Windows installer, change installed filename
extensions from .rtf to .txt (assume a text editor is installed that
understands LF) and fix the "update url" to be http://bluestop.org/fio/
since that's where new Windows installers are.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd section 9 to HOWTO table of contents.
Bruce Cran [Sun, 24 Feb 2013 11:09:11 +0000 (11:09 +0000)]
Add section 9 to HOWTO table of contents.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd --eta-newline option
Jens Axboe [Fri, 22 Feb 2013 19:48:56 +0000 (20:48 +0100)]
Add --eta-newline option

For certain situations, it's handy to force a new line in
the ETA output. It allows you to easily track what
happened, without having to do detailed logging.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' into gfio
Jens Axboe [Thu, 21 Feb 2013 19:13:57 +0000 (20:13 +0100)]
Merge branch 'master' into gfio

7 years agoFio 2.0.14 fio-2.0.14
Jens Axboe [Thu, 21 Feb 2013 18:12:32 +0000 (19:12 +0100)]
Fio 2.0.14

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix problem with terminating on unaligned sizes
Jens Axboe [Thu, 21 Feb 2013 14:18:17 +0000 (15:18 +0100)]
Fix problem with terminating on unaligned sizes

Three separate little issues:

- Don't round up the number of blocks needed. That means we end
  up with potential partial blocks, which we can never do IO to.

- Fix an axmap bug where we only check against the specific bit
  in the mask, but we really want to check for "this bit or any
  higher bit". This makes axmap_next_free() behave more like it
  should, instead of failing way too often.

- For the almost-full case, we can return a next available bit
  that is just outside the allowed range. Punt to first-free for
  that case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd FreeBSD cpu affinity support.
Bruce Cran [Thu, 21 Feb 2013 12:39:24 +0000 (12:39 +0000)]
Add FreeBSD cpu affinity support.

Since FreeBSD has different thread and process IDs, assume that fio_setaffinity
will always be called with a tid and fio_getaffinity with a pid.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoDeclare 'prev' and 'this' outside the loop to avoid clang warning about 'prev' being...
Bruce Cran [Thu, 21 Feb 2013 13:16:17 +0000 (14:16 +0100)]
Declare 'prev' and 'this' outside the loop to avoid clang warning about 'prev' being uninitialized

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFixup ia64 32-bit register clock cycle read
Jens Axboe [Thu, 21 Feb 2013 11:57:56 +0000 (12:57 +0100)]
Fixup ia64 32-bit register clock cycle read

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix x86 32-bit wrong variable type for rdtsc
Jens Axboe [Thu, 21 Feb 2013 11:39:42 +0000 (12:39 +0100)]
Fix x86 32-bit wrong variable type for rdtsc

Bruce points out, that we are putting the result into a 32-bit
variable. That is incorrect, make it a proper ULL instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoREADME: kill reference to non-functional SUSE package page
Jens Axboe [Thu, 21 Feb 2013 10:03:28 +0000 (11:03 +0100)]
README: kill reference to non-functional SUSE package page

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd recommendation that fio be run in a native console on Windows.
Bruce Cran [Wed, 20 Feb 2013 20:35:16 +0000 (20:35 +0000)]
Add recommendation that fio be run in a native console on Windows.

Windows console applications often don't run correctly under Cygwin shells such
as mintty because of the way they're implemented. Add a recommendation to the
README to run fio under a native console such as the Command Prompt or console2
to avoid these problems.

http://code.google.com/p/mintty/issues/detail?id=56 documents the problem.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUpdate all Windows files except dobuild.cmd to use LF line endings
Bruce Cran [Thu, 21 Feb 2013 10:02:41 +0000 (11:02 +0100)]
Update all Windows files except dobuild.cmd to use LF line endings

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agocheck output_format before calling show_idle_prof_stats()
Huadong Liu [Thu, 21 Feb 2013 09:24:20 +0000 (10:24 +0100)]
check output_format before calling show_idle_prof_stats()

Do the proper checks for output format type and whether we are
a server backend or not.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agot/axmap: add pre/post bit set checks
Jens Axboe [Thu, 21 Feb 2013 09:23:51 +0000 (10:23 +0100)]
t/axmap: add pre/post bit set checks

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agocheck output_format before calling show_idle_prof_stats()
Huadong Liu [Thu, 21 Feb 2013 09:24:20 +0000 (10:24 +0100)]
check output_format before calling show_idle_prof_stats()

Do the proper checks for output format type and whether we are
a server backend or not.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agot/axmap: add pre/post bit set checks
Jens Axboe [Thu, 21 Feb 2013 09:23:51 +0000 (10:23 +0100)]
t/axmap: add pre/post bit set checks

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' into gfio
Jens Axboe [Wed, 20 Feb 2013 10:52:28 +0000 (11:52 +0100)]
Merge branch 'master' into gfio

Conflicts:
configure
fio.c
init.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Mon, 18 Feb 2013 19:09:28 +0000 (20:09 +0100)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

7 years agoFix replay of IO pattern with multiple open files
Nikolaus Jeremic [Mon, 18 Feb 2013 18:44:41 +0000 (19:44 +0100)]
Fix replay of IO pattern with multiple open files

Nikolaus writes:

I've tried to replay I/Os from a textual iolog file to a couple of files
using fio 2.0.13 on Linux. However, all I/Os were performed to the last
opened file, which is '/mnt/ssd5/rand-write.1.0_1_2' in this example .
My jobfile and iolog files contain the following:

[global]
name=replay_filetraces
description=Replay file traces
ioengine=libaio
iodepth=6
direct=1
continue_on_error=1

[replay]
numjobs=1
read_iolog=file_io_patterns
unlink=0

fio version 2 iolog
/mnt/ssd0/rand-write.1.0_0_0 add
/mnt/ssd1/rand-write.1.0_0_1 add
/mnt/ssd2/rand-write.1.0_0_2 add
/mnt/ssd3/rand-write.1.0_1_0 add
/mnt/ssd4/rand-write.1.0_1_1 add
/mnt/ssd5/rand-write.1.0_1_2 add
/mnt/ssd0/rand-write.1.0_0_0 open
/mnt/ssd1/rand-write.1.0_0_1 open
/mnt/ssd2/rand-write.1.0_0_2 open
/mnt/ssd3/rand-write.1.0_1_0 open
/mnt/ssd4/rand-write.1.0_1_1 open
/mnt/ssd5/rand-write.1.0_1_2 open
/mnt/ssd0/rand-write.1.0_0_0 write 0 65536
/mnt/ssd1/rand-write.1.0_0_1 write 0 65536
/mnt/ssd2/rand-write.1.0_0_2 write 0 65536
/mnt/ssd3/rand-write.1.0_1_0 write 0 65536
/mnt/ssd4/rand-write.1.0_1_1 write 0 65536
/mnt/ssd5/rand-write.1.0_1_2 write 0 65536
/mnt/ssd0/rand-write.1.0_0_0 write 65536 65536
/mnt/ssd1/rand-write.1.0_0_1 write 65536 65536
/mnt/ssd2/rand-write.1.0_0_2 write 65536 65536
/mnt/ssd3/rand-write.1.0_1_0 write 65536 65536
/mnt/ssd4/rand-write.1.0_1_1 write 65536 65536
/mnt/ssd5/rand-write.1.0_1_2 write 65536 65536
/mnt/ssd0/rand-write.1.0_0_0 close
/mnt/ssd1/rand-write.1.0_0_1 close
/mnt/ssd2/rand-write.1.0_0_2 close
/mnt/ssd3/rand-write.1.0_1_0 close
/mnt/ssd4/rand-write.1.0_1_1 close
/mnt/ssd5/rand-write.1.0_1_2 close

I've turned on debugging and added some extra printf in read_iolog2()
within iolog.c.

file     24594 add file /mnt/ssd0/rand-write.1.0_0_0
file     24594 resize file array to 2 files
file     24594 file 0x7fbe03f53090 "/mnt/ssd0/rand-write.1.0_0_0" added at 0
file     24594 add file /mnt/ssd1/rand-write.1.0_0_1
file     24594 file 0x7fbe03f53230 "/mnt/ssd1/rand-write.1.0_0_1" added at 1
file     24594 add file /mnt/ssd2/rand-write.1.0_0_2
file     24594 resize file array to 4 files
file     24594 file 0x7fbe03f533d0 "/mnt/ssd2/rand-write.1.0_0_2" added at 2
file     24594 add file /mnt/ssd3/rand-write.1.0_1_0
file     24594 file 0x7fbe03f53570 "/mnt/ssd3/rand-write.1.0_1_0" added at 3
file     24594 add file /mnt/ssd4/rand-write.1.0_1_1
file     24594 resize file array to 6 files
file     24594 file 0x7fbe03f53710 "/mnt/ssd4/rand-write.1.0_1_1" added at 4
file     24594 add file /mnt/ssd5/rand-write.1.0_1_2
file     24594 file 0x7fbe03f538b0 "/mnt/ssd5/rand-write.1.0_1_2" added at 5
io       24594 read_iolog2: Making note of file.
offset=7310584035475809390 bytes=811544671 fileno=0 file_action=1
fname=/mnt/ssd0/rand-write.1.0_0_0 act=open
io       24594 read_iolog2: Making note of file.
offset=7310584035475809390 bytes=811544671 fileno=1 file_action=1
fname=/mnt/ssd1/rand-write.1.0_0_1 act=open
io       24594 read_iolog2: Making note of file.
offset=7310584035475809390 bytes=811544671 fileno=2 file_action=1
fname=/mnt/ssd2/rand-write.1.0_0_2 act=open
io       24594 read_iolog2: Making note of file.
offset=7310584035475809390 bytes=811544671 fileno=3 file_action=1
fname=/mnt/ssd3/rand-write.1.0_1_0 act=open
io       24594 read_iolog2: Making note of file.
offset=7310584035475809390 bytes=811544671 fileno=4 file_action=1
fname=/mnt/ssd4/rand-write.1.0_1_1 act=open
io       24594 read_iolog2: Making note of file.
offset=7310584035475809390 bytes=811544671 fileno=5 file_action=1
fname=/mnt/ssd5/rand-write.1.0_1_2 act=open
io       24594 read_iolog2: Making note of file. offset=0 bytes=65536
fileno=5 file_action=1 fname=/mnt/ssd0/rand-write.1.0_0_0 act=write
io       24594 read_iolog2: Making note of file. offset=0 bytes=65536
fileno=5 file_action=1 fname=/mnt/ssd1/rand-write.1.0_0_1 act=write
io       24594 read_iolog2: Making note of file. offset=0 bytes=65536
fileno=5 file_action=1 fname=/mnt/ssd2/rand-write.1.0_0_2 act=write
io       24594 read_iolog2: Making note of file. offset=0 bytes=65536
fileno=5 file_action=1 fname=/mnt/ssd3/rand-write.1.0_1_0 act=write
io       24594 read_iolog2: Making note of file. offset=0 bytes=65536
fileno=5 file_action=1 fname=/mnt/ssd4/rand-write.1.0_1_1 act=write
io       24594 read_iolog2: Making note of file. offset=0 bytes=65536
fileno=5 file_action=1 fname=/mnt/ssd5/rand-write.1.0_1_2 act=write
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=5 file_action=1 fname=/mnt/ssd0/rand-write.1.0_0_0
act=write
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=5 file_action=1 fname=/mnt/ssd1/rand-write.1.0_0_1
act=write
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=5 file_action=1 fname=/mnt/ssd2/rand-write.1.0_0_2
act=write
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=5 file_action=1 fname=/mnt/ssd3/rand-write.1.0_1_0
act=write
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=5 file_action=1 fname=/mnt/ssd4/rand-write.1.0_1_1
act=write
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=5 file_action=1 fname=/mnt/ssd5/rand-write.1.0_1_2
act=write
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=0 file_action=2 fname=/mnt/ssd0/rand-write.1.0_0_0
act=close
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=1 file_action=2 fname=/mnt/ssd1/rand-write.1.0_0_1
act=close
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=2 file_action=2 fname=/mnt/ssd2/rand-write.1.0_0_2
act=close
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=3 file_action=2 fname=/mnt/ssd3/rand-write.1.0_1_0
act=close
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=4 file_action=2 fname=/mnt/ssd4/rand-write.1.0_1_1
act=close
io       24594 read_iolog2: Making note of file. offset=65536
bytes=65536 fileno=5 file_action=2 fname=/mnt/ssd5/rand-write.1.0_1_2
act=close

Taking a look to the debugging output shows that on write action the
'fileno' variable is always set to 5 regardless of the file that should
be written. However, for open and close we have a different fileno value
for each file. Taking a look into the source code of read_iolog2()
reveals why: In case of actions 'open' or 'close', 'fileno' is looked up
through 'get_fileno(td, fname)', but for 'write', 'read' and the other 4
argument actions 'fileno' is not looked up. When I add a 'fileno =
get_fileno(td, fname);' at the end of the '(r == 4)' branch, I/Os are
replayed to the designated files.

                 if (r == 4) {
                         /*
                          * Check action first
                          */
                         if (!strcmp(act, "wait"))
                                 rw = DDIR_WAIT;
                         else if (!strcmp(act, "read"))
                                 rw = DDIR_READ;
                         else if (!strcmp(act, "write"))
                                 rw = DDIR_WRITE;
                         else if (!strcmp(act, "sync"))
                                 rw = DDIR_SYNC;
                         else if (!strcmp(act, "datasync"))
                                 rw = DDIR_DATASYNC;
                         else if (!strcmp(act, "trim"))
                                 rw = DDIR_TRIM;
                         else {
                                 log_err("fio: bad iolog file action: %s\n",
act);
                                 continue;
                         }
                 } else if (r == 2) {
                         rw = DDIR_INVAL;
                         if (!strcmp(act, "add")) {
                                 td->o.nr_files++;
                                 fileno = add_file(td, fname);
                                 file_action = FIO_LOG_ADD_FILE;
                                 continue;
                         } else if (!strcmp(act, "open")) {
                                 fileno = get_fileno(td, fname);
                                 file_action = FIO_LOG_OPEN_FILE;
                         } else if (!strcmp(act, "close")) {
                                 fileno = get_fileno(td, fname);
                                 file_action = FIO_LOG_CLOSE_FILE;
                         } else {
                                 log_err("fio: bad iolog file action: %s\n",
act);
                                 continue;
                         }
                 } else {
                         log_err("bad iolog2: %s", p);
                         continue;
                 }

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoconfigure: missing Android config options
Aaron Carroll [Mon, 18 Feb 2013 07:52:18 +0000 (08:52 +0100)]
configure: missing Android config options

We need this for the run-time endianness check to pass.  Add supported
clock monotonic while we're at it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoconfigure: attempt to get Android going again
Aaron Carroll [Tue, 12 Feb 2013 08:58:14 +0000 (09:58 +0100)]
configure: attempt to get Android going again

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoInitialize global and non-static variables
Jens Axboe [Mon, 11 Feb 2013 15:33:25 +0000 (16:33 +0100)]
Initialize global and non-static variables

Don't assume they get zeroed. The AIX linker throws a hissy fit on
these, refusing to link the program.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoTurn f->engine_data into a 64-bit value
Jens Axboe [Mon, 11 Feb 2013 13:35:43 +0000 (14:35 +0100)]
Turn f->engine_data into a 64-bit value

Even on 32-bit platforms, we store (uint64_t) -1 in there
for some cases to make it be 'invalid'.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoconfigure: ensure that fatal errors kill config-host.h and mak files
Jens Axboe [Mon, 11 Feb 2013 12:21:54 +0000 (13:21 +0100)]
configure: ensure that fatal errors kill config-host.h and mak files

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoHandle end_fsync if a file is closed
Jens Axboe [Mon, 11 Feb 2013 10:26:55 +0000 (11:26 +0100)]
Handle end_fsync if a file is closed

Should not happen, but better be careful since it could have
an impact on data integrity.

Additionally, add a log_err() error if we do fail it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoconfigure: allow --cpu to be specified
Jens Axboe [Sun, 10 Feb 2013 14:48:34 +0000 (15:48 +0100)]
configure: allow --cpu to be specified

And don't fail if we don't detect, it's not critical
for configure.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix rate limiting
Jens Axboe [Fri, 8 Feb 2013 18:05:25 +0000 (19:05 +0100)]
Fix rate limiting

Commit e0224c6b accidentally broke rate limiting, since
it bypassed the sleep logic. Rework it slightly to not
skip that.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoWindows: io cancellation often fails and causes crashes, so remove it.
Bruce Cran [Fri, 8 Feb 2013 01:45:17 +0000 (01:45 +0000)]
Windows: io cancellation often fails and causes crashes, so remove it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix bug with rate and read/write mixed workloads at 100% bias
Jens Axboe [Thu, 7 Feb 2013 18:55:24 +0000 (19:55 +0100)]
Fix bug with rate and read/write mixed workloads at 100% bias

The rate_ddir() might sometimes decide to send the other data
direction, even if we have a workload with rw=randrw and
rwmixread=0. Check for this condition.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' into gfio
Jens Axboe [Thu, 7 Feb 2013 14:45:39 +0000 (15:45 +0100)]
Merge branch 'master' into gfio

Conflicts:
Makefile
configure
fio.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agot/log: include minmax.h for min()
Jens Axboe [Thu, 7 Feb 2013 14:40:48 +0000 (15:40 +0100)]
t/log: include minmax.h for min()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMakfile: use LINK for the link phase
Jens Axboe [Thu, 7 Feb 2013 14:40:37 +0000 (15:40 +0100)]
Makfile: use LINK for the link phase

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix zones for numjobs=x, where x > 1
Jens Axboe [Thu, 7 Feb 2013 12:06:18 +0000 (13:06 +0100)]
Fix zones for numjobs=x, where x > 1

In the options fixup, we auto-set open_files to 1 if it hasn't been set.
So any duplicated jobs will clear the zone. Fixup the check to properly
check for > 1 open files.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agostat: fix wrong type used for 32-bit compiles
Jens Axboe [Thu, 7 Feb 2013 11:56:09 +0000 (12:56 +0100)]
stat: fix wrong type used for 32-bit compiles

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix crash and precision of ETA with zones
Jens Axboe [Thu, 7 Feb 2013 11:54:10 +0000 (12:54 +0100)]
Fix crash and precision of ETA with zones

If zonesize was bigger than zoneskip, we could encounter a divide by
zero when calculating the number of bytes.

Additionally, the math was just wrong for most cases of zone settings.
Improve that.

Signed-off-by: Jens Axboe <axboe@kernel.dk>