fio.git
6 years agoeta: improve accuracy for rwmix workloads and verify
Jens Axboe [Wed, 23 Jan 2013 15:40:11 +0000 (08:40 -0700)]
eta: improve accuracy for rwmix workloads and verify

We assume 50/50 for mixed workloads. That's silly, use the actual
number specified instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUse unsigned long long for the uint64_t type
Jens Axboe [Mon, 21 Jan 2013 19:30:34 +0000 (12:30 -0700)]
Use unsigned long long for the uint64_t type

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agowindows: use hweight64(), it's a 64-bit type
Jens Axboe [Mon, 21 Jan 2013 19:28:42 +0000 (12:28 -0700)]
windows: use hweight64(), it's a 64-bit type

Courtesy of Bruce.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd hweight64()
Jens Axboe [Mon, 21 Jan 2013 19:28:23 +0000 (12:28 -0700)]
Add hweight64()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix _SC_NPROCESSORS_ONLN on Windows
Bruce Cran [Mon, 21 Jan 2013 17:57:59 +0000 (10:57 -0700)]
Fix _SC_NPROCESSORS_ONLN on Windows

_SC_NPROCESSORS_ONLN was returning the wrong value on Windows because
GetLogicalProcessorInformation() returns data about caches, nodes and
packages in addition to processors.

Modified by Jens to use hweight32()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd generic hweight helpers
Jens Axboe [Mon, 21 Jan 2013 17:55:02 +0000 (10:55 -0700)]
Add generic hweight helpers

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoverify: add new experimental mode that requires no meta data
Jens Axboe [Mon, 21 Jan 2013 16:47:03 +0000 (09:47 -0700)]
verify: add new experimental mode that requires no meta data

Should work fine, but we need to account and track trims to know
which blocks NOT to verify (or verify as zero).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRe-seed random generator when a file is reset
Jens Axboe [Mon, 21 Jan 2013 16:46:06 +0000 (09:46 -0700)]
Re-seed random generator when a file is reset

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoaxmap: clear map->first_free when we reset it
Jens Axboe [Mon, 21 Jan 2013 16:43:30 +0000 (09:43 -0700)]
axmap: clear map->first_free when we reset it

Otherwise it could be left at an invalid value. When we clear the map,
we know the first free bit is 0.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agolfsr: add lfsr_reset()
Jens Axboe [Mon, 21 Jan 2013 16:42:49 +0000 (09:42 -0700)]
lfsr: add lfsr_reset()

This enables us to restart a sequence.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: use proper uint64_t types where needed
Jens Axboe [Mon, 21 Jan 2013 12:38:22 +0000 (05:38 -0700)]
gettime: use proper uint64_t types where needed

Windows has 32-bit longs even on 64-bit, so we risk overflowing.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd info log on whether tsc is reliable or not for --cpuclock-test
Jens Axboe [Fri, 18 Jan 2013 19:13:45 +0000 (20:13 +0100)]
Add info log on whether tsc is reliable or not for --cpuclock-test

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoWindows: fix off-by-one error in fio_cpu_set and add affinity error messages.
Bruce Cran [Fri, 18 Jan 2013 13:13:13 +0000 (13:13 +0000)]
Windows: fix off-by-one error in fio_cpu_set and add affinity error messages.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMake hugetlbfs maping work
Shaohua Li [Thu, 17 Jan 2013 12:28:15 +0000 (13:28 +0100)]
Make hugetlbfs maping work

If it's a file mapping, flags must be MAP_HARED; If it's an anonymous
mapping, flags must be MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB (can be
shared, but doesn't matter here).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agofusion-aw: make configure and compile actually work
Jens Axboe [Mon, 14 Jan 2013 18:33:40 +0000 (19:33 +0100)]
fusion-aw: make configure and compile actually work

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agofusion-aw: fixes
Jens Axboe [Mon, 14 Jan 2013 18:22:15 +0000 (19:22 +0100)]
fusion-aw: fixes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoconfigure: use exit 0 instead of just exit
Jens Axboe [Mon, 14 Jan 2013 18:21:55 +0000 (19:21 +0100)]
configure: use exit 0 instead of just exit

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix typo (READ/DDIR_READ).
Bruce Cran [Sun, 13 Jan 2013 17:21:58 +0000 (17:21 +0000)]
Fix typo (READ/DDIR_READ).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix configure script for 64-bit Windows.
Bruce Cran [Sun, 13 Jan 2013 17:10:25 +0000 (17:10 +0000)]
Fix configure script for 64-bit Windows.

Hard-code config options when building using MinGW-w64 on Windows.
For 32-bit builds, CC and CONFIG_32BIT/CONFIG_64BIT will need to be
updated manually.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd CONFIG_64BIT_LLP64, CONFIG_POSIX_FALLOCATE and CONFIG_LINUX_FALLOCATE to Makefile.
Bruce Cran [Sun, 13 Jan 2013 17:09:22 +0000 (17:09 +0000)]
Add CONFIG_64BIT_LLP64, CONFIG_POSIX_FALLOCATE and CONFIG_LINUX_FALLOCATE to Makefile.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoImprove documentation for end_fsync
Jens Axboe [Sun, 13 Jan 2013 16:16:46 +0000 (17:16 +0100)]
Improve documentation for end_fsync

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoconfigure: enable e4defrag engine regardless of MOVE_EXTENT compile test
Jens Axboe [Sat, 12 Jan 2013 09:27:53 +0000 (10:27 +0100)]
configure: enable e4defrag engine regardless of MOVE_EXTENT compile test

It may exist in the running kernel, even if the defines
and structures haven't bubbled back into userspace yet.
The engine carries it's own check and structure define
for that case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoPre-load and sort random blocks for pure read verify workloads
Jens Axboe [Sat, 12 Jan 2013 08:44:15 +0000 (01:44 -0700)]
Pre-load and sort random blocks for pure read verify workloads

If fio is run with a write phase before a read phase and the
IO type is random, the end read verify phase will get sorted
blocks to read back. But if fio is asked to verify something
that was previously randomly written, it will generate the
same random offsets in random order and verify those. This is
usually much slower than a sorted read back.

So add a verifysort_nr option that allows the user to specify
a number of random offsets to pre-generate and sort, before
reading them and verifying the contents. This greatly speeds
up pure read verify workloads.

Default to 1024, and put a max of 64K entries on the option.
We do a merge list sort on the entries, so we don't want a
huge amount of backlog.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agolfsr: ensure we don't generate an offset + buflen that exceeds the max size
Jens Axboe [Fri, 11 Jan 2013 13:03:25 +0000 (14:03 +0100)]
lfsr: ensure we don't generate an offset + buflen that exceeds the max size

Currently we check for the max value, but that doesn't always
work since it may not fit the minimum block size (even if it
is guaranteed to be smaller than the max offset). Pass in the
last valid block.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agofilesetup: limit rand map to the actual IO size, if needed
Jens Axboe [Fri, 11 Jan 2013 13:03:05 +0000 (14:03 +0100)]
filesetup: limit rand map to the actual IO size, if needed

No need allocating a larger map than we need.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd a check avoid segfault
Jianpeng Ma [Fri, 11 Jan 2013 07:52:55 +0000 (08:52 +0100)]
Add a check avoid segfault

If new_size was zero or realloc failed, it would be segment fault.
So add a check.

Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRemove repeated --readonly options in documentation
Jianpeng Ma [Thu, 10 Jan 2013 12:20:02 +0000 (13:20 +0100)]
Remove repeated --readonly options in documentation

Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix a potential integer overflow
Jianpeng Ma [Thu, 10 Jan 2013 12:19:27 +0000 (13:19 +0100)]
Fix a potential integer overflow

Because nr_files and new_size are not same types.It can cause new_size
less than zero.Then realloc will be failed.

Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd configure script
Jens Axboe [Thu, 10 Jan 2013 10:23:19 +0000 (11:23 +0100)]
Add configure script

Get rid of all the fragile guessing and checking of features,
and roll a configure script instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix variable type warning
Jens Axboe [Wed, 9 Jan 2013 10:46:01 +0000 (11:46 +0100)]
Fix variable type warning

Bruce reports:

With 2.0.13 on FreeBSD there's also a warning from client.c:

client.c:24:18: warning: field 'eta' with variable sized type 'struct
jobs_eta' not at the end of a struct or class is a GNU extension [-Wgnu]
         struct jobs_eta eta;

Re-order ->eta and ->pending.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.13 fio-2.0.13
Jens Axboe [Wed, 9 Jan 2013 10:01:55 +0000 (11:01 +0100)]
Fio 2.0.13

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix potential null pointer dereference on verify and requeue events
Jens Axboe [Tue, 8 Jan 2013 20:02:14 +0000 (21:02 +0100)]
Fix potential null pointer dereference on verify and requeue events

We clear io_u to NULL, but later we derefence io_u to check
the data direction. Do as we do in the main IO loop - store
the data direction in a local variable, and use that after
the main switch.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoserver: move fio_net_cmd to the end of fio_net_int_cmd
Jens Axboe [Tue, 8 Jan 2013 12:43:14 +0000 (13:43 +0100)]
server: move fio_net_cmd to the end of fio_net_int_cmd

fio_net_cmd is variably sized. We don't use any transfer space
for the internal command, but the compiler might complain about
this (since it has no way to know this fact). Move fio_net_cmd
to the end of the internal command to both avoid a warning on
clang and to make this more idiot proof.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUse variable[] instead of GCC variable[0] syntax
Jens Axboe [Tue, 8 Jan 2013 12:42:41 +0000 (13:42 +0100)]
Use variable[] instead of GCC variable[0] syntax

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoEnable -ffast-math
Jens Axboe [Tue, 8 Jan 2013 12:41:35 +0000 (13:41 +0100)]
Enable -ffast-math

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agotime: convert to uint64_t
Jens Axboe [Fri, 4 Jan 2013 12:24:52 +0000 (13:24 +0100)]
time: convert to uint64_t

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUse uintptr_t for engine data
Jens Axboe [Fri, 4 Jan 2013 12:21:23 +0000 (13:21 +0100)]
Use uintptr_t for engine data

Gets rid of a warning on 32-bit builds on Linux.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoStart of transition from unsigned long long to uint64_t
Jens Axboe [Fri, 4 Jan 2013 12:16:54 +0000 (13:16 +0100)]
Start of transition from unsigned long long to uint64_t

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agozipf/pareto: use min of file size and io size for init
Jens Axboe [Fri, 4 Jan 2013 12:09:29 +0000 (13:09 +0100)]
zipf/pareto: use min of file size and io size for init

It is valid to use size=foo, where foo is larger than the size
of the file or device. For that case, we don't want offsets
larger than the size of the file. So use the minimum of the
real or given size.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Fri, 4 Jan 2013 07:27:06 +0000 (08:27 +0100)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

7 years agozipf/pareto: use size= if given, not always device size
Jens Axboe [Fri, 4 Jan 2013 07:26:10 +0000 (08:26 +0100)]
zipf/pareto: use size= if given, not always device size

We used the full device/file size for the zipf or pareto init,
this breaks if size= is specifically given. This causes fio
to generate offsets outside the specified size= range, and fio
will then just exit.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agolibaio: use container_of() instead of silly casting
Jens Axboe [Wed, 2 Jan 2013 11:55:41 +0000 (12:55 +0100)]
libaio: use container_of() instead of silly casting

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Conflicts:
engines/libaio.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: even rounding, don't always round up
Jens Axboe [Tue, 1 Jan 2013 12:38:18 +0000 (13:38 +0100)]
gettime: even rounding, don't always round up

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMove 'tsc_reliable' outside of ARCH_HAVE_CPU_CLOCK
Jens Axboe [Tue, 1 Jan 2013 09:59:04 +0000 (10:59 +0100)]
Move 'tsc_reliable' outside of ARCH_HAVE_CPU_CLOCK

Otherwise we fail building on architectures that do not define
it, as reported by Dan:

cc -o gettime.o -c -std=gnu99 -Wwrite-strings -Wall -O3 -g -O2 -g -pipe
-Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m31 -march=z9-109 -mtune=z10
-DFIO_VERSION='"fio-2.0.12.2"' -D_GNU_SOURCE -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -DFIO_INC_DEBUG gettime.c
gettime.c: In function 'fio_clock_init':
gettime.c:317:6: error: 'tsc_reliable' undeclared (first use in this function)
gettime.c:317:6: note: each undeclared identifier is reported only once for each function it appears in
make: *** [gettime.o] Error 1

Reported-by: Dan Horák <dan@danny.cz>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoimplement get_cpu_clock() for s390/s390x
Dan Horák [Sun, 30 Dec 2012 22:03:36 +0000 (17:03 -0500)]
implement get_cpu_clock() for s390/s390x

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoclock: turn expensive division into multiply + cheap division
Jens Axboe [Fri, 21 Dec 2012 21:54:56 +0000 (22:54 +0100)]
clock: turn expensive division into multiply + cheap division

On x86-64, dividing by a variable turns into a hugely expensive
divq. It's much cheaper to invert the division. Instead of
dividing clocks by clocks-per-usec, multiply by a 16M/clocks-per-usec
constant instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agot/axmap: update tester for lfsr_init() taking a seed argument
Jens Axboe [Fri, 21 Dec 2012 20:32:50 +0000 (21:32 +0100)]
t/axmap: update tester for lfsr_init() taking a seed argument

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoInclude <stdint.h> before checking __WORDSIZE
Jens Axboe [Fri, 21 Dec 2012 20:29:30 +0000 (21:29 +0100)]
Include <stdint.h> before checking __WORDSIZE

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.12.2 fio-2.0.12.2
Jens Axboe [Thu, 20 Dec 2012 14:19:57 +0000 (15:19 +0100)]
Fio 2.0.12.2

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd return statements to arch_init functions for ia64 and ppc.
Bruce Cran [Thu, 20 Dec 2012 13:59:56 +0000 (13:59 +0000)]
Add return statements to arch_init functions for ia64 and ppc.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoUse clock_gettime() for CPU clock calibration
Jens Axboe [Thu, 20 Dec 2012 14:17:57 +0000 (15:17 +0100)]
Use clock_gettime() for CPU clock calibration

gettimeofday() has very poor resolution on some platforms. So
use clock_gettime() instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoConsistently indent using tabs in Windows files and remove trailing spaces.
Bruce Cran [Wed, 19 Dec 2012 09:47:29 +0000 (10:47 +0100)]
Consistently indent using tabs in Windows files and remove trailing spaces.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoCreate a new Windows installer product GUID for 2.0.12.
Bruce Cran [Wed, 19 Dec 2012 09:46:57 +0000 (10:46 +0100)]
Create a new Windows installer product GUID for 2.0.12.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix $(CC) override: use system compiler except on HP-UX and Solaris.
Bruce Cran [Tue, 18 Dec 2012 23:07:47 +0000 (23:07 +0000)]
Fix $(CC) override: use system compiler except on HP-UX and Solaris.

$(CC) was being overridden, preventing the system compiler from
being used. This caused a problem on FreeBSD systems where gcc has
been removed.  Since fio requires gcc on HP-UX and Solaris, always
set $(CC) on those platforms.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.12.1 fio-2.0.12.1
Jens Axboe [Tue, 18 Dec 2012 18:55:12 +0000 (19:55 +0100)]
Fio 2.0.12.1

Fix issue with 32-bit x86 build.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFixup CPUID for 32-bit x86
Jens Axboe [Tue, 18 Dec 2012 18:54:40 +0000 (19:54 +0100)]
Fixup CPUID for 32-bit x86

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFio 2.0.12 fio-2.0.12
Jens Axboe [Tue, 18 Dec 2012 18:32:20 +0000 (19:32 +0100)]
Fio 2.0.12

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agowindowsaio: fix typo
Jens Axboe [Tue, 18 Dec 2012 10:53:46 +0000 (11:53 +0100)]
windowsaio: fix typo

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: make last_cycles thread local too
Jens Axboe [Tue, 18 Dec 2012 08:41:59 +0000 (09:41 +0100)]
gettime: make last_cycles thread local too

Avoids a lot of cache line bouncing for threads.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: use pthread_{set,get}specific() for TLS
Jens Axboe [Tue, 18 Dec 2012 08:12:27 +0000 (09:12 +0100)]
gettime: use pthread_{set,get}specific() for TLS

__thread doesn't work on OSX at least (thanks...), so resort to
using the pthread functions to maintaining a per-thread clock
check.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: fix race/bug with threads and time keeping
Jens Axboe [Tue, 18 Dec 2012 07:23:44 +0000 (08:23 +0100)]
gettime: fix race/bug with threads and time keeping

Sam writes:

First, the 1 second latency events come in batches and those batches
occur suspiciously close to a usec wrap (0.999999 us -> 1.000000 us).
Second, if you subtract exactly 1 second from these outlier latencies,
the remaining amount is very close to what our instrumented low level
driver records for the IO latency and consistent with the expected
latencies of our SSD.  Similarly, the tv_usec portion of the timeval
structs shows increasing values.  See snippet below.

Format is like    start: <start_time.tv_sec>.<start_time.tv_usec>

latency: 1004657 us, lba: 1111289192, start: 1355776806.995294 issue: 1355776806.995312 complete: 1355776807.999969
latency: 1000494 us, lba: 203093568, start: 1355776806.999456 issue: 1355776806.999475 complete: 1355776807.999969
latency: 1000404 us, lba: 551350992, start: 1355776806.999546 issue: 1355776806.999565 complete: 1355776807.999969
latency: 1000477 us, lba: 449672928, start: 1355776806.999484 issue: 1355776806.999492 complete: 1355776807.999969

All this pointed to the time collection code being buggy.  Reviewing the
code, I spotted this in fio_gettime():

/*
 * If Linux is using the tsc clock on non-synced processors,
 * sometimes time can appear to drift backwards. Fix that up.
 */
if (last_tv_valid) {
if (tp->tv_sec < last_tv.tv_sec)
tp->tv_sec = last_tv.tv_sec;
else if (last_tv.tv_sec == tp->tv_sec &&
 tp->tv_usec < last_tv.tv_usec)
tp->tv_usec = last_tv.tv_usec;
}
last_tv_valid = 1;
memcpy(&last_tv, tp, sizeof(*tp));

This does not appear to be multi-thread safe.  Pre-emption can occur
between either comparison and the subsequent update.  Commenting it out
makes the problem go away (at the expense of being subject to drift).
How about making last_tv & last_tv_valid thread-local?

Reported-by: Sam Bradshaw <sbradshaw@micron.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd --cpuclock-test and clocksource= option help
Jens Axboe [Mon, 17 Dec 2012 13:44:08 +0000 (14:44 +0100)]
Add --cpuclock-test and clocksource= option help

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: include per-cpu clock calibration in cpu clock test
Jens Axboe [Mon, 17 Dec 2012 13:29:28 +0000 (14:29 +0100)]
gettime: include per-cpu clock calibration in cpu clock test

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: locking fix and debug check for identical sequence
Jens Axboe [Mon, 17 Dec 2012 13:23:47 +0000 (14:23 +0100)]
gettime: locking fix and debug check for identical sequence

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agocpu clock: add independent test for monotonic/sane TSC
Jens Axboe [Mon, 17 Dec 2012 11:03:29 +0000 (12:03 +0100)]
cpu clock: add independent test for monotonic/sane TSC

Fire up all threads on the system, checking the clock 100,000
times on each. Then collect and compare results, to ensure we
never have a clock that goes backwards.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoclock: ensure that we re-init if the clocksource changes from the default
Jens Axboe [Fri, 14 Dec 2012 19:37:06 +0000 (20:37 +0100)]
clock: ensure that we re-init if the clocksource changes from the default

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agobinject: fixup ->file_data breakage
Jens Axboe [Wed, 12 Dec 2012 08:07:00 +0000 (09:07 +0100)]
binject: fixup ->file_data breakage

Not sure how that was done, but I missed binject in doing
the ->file_data/->file_pos unification.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agofile: unify ->file_data and ->file_pos
Jens Axboe [Wed, 12 Dec 2012 07:34:42 +0000 (08:34 +0100)]
file: unify ->file_data and ->file_pos

The only real use case of ->file_pos was by the sync engine to
avoid an lseek() if the offset was already correct. The e4defrag,
falloc, and fusion-aw engine also set ->file_pos, but that looks
like a case of copy-paste, since they don't actually use it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agofile hash: don't close file fd on lookup/add race
Jens Axboe [Wed, 12 Dec 2012 07:16:27 +0000 (08:16 +0100)]
file hash: don't close file fd on lookup/add race

On Linux, udev often has a rule that triggers blkid to read data off
the device, if the device has been opened for writing. This triggers
when the device is closed.

Fio has an internal file hash for lookup of files. When threads start,
it can happen that one thread does a hash lookup without finding the
file, but when it has opened the file and attempts to insert it into
the hash, another thread has already completed that operation. When
that race happens, fio closes the file handle and does the lookup
again. That then triggers blkid to read pages off the device. As data
in the cache is invalidated on open of the device, we know have page
cache entries for the device again.

This is a problem for unbuffered workloads, where the existance of
page cache pages slows it down due to having to check for invalidation.
The user observed problem is that fio exhibits bi-modal performance
results, depending on whether the file hash race was hit during setup
or not.

Fix this by NOT closing the file if we hit this race, but instead wait
until the file is closed after the run.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Mon, 10 Dec 2012 09:16:45 +0000 (10:16 +0100)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

7 years agocpu clock: round up when dividing by samples
Jens Axboe [Mon, 10 Dec 2012 09:15:59 +0000 (10:15 +0100)]
cpu clock: round up when dividing by samples

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoDefine TSC arch_init() for PPC and IA64
Jens Axboe [Mon, 10 Dec 2012 07:36:35 +0000 (08:36 +0100)]
Define TSC arch_init() for PPC and IA64

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: fix CPU calibration reported mean
Jens Axboe [Mon, 10 Dec 2012 07:29:56 +0000 (08:29 +0100)]
gettime: fix CPU calibration reported mean

Bug introduced in fa80feae. It only affected the reported mean
if --debug=time was used, cosmetic issue.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agogettime: calibration rounding error
Jens Axboe [Mon, 10 Dec 2012 07:29:03 +0000 (08:29 +0100)]
gettime: calibration rounding error

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoIncrease CPU clock calibration accuracy
Jens Axboe [Mon, 10 Dec 2012 07:07:14 +0000 (08:07 +0100)]
Increase CPU clock calibration accuracy

Lets throw some more loops at it, it reduces the noise.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoAdd check for invariant TSC on x86 and use TSC is default clock if reliable
Jens Axboe [Sun, 9 Dec 2012 19:29:00 +0000 (20:29 +0100)]
Add check for invariant TSC on x86 and use TSC is default clock if reliable

TSC is by far the fastest clock we can use. Check the CPUID bits for
whether it is both constant rate AND synced across cores. If it is,
we can use it as our default clock source.

Fio will default to this clock source on x86 if no other clock source
is specifically given with clocksource= in the job file.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agowindowsaio: create a single completion port during init, associate files during open.
Bruce Cran [Thu, 6 Dec 2012 19:59:42 +0000 (19:59 +0000)]
windowsaio: create a single completion port during init, associate files during open.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoDocument the ioengine=net pingpong= option
Jens Axboe [Thu, 6 Dec 2012 19:53:57 +0000 (20:53 +0100)]
Document the ioengine=net pingpong= option

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix windows out-of-memory handling
Bruce Cran [Thu, 6 Dec 2012 19:33:14 +0000 (20:33 +0100)]
Fix windows out-of-memory handling

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFree io_u related structures before killing IO engine
Bruce Cran [Thu, 6 Dec 2012 19:32:40 +0000 (20:32 +0100)]
Free io_u related structures before killing IO engine

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agowindowsaio: initialize and map windowsaio IO structure to io_u
Jens Axboe [Thu, 6 Dec 2012 19:30:38 +0000 (20:30 +0100)]
windowsaio: initialize and map windowsaio IO structure to io_u

Instead of searching for a free entry everytime we enter
queue, do this when the io_u is setup. It's cleaner and
faster.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoparser: always match the correct option length for posval options
Jens Axboe [Thu, 6 Dec 2012 16:34:57 +0000 (17:34 +0100)]
parser: always match the correct option length for posval options

Right now we match rw=randr as rw=randrw, since we use the length
of the passed in option. Use a correct max of the ival and string
passed in, so that we fail if a full match isn't present.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix man page indentation
Jens Axboe [Wed, 5 Dec 2012 20:15:01 +0000 (21:15 +0100)]
Fix man page indentation

Was broken after the io engines.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agolfsr: add HOWTO and man page documentation
Jens Axboe [Wed, 5 Dec 2012 09:28:08 +0000 (10:28 +0100)]
lfsr: add HOWTO and man page documentation

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agolfsr: ensure that the cycle follows the randrepeat= setting
Jens Axboe [Tue, 4 Dec 2012 12:12:25 +0000 (13:12 +0100)]
lfsr: ensure that the cycle follows the randrepeat= setting

Use the regular block offset seed to "seed" the lfsr generator,
so that it obeys randrepeat=0/1 as well.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoName the various random offsets we use
Jens Axboe [Tue, 4 Dec 2012 12:10:29 +0000 (13:10 +0100)]
Name the various random offsets we use

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoRevert "Change preferred default clocksource to gettimeofday()"
Jens Axboe [Sat, 1 Dec 2012 14:02:11 +0000 (15:02 +0100)]
Revert "Change preferred default clocksource to gettimeofday()"

This reverts commit f7161017b75f5d3526147d42d3b583973922c444.

clock_gettime() is monotonic, so a superior clock source even
if it is slower.

7 years agoengines/net.c: fix compilation error due to missing signal.h include
Steven Noonan [Sat, 1 Dec 2012 08:16:55 +0000 (08:16 +0000)]
engines/net.c: fix compilation error due to missing signal.h include

Corrects this build failure:

engines/net.c: In function 'fio_netio_terminate':
engines/net.c:900:2: warning: implicit declaration of function 'kill' [-Wimplicit-function-declaration]
engines/net.c:900:16: error: 'SIGUSR2' undeclared (first use in this function)
engines/net.c:900:16: note: each undeclared identifier is reported only once for each function it appears in

Signed-off-by: Steven Noonan <snoonan@amazon.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoWire up SIGUSR2 to kill blocking threads
Jens Axboe [Fri, 30 Nov 2012 20:46:06 +0000 (21:46 +0100)]
Wire up SIGUSR2 to kill blocking threads

Get rid of io engine ops SIGTERM flag, it didn't really
work.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoChange preferred default clocksource to gettimeofday()
Jens Axboe [Fri, 30 Nov 2012 18:43:31 +0000 (19:43 +0100)]
Change preferred default clocksource to gettimeofday()

It's faster than clock_gettime()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoCache layout improvements
Jens Axboe [Fri, 30 Nov 2012 18:37:46 +0000 (19:37 +0100)]
Cache layout improvements

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agonet: fix receiver start time
Jens Axboe [Fri, 30 Nov 2012 15:22:31 +0000 (16:22 +0100)]
net: fix receiver start time

Reset epoch/start time to when we get a connection.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agonet: add basic ping/pong type workload support
Jens Axboe [Fri, 30 Nov 2012 08:59:20 +0000 (09:59 +0100)]
net: add basic ping/pong type workload support

If you use ioengine=net and then set pingpong=1, fio will operate
in a ping pong type fashion. When the reader receives a package
from the writer, it will return the package to the writer again.
The writer, in turn, will wait for this package before writing
a new package. This makes fio useful for roundrobin measurements
on networks.

The submission latency then measures how long it took to submit
a packet to the local system, while the completion latency is how
long it takes for the package to be received and returned by
the other end.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoverify: treat as failure if given verify type is different from media
Jens Axboe [Fri, 30 Nov 2012 08:39:02 +0000 (09:39 +0100)]
verify: treat as failure if given verify type is different from media

When fio verifies right now, it'll look at the on media stored type
and verify that type. This means that if verify=foo set and we
find older blocks that are written with verify=bar, we will
verify those against bar.

Change this so that if a specific verify type is given, that type
is ALWAYS used for verification. If none is given but asked to
verify, we retain the old behaviour of just verifying based on
the type given in the on-media block.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agonet: sent udp open messages
Jens Axboe [Fri, 30 Nov 2012 07:27:46 +0000 (08:27 +0100)]
net: sent udp open messages

The receiver goes directly into recvfrom() when it starts up,
which greatly skews the first submission latency (since it's
measure from when it enters recvfrom() and until we finally
get the first packet from the client). Add a file open
handshake to avoid this issue.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoMerge branch 'rand-map'
Jens Axboe [Thu, 29 Nov 2012 20:56:06 +0000 (21:56 +0100)]
Merge branch 'rand-map'

7 years agosolaris: enable fadvise()
Jens Axboe [Thu, 29 Nov 2012 20:48:55 +0000 (21:48 +0100)]
solaris: enable fadvise()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoFix broken 'norandommap'
Jens Axboe [Thu, 29 Nov 2012 20:37:11 +0000 (21:37 +0100)]
Fix broken 'norandommap'

commit 8055e41d broke the option, always falling through to
the failure case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoparse: fix wrong "might be used unitialized" warning on some compilers
Jens Axboe [Thu, 29 Nov 2012 20:33:03 +0000 (21:33 +0100)]
parse: fix wrong "might be used unitialized" warning on some compilers

GCC 3.4.3 on Solaris, I'm looking at you. Apparently:

if (a || b) {
if (a)
c = foo;
else if (b)
c = bar;

*c = foobar;
}

is too hard to figure out.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 years agoWire up _weak posix_fadvise()
Jens Axboe [Thu, 29 Nov 2012 20:27:20 +0000 (21:27 +0100)]
Wire up _weak posix_fadvise()

The commit adding Android support included a botched
variant of posix_fadvise().

Signed-off-by: Jens Axboe <axboe@kernel.dk>