Jens Axboe [Tue, 11 Feb 2014 15:31:13 +0000 (08:31 -0700)]
Fix issue with openfiles= and file sizing
Fill the given size as well as we can, given the size and nr
of files.
Fix an issue where we don't properly honor how many files to
keep open at any point in time.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 10 Feb 2014 20:57:09 +0000 (13:57 -0700)]
Fix issue with td->mutex being used-after-free
Depending on how the OS schedules the threads, it's not necessarily
safe to immediately free a mutex after we have been woken up.
Remove the td->mutex freeing to normal cleanup after the job is done.
Reported-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 8 Feb 2014 03:54:39 +0000 (20:54 -0700)]
crc: add option to list possible crc types
=help or =list will show them.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 8 Feb 2014 03:52:32 +0000 (20:52 -0700)]
crc: ensure we properly match test name
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 8 Feb 2014 03:48:29 +0000 (20:48 -0700)]
README: update for crctest
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 7 Feb 2014 21:39:33 +0000 (14:39 -0700)]
Add support for testing checksumming speed
fio --crctest will test all of them, --crctest=md5,crc32c will
test md5 and crc32c, for example.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 7 Feb 2014 17:57:13 +0000 (10:57 -0700)]
Prioritize lockfile= option
This needs to be run before adding files, or we'll potentially
make a mess of things.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 7 Feb 2014 17:56:15 +0000 (10:56 -0700)]
Fix crash with file locking and dup'ed files
Ensure that we properly inherit the file locks when
duplicating files.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 6 Feb 2014 19:17:37 +0000 (12:17 -0700)]
verify: always log IO in the order they are issued
We currently log verify_backlog verifies when they complete,
which means the sequence of verify and issue might be different.
Change this to log in one spot, prior to issue, and track the
completion state of the logged unit instead. This unifies the
handling of verifies.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 5 Feb 2014 20:15:22 +0000 (13:15 -0700)]
More precise fix for verify_backlog verification failure
Commit
c2703bf3 could be a bit more precise - we only need to
add to the byte count if verify_backlog is enabled, since that
is the case where do_io() will run both the IO and verify IO.
Signed-off-by: Jens Axboe <axboe@fb.com>
Puthikorn Voravootivat [Wed, 5 Feb 2014 18:28:15 +0000 (10:28 -0800)]
Fix verify_backlog start verification before finish writing
Due to recent fix for checking rand_seed in verify phase. In write phase,
the IO transaction will be logged before completing the actual data write.
If verify_backlog is enabled, verification may start before all data has
been written. This patch disable that fix if the verify_backlog is enabled.
Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 5 Feb 2014 19:36:02 +0000 (12:36 -0700)]
Fix for prematurely stopping on verify
Commit
20876c53 added a total byte check, but forgot to
account for potential verification bytes. Fix that.
Signed-off-by: Jens Axboe <axboe@fb.com>
Puthikorn Voravootivat [Wed, 5 Feb 2014 17:17:11 +0000 (10:17 -0700)]
Fix rand_seed mismatches in verify phase
In verify phase, the rand_seed generated on replay does not match
the written rand_seed.
Multiple problems are causing this:
1. In verify phase fio does not set io_u->rand_seed to compare with
hdr->rand_seed
2. In randrw scenario, fio log is stored in red-black tree in "sorted by LBA"
order. Thus, it is imposible to replay the written order, or rather
generate the seeds again in the same order.
3. In write phase, the code currently will generate rand_seed, write data
and log rand_seed. When queuedepth > 1, it's possible the writes complete
in a different order than rand_seed was generated. Thus when replaying
the log, the generated rand_seed might not match what was written.
4. verify_backlog option will start verification before all the data has been
written and it make rand_seed replay code broken with current design.
Proposed fixes:
1. Use of existing verify_state to generate verify header.
(and assumes this was the original intention of verify_state). And also
adds code to replay rand_seed in verify phase.
2. If verifysort option is not enabled, store the write log in a list instead
of the red-black tree. Otherwise, don't attempt to verify the rand_seed
in the header.
3. In write phase, generate rand_seed, log rand_seed, write data. I.e. log
IO transactions in the order generated, not completed.
4. Don't verify rand_seed when verify_backlog is enabled.
Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Stefan Hajnoczi [Tue, 4 Feb 2014 13:27:11 +0000 (14:27 +0100)]
init: log error on missing --output-format argument
strcmp(optarg, "foo") will crash if --output-format was given without an
argument. Log an error and exit properly instead of crashing.
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
Stefan Hajnoczi [Tue, 4 Feb 2014 13:27:10 +0000 (14:27 +0100)]
init: --minimal does not take an optional argument
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 29 Jan 2014 23:39:04 +0000 (16:39 -0700)]
init: update email address
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bruce Cran [Wed, 29 Jan 2014 00:34:08 +0000 (17:34 -0700)]
Enable IPv6 in the Windows build
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Grant Grundler [Tue, 28 Jan 2014 22:11:23 +0000 (15:11 -0700)]
fio: consolidate rand_seed to uint64_t
csscope showed rand_seed was defined 4 times - each a different type.
While they are used differently, the places where they overlap
should reference the same type. This patch makes three of them the same
as suggested by Jens Axboe.
Signed-off-by: Grant Grundler <grundler@chromium.org>
Updated by me to change rand_seed option to unsigned long long
and changes related to that.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bruce Cran [Tue, 28 Jan 2014 19:56:53 +0000 (12:56 -0700)]
Update Windows build for new threading library
MinGW now uses libwinpthread-1.dll instead of pthreadGC2.dll.
To simplify things, create a static executable so users can run fio
from anywhere without needing to copy the dll.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bruce Cran [Tue, 28 Jan 2014 19:54:17 +0000 (12:54 -0700)]
Fix Windows headers for IPv6
MinGW includes all the necessary IPv6 definitions, so include ws2tcpip.h
and remove the custom definitions.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 25 Jan 2014 02:59:15 +0000 (18:59 -0800)]
engine: protect net engine IPv6 support with configure check
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 25 Jan 2014 02:56:56 +0000 (18:56 -0800)]
client/server: don't reset ipv6 expectations
fio_server_parse_host() no longer falls back to ipv4, so just
pass in the actual value, not a reference.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 25 Jan 2014 02:56:34 +0000 (18:56 -0800)]
configure: add test for whether required IPv6 helpers
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 25 Jan 2014 02:52:16 +0000 (18:52 -0800)]
server: fix wrong error return on host lookup failure
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 24 Jan 2014 20:15:07 +0000 (12:15 -0800)]
Add verify_only to man page
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Juan Casse [Tue, 17 Sep 2013 21:06:14 +0000 (14:06 -0700)]
Adds check for rand_seed during verify phase.
Improve data integrity checking of header (meta) data.
verify_header() will now return an additional error:
"verify: bad header rand seed ..."
The addition of the check of rand_seed helps detect stale data from
previous fio runs.
This patch also disambiguates the different data mismatches by returning
different error codes from verify_header().
Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Juan Casse [Tue, 17 Sep 2013 21:06:13 +0000 (14:06 -0700)]
Adds verify_only option
When this option is set, a dry run (no actual io is performed) of the
workload will be done in order to compute the numberio for each block
header without overwriting the data on disk. Then, do_verify() will be
effectively verifying data that was written in a previous fio run.
In the case that "loops" is set to more than 1, do_verify() will delay
the verification of numberio to the last iteration when the same
numberio state that would have been written to disk in a previous
fio run has been reached.
Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Juan Casse [Tue, 17 Sep 2013 21:06:12 +0000 (14:06 -0700)]
Adds check for numberio during verify phase.
Currently, fio checks the block offset number in a block's header during
the verify phase.
We add a check for the io number (numberio) to detect stale blocks. This
check is performed only on workloads that write data, as those workloads
know what numberio was written to each block.
td->io_issues[ddir] = 0; was removed so that numberio does not get reset
at each iteration; we want numberio to keep incrementing to reflect
how many times the same data was written.
Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Fixed typo.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 24 Jan 2014 15:36:43 +0000 (07:36 -0800)]
engine: more checks on IPv6 and multicast
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 24 Jan 2014 04:42:06 +0000 (20:42 -0800)]
server: make the connect code fully IPv6
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 24 Jan 2014 04:41:33 +0000 (20:41 -0800)]
engine: pass in right protocol family on host lookup
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 24 Jan 2014 04:19:17 +0000 (20:19 -0800)]
engine: IPv6 net engine updates
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 24 Jan 2014 00:49:37 +0000 (16:49 -0800)]
engine: basic support for IPv6 in the net IO engine
Adds 'tcpv6' and 'udpv6' protocol entries.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Jan 2014 00:27:32 +0000 (16:27 -0800)]
Fix potential oops in file unlocking at free time
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 15 Jan 2014 16:41:31 +0000 (09:41 -0700)]
server: convert buffer pattern settings across client/server
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 14 Jan 2014 22:35:58 +0000 (15:35 -0700)]
Add option for specifically setting buffer contents
Fio can use zeroes, slightly scrambled data, full random data,
or specifically compressible data. With this option, the user
can now fully control the pattern written, similarly to how
verify_pattern works for verify=meta.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 11 Jan 2014 03:57:01 +0000 (20:57 -0700)]
Add option to manually seed the random generators
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 30 Dec 2013 02:42:06 +0000 (19:42 -0700)]
Update README to reflect that fio now runs on OpenBSD
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 30 Dec 2013 02:22:46 +0000 (19:22 -0700)]
libfio: add missing OS's to fio_os_strings[]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
John [Mon, 30 Dec 2013 02:20:35 +0000 (19:20 -0700)]
Add OpenBSD support
I managed to make a clean compile and a few test runs using the
attached patches.
These work on OpenBSD 5.4 at least. I used the 2.1.4 release of
fio as a basis.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Castor Fu [Fri, 20 Dec 2013 07:00:46 +0000 (23:00 -0800)]
Make configure tests pass pickier compiler options.
Signed-off-by: Castor Fu <castor@alumni.caltech.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 9 Dec 2013 19:38:53 +0000 (12:38 -0700)]
Add options to have fio latency profile a device
This adds three new options:
- latency_target. This defines a specific latency target, in usec.
- latency_window. This defines the period over which fio samples.
- latency_percentile. This defines the percentage of IOs that must
meet the criteria specified by latency_target/latency_window.
With these options set, fio will run the described workload and
vary the queue depth between 1 and iodepth= to find the best
performing spot that meets the criteria specified by the three
options.
A sample job file is also added to demonstrate how to use this.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 6 Dec 2013 19:02:10 +0000 (12:02 -0700)]
Calloc() cleanup
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 27 Nov 2013 03:40:00 +0000 (20:40 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Wed, 27 Nov 2013 03:39:35 +0000 (20:39 -0700)]
init: escape description option string
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 27 Nov 2013 03:19:59 +0000 (20:19 -0700)]
init: allow FIO_OPT_STR_SET options to have an optional argument
Makes them work like a bool.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bruce Cran [Thu, 7 Nov 2013 14:04:00 +0000 (14:04 +0000)]
Windows: add CONFIG_TLS_THREAD=y and swap pthreadGC2 to libwinpthread-1.
Since GCC 4.4, MinGW has supported TLS via the __thread keyword, so
generate config-host.mak with CONFIG_TLS_THREAD=y.
MinGW now uses a new threading library libwinpthread-1.dll: install it
instead of pthreadGC2.dll.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 21 Nov 2013 18:13:12 +0000 (11:13 -0700)]
Error out gracefully if we don't find the replay device for log replay
It would segfault before because no files got added to replay
against. Handle this more gracefully and ensure that fio exits
with a non-zero exit status.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 21 Nov 2013 16:55:49 +0000 (09:55 -0700)]
blktrace: add support for non-native endian format
The blktrace format is stored in the native endianness of
the machine it is run on. So to reply traces on a machine
with a different endianness, we need to swap the trace
fields. Detect and do this automatically.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 16 Nov 2013 18:15:12 +0000 (11:15 -0700)]
Fio 2.1.4
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Chris Mason [Fri, 15 Nov 2013 22:52:58 +0000 (15:52 -0700)]
Add support for O_ATOMIC
O_ATOMIC makes writes atomic, meaning that they are fully stable
on media (in the event of a power cut) when acknowledged by the
device and OS.
This only truly works on Linux with the pending patches to
add O_ATOMIC.
Updated by Jens to:
- Add man page and HOWTO description of the option
- Make O_ATOMIC imply O_DIRECT, so that it actually works if you
don't set O_DIRECT manually.
- Add the option to the conversion list so it works for
client/server.
- Error handling so that if atomic=1 is set and the OS does not
support it, error out instead of just pretending it works.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 8 Nov 2013 17:57:08 +0000 (10:57 -0700)]
Fix access-outside-array of o->rwmix[]
gcc 4.9 rightfully complains about it, though in a very obtuse way:
CC cconv.o
In file included from thread_options.h:5:0,
from cconv.c:3:
cconv.c: In function 'convert_thread_options_to_cpu':
os/os.h:197:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
__le32_to_cpu(*__val); \
^
os/os.h:176:28: note: in definition of macro '__le32_to_cpu'
#define __le32_to_cpu(x) (x)
^
cconv.c:78:17: note: in expansion of macro 'le32_to_cpu'
o->rwmix[i] = le32_to_cpu(top->rwmix[i]);
^
cconv.c:63:2: note: containing loop
for (i = 0; i < DDIR_RWDIR_CNT; i++) {
^
In file included from thread_options.h:5:0,
from cconv.c:3:
cconv.c: In function 'convert_thread_options_to_net':
os/os.h:209:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
__cpu_to_le32(*__val); \
^
os/os.h:179:28: note: in definition of macro '__cpu_to_le32'
#define __cpu_to_le32(x) (x)
^
cconv.c:372:19: note: in expansion of macro 'cpu_to_le32'
top->rwmix[i] = cpu_to_le32(o->rwmix[i]);
^
cconv.c:352:2: note: containing loop
for (i = 0; i < DDIR_RWDIR_CNT; i++) {
^
In file included from thread_options.h:5:0,
from cconv.c:3:
cconv.c: In function 'fio_test_cconv':
os/os.h:209:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
__cpu_to_le32(*__val); \
^
os/os.h:179:28: note: in definition of macro '__cpu_to_le32'
#define __cpu_to_le32(x) (x)
^
cconv.c:372:19: note: in expansion of macro 'cpu_to_le32'
top->rwmix[i] = cpu_to_le32(o->rwmix[i]);
^
cconv.c:352:2: note: containing loop
for (i = 0; i < DDIR_RWDIR_CNT; i++) {
^
In file included from thread_options.h:5:0,
from cconv.c:3:
os/os.h:197:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
__le32_to_cpu(*__val); \
^
os/os.h:176:28: note: in definition of macro '__le32_to_cpu'
#define __le32_to_cpu(x) (x)
^
cconv.c:78:17: note: in expansion of macro 'le32_to_cpu'
o->rwmix[i] = le32_to_cpu(top->rwmix[i]);
^
cconv.c:63:2: note: containing loop
for (i = 0; i < DDIR_RWDIR_CNT; i++) {
^
Fix it by extending the rwmix indexes to the full rwdir count,
even though it doesn't support trim as part of the mix (yet).
Reported-by: Bruce Cran <bruce@cran.org.uk>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 7 Nov 2013 20:37:09 +0000 (13:37 -0700)]
server: make job/connection lists private
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 6 Nov 2013 22:37:35 +0000 (15:37 -0700)]
solaris: ensure that -D_REENTRANT gets set
Apparently some Solaris' require this for threadsafe
errno.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 6 Nov 2013 21:47:22 +0000 (14:47 -0700)]
posixaio: fix thread problem with using errno
If we fail queueing a read or a write, use aio_error() to
retrieve the right error value. This fixes an issue on
(at least) Solaris where we get EAGAIN due to system
shortage of resources, but treat that as a random type
of error due to using errno.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 6 Nov 2013 18:19:42 +0000 (11:19 -0700)]
server: get rid of compile warning on platforms where pid_t isn't int
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bruce Cran [Mon, 4 Nov 2013 11:59:43 +0000 (11:59 +0000)]
Fix a couple of typos in fio2gnuplot.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bruce Cran [Mon, 4 Nov 2013 11:52:08 +0000 (11:52 +0000)]
Use #!/usr/bin/env to allow for interpreters in different directories.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bruce Cran [Fri, 1 Nov 2013 18:32:40 +0000 (12:32 -0600)]
server: ensure that fio_time_init() is called before option parsing
When run in server mode, jobs are processed before fio_time_init()
is called, which causes pthread_setspecific() to be called before
pthread_key_create(). Work around this by calling it in thread_main()
too.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bruce Cran [Fri, 1 Nov 2013 18:31:03 +0000 (12:31 -0600)]
Implement writev on Windows
The network code depends on writev(), so implement it for Windows.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bruce Cran [Tue, 8 Oct 2013 14:05:27 +0000 (15:05 +0100)]
Update Windows code to fix build breakage
MinGW has been updated: this causes problems due to changes in header
files, including COM stuff being pulled in resulting in 'interface' being
a reserved keyword, and changes to a few POSIX definitions.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Castor Fu [Thu, 31 Oct 2013 17:00:54 +0000 (11:00 -0600)]
configure: add option to disable libnuma usage
Signed-off-by: Castor Fu <castor@alumni.caltech.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Castor Fu [Thu, 31 Oct 2013 17:00:34 +0000 (11:00 -0600)]
Add json output for client/server mode
In client/server mode, this adds support for json mode. Each job's
details are gradually accumulated and upon completion the client
dumps the full json tree. The tree is annotated with the server
host and port.
Signed-off-by: Castor Fu <castor@alumni.caltech.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 31 Oct 2013 16:56:53 +0000 (10:56 -0600)]
Fix bad asm constrants for x86-64 cpuid()
Bruce reports:
It seems there's a bug in the x86_64 version of do_cpuid() that causes
fio to segfault when built with clang at lower optimization levels:
static inline void do_cpuid(unsigned int *eax, unsigned int *ebx,
unsigned int *ecx, unsigned int *edx)
{
asm volatile("cpuid"
: "=a" (*eax), "=b" (*ebx), "=r" (*ecx), "=d" (*edx)
: "0" (*eax), "2" (*ecx)
: "memory");
}
via Tijl Coosemans:
Should be: "=c" (*ecx)
But you can also use the '+' modifier and remove the input operands:
: "+a" (*eax), "=b" (*ebx), "+c" (*ecx), "=d" (*edx)
:
: "memory"
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Hong Zhiguo [Wed, 16 Oct 2013 14:35:12 +0000 (08:35 -0600)]
client: fix free of wrong pointer
Signed-off-by: Hong Zhiguo <zhiguohong@tencent.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 11 Oct 2013 16:27:28 +0000 (10:27 -0600)]
ppc: disable CPU clock until we can detect whether we have it or not
The child segfault test should catch it, however it does not on
AIX at least.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Akash Verma [Fri, 11 Oct 2013 16:24:26 +0000 (10:24 -0600)]
arch-ppc.h: Fix typo in header guard
Signed-off-by: Akash Verma <akashv@google.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Akinobu Mita [Wed, 9 Oct 2013 15:32:34 +0000 (09:32 -0600)]
make ioengine specific command line options work
If the ioengine specific parameter is given on the command line,
unrelated value in thread_data.o will be corrupted unexpectedly.
Specifically, if libaio specific parameter userspace_reap is given
on the command line, thread_data.o.description will be set to 1.
This fixes it by passing the correct pointer of ioengine specific
options to the command line parser.
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 8 Oct 2013 14:49:25 +0000 (08:49 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio
Hong Zhiguo [Tue, 8 Oct 2013 14:48:47 +0000 (08:48 -0600)]
fix crash: free of wrong pointer in bssplit_ddir
I got SIGABRT when I passed "bssplit=4k/10:64k/50:/40" to fio.
It should be "bssplit=4k/10:64k/50:32k/40" but I missed "32k" in
my typing. The bug turns out to be another typo just like mine:)
*** glibc detected *** /viki/lkml/fio/fio: free(): invalid pointer:
0x000000000071fd30 ***
======= Backtrace: =========
/lib/libc.so.6(+0x78bb6)[0x7ffff700cbb6]
/lib/libc.so.6(cfree+0x73)[0x7ffff7013483]
/viki/lkml/fio/fio[0x42c108]
/viki/lkml/fio/fio[0x42c475]
/viki/lkml/fio/fio[0x429194]
/viki/lkml/fio/fio[0x42a888]
/viki/lkml/fio/fio(parse_option+0x101)[0x42ac47]
/viki/lkml/fio/fio(fio_options_parse+0x95)[0x42e5b3]
/viki/lkml/fio/fio(parse_jobs_ini+0x3ae)[0x4103ae]
/viki/lkml/fio/fio(parse_options+0x112)[0x41198a]
/viki/lkml/fio/fio(main+0x3b)[0x460f5b]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7ffff6fb2c4d]
/viki/lkml/fio/fio[0x40a7c9]
Signed-off-by: Hong Zhiguo <zhiguohong@tencent.com>
Tested-by: Hong Zhiguo <zhiguohong@tencent.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Sitsofe Wheeler [Fri, 4 Oct 2013 22:09:42 +0000 (23:09 +0100)]
howto: Fix redundant entries
Reported-by: Konstantin Ruvinsky <Konstantin.Ruvinsky@emc.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Fri, 4 Oct 2013 21:07:23 +0000 (22:07 +0100)]
man page: Bump up release date
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Fri, 4 Oct 2013 21:07:23 +0000 (22:07 +0100)]
man page: fix broken formatting before OUTPUT section
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Fri, 4 Oct 2013 21:07:23 +0000 (22:07 +0100)]
man page: manpagify things that look like syscalls.
Use the bold style with section number to mark syscalls.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Fri, 4 Oct 2013 21:07:23 +0000 (22:07 +0100)]
man page: Fix typos
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Fri, 4 Oct 2013 18:33:11 +0000 (12:33 -0600)]
man page: correct typoe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 4 Oct 2013 15:48:15 +0000 (09:48 -0600)]
Merge branch 'manpage' of https://github.com/sitsofe/fio
Jens Axboe [Mon, 30 Sep 2013 18:17:34 +0000 (12:17 -0600)]
log: disable log, if realloc fails
Right now we just segfault, if realloc() returns NULL. Handle
this a bit more gracefully, by just disabling logging if that
happens. Fio will also print an error for that case.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Sitsofe Wheeler [Fri, 27 Sep 2013 12:17:59 +0000 (13:17 +0100)]
Update man page
Many of fio's options are not described in the man page. Copy the short
text to the man page for those commands listed by:
./fio --cmdhelp=all | sed 's/ *:.*$//' | sed 's/ \+//' \
| while read line; do grep "\b$line\b" -q fio.1; if [[ $? -eq 1 ]]; then \
echo $line; fi; done
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Fri, 27 Sep 2013 12:17:59 +0000 (13:17 +0100)]
Update man page
Fix verifysort typo, broken formatting after roundrobin and tidy up some
grammar.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Anatol Pomozov [Thu, 26 Sep 2013 23:31:34 +0000 (16:31 -0700)]
Fix misspellings using 'codespell' tool
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 24 Sep 2013 14:42:24 +0000 (08:42 -0600)]
Fio 2.1.3
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Juan Casse [Fri, 20 Sep 2013 15:20:52 +0000 (09:20 -0600)]
Fixes bug: stale LAST_POS(f) is not being reset
Problem:
When running fio with ioengine=sync, LAST_POS(f) state is not
reset after the file is closed. This causes workloads with
readwrite=randread and loops > 1 to fail verification if the
state of LAST_POS(f) at the beginning of the next loop is the
same as the io_u->offset. If that is the case, lseek in not
invoked, but the file position is at 0 and not at io_u->offset.
Proposed Solution:
Other ioengines, such as binject and fusion-aw, set f->engine_data
to 0 when closing the file. The sync ioengin uses f->engine_data to
store the LAST_POS(f) state. The proposed solution is to set
f->engine_data = 0 when closing a file in generic_close_file().
Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Juan Casse [Mon, 16 Sep 2013 16:22:10 +0000 (09:22 -0700)]
Add condition to stop issuing io in do_io().
Problem:
When specifying iodepth > 1 while doing random async io
without a random map (and using the default random generator), the
main while loop in do_io() will issue extra io in the amount of io
allowed to accumulate in the queue. For example, a job with size=24k
and bs=8k will perform 4 io instead of the expected 3.
Reason:
The above behavior occurs because the while loop in do_io()
will continue until the amount of "completed io" >= size, but io
in the queue are not accounted for because they have not been
completed by io_u_queued_complete().
Exceptions:
The above behavior does not manifest itself when:
using random_generator=lfsr bacause lfsr knows when to stop
generating, and get_io_u() inside the while loop returns NULL,
breaking out of the loop.
using random map because random map ensures that all blocks
are hit, and after they are all hit, no more are generated.
Proposed Solution:
Stop while loop based on bytes issued instead of bytes completed.
Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 11 Sep 2013 14:23:45 +0000 (08:23 -0600)]
debug: ensure that __dprint() is also logged over the network
We force local logging now, but that's not terribly useful
if you can't see the output from the backend. Make it follow
the usual rules of logging over the network if a backend, to
syslog, or locally.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 9 Sep 2013 21:16:28 +0000 (15:16 -0600)]
Fix broken requeue of io_us
Suresh reports:
-----
Does a requeued io_u work get submitted again through __get_io_u() ? If
so, the io_u item is taken out of requeue list & then the io_u->file is
set to NULL. We don't make a distinction about item being already setup
Vs a new io_u in __get_io_u(). In get_io_u(), when an io_u may come out
of requeue list, we depend on io_u->file to be non NULL to detect the
work as old pending one.
Am I missing something here or is this a bug?
1274 io_u = __get_io_u(td);
1275 if (!io_u) {
1276 dprint(FD_IO, "__get_io_u failed\n");
1277 return NULL;
1278 }
1279
1280 if (check_get_verify(td, io_u))
1281 goto out;
1282 if (check_get_trim(td, io_u))
1283 goto out;
1284
1285 /*
1286 * from a requeue, io_u already setup
1287 */
1288 if (io_u->file)
1289 goto out;
1290
"io_u.c" line 1286 of 1667 --77%-- col 3-10
1130 struct io_u *__get_io_u(struct thread_data *td)
1131 {
1132 struct io_u *io_u;
1133
1134 td_io_u_lock(td);
1135
1136 again:
1137 if (!io_u_rempty(&td->io_u_requeues))
1138 io_u = io_u_rpop(&td->io_u_requeues);
1139 else if (!io_u_qempty(&td->io_u_freelist))
1140 io_u = io_u_qpop(&td->io_u_freelist);
1141
1142 if (io_u) {
1143 io_u->buflen = 0;
1144 io_u->resid = 0;
1145 io_u->file = NULL;
1146 io_u->end_io = NULL;
1147 }
"io_u.c" line 1130 of 1667 --67%-- col 1
-----
We want a retry of a requeue to be identical, not some
new chunk (and drop the latter one on the floor). This got broken with
the introduction of the rings instead of lists, where a single if
section was split into two.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 6 Sep 2013 16:07:37 +0000 (10:07 -0600)]
Catch more places where the callback should just return on parse/check
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 6 Sep 2013 15:59:56 +0000 (09:59 -0600)]
Fix problem with --showcmd and callbacks that verify
David reports:
I'm using "fio --showcmd file.job" to convert a script to plain
commandline arguments and this fails if the directory specified in job
file does not exist. While this has to be an error if the job is being
executed, it should not be in context of --showcmd.
To reproduce:
$ cat job.fio
[global]
directory=/x
$ fio --showcmd job.fio
fio: /x is not a directory
fio: failed parsing directory=/x
fio: job global dropped
Expected output:
fio --directory=/x
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Grant Grundler [Wed, 4 Sep 2013 20:12:09 +0000 (14:12 -0600)]
Remove duplicate LDFLAGS on -o lines
Only need to list LDFLAGS once on each line.
Signed-off-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Grant Grundler [Wed, 4 Sep 2013 14:26:52 +0000 (08:26 -0600)]
Rework descriptions for mailing list, options, job files
In general, make the existing document less ambiguous:
o Use consistent formatting/layout when describing options
o always refer to options with "--" prefix.
o reduce use of pronouns: "its" and "this" are often ambigous.
o remove use of "you" to describe user actions/choices
o add reference to examples/ directory
Signed-off-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 30 Aug 2013 16:13:36 +0000 (10:13 -0600)]
client: fix compile warning on platforms where pfds->fd is not an int
It's apparently a long on some platforms.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Alex Pyrgiotis [Tue, 27 Aug 2013 16:58:34 +0000 (10:58 -0600)]
LFSR: Do not ignore returning the seed
Fix a situation where we would spin prematurely and would hop over the
seed value.
Also, add some more comments to the code.
Signed-off-by: Alex Pyrgiotis <apyrgio@grnet.gr>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Martin Steigerwald [Tue, 27 Aug 2013 15:33:35 +0000 (09:33 -0600)]
Fix spelling error in fio man page
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Aug 2013 20:31:06 +0000 (14:31 -0600)]
verify: fix problem with hole punching on newer Linux kernels
Jeff Moyer writes:
Our QE team noticed fio failures on recent kernels. I simplified the
job file to the following:
[global]
name=fio-mmap
rw=write
bs=4K
direct=1
end_fsync=1
verify=crc32
[file3]
size=100M
ioengine=mmap
mem=malloc
direct=1
After fio completes (and returns verify errors), the file is completely
full of zeroes.
Here is what the verify logic is doing:
static void do_verify(struct thread_data *td, uint64_t verify_bytes)
{
...
/*
* sync io first and invalidate cache, to make sure we really
* read from disk.
*/
for_each_file(td, f, i) {
if (!fio_file_open(f))
continue;
if (fio_io_sync(td, f))
break;
if (file_invalidate_cache(td, f)) <--------
break;
}
That invalidate cache call looks like so:
static int __file_invalidate_cache(struct thread_data *td, struct fio_file *f,
unsigned long long off,
unsigned long long len)
{
...
/*
* FIXME: add blockdev flushing too
*/
if (f->mmap_ptr) {
ret = posix_madvise(f->mmap_ptr, f->mmap_sz, POSIX_MADV_DONTNEED);
(void) posix_madvise(f->mmap_ptr, f->mmap_sz, FIO_MADV_FREE); <-------
FIO_MADV_FREE can be defined as MADV_REMOVE, which will actually punch a
hole in the file (a hole the size of the entire file, btw). Of course,
unallocated blocks are returned as zeroes by the file system, so that
explains that!
---
For now, only do the MADV_FREE on non-files to avoid punching holes.
Further investigation is needed on the blockdev side, but at least
this should cure the immediate problem.
Reported-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Aug 2013 18:53:44 +0000 (12:53 -0600)]
Add number_ios= option
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Aug 2013 15:17:09 +0000 (09:17 -0600)]
powerpc: disable use of ATBU clock
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Erwan Velu [Thu, 8 Aug 2013 08:59:46 +0000 (10:59 +0200)]
genfio: Don't consider only /dev/ disk device
Prior this patch, genfio was considering the disks __NOT__ having the "/dev"
name in it.
That's a serious limitation as
- we can't test regular files
- that doesn't works for complex /dev/ naming like /dev/disk/by-id/...
This patch removes this requirement and let user giving the complete
path.
Erwan Velu [Wed, 7 Aug 2013 14:02:25 +0000 (16:02 +0200)]
fio2gnuplot: Adding manpage
As requested by users, let's add a manpage to make things easier to
understand.
I'm providing here the raw file used to generate the manpage. It can be
converted to a real manpage by using txt2man.
I'm also providing the manpage file to avoid dependencies at build time,
anyway the makefile target can be easily used to rebuild it if
necessary.
Erwan Velu [Wed, 7 Aug 2013 13:32:02 +0000 (15:32 +0200)]
fio2gnuplot: Removing .py extension
It's pretty boring having the .py extension to the tool.
Let's remove it !
Erwan Velu [Wed, 7 Aug 2013 12:32:50 +0000 (14:32 +0200)]
fio2gnuplot: Fixing help message
Adding missing features & fix indent