fio.git
5 years agoFio 2.1.4 fio-2.1.4
Jens Axboe [Sat, 16 Nov 2013 18:15:12 +0000 (11:15 -0700)]
Fio 2.1.4

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoAdd support for O_ATOMIC
Chris Mason [Fri, 15 Nov 2013 22:52:58 +0000 (15:52 -0700)]
Add support for O_ATOMIC

O_ATOMIC makes writes atomic, meaning that they are fully stable
on media (in the event of a power cut) when acknowledged by the
device and OS.

This only truly works on Linux with the pending patches to
add O_ATOMIC.

Updated by Jens to:

- Add man page and HOWTO description of the option
- Make O_ATOMIC imply O_DIRECT, so that it actually works if you
  don't set O_DIRECT manually.
- Add the option to the conversion list so it works for
  client/server.
- Error handling so that if atomic=1 is set and the OS does not
  support it, error out instead of just pretending it works.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoFix access-outside-array of o->rwmix[]
Jens Axboe [Fri, 8 Nov 2013 17:57:08 +0000 (10:57 -0700)]
Fix access-outside-array of o->rwmix[]

gcc 4.9 rightfully complains about it, though in a very obtuse way:

     CC cconv.o
In file included from thread_options.h:5:0,
                  from cconv.c:3:
cconv.c: In function 'convert_thread_options_to_cpu':
os/os.h:197:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
   __le32_to_cpu(*__val);   \
                 ^
os/os.h:176:28: note: in definition of macro '__le32_to_cpu'
  #define __le32_to_cpu(x)  (x)
                             ^
cconv.c:78:17: note: in expansion of macro 'le32_to_cpu'
    o->rwmix[i] = le32_to_cpu(top->rwmix[i]);
                  ^
cconv.c:63:2: note: containing loop
   for (i = 0; i < DDIR_RWDIR_CNT; i++) {
   ^
In file included from thread_options.h:5:0,
                  from cconv.c:3:
cconv.c: In function 'convert_thread_options_to_net':
os/os.h:209:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
   __cpu_to_le32(*__val);   \
                 ^
os/os.h:179:28: note: in definition of macro '__cpu_to_le32'
  #define __cpu_to_le32(x)  (x)
                             ^
cconv.c:372:19: note: in expansion of macro 'cpu_to_le32'
    top->rwmix[i] = cpu_to_le32(o->rwmix[i]);
                    ^
cconv.c:352:2: note: containing loop
   for (i = 0; i < DDIR_RWDIR_CNT; i++) {
   ^
In file included from thread_options.h:5:0,
                  from cconv.c:3:
cconv.c: In function 'fio_test_cconv':
os/os.h:209:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
   __cpu_to_le32(*__val);   \
                 ^
os/os.h:179:28: note: in definition of macro '__cpu_to_le32'
  #define __cpu_to_le32(x)  (x)
                             ^
cconv.c:372:19: note: in expansion of macro 'cpu_to_le32'
    top->rwmix[i] = cpu_to_le32(o->rwmix[i]);
                    ^
cconv.c:352:2: note: containing loop
   for (i = 0; i < DDIR_RWDIR_CNT; i++) {
   ^
In file included from thread_options.h:5:0,
                  from cconv.c:3:
os/os.h:197:16: warning: iteration 2u invokes undefined behavior
[-Waggressive-loop-optimizations]
   __le32_to_cpu(*__val);   \
                 ^
os/os.h:176:28: note: in definition of macro '__le32_to_cpu'
  #define __le32_to_cpu(x)  (x)
                             ^
cconv.c:78:17: note: in expansion of macro 'le32_to_cpu'
    o->rwmix[i] = le32_to_cpu(top->rwmix[i]);
                  ^
cconv.c:63:2: note: containing loop
   for (i = 0; i < DDIR_RWDIR_CNT; i++) {
   ^

Fix it by extending the rwmix indexes to the full rwdir count,
even though it doesn't support trim as part of the mix (yet).

Reported-by: Bruce Cran <bruce@cran.org.uk>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoserver: make job/connection lists private
Jens Axboe [Thu, 7 Nov 2013 20:37:09 +0000 (13:37 -0700)]
server: make job/connection lists private

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agosolaris: ensure that -D_REENTRANT gets set
Jens Axboe [Wed, 6 Nov 2013 22:37:35 +0000 (15:37 -0700)]
solaris: ensure that -D_REENTRANT gets set

Apparently some Solaris' require this for threadsafe
errno.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoposixaio: fix thread problem with using errno
Jens Axboe [Wed, 6 Nov 2013 21:47:22 +0000 (14:47 -0700)]
posixaio: fix thread problem with using errno

If we fail queueing a read or a write, use aio_error() to
retrieve the right error value. This fixes an issue on
(at least) Solaris where we get EAGAIN due to system
shortage of resources, but treat that as a random type
of error due to using errno.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoserver: get rid of compile warning on platforms where pid_t isn't int
Jens Axboe [Wed, 6 Nov 2013 18:19:42 +0000 (11:19 -0700)]
server: get rid of compile warning on platforms where pid_t isn't int

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoFix a couple of typos in fio2gnuplot.
Bruce Cran [Mon, 4 Nov 2013 11:59:43 +0000 (11:59 +0000)]
Fix a couple of typos in fio2gnuplot.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoUse #!/usr/bin/env to allow for interpreters in different directories.
Bruce Cran [Mon, 4 Nov 2013 11:52:08 +0000 (11:52 +0000)]
Use #!/usr/bin/env to allow for interpreters in different directories.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoserver: ensure that fio_time_init() is called before option parsing
Bruce Cran [Fri, 1 Nov 2013 18:32:40 +0000 (12:32 -0600)]
server: ensure that fio_time_init() is called before option parsing

When run in server mode, jobs are processed before fio_time_init()
is called, which causes pthread_setspecific() to be called before
pthread_key_create(). Work around this by calling it in thread_main()
too.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoImplement writev on Windows
Bruce Cran [Fri, 1 Nov 2013 18:31:03 +0000 (12:31 -0600)]
Implement writev on Windows

The network code depends on writev(), so implement it for Windows.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoUpdate Windows code to fix build breakage
Bruce Cran [Tue, 8 Oct 2013 14:05:27 +0000 (15:05 +0100)]
Update Windows code to fix build breakage

MinGW has been updated: this causes problems due to changes in header
files, including COM stuff being pulled in resulting in 'interface' being
a reserved keyword, and changes to a few POSIX definitions.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoconfigure: add option to disable libnuma usage
Castor Fu [Thu, 31 Oct 2013 17:00:54 +0000 (11:00 -0600)]
configure: add option to disable libnuma usage

Signed-off-by: Castor Fu <castor@alumni.caltech.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoAdd json output for client/server mode
Castor Fu [Thu, 31 Oct 2013 17:00:34 +0000 (11:00 -0600)]
Add json output for client/server mode

In client/server mode, this adds support for json mode.  Each job's
details are gradually accumulated and upon completion the client
dumps the full json tree.  The tree is annotated with the server
host and port.

Signed-off-by: Castor Fu <castor@alumni.caltech.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoFix bad asm constrants for x86-64 cpuid()
Jens Axboe [Thu, 31 Oct 2013 16:56:53 +0000 (10:56 -0600)]
Fix bad asm constrants for x86-64 cpuid()

Bruce reports:

It seems there's a bug in the x86_64 version of do_cpuid() that causes
fio to segfault when built with clang at lower optimization levels:

static inline void do_cpuid(unsigned int *eax, unsigned int *ebx,
                 unsigned int *ecx, unsigned int *edx)
{
     asm volatile("cpuid"
         : "=a" (*eax), "=b" (*ebx), "=r" (*ecx), "=d" (*edx)
         : "0" (*eax), "2" (*ecx)
         : "memory");
}

via Tijl Coosemans:

Should be: "=c" (*ecx)

But you can also use the '+' modifier and remove the input operands:

: "+a" (*eax), "=b" (*ebx), "+c" (*ecx), "=d" (*edx)
:
: "memory"

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoclient: fix free of wrong pointer
Hong Zhiguo [Wed, 16 Oct 2013 14:35:12 +0000 (08:35 -0600)]
client: fix free of wrong pointer

Signed-off-by: Hong Zhiguo <zhiguohong@tencent.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoppc: disable CPU clock until we can detect whether we have it or not
Jens Axboe [Fri, 11 Oct 2013 16:27:28 +0000 (10:27 -0600)]
ppc: disable CPU clock until we can detect whether we have it or not

The child segfault test should catch it, however it does not on
AIX at least.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoarch-ppc.h: Fix typo in header guard
Akash Verma [Fri, 11 Oct 2013 16:24:26 +0000 (10:24 -0600)]
arch-ppc.h: Fix typo in header guard

Signed-off-by: Akash Verma <akashv@google.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agomake ioengine specific command line options work
Akinobu Mita [Wed, 9 Oct 2013 15:32:34 +0000 (09:32 -0600)]
make ioengine specific command line options work

If the ioengine specific parameter is given on the command line,
unrelated value in thread_data.o will be corrupted unexpectedly.
Specifically, if libaio specific parameter userspace_reap is given
on the command line, thread_data.o.description will be set to 1.

This fixes it by passing the correct pointer of ioengine specific
options to the command line parser.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Tue, 8 Oct 2013 14:49:25 +0000 (08:49 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

5 years agofix crash: free of wrong pointer in bssplit_ddir
Hong Zhiguo [Tue, 8 Oct 2013 14:48:47 +0000 (08:48 -0600)]
fix crash: free of wrong pointer in bssplit_ddir

I got SIGABRT when I passed "bssplit=4k/10:64k/50:/40" to fio.
It should be "bssplit=4k/10:64k/50:32k/40" but I missed "32k" in
my typing. The bug turns out to be another typo just like mine:)

*** glibc detected *** /viki/lkml/fio/fio: free(): invalid pointer:
0x000000000071fd30 ***
======= Backtrace: =========
/lib/libc.so.6(+0x78bb6)[0x7ffff700cbb6]
/lib/libc.so.6(cfree+0x73)[0x7ffff7013483]
/viki/lkml/fio/fio[0x42c108]
/viki/lkml/fio/fio[0x42c475]
/viki/lkml/fio/fio[0x429194]
/viki/lkml/fio/fio[0x42a888]
/viki/lkml/fio/fio(parse_option+0x101)[0x42ac47]
/viki/lkml/fio/fio(fio_options_parse+0x95)[0x42e5b3]
/viki/lkml/fio/fio(parse_jobs_ini+0x3ae)[0x4103ae]
/viki/lkml/fio/fio(parse_options+0x112)[0x41198a]
/viki/lkml/fio/fio(main+0x3b)[0x460f5b]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7ffff6fb2c4d]
/viki/lkml/fio/fio[0x40a7c9]

Signed-off-by: Hong Zhiguo <zhiguohong@tencent.com>
Tested-by: Hong Zhiguo <zhiguohong@tencent.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agohowto: Fix redundant entries
Sitsofe Wheeler [Fri, 4 Oct 2013 22:09:42 +0000 (23:09 +0100)]
howto: Fix redundant entries

Reported-by: Konstantin Ruvinsky <Konstantin.Ruvinsky@emc.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
5 years agoman page: Bump up release date
Sitsofe Wheeler [Fri, 4 Oct 2013 21:07:23 +0000 (22:07 +0100)]
man page: Bump up release date

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
5 years agoman page: fix broken formatting before OUTPUT section
Sitsofe Wheeler [Fri, 4 Oct 2013 21:07:23 +0000 (22:07 +0100)]
man page: fix broken formatting before OUTPUT section

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
5 years agoman page: manpagify things that look like syscalls.
Sitsofe Wheeler [Fri, 4 Oct 2013 21:07:23 +0000 (22:07 +0100)]
man page: manpagify things that look like syscalls.

Use the bold style with section number to mark syscalls.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
5 years agoman page: Fix typos
Sitsofe Wheeler [Fri, 4 Oct 2013 21:07:23 +0000 (22:07 +0100)]
man page: Fix typos

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
5 years agoman page: correct typoe
Jens Axboe [Fri, 4 Oct 2013 18:33:11 +0000 (12:33 -0600)]
man page: correct typoe

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoMerge branch 'manpage' of https://github.com/sitsofe/fio
Jens Axboe [Fri, 4 Oct 2013 15:48:15 +0000 (09:48 -0600)]
Merge branch 'manpage' of https://github.com/sitsofe/fio

5 years agolog: disable log, if realloc fails
Jens Axboe [Mon, 30 Sep 2013 18:17:34 +0000 (12:17 -0600)]
log: disable log, if realloc fails

Right now we just segfault, if realloc() returns NULL. Handle
this a bit more gracefully, by just disabling logging if that
happens. Fio will also print an error for that case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoUpdate man page
Sitsofe Wheeler [Fri, 27 Sep 2013 12:17:59 +0000 (13:17 +0100)]
Update man page

Many of fio's options are not described in the man page. Copy the short
text to the man page for those commands listed by:

./fio --cmdhelp=all | sed 's/ *:.*$//' | sed 's/ \+//' \
| while read line; do grep "\b$line\b" -q fio.1; if [[ $? -eq 1 ]]; then \
echo $line; fi; done

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
5 years agoUpdate man page
Sitsofe Wheeler [Fri, 27 Sep 2013 12:17:59 +0000 (13:17 +0100)]
Update man page

Fix verifysort typo, broken formatting after roundrobin and tidy up some
grammar.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
5 years agoFix misspellings using 'codespell' tool
Anatol Pomozov [Thu, 26 Sep 2013 23:31:34 +0000 (16:31 -0700)]
Fix misspellings using 'codespell' tool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoFio 2.1.3 fio-2.1.3
Jens Axboe [Tue, 24 Sep 2013 14:42:24 +0000 (08:42 -0600)]
Fio 2.1.3

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 years agoFixes bug: stale LAST_POS(f) is not being reset
Juan Casse [Fri, 20 Sep 2013 15:20:52 +0000 (09:20 -0600)]
Fixes bug: stale LAST_POS(f) is not being reset

Problem:
When running fio with ioengine=sync, LAST_POS(f) state is not
reset after the file is closed. This causes workloads with
readwrite=randread and loops > 1 to fail verification if the
state of LAST_POS(f) at the beginning of the next loop is the
same as the io_u->offset. If that is the case, lseek in not
invoked, but the file position is at 0 and not at io_u->offset.

Proposed Solution:
Other ioengines, such as binject and fusion-aw, set f->engine_data
to 0 when closing the file. The sync ioengin uses f->engine_data to
store the LAST_POS(f) state. The proposed solution is to set
f->engine_data = 0 when closing a file in generic_close_file().

Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd condition to stop issuing io in do_io().
Juan Casse [Mon, 16 Sep 2013 16:22:10 +0000 (09:22 -0700)]
Add condition to stop issuing io in do_io().

Problem:
When specifying iodepth > 1 while doing random async io
without a random map (and using the default random generator), the
main while loop in do_io() will issue extra io in the amount of io
allowed to accumulate in the queue. For example, a job with size=24k
and bs=8k will perform 4 io instead of the expected 3.

Reason:
The above behavior occurs because the while loop in do_io()
will continue until the amount of "completed io" >= size, but io
in the queue are not accounted for because they have not been
completed by io_u_queued_complete().

Exceptions:
The above behavior does not manifest itself when:
using random_generator=lfsr bacause lfsr knows when to stop
generating, and get_io_u() inside the while loop returns NULL,
breaking out of the loop.
using random map because random map ensures that all blocks
are hit, and after they are all hit, no more are generated.

Proposed Solution:
Stop while loop based on bytes issued instead of bytes completed.

Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agodebug: ensure that __dprint() is also logged over the network
Jens Axboe [Wed, 11 Sep 2013 14:23:45 +0000 (08:23 -0600)]
debug: ensure that __dprint() is also logged over the network

We force local logging now, but that's not terribly useful
if you can't see the output from the backend. Make it follow
the usual rules of logging over the network if a backend, to
syslog, or locally.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix broken requeue of io_us
Jens Axboe [Mon, 9 Sep 2013 21:16:28 +0000 (15:16 -0600)]
Fix broken requeue of io_us

Suresh reports:

-----

Does a requeued io_u work get submitted again through __get_io_u() ? If
so, the io_u item is taken out of requeue list & then the io_u->file is
set to NULL. We don't make a distinction about item being already setup
Vs a new io_u in __get_io_u(). In get_io_u(), when an io_u may come out
of requeue list, we depend on io_u->file to be non NULL to detect the
work as old pending one.

Am I missing something here or is this a bug?

1274         io_u = __get_io_u(td);
1275         if (!io_u) {
1276                 dprint(FD_IO, "__get_io_u failed\n");
1277                 return NULL;
1278         }
1279
1280         if (check_get_verify(td, io_u))
1281                 goto out;
1282         if (check_get_trim(td, io_u))
1283                 goto out;
1284
1285         /*
1286          * from a requeue, io_u already setup
1287          */
1288         if (io_u->file)
1289                 goto out;
1290
"io_u.c" line 1286 of 1667 --77%-- col 3-10

1130 struct io_u *__get_io_u(struct thread_data *td)
1131 {
1132         struct io_u *io_u;
1133
1134         td_io_u_lock(td);
1135
1136 again:
1137         if (!io_u_rempty(&td->io_u_requeues))
1138                 io_u = io_u_rpop(&td->io_u_requeues);
1139         else if (!io_u_qempty(&td->io_u_freelist))
1140                 io_u = io_u_qpop(&td->io_u_freelist);
1141
1142         if (io_u) {
1143                 io_u->buflen = 0;
1144                 io_u->resid = 0;
1145                 io_u->file = NULL;
1146                 io_u->end_io = NULL;
1147         }
"io_u.c" line 1130 of 1667 --67%-- col 1

-----

We want a retry of a requeue to be identical, not some
new chunk (and drop the latter one on the floor). This got broken with
the introduction of the rings instead of lists, where a single if
section was split into two.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoCatch more places where the callback should just return on parse/check
Jens Axboe [Fri, 6 Sep 2013 16:07:37 +0000 (10:07 -0600)]
Catch more places where the callback should just return on parse/check

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix problem with --showcmd and callbacks that verify
Jens Axboe [Fri, 6 Sep 2013 15:59:56 +0000 (09:59 -0600)]
Fix problem with --showcmd and callbacks that verify

David reports:

I'm using "fio --showcmd file.job" to convert a script to plain
commandline arguments and this fails if the directory specified in job
file does not exist. While this has to be an error if the job is being
executed, it should not be in context of --showcmd.

To reproduce:
$ cat job.fio
[global]
directory=/x

$ fio --showcmd job.fio
fio: /x is not a directory
fio: failed parsing directory=/x
fio: job global dropped

Expected output:
fio --directory=/x

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoRemove duplicate LDFLAGS on -o lines
Grant Grundler [Wed, 4 Sep 2013 20:12:09 +0000 (14:12 -0600)]
Remove duplicate LDFLAGS on -o lines

Only need to list LDFLAGS once on each line.

Signed-off-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoRework descriptions for mailing list, options, job files
Grant Grundler [Wed, 4 Sep 2013 14:26:52 +0000 (08:26 -0600)]
Rework descriptions for mailing list, options, job files

In general, make the existing document less ambiguous:

o Use consistent formatting/layout when describing options
o always refer to options with "--" prefix.
o reduce use of pronouns: "its" and "this" are often ambigous.
o remove use of "you" to describe user actions/choices
o add reference to examples/ directory

Signed-off-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoclient: fix compile warning on platforms where pfds->fd is not an int
Jens Axboe [Fri, 30 Aug 2013 16:13:36 +0000 (10:13 -0600)]
client: fix compile warning on platforms where pfds->fd is not an int

It's apparently a long on some platforms.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoLFSR: Do not ignore returning the seed
Alex Pyrgiotis [Tue, 27 Aug 2013 16:58:34 +0000 (10:58 -0600)]
LFSR: Do not ignore returning the seed

Fix a situation where we would spin prematurely and would hop over the
seed value.

Also, add some more comments to the code.

Signed-off-by: Alex Pyrgiotis <apyrgio@grnet.gr>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix spelling error in fio man page
Martin Steigerwald [Tue, 27 Aug 2013 15:33:35 +0000 (09:33 -0600)]
Fix spelling error in fio man page

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoverify: fix problem with hole punching on newer Linux kernels
Jens Axboe [Fri, 9 Aug 2013 20:31:06 +0000 (14:31 -0600)]
verify: fix problem with hole punching on newer Linux kernels

Jeff Moyer writes:

Our QE team noticed fio failures on recent kernels.  I simplified the
job file to the following:

[global]
name=fio-mmap
rw=write
bs=4K
direct=1
end_fsync=1
verify=crc32

[file3]
size=100M
ioengine=mmap
mem=malloc
direct=1

After fio completes (and returns verify errors), the file is completely
full of zeroes.

Here is what the verify logic is doing:

static void do_verify(struct thread_data *td, uint64_t verify_bytes)
{
...
        /*
         * sync io first and invalidate cache, to make sure we really
         * read from disk.
         */
        for_each_file(td, f, i) {
                if (!fio_file_open(f))
                        continue;
                if (fio_io_sync(td, f))
                        break;
                if (file_invalidate_cache(td, f)) <--------
                        break;
        }

That invalidate cache call looks like so:

static int __file_invalidate_cache(struct thread_data *td, struct fio_file *f,
                                   unsigned long long off,
                                   unsigned long long len)
{
...
        /*
         * FIXME: add blockdev flushing too
         */
        if (f->mmap_ptr) {
                ret = posix_madvise(f->mmap_ptr, f->mmap_sz, POSIX_MADV_DONTNEED);
                (void) posix_madvise(f->mmap_ptr, f->mmap_sz, FIO_MADV_FREE); <-------

FIO_MADV_FREE can be defined as MADV_REMOVE, which will actually punch a
hole in the file (a hole the size of the entire file, btw).  Of course,
unallocated blocks are returned as zeroes by the file system, so that
explains that!

---

For now, only do the MADV_FREE on non-files to avoid punching holes.
Further investigation is needed on the blockdev side, but at least
this should cure the immediate problem.

Reported-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd number_ios= option
Jens Axboe [Fri, 9 Aug 2013 18:53:44 +0000 (12:53 -0600)]
Add number_ios= option

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agopowerpc: disable use of ATBU clock
Jens Axboe [Fri, 9 Aug 2013 15:17:09 +0000 (09:17 -0600)]
powerpc: disable use of ATBU clock

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agogenfio: Don't consider only /dev/ disk device
Erwan Velu [Thu, 8 Aug 2013 08:59:46 +0000 (10:59 +0200)]
genfio: Don't consider only /dev/ disk device

Prior this patch, genfio was considering the disks __NOT__ having the "/dev"
name in it.

That's a serious limitation as
- we can't test regular files
- that doesn't works for complex /dev/ naming like /dev/disk/by-id/...

This patch removes this requirement and let user giving the complete
path.

6 years agofio2gnuplot: Adding manpage
Erwan Velu [Wed, 7 Aug 2013 14:02:25 +0000 (16:02 +0200)]
fio2gnuplot: Adding manpage

As requested by users, let's add a manpage to make things easier to
understand.

I'm providing here the raw file used to generate the manpage. It can be
converted to a real manpage by using txt2man.

I'm also providing the manpage file to avoid dependencies at build time,
anyway the makefile target can be easily used to rebuild it if
necessary.

6 years agofio2gnuplot: Removing .py extension
Erwan Velu [Wed, 7 Aug 2013 13:32:02 +0000 (15:32 +0200)]
fio2gnuplot: Removing .py extension

It's pretty boring having the .py extension to the tool.
Let's remove it !

6 years agofio2gnuplot: Fixing help message
Erwan Velu [Wed, 7 Aug 2013 12:32:50 +0000 (14:32 +0200)]
fio2gnuplot: Fixing help message

Adding missing features & fix indent

6 years agofio2gnuplot: Managing temporary files in a better way
Erwan Velu [Wed, 7 Aug 2013 12:23:43 +0000 (14:23 +0200)]
fio2gnuplot: Managing temporary files in a better way

If gnuplot success at rendering, let's delete all temporary files.
If gnuplot fails at rendering, let's keep all temporary files.
If user don't want to run gnuplot, let's keep temporary files.
If user ask to keep the temporary files (option -k), let's always keep
them.

6 years agofio2gnuplot: Deleting temporary files by default
Erwan Velu [Wed, 7 Aug 2013 12:11:38 +0000 (14:11 +0200)]
fio2gnuplot: Deleting temporary files by default

6 years agofio2gnuplot: Adding verbose option
Erwan Velu [Wed, 7 Aug 2013 11:28:58 +0000 (13:28 +0200)]
fio2gnuplot: Adding verbose option

Reducing default's verbosity and add a -v option to increase it.
This avoids flooding end-users and let an option for thoses who need to
get a complete trace.

6 years agofio2gnuplot: Fixing typo in error message
Erwan Velu [Wed, 7 Aug 2013 11:20:56 +0000 (13:20 +0200)]
fio2gnuplot: Fixing typo in error message

6 years agofio2gnuplot: Adding getoplong support for cmdline
Erwan Velu [Wed, 7 Aug 2013 08:52:05 +0000 (10:52 +0200)]
fio2gnuplot: Adding getoplong support for cmdline

Help message was misleading since long options were not supported....
This patch adds all the long options to be supported.

6 years agofio2gnplot: Print specific message if cmdline parsing fails
Erwan Velu [Wed, 7 Aug 2013 08:46:55 +0000 (10:46 +0200)]
fio2gnplot: Print specific message if cmdline parsing fails

If the cmdline isn't consistent regarding the getopt, let's print a
specific message to inform user about this situation instead of printing
the help.

Showing the help without any specific message was very confusing.

6 years agoFio 2.1.2 fio-2.1.2
Jens Axboe [Tue, 6 Aug 2013 15:39:54 +0000 (09:39 -0600)]
Fio 2.1.2

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoiolog: Logging blocksize in IOPS traces
Erwan Velu [Fri, 2 Aug 2013 14:39:40 +0000 (16:39 +0200)]
iolog: Logging blocksize in IOPS traces

When saving bandwidth's with write_bw_log option, the last column
provides the blocksize used during the run.

When doing the same with write_io_log option, the last column was always
set to 0 which is confusing when reading the log file later.

This patch put both write_bw_log & write_io_log using the same semantic
by reporting the blocksize for every sample.

6 years agofio2gnuplot: Display average value in label
Erwan Velu [Fri, 2 Aug 2013 12:29:00 +0000 (14:29 +0200)]
fio2gnuplot: Display average value in label

When plotting 2D or mathematical graphs, let's show the numerical value
of the average value to ease the reading.

6 years agofio2gnuplot: Plotting average value for compare graphs
Erwan Velu [Fri, 2 Aug 2013 12:16:14 +0000 (14:16 +0200)]
fio2gnuplot: Plotting average value for compare graphs

When comparing in 2D several plots, it's pretty useful to have the
average value printed in the same graph.

6 years agofio2gnuplot: Fixing plotting issues on shorten files
Erwan Velu [Fri, 2 Aug 2013 08:52:50 +0000 (10:52 +0200)]
fio2gnuplot: Fixing plotting issues on shorten files

fio2gnuplot was mixing files when some got shorten.

The position inside the array was used to remember what file was read
leading to mistakes once some values got ignored.

As a fix, we shall count the index for every single line we read and add
to the time & perf we keep. While writing data to the temporary files,
the extraction of the index insure we write to the proper file.

Before this patch, we observed at the end of the traces some stupid artefacts like loops... was
looking like a 4years old kid drawing...

That's now fixed.

6 years agofio2gnplot: Renaming file variable to myfile
Erwan Velu [Fri, 2 Aug 2013 08:51:25 +0000 (10:51 +0200)]
fio2gnplot: Renaming file variable to myfile

'file' could be reserved on some modules or in the python.
It's better to rename it to avoid troubles.

6 years agofio2gnuplot: Adding time selection options
Erwan Velu [Thu, 1 Aug 2013 13:11:23 +0000 (15:11 +0200)]
fio2gnuplot: Adding time selection options

It's could be very handy being able to extract a portion of a trace to
plot it.

This patch adds two new options to select the starting time and the end
time were the logs shall be considered for plotting.

6 years agofio2gnuplot: Print line if int() conversion fails
Erwan Velu [Thu, 1 Aug 2013 11:41:02 +0000 (13:41 +0200)]
fio2gnuplot: Print line if int() conversion fails

When data aren't properly formated the int() conversion could fails but
doesn't print the impacted line which is pretty difficult to estimate
where the bug is.

This patch simply reports the line that cause the trouble.

6 years agofio2gnuplot: Fixing typo when inserting fake data
Erwan Velu [Thu, 1 Aug 2013 11:39:32 +0000 (13:39 +0200)]
fio2gnuplot: Fixing typo when inserting fake data

Commit f9846c392d176582916c39b1925a436e79f23de1 introduced fake data
when matched files doesn't have the exact same amount of traces. This
patch introduced a typo that prevented the int comparison to work
properly.

6 years agoMake test for gnuplot work with empty strings.
Martin Steigerwald [Mon, 5 Aug 2013 14:57:33 +0000 (08:57 -0600)]
Make test for gnuplot work with empty strings.

When I launch fio_generate_plots on a system where gnuplot is not installed I
get this error :

$ fio_generate_plots test
Making bw logs
/usr/bin/fio_generate_plots: 42: /usr/bin/fio_generate_plots: -: not found

That's because the test checking whether gnuplot is installed is failing
because of an empty variable :
GNUPLOT=$(which gnuplot)
if [ ! -x $GNUPLOT ]; then
        echo You need gnuplot installed to generate graphs
        exit 1
fi

Indeed the command "test -x" is exiting with code 0.

To correct this we should enclose the variable with quotes :
if [ ! -x "$GNUPLOT" ]; then

Then the script is going into the test properly :
$ fio_generate_plots test
You need gnuplot installed to generate graphs

I copied problem description by Hervé from the Debian bug report below.

Fixes:
Bug#700580: /usr/bin/fio_generate_plots: -: not found
http://bugs.debian.org/700580

Reported-By: Hervé Werner <hwerner@score-md.com>
Tested-By: Hervé Werner <hwerner@score-md.com>
Tested-By: Martin Steigerwald <ms@teamix.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agofio2gnuplot: Adding filename as title for 2D graphs
Erwan Velu [Wed, 31 Jul 2013 10:32:40 +0000 (12:32 +0200)]
fio2gnuplot: Adding filename as title for 2D graphs

While plotting 2D graphs, it's pretty handy to keep the filename as
title for the plotting trace. This way, the png file still keep the
original filename used for plotting.

6 years agofio2gnuplot: Adding information steps
Erwan Velu [Wed, 31 Jul 2013 10:21:10 +0000 (12:21 +0200)]
fio2gnuplot: Adding information steps

While executing fio2gnuplot, we were lacking of some information of
what's going on.

This patch makes things easier to understand/follow for end users.

6 years agofio2gnuplot: Fixing help message
Erwan Velu [Wed, 31 Jul 2013 10:15:24 +0000 (12:15 +0200)]
fio2gnuplot: Fixing help message

Some options were missing on the main description.

6 years agofio2gnuplot: Adding comparing 2D graphs
Erwan Velu [Wed, 31 Jul 2013 10:07:32 +0000 (12:07 +0200)]
fio2gnuplot: Adding comparing 2D graphs

It's a pretty common need to plot several files on the same graph to
compare behavior during various benchs.

If 2 or more files are selected by the pattern, new plots will be
generated to compare the traces into a single graph. 3 kinds of graphs
will be generated (raw, smooth and trends) like for the individual
graphs.

The output filename will be prefixed with "compare-" like :
compare-vm1-1-4m-vdb-read-para.results_bw-2Draw.png
compare-vm1-1-4m-vdb-read-para.results_bw-2Dsmooth.png
compare-vm1-1-4m-vdb-read-para.results_bw-2Dtrend.png

6 years agofio2gnuplot: Printing number of selected files
Erwan Velu [Tue, 30 Jul 2013 12:13:51 +0000 (14:13 +0200)]
fio2gnuplot: Printing number of selected files

when using pattern matching on a big set of files, it's pretty hard to
estimate if we missed some or not.

Let's print the number of matched items with the selected pattern to
help user estimate if it's ok or not.

6 years agofio2gnuplot: Using default install dir for gpm files
Erwan Velu [Tue, 30 Jul 2013 11:50:07 +0000 (13:50 +0200)]
fio2gnuplot: Using default install dir for gpm files

That was pretty boring trying to read the gpm files from the same dir as
the logs. Since gpm files are now installed in a specific dir at make
install time, let's use them from here.

Two locations will be considered at runtime : /usr/share/fio and
/usr/local/share/fio

6 years agoTree: Moving fio_generate_plots in tools/
Erwan Velu [Tue, 30 Jul 2013 09:54:45 +0000 (11:54 +0200)]
Tree: Moving fio_generate_plots in tools/

As fio2gnplot.py & genfio are located in tools, it sounds better to put
all the additional tools in the same directory.

6 years agoMakefile: Installing genfio at install time
Erwan Velu [Tue, 30 Jul 2013 09:47:16 +0000 (11:47 +0200)]
Makefile: Installing genfio at install time

genfio wasn't installed at all during 'make install'.

6 years agoMakefile: Installing fio2gnuplot & sons
Erwan Velu [Tue, 30 Jul 2013 09:45:18 +0000 (11:45 +0200)]
Makefile: Installing fio2gnuplot & sons

Let's install fio2gnuplot & other gnuplot files (.gpm) while installing
fio.

Gnuplot files are installed into $(prefix}/share/fio/ subdirectory

6 years agofio2gnuplot: Option to select gnuplot's output dir
Erwan Velu [Tue, 30 Jul 2013 09:34:53 +0000 (11:34 +0200)]
fio2gnuplot: Option to select gnuplot's output dir

This patch add the '-d' option to select where gnuplot shall render its
files.

When plotting big series of files, it's pretty anoying having all the
files in the current directory as there is many many of them.

So let's use this option to allow users specifying where all the
graphics shall be rendered

6 years agoUpdate sample atomic write job file
Jens Axboe [Fri, 26 Jul 2013 19:11:52 +0000 (13:11 -0600)]
Update sample atomic write job file

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMake it clear in job output whether we are using bs_is_seq_rand or not
Jens Axboe [Thu, 25 Jul 2013 19:01:11 +0000 (13:01 -0600)]
Make it clear in job output whether we are using bs_is_seq_rand or not

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd support for bs_is_seq_rand
Jens Axboe [Thu, 25 Jul 2013 18:45:26 +0000 (12:45 -0600)]
Add support for bs_is_seq_rand

This option will switch the meaning of the read,write part of any
blocksize setting to mean sequential,random instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoinit: remove leftover unused variable
Jens Axboe [Thu, 25 Jul 2013 18:30:06 +0000 (12:30 -0600)]
init: remove leftover unused variable

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd support for randomness of any IO direction
Jens Axboe [Thu, 25 Jul 2013 16:20:45 +0000 (10:20 -0600)]
Add support for randomness of any IO direction

sequential_random used to be applied to all of the IO
directions, with no possibility of having different settings
for reads, writes, and trims. Now it supports setting each of
them individually.

By default, if you do:

sequential_random=50

it will still apply to all three. If you do:

sequential_random=10,90,80

you would get reads 10% random, writes 90% random, and trims
80% random.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoUpdate for RDMA io engine's compatibility
Yufei Ren [Tue, 23 Jul 2013 02:58:05 +0000 (20:58 -0600)]
Update for RDMA io engine's compatibility

Since the RDMA engine is not able to be successfully compiled, this
patch accommodates it to the upstream version of fio.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agocore: Insure "il" is initialized in all cases
Erwan Velu [Mon, 22 Jul 2013 21:48:48 +0000 (23:48 +0200)]
core: Insure "il" is initialized in all cases

Prior this commit, "il" was not initialized and check_int() was supposed
to setup it's value before usage.

In fact, check_int() can let "il" uninitialized leading to a dprint()
with a unintialized value.

6 years agocore: Don't leak memory if error is unknown
Erwan Velu [Mon, 22 Jul 2013 21:46:10 +0000 (23:46 +0200)]
core: Don't leak memory if error is unknown

If error is unknown, the "return 1" call was not freeing error which was
malloced at the beginning of the function.

6 years agocore: Removing duplicated code
Erwan Velu [Mon, 22 Jul 2013 21:39:18 +0000 (23:39 +0200)]
core: Removing duplicated code

Some variable are assigned during initialization and another time with
the same value a very few later.

That's clearly useless and could lead to an error if only one is changed.

6 years agofio2gnuplot: Don't plot fake data
Erwan Velu [Mon, 22 Jul 2013 10:59:04 +0000 (12:59 +0200)]
fio2gnuplot: Don't plot fake data

As we did add some fake data to keep as long a possible the coherency
between all the fio files, let's ignore them when rendering.

As a consequence, we are keeping the time associated to each
performance we store instead of a single time for all performance
traces. This will increase precision while plotting.

6 years agofio2gnuplot: Don't truncate fio log files
Erwan Velu [Mon, 22 Jul 2013 10:56:26 +0000 (12:56 +0200)]
fio2gnuplot: Don't truncate fio log files

The code was stopping at reading once the shorter stopped. But in the
case of a very big difference of performances between hosts, some could
be much longer/shorter than others.

This patch does insure that until _all_ input files got read complety we
do had fake data with a negative time.

This way, all the traces will be complete as per fio's output.

6 years agofio2gnuplot: Keep original filename in temp. files
Erwan Velu [Mon, 22 Jul 2013 10:54:39 +0000 (12:54 +0200)]
fio2gnuplot: Keep original filename in temp. files

It was pretty hard to understand what trace file was linked to which fio
output file. This simple commit add a simpler header to inform the
debugger which fio outputfile got used.

6 years agonet: Add UDP multicast example job file
Shawn Bohrer [Mon, 22 Jul 2013 14:57:44 +0000 (09:57 -0500)]
net: Add UDP multicast example job file

This adds a pingpong UDP multicast example with two readers.

Signed-off-by: Shawn Bohrer <sbohrer@rgmadvisors.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agonet: Add option to set outgoing multicast TTL
Shawn Bohrer [Fri, 19 Jul 2013 18:24:08 +0000 (13:24 -0500)]
net: Add option to set outgoing multicast TTL

Signed-off-by: Shawn Bohrer <sbohrer@rgmadvisors.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agonet: Allow setting network interface to use for multicast
Shawn Bohrer [Fri, 19 Jul 2013 18:24:07 +0000 (13:24 -0500)]
net: Allow setting network interface to use for multicast

Signed-off-by: Shawn Bohrer <sbohrer@rgmadvisors.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agonet: Add UDP multicast support
Shawn Bohrer [Fri, 19 Jul 2013 18:24:06 +0000 (13:24 -0500)]
net: Add UDP multicast support

Allow UDP readers to listen to UDP multicast traffic if hostname is set
to a valid UDP multicast address.

Signed-off-by: Shawn Bohrer <sbohrer@rgmadvisors.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agofio2gnuplot: Fixing Yaxis rendering for 2D plots
Erwan Velu [Tue, 16 Jul 2013 09:07:34 +0000 (11:07 +0200)]
fio2gnuplot: Fixing Yaxis rendering for 2D plots

With many values, gnuplot was complaining about too many ticks like :

"graph2D.gpm", line 26: warning: Too many axis ticks requested (>9e+03)

Using the "auto" command, that works much more better and axis are now
readable

6 years agogenfio: Protecting parse_cmdline from spaces
Erwan Velu [Mon, 15 Jul 2013 10:33:44 +0000 (12:33 +0200)]
genfio: Protecting parse_cmdline from spaces

If you want to provide arguments with spaces in it, $@ shall be
protected with quotes.

That's required to manage complex cmdline for -A or -B options

6 years agotools: Adding gitignore
Erwan Velu [Mon, 15 Jul 2013 10:05:01 +0000 (12:05 +0200)]
tools: Adding gitignore

Nothing like a clean git status ;o)

6 years agogenfio: Use dynamic default values in help
Erwan Velu [Mon, 15 Jul 2013 10:03:55 +0000 (12:03 +0200)]
genfio: Use dynamic default values in help

To avoid mismatch between default values and help, let's print directly
the default value of the option.

6 years agogenfio: Don't use cat EOF for small prints
Erwan Velu [Mon, 15 Jul 2013 09:14:39 +0000 (11:14 +0200)]
genfio: Don't use cat EOF for small prints

That's just bother the reading when very few data is involved. A simple
print is enough

6 years agogenfio: Adding exec_{pre|post}run support
Erwan Velu [Mon, 15 Jul 2013 09:10:27 +0000 (11:10 +0200)]
genfio: Adding exec_{pre|post}run support

It's pretty useful being able to run some commands before or after a
job. That could be use to ask some external storage array to loose its
cache or return in a particular configuration state.

The -A (stands for After) or -B (stands for Before) allow user to define
what commands shall be run before or after a job.

6 years agogenfio: Splitting gen_template in 2 parts
Erwan Velu [Mon, 15 Jul 2013 08:44:58 +0000 (10:44 +0200)]
genfio: Splitting gen_template in 2 parts

gen_template can be split in to two parts:
- the static headers
- the dynamic ones linked to the arguments from cmdline

The static ones are now in gen_template() call before parsing the
cmdline.

The dynamic ones are now call in finish_template() after parsing the
cmdline.

That also open the ability of feeding the fio header during the parsing
removing the need of specific variables to remember the values.