fio.git
3 years agoChange arch_i386 to arch_x86
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:20 +0000 (22:37 +0900)]
Change arch_i386 to arch_x86

fio_arch_strings[arch_i386] is "x86" (not "i386"),
so change arch_i386 to arch_x86.

Other than arch_i386, "i386" is only used in __i386__.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing archs in fio_arch_strings[]
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:19 +0000 (22:37 +0900)]
Add missing archs in fio_arch_strings[]

Add MIPS and AArch64 after PARISC.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRename exists_and_not_file() to exists_and_not_regfile()
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:18 +0000 (22:37 +0900)]
Rename exists_and_not_file() to exists_and_not_regfile()

Renaming makes the purpose of this function more clear IMO.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix stat(2) related bugs introduced by changes made for Windows
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:17 +0000 (22:37 +0900)]
Fix stat(2) related bugs introduced by changes made for Windows

Non-Windows could have regular files that start with "\\\\.\\",
so the conditionals added by ecc314ba and 3892182a in 2011-2012
introduced corner case bugs, though in reality I doubt anyone
would want to create such a file.

This commit brings back the original conditional that was there
before ecc314ba for non-Windows platforms. It also adds WIN32
guard for a change made by 3892182a.

--
 # uname
 Linux
 # cat ./test1.c
 #include <stdio.h>
 #include <string.h>
 #include <sys/types.h>
 #include <sys/stat.h>
 /* copied from fio/init.c */
 static int exists_and_not_file(const char *filename)
 {
  struct stat sb;
  if (lstat(filename, &sb) == -1)
  return 0;
  /* \\.\ is the device namespace in Windows, where every file
   * is a device node */
  if (S_ISREG(sb.st_mode) && strncmp(filename, "\\\\.\\", 4) != 0)
  return 0;
  return 1;
 }
 int main(int argc, char **argv) {
  printf("exists_and_not_file(\"%s\") = %d\n",
  argv[1], exists_and_not_file(argv[1]));
  return 0;
 }
 # gcc -Wall -g ./test1.c -o test1
 # ls /dev/sda
 /dev/sda
 # ./test1 /dev/sda
 exists_and_not_file("/dev/sda") = 1  /* the existing blkdev isn't a file */
 # touch xxxxx
 # ./test1 ./xxxxx
 exists_and_not_file("./xxxxx") = 0  /* the existing regfile is not not a file */
 # touch "\\\\.\\xxxxx"
 # ./test1 "\\\\.\\xxxxx"
 exists_and_not_file("\\.\xxxxx") = 1  /* XXX */
 # stat "\\\\.\\xxxxx" | head -2
   File: `\\\\.\\xxxxx'
   Size: 0               Blocks: 0          IO Block: 4096   regular empty file

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDon't malloc/memcpy ioengine_ops on td initialization
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:16 +0000 (22:37 +0900)]
Don't malloc/memcpy ioengine_ops on td initialization

The only reason load_ioengine() needs to malloc extra space for
struct ioengine_ops and memcpy from the original ops is because td
has per-thread/process ioengine private data which is a part of
this structure.

If data (and dlhandle) are separated from struct ioengine_ops,
load_ioengine() no longer needs to malloc, as the rest of structure
fields such as function pointers and string literal are static.
Td only needs a pointer to the original ops. Also avoid memcpying
a list_head from the original ops.

This commit moves/renames data and dlhandle to struct thread_data
as thread_data::io_ops_data and thread_data::io_ops_dlhandle.
engines/*.c which access ioengine private data by dereferencing
td->io_ops->... are modified accordingly, but nothing more than that.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing --cmdhelp type string for FIO_OPT_UNSUPPORTED
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:15 +0000 (22:37 +0900)]
Add missing --cmdhelp type string for FIO_OPT_UNSUPPORTED

Without "unsupported" at index 12 (FIO_OPT_UNSUPPORTED) of typehelp[],
unsupported commands print irrelevant binary at typehelp[12].

 # fio --cmdhelp=numa_cpu_nodes
       numa_cpu_nodes: Build fio with libnuma-dev(el) to enable this option
                 type: ÏÿF
              default: no default

With this commit.
 # fio --cmdhelp=numa_cpu_nodes
       numa_cpu_nodes: Build fio with libnuma-dev(el) to enable this option
                 type: unsupported
              default: no default

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse correct I/O engine name "cpuio" instead of "cpu"
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:14 +0000 (22:37 +0900)]
Use correct I/O engine name "cpuio" instead of "cpu"

"cpuio" is the name of the engine/option, though the filename is cpu.c.

 # grep "ioengine =" engines/cpu.c -A1
 static struct ioengine_ops ioengine = {
  .name           = "cpuio",

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoIgnore exit_io_done= option if no I/O threads are configured
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:13 +0000 (22:37 +0900)]
Ignore exit_io_done= option if no I/O threads are configured

The cpuio engine option exit_io_done= should be effective only if
there is at least one real I/O thread configured.

The existing fio_cpuio_queue() with exit_io_done= option enabled
lets cpuio threads exit even if cpuio threads are the only threads
configured.

Since this option is supposed to mean "exit when I/O threads are done",
cpuio threads exiting (but only after the first spin cycle is done)
when no threads have done any I/O is a bit confusing.

> [cpu] exit_on_io_done=bool Detect when IO threads are done, then exit.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMention cpuio never finishes without real I/O in documentation
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:12 +0000 (22:37 +0900)]
Mention cpuio never finishes without real I/O in documentation

reap_threads() terminates cpuio threads only when there is
at least one real I/O thread that is exiting.

> if (*nr_running == cputhreads && !pending && realthreads)
> fio_terminate_threads(TERMINATE_ALL);

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMention default values for readwrite=/ioengine=/mem= in documentation
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:11 +0000 (22:37 +0900)]
Mention default values for readwrite=/ioengine=/mem= in documentation

HOWTO/fio(1) mention defaults for many of the available options.
This commit adds defaults for the following options.

 readwrite=
 ioengine=
 mem=

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse sizeof(char*) instead of sizeof(void*)
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:10 +0000 (22:37 +0900)]
Use sizeof(char*) instead of sizeof(void*)

Conform to the exact data type though both pointers are
normally expected to be the same size on various archs.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse default CPU_COUNT() function in DragonFlyBSD
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:08 +0000 (22:37 +0900)]
Use default CPU_COUNT() function in DragonFlyBSD

os/os.h has a default CPU_COUNT() inline function for those that
are expected to have CPU_COUNT() macro, but not in certain versions.
This seems to have been added for RHEL5 (or older?) and variants,
but use this in os/os-dragonfly.h since it's essentially the same
as its own fio_cpu_count() implementation.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix typos in log_err() message
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:07 +0000 (22:37 +0900)]
Fix typos in log_err() message

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMake return value type of fio_getaffinity() consistent
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:06 +0000 (22:37 +0900)]
Make return value type of fio_getaffinity() consistent

Return type of fio_getaffinity() isn't consistent among supported OS.
Windows and DragonFlyBSD return void while FreeBSD version is int.
The default version for those that don't support is do{}while(0).

Linux version is a macro for sched_getaffinity(2) which returns 0
on success and -1 otherwise, so others should basically follow that.

Note that I haven't compiled this on Windows, but it shouldn't fail.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix memory leak in _fio_setup_rbd_data()
Jevon Qiao [Wed, 27 Jul 2016 07:36:36 +0000 (15:36 +0800)]
Fix memory leak in _fio_setup_rbd_data()

Signed-off-by: Jevon Qiao <scaleqiao@gmail.com>
3 years agoAdd sample job file showing how to read backwards
Jens Axboe [Tue, 26 Jul 2016 20:50:02 +0000 (14:50 -0600)]
Add sample job file showing how to read backwards

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRevert "Revert "fio: Simplify forking of processes""
Jens Axboe [Tue, 26 Jul 2016 14:17:42 +0000 (08:17 -0600)]
Revert "Revert "fio: Simplify forking of processes""

This reverts commit cb92ecf8118b1a8b13909ab18d8a646e24ac79f2.

3 years agoFio 2.13 fio-2.13
Jens Axboe [Fri, 22 Jul 2016 19:43:56 +0000 (13:43 -0600)]
Fio 2.13

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: fix averaged latency logging
Jens Axboe [Fri, 22 Jul 2016 19:19:31 +0000 (13:19 -0600)]
log: fix averaged latency logging

We inadvertently broke that in the previous commit, fix it up.

Fixes: d454a205f81e ("log: fix missing entries")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: fix missing entries
Jens Axboe [Fri, 22 Jul 2016 18:34:53 +0000 (12:34 -0600)]
log: fix missing entries

If the timing is just off, we could miss adding entries to
an averaged iops or bandwidth log. Allow a bit of slack for
that, similarly to what we do elsewhere.

Fixes: a47591e4923f ("Improve logging accuracy")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix "exitall_on_error" option
YukiKita [Fri, 22 Jul 2016 09:18:14 +0000 (18:18 +0900)]
Fix "exitall_on_error" option
"exitall_on_error" option should be enabled without any argument.

3 years agodrifting in output of interval-averaged values was eventually causing IOP samples...
Karl Cronburg [Wed, 20 Jul 2016 20:21:55 +0000 (16:21 -0400)]
drifting in output of interval-averaged values was eventually causing IOP samples to be dropped.

3 years agoAdd missing header inclusion for Android from 1c764dbe
Tomohiro Kusumi [Tue, 19 Jul 2016 22:30:42 +0000 (07:30 +0900)]
Add missing header inclusion for Android from 1c764dbe

1c764dbe (which just got committed) was missing below for Android,
sorry about that.
 #include "./os-linux-syscall.h"

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd os/os-linux-syscall.h to separate syscall NR from arch headers
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:47 +0000 (01:33 +0900)]
Add os/os-linux-syscall.h to separate syscall NR from arch headers

Linux syscall NR should be defined under os/ with ifdef(arch)
switch rather than arch/ where headers are included not only by
Linux but also all other supported OS that don't need these NR.
(It'll get worse and worse if someone wants to add something
similar under arch/ for other supported OS)

Not sure if you like the way I separated these by adding
os/os-linux-syscall.h, but doing something like
 #if FIO_OS is (Linux or Android)
 ...
 #endif
in arch/arch-*.h doesn't work because arch headers are to be
included prior to os/os-*.h.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoChange ARCH_X86_64_h to ARCH_X86_64_H
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:46 +0000 (01:33 +0900)]
Change ARCH_X86_64_h to ARCH_X86_64_H

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd ioprio_set() support for DragonFlyBSD
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:45 +0000 (01:33 +0900)]
Add ioprio_set() support for DragonFlyBSD

It basically has the same interface as Linux kernel's ioprio_set(2),
but needed to workaround a compile issue by defining it as a macro
as mentioned in a comment (it can be called without syscall(NR, ...)).

This commit is based on the last two commits which made I/O priority
option more generic for non Linux environment.

 # uname
 DragonFly
 # fio --cmdhelp | grep prioclass -B1
 prio                    : Set job IO priority value
 prioclass               : Your platform does not support IO priority classes

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMake I/O priority option generic for non-Linux environment [2/2]
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:44 +0000 (01:33 +0900)]
Make I/O priority option generic for non-Linux environment [2/2]

There is a kernel that supports I/O priority with a syscall similar
to Linux (e.g. DragonFlyBSD's ioprio_set(2) which apparently seems
to have been inspired by ioprio_set(2) in Linux kernel), however
the idea of class within the syscall may not exist depending on the
I/O scheduler and its design, so "prioclass" option should be an
optional one for "prio" option.

This commit adds FIO_HAVE_IOPRIO_CLASS to separate "prioclass" from
"prio" on compile-time for those that do support priority itself,
but not priority classes.

If the platform supports I/O priority, it defines FIO_HAVE_IOPRIO
like it did before. If the platform supports I/O priority classes,
it defines FIO_HAVE_IOPRIO_CLASS in addition to above.
If FIO_HAVE_IOPRIO_CLASS is enabled, FIO_HAVE_IOPRIO must also be
enabled since FIO_HAVE_IOPRIO defines io_prioset(). This is also
checked on compile-time.

Linux (incl Android) has both of these macros enabled, so there is
no functional changes made by this commit.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMake I/O priority option generic for non-Linux environment [1/2]
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:43 +0000 (01:33 +0900)]
Make I/O priority option generic for non-Linux environment [1/2]

There is a kernel that supports I/O priority with a syscall similar
to Linux (e.g. DragonFlyBSD's ioprio_set(2) which apparently seems
to have been inspired by ioprio_set(2) in Linux kernel), however
min/max value of the priority may differ, so use OS specific macros
instead of hardcoded values that are designed to work on Linux.

This commit adds IOPRIO_MIN|MAX_PRIO and IOPRIO_MIN|MAX_PRIO_CLASS
to Linux (incl Android), but no functional changes.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd CPU affinity support for DragonFlyBSD
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:42 +0000 (01:33 +0900)]
Add CPU affinity support for DragonFlyBSD

Confirmed that the cpu affinity works using cpuio engine.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix wrong cpuio option name in documentation
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:41 +0000 (01:33 +0900)]
Fix wrong cpuio option name in documentation

The name of an option for cpuio engine is "cpuchunks",
though the code internally uses "cpucycle" for variables.

(If the option name "cpuchunks" is the incorrect one,
I'll resubmit a patch that corrects the option name)

 # fio --enghelp=cpuio
 cpuload                 : Use this percentage of CPU
   cpuchunks             : Length of the CPU burn cycles (usecs)
   exit_on_io_done         : Exit when IO threads finish

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoplot: indicate that the pattern is a glob
Sitsofe Wheeler [Sun, 17 Jul 2016 07:49:12 +0000 (08:49 +0100)]
plot: indicate that the pattern is a glob

Rename the word regex to glob in fio2gnuplot because its pattern parameter was
incorrectly called a regex when it was always really a glob.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoplot: add gnuplot 5 support
Sitsofe Wheeler [Sun, 17 Jul 2016 07:35:14 +0000 (08:35 +0100)]
plot: add gnuplot 5 support

gnuplot 5 no longer supports using old-style ($1, $2 etc.) gnuplot 4 parameter
substituion causing fio2gnuplot to break on newer Linux distributions.

Fix this by using new-style arg variables everywhere and conditionally convert
old-style substituions to new-style variables. Add some casting to the average
variable so it is treated as a number or string appropriately and bail out with
an error message when too few parameters are passed in.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agopthread: bump min stack size
Jens Axboe [Thu, 14 Jul 2016 17:36:12 +0000 (10:36 -0700)]
pthread: bump min stack size

For some reason fio fails to create a thread with the stack size
set to the minimum. For now, let's just bump it to 2x the min size,
that seems to work...

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd os_trim() support for FreeBSD
Tomohiro Kusumi [Thu, 14 Jul 2016 12:27:43 +0000 (21:27 +0900)]
Add os_trim() support for FreeBSD

It has the same interface as Linux kernel with a different ioctl name.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd os_trim() support for DragonFlyBSD
Tomohiro Kusumi [Thu, 14 Jul 2016 12:27:42 +0000 (21:27 +0900)]
Add os_trim() support for DragonFlyBSD

It has the same interface as Linux kernel with a different ioctl name.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'fix_verify' of https://github.com/charles-jacobsen/fio
Jens Axboe [Wed, 13 Jul 2016 16:13:46 +0000 (09:13 -0700)]
Merge branch 'fix_verify' of https://github.com/charles-jacobsen/fio

3 years agoverify: Reset verify_state before verification phase.
Charlie Jacobsen [Fri, 8 Jul 2016 17:22:29 +0000 (11:22 -0600)]
verify: Reset verify_state before verification phase.

verify_state needs to be reset in order for random seeds in
the verification headers and expected random seeds to match.

Signed-off-by: Charlie Jacobsen <charles.jacobsen@primarydata.com>
3 years agoworkqueue: rename private to priv for compiling as c++
Casey Bodley [Mon, 11 Jul 2016 20:47:47 +0000 (16:47 -0400)]
workqueue: rename private to priv for compiling as c++

Given the main.cc source file:
  #include <fio.h>
  int main() { return 0; }

And the gcc command line:
  gcc main.cc -Ifio -DCONFIG_HAVE_BOOL

In file included from ./ioengine.h:10:0,
                 from ./iolog.h:7,
                 from ./stat.h:4,
                 from ./thread_options.h:7,
                 from ./fio.h:18,
                 from main.cc:1:
./workqueue.h:19:8: error: expected unqualified-id before ‘private’
  void *private;
        ^
./workqueue.h:19:7: error: expected ‘;’ at end of member declaration
  void *private;
       ^
./workqueue.h:19:15: error: expected ‘:’ before ‘;’ token
  void *private;
               ^

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agoiolog: flush_log() can be bool
Jens Axboe [Mon, 11 Jul 2016 18:51:58 +0000 (11:51 -0700)]
iolog: flush_log() can be bool

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosamples being added to the pending log were silently dropped because we failed to...
Karl Cronburg [Wed, 6 Jul 2016 19:54:10 +0000 (15:54 -0400)]
samples being added to the pending log were silently dropped because we failed to set nr_samples in the new log they get copied into

3 years agogfio: call g_thread_init() for <= 2.31.0
Jens Axboe [Tue, 5 Jul 2016 20:23:56 +0000 (14:23 -0600)]
gfio: call g_thread_init() for <= 2.31.0

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoHOWTO: remove old use cases for the net IO engine
Jens Axboe [Wed, 29 Jun 2016 19:09:15 +0000 (13:09 -0600)]
HOWTO: remove old use cases for the net IO engine

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agohelper_thread: remove impossible branch
Vincent Fu [Thu, 16 Jun 2016 18:31:52 +0000 (14:31 -0400)]
helper_thread: remove impossible branch

I believe that the intention is to call update_io_ticks every
DISK_UTIL_MSEC. Make helper_thread_main do this.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove hard-coded precision for printing JSON float values
Vincent Fu [Mon, 13 Jun 2016 16:37:24 +0000 (12:37 -0400)]
Remove hard-coded precision for printing JSON float values

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFio 2.12 fio-2.12
Jens Axboe [Mon, 13 Jun 2016 21:42:44 +0000 (15:42 -0600)]
Fio 2.12

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agostat: treat !per_unit_logs() like IO offload mode
Jens Axboe [Mon, 13 Jun 2016 19:32:55 +0000 (13:32 -0600)]
stat: treat !per_unit_logs() like IO offload mode

For IO offload mode, we regrow the logs inline. We need to do the
same for !per_unit_logs(), since we log identically for those.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: fix 'cur_log' leaks
Jens Axboe [Sun, 12 Jun 2016 03:41:13 +0000 (21:41 -0600)]
iolog: fix 'cur_log' leaks

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: allocate 'cur_log's out of shared pool
Jens Axboe [Sun, 12 Jun 2016 03:38:51 +0000 (21:38 -0600)]
iolog: allocate 'cur_log's out of shared pool

Different process, so we need to ensure we can write to them
properly.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRevert "fio: Simplify forking of processes"
Jens Axboe [Sun, 12 Jun 2016 03:27:29 +0000 (21:27 -0600)]
Revert "fio: Simplify forking of processes"

This reverts commit d7982dd0ab2a1a315b5f9859c67a02414ce6274f.

Causes a regression in the logging, looks like a problem with
shared memory. Will need to investigate further, but it's
reproducibly broken.

3 years agooptions: fix typos
Vincent Fu [Thu, 9 Jun 2016 19:30:52 +0000 (13:30 -0600)]
options: fix typos

Fixes: a275c37ab000 ("options: mark unsupported options as such")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: mark unsupported options as such
Jens Axboe [Wed, 8 Jun 2016 17:13:08 +0000 (11:13 -0600)]
options: mark unsupported options as such

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoparse: add support for unsupported options
Jens Axboe [Wed, 8 Jun 2016 16:58:07 +0000 (10:58 -0600)]
parse: add support for unsupported options

Some options are only available if fio is built with a certain
set of libraries. This can confuse the user. Add support for
listing an unsupported option, so fio can print some related help
for that option.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agostat: fix reversed check for ramp time
Jens Axboe [Tue, 7 Jun 2016 03:23:53 +0000 (21:23 -0600)]
stat: fix reversed check for ramp time

A previous commit flipped the return value of checking for
ramp time or not. Fix it up.

Fixes: 356014ff351c ("Fix ramp time breakage")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agostat: remove redundant unit log check
Jens Axboe [Mon, 6 Jun 2016 15:28:13 +0000 (09:28 -0600)]
stat: remove redundant unit log check

We only call the two functions if we know that unit logging is
disabled.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix ramp time breakage
Jens Axboe [Mon, 6 Jun 2016 15:10:24 +0000 (09:10 -0600)]
Fix ramp time breakage

A previous commit inadvertently used ramp_time_over() instead of
in_ramp_time(), which breaks some jobs with ramp time. This is
because the former function has side effects, where we really
just want to check if we're in ramp time or not.

Fixes: a47591e4923f ("Improve logging accuracy")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDocumentation: fix psyncv2 typo
Jens Axboe [Fri, 3 Jun 2016 15:00:49 +0000 (09:00 -0600)]
Documentation: fix psyncv2 typo

It's pvsync2, not psyncv2 (or other variants).

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agotools/fio_latency2csv.py: add tool that converts json+ to CSV
Vincent Fu [Fri, 3 Jun 2016 14:57:19 +0000 (08:57 -0600)]
tools/fio_latency2csv.py: add tool that converts json+ to CSV

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDocumentation update
Jens Axboe [Thu, 2 Jun 2016 22:57:20 +0000 (16:57 -0600)]
Documentation update

Ensure that pvsync2 is mentioned, and kill remnants of the syslet
engine, it is no more (and never made it to Linux mainline).

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoarch: wire up preadv2/pwritev2 for more architectures
Jens Axboe [Thu, 2 Jun 2016 22:51:57 +0000 (16:51 -0600)]
arch: wire up preadv2/pwritev2 for more architectures

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix iodepth_batch=0
Omar Sandoval [Wed, 1 Jun 2016 07:04:48 +0000 (00:04 -0700)]
Fix iodepth_batch=0

The man page claims that iodepth_batch=0 falls back to whatever was
specified for iodepth, but the enforced minimum of 1 means that 0 is not
actually valid.

Fixes: a2e6f8ac56a9 ("Make iodepth_batch=1 by default")
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoEnable preadv2/pwritev2 engines by default on Linux
Jens Axboe [Thu, 2 Jun 2016 16:54:24 +0000 (10:54 -0600)]
Enable preadv2/pwritev2 engines by default on Linux

We don't have defines in libc yet, but let's wire them up so we
can use them. Only enabled for x86 and x86-64, adding for other
architectures would be as simple as adding the right defines
for the two syscall numbers.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdded millisecond-accurate timestamp to JSON output
Ryan Hardin [Thu, 26 May 2016 20:39:41 +0000 (16:39 -0400)]
Added millisecond-accurate timestamp to JSON output

Kept original value for backward compatibility.

3 years agoserver: ensure that we flush compressed logs correctly
Jens Axboe [Fri, 27 May 2016 17:01:15 +0000 (11:01 -0600)]
server: ensure that we flush compressed logs correctly

Do chunkwise block compression, and flush at the end, adding more space
as needed.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agomutex: abstract out cond/lock pshared init
Jens Axboe [Wed, 25 May 2016 19:55:48 +0000 (13:55 -0600)]
mutex: abstract out cond/lock pshared init

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix occasional hangs on mutexes
Jan Kara [Tue, 24 May 2016 15:03:22 +0000 (17:03 +0200)]
Fix occasional hangs on mutexes

When running xfstest generic/299 using fio on my test machine using
ramdisk as a backing store, I have noticed that fio often hangs waiting
for td->io_u_lock. After some debugging I have found out the reason is
that mutexes are created as process-private by default and but this
mutex is actually manipulated from several processes. The hang is not
obvious immediately as the mutex is located in shared memory and thus
while the locking is resolved in userspace, everything works as
expected. Only once we use kernel futexes, the process is not properly
woken up when futex is released.

Fix the problem by marking all mutexes and conditional variables that
are located in shared memory as shared.

Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: Simplify forking of processes
Jan Kara [Tue, 24 May 2016 15:03:21 +0000 (17:03 +0200)]
fio: Simplify forking of processes

There is no reason to re-attach to shared memory segments after fork(2).
shmat(2) manpage specifically says:

After a fork(2), the child inherits the attached shared memory segments.

So get rid of some unnecessary code.

Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agohash: make 64-bit even on 32-bit
Jens Axboe [Wed, 25 May 2016 19:19:16 +0000 (13:19 -0600)]
hash: make 64-bit even on 32-bit

Fixes this warning on Windows, where unsigned long is 32-bit:

hash.h: In function ‘__hash_long’:
hash.h:58:2: warning: left shift count >= width of type
  n <<= 33;
  ^

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFio 2.11 fio-2.11
Jens Axboe [Wed, 25 May 2016 00:42:04 +0000 (18:42 -0600)]
Fio 2.11

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: regrow logs for sync IO engines as well
Jens Axboe [Wed, 25 May 2016 00:39:45 +0000 (18:39 -0600)]
backend: regrow logs for sync IO engines as well

Fixes: 1fed20802961 ("iolog: regrow log out-of-line")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoremove numpy and scipy dependency
Mark Nelson [Tue, 24 May 2016 16:01:04 +0000 (11:01 -0500)]
remove numpy and scipy dependency

Signed-off-by: Mark Nelson <mnelson@redhat.com>
3 years agoSpelling fix. Reported by Debian´s lintian.
Martin Steigerwald [Tue, 24 May 2016 09:17:20 +0000 (11:17 +0200)]
Spelling fix. Reported by Debian´s lintian.

3 years agoSpelling fix.
Martin Steigerwald [Tue, 24 May 2016 09:08:37 +0000 (11:08 +0200)]
Spelling fix.

3 years agoSpelling fix. Reported by Debian´s lintian.
Martin Steigerwald [Tue, 24 May 2016 09:08:06 +0000 (11:08 +0200)]
Spelling fix. Reported by Debian´s lintian.

3 years agoSpelling fix. Reported by Debian´s lintian.
Martin Steigerwald [Tue, 24 May 2016 09:06:49 +0000 (11:06 +0200)]
Spelling fix. Reported by Debian´s lintian.

3 years agoSpelling fix. Reported by Debian´s lintian.
Martin Steigerwald [Tue, 24 May 2016 09:06:04 +0000 (11:06 +0200)]
Spelling fix. Reported by Debian´s lintian.

3 years agoparse: warn if option is missing a long option variant
Jens Axboe [Mon, 23 May 2016 16:39:16 +0000 (10:39 -0600)]
parse: warn if option is missing a long option variant

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: add missing long option names
Jens Axboe [Mon, 23 May 2016 16:38:35 +0000 (10:38 -0600)]
options: add missing long option names

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agocconv: wire up conversion of unique_filename
Jens Axboe [Mon, 23 May 2016 14:39:41 +0000 (08:39 -0600)]
cconv: wire up conversion of unique_filename

Also bump server version, since thread_options changed.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: add 'unique_filename'
Jens Axboe [Mon, 23 May 2016 14:37:19 +0000 (08:37 -0600)]
options: add 'unique_filename'

Allows the user to disable the prefixing of network clients source
IP.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFio 2.10 fio-2.10
Jens Axboe [Sat, 21 May 2016 15:00:54 +0000 (09:00 -0600)]
Fio 2.10

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: fix potential oops in iolog disabling
Jens Axboe [Fri, 20 May 2016 20:36:34 +0000 (14:36 -0600)]
iolog: fix potential oops in iolog disabling

Do it in the function, not in the caller. The log may be NULL.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: fix two bugs in deferred growing
Jens Axboe [Fri, 20 May 2016 19:57:42 +0000 (13:57 -0600)]
iolog: fix two bugs in deferred growing

1) Setup a new current log, if the log doesn't have one.
2) Bump size of pending log. Should be depth + 1, let's just
   use the default size of a log (1024).

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'master' of https://github.com/davidzengxhsh/fio
Jens Axboe [Fri, 20 May 2016 17:22:37 +0000 (11:22 -0600)]
Merge branch 'master' of https://github.com/davidzengxhsh/fio

3 years agoiolog: remove dead define
Jens Axboe [Fri, 20 May 2016 17:17:37 +0000 (11:17 -0600)]
iolog: remove dead define

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: regrow log out-of-line
Jens Axboe [Fri, 20 May 2016 17:02:28 +0000 (11:02 -0600)]
iolog: regrow log out-of-line

From the completion side, when we run out of entries, store in a
temporary location that we size at log init time. Mark 'td' as
needing log regrow, and handle that on the submission side. We
can't easily quiesce and grow logs at completion time, since
it'll potentially recurse. On top of that, the IO engines rely
on the fact that we serialize getevents/completions, we can't
nest them.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoThe fixed CPU architecture in the Makefile will make failure on ppc64le.
David Zeng [Fri, 20 May 2016 09:41:14 +0000 (17:41 +0800)]
The fixed CPU architecture in the Makefile will make failure on ppc64le.

3 years agoiolog: fix duplicate handling of compression end
Jens Axboe [Thu, 19 May 2016 21:49:57 +0000 (15:49 -0600)]
iolog: fix duplicate handling of compression end

Also handle an error return from deflate() with Z_FINISH.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: fix bug with ret != Z_STREAM_END
Jens Axboe [Thu, 19 May 2016 21:35:18 +0000 (15:35 -0600)]
iolog: fix bug with ret != Z_STREAM_END

Before adding more chunks, adjust the previous chunk size. This
fixes an inflate issue, where we'd get Z_DATA_ERROR because the
middle chunk did not have the correct length.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: more compression debugging/fixes
Jens Axboe [Thu, 19 May 2016 21:00:21 +0000 (15:00 -0600)]
iolog: more compression debugging/fixes

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: sum last chunk length to total
Jens Axboe [Thu, 19 May 2016 20:36:21 +0000 (14:36 -0600)]
iolog: sum last chunk length to total

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: memset() zstream at init time
Jens Axboe [Thu, 19 May 2016 18:44:06 +0000 (12:44 -0600)]
iolog: memset() zstream at init time

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: fix missing new-line in inflate debug statement
Jens Axboe [Thu, 19 May 2016 18:43:19 +0000 (12:43 -0600)]
iolog: fix missing new-line in inflate debug statement

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: move iolog compression init before CPU affinity settings
Jens Axboe [Thu, 19 May 2016 17:41:28 +0000 (11:41 -0600)]
backend: move iolog compression init before CPU affinity settings

Previously we did this after we had set CPU affinity for the IO
worker. But if we do that, then we limit the log compression
to the exact CPUs that we don't want it to run on.

So let's do this earlier, which means the compression threads can
run anyway. We'll trust the OS scheduler to keep them out of our
way.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: don't quiesce on completion
Jens Axboe [Thu, 19 May 2016 17:24:30 +0000 (11:24 -0600)]
iolog: don't quiesce on completion

We can infinitely recurse if we do that, if log_compression= is set.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: switch to list based scheme
Jens Axboe [Thu, 19 May 2016 15:20:16 +0000 (09:20 -0600)]
iolog: switch to list based scheme

The previous iolog implementation had one big log, that we continually
resized when we ran out of space. The resize involved a realloc()
of the data, which could potentially be slow for big logs. This could
introduce long latency outliers for workloads, since fio did not
quiesce IO before resizing the log.

Rewrite the logging code to have a list of logs, and cap each of them
at a reasonable size. This means we never have to realloc data, and
that we allocate smaller chunks instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: dump state of stuck thread
Jens Axboe [Thu, 19 May 2016 15:19:56 +0000 (09:19 -0600)]
backend: dump state of stuck thread

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: mark the thread as finishing, when we are out of the IO loop
Jens Axboe [Thu, 19 May 2016 15:19:03 +0000 (09:19 -0600)]
backend: mark the thread as finishing, when we are out of the IO loop

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: only do forceful timeout exit if the job isn't actively finishing
Jens Axboe [Thu, 19 May 2016 14:49:31 +0000 (08:49 -0600)]
backend: only do forceful timeout exit if the job isn't actively finishing

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoflist: add flist_last_entry()
Jens Axboe [Thu, 19 May 2016 14:25:05 +0000 (08:25 -0600)]
flist: add flist_last_entry()

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofilesetup: align a size given as a percentage to the block size
Jens Axboe [Wed, 18 May 2016 21:06:05 +0000 (15:06 -0600)]
filesetup: align a size given as a percentage to the block size

We want it to be a nice multiple of (at least) our minimum block
size.

Signed-off-by: Jens Axboe <axboe@fb.com>