fio.git
5 years agofio: bump max jobs to 4k
Jens Axboe [Mon, 26 Sep 2016 07:29:25 +0000 (01:29 -0600)]
fio: bump max jobs to 4k

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agosmalloc: OOM fixups
Jens Axboe [Sun, 25 Sep 2016 20:22:56 +0000 (14:22 -0600)]
smalloc: OOM fixups

Warn directly in smalloc() instead of having the callers do it.
Add a hint on how to solve a situation where we run out of
shared memory.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobloom: don't enforce minimum entry count
Jens Axboe [Sun, 25 Sep 2016 20:10:24 +0000 (14:10 -0600)]
bloom: don't enforce minimum entry count

We defaulted to 1G of entries, which is crazy. Assume the caller
passes in the right amount.

Fixes: 1b2a83dcda75 ("file: add bloom filter to avoid quadratic lookup behavior")
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofile: add bloom filter to avoid quadratic lookup behavior
Jens Axboe [Sun, 25 Sep 2016 19:57:32 +0000 (13:57 -0600)]
file: add bloom filter to avoid quadratic lookup behavior

If we have a lot of jobs, with a lot of subjobs, and they each
have a lot of files, we get pathetic startup performance
because each file add will check all the previously added
files.

Fixes: bcbfeefa7bce ("fio: add multi directory support")
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoinit: re-call sinit() if we change the smallc pool size
Jens Axboe [Sun, 25 Sep 2016 19:56:59 +0000 (13:56 -0600)]
init: re-call sinit() if we change the smallc pool size

Just adds more pools, but that should be fine at init time.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agosmalloc: fixup --alloc-size
Jens Axboe [Sun, 25 Sep 2016 19:45:59 +0000 (13:45 -0600)]
smalloc: fixup --alloc-size

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobloom: allow to pass in whether to set bits for strings
Jens Axboe [Sun, 25 Sep 2016 19:43:51 +0000 (13:43 -0600)]
bloom: allow to pass in whether to set bits for strings

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobloom: add string version
Jens Axboe [Fri, 23 Sep 2016 17:57:00 +0000 (11:57 -0600)]
bloom: add string version

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobloom: hashes take byte lengths, not nwords
Jens Axboe [Fri, 23 Sep 2016 17:38:58 +0000 (11:38 -0600)]
bloom: hashes take byte lengths, not nwords

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobloom: use bool
Jens Axboe [Fri, 23 Sep 2016 17:33:02 +0000 (11:33 -0600)]
bloom: use bool

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agostat: check if ctime_r() ends in a newline before stripping
Jens Axboe [Wed, 21 Sep 2016 03:58:34 +0000 (21:58 -0600)]
stat: check if ctime_r() ends in a newline before stripping

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoos/windows/posix.c: ensure that ctime_r() adds a newline
Jens Axboe [Wed, 21 Sep 2016 03:57:25 +0000 (21:57 -0600)]
os/windows/posix.c: ensure that ctime_r() adds a newline

The normal POSIX versions do that. And as fio proper strips the
newline, we end up losing the last digit if the Windows version
does not add the newline.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoWindows: fix Time_tToSystemTime function to be 64-bit compliant
Josh Sinykin [Mon, 19 Sep 2016 03:17:30 +0000 (04:17 +0100)]
Windows: fix Time_tToSystemTime function to be 64-bit compliant

Use LONGLONG instead of LARGE_INTEGER to avoid problems on 64-bit
systems causing a crash.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix garbage characters in json output caused by time_buf being uninitialized
Josh Sinykin [Mon, 19 Sep 2016 03:25:02 +0000 (04:25 +0100)]
Fix garbage characters in json output caused by time_buf being uninitialized

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoworkqueue: kill SW_F_EXITED
Jens Axboe [Fri, 16 Sep 2016 18:48:32 +0000 (12:48 -0600)]
workqueue: kill SW_F_EXITED

We don't use the flag, it's set but never tested. On exit, we use
pthread_join() to wait for the thread to exit.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: dprint() casts for 32-bit warnings
Jens Axboe [Fri, 16 Sep 2016 17:22:09 +0000 (11:22 -0600)]
iolog: dprint() casts for 32-bit warnings

Cast size_t to unsigned long before printing.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFixup two compile warnings
Jens Axboe [Fri, 16 Sep 2016 17:15:59 +0000 (11:15 -0600)]
Fixup two compile warnings

- Don't put code before variables.

- uintptr_t cast to fix a warning on 32-bit

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFio 2.14 fio-2.14
Jens Axboe [Fri, 16 Sep 2016 17:02:09 +0000 (11:02 -0600)]
Fio 2.14

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoconfigure: harden clockid_t test
Jens Axboe [Fri, 16 Sep 2016 16:08:15 +0000 (10:08 -0600)]
configure: harden clockid_t test

Don't assume we can assign '0' to a clockid_t type, let's just
memset it to be on the safe side.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agomac: fix for 10.12 having clockid_t
Jens Axboe [Fri, 16 Sep 2016 16:05:44 +0000 (10:05 -0600)]
mac: fix for 10.12 having clockid_t

Previous versions did not have this type, so we defined our own.
As of 10.12 it does seem to, so add a configure test for this.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoclient: coding style fixups
Jens Axboe [Fri, 16 Sep 2016 15:27:41 +0000 (09:27 -0600)]
client: coding style fixups

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'client-unique-log-names' of https://github.com/bengland2/fio into log...
Jens Axboe [Fri, 16 Sep 2016 15:26:30 +0000 (09:26 -0600)]
Merge branch 'client-unique-log-names' of https://github.com/bengland2/fio into log-unique

5 years agoio_u: fix overflow in 64-bit bssplit calculation
Jens Axboe [Fri, 16 Sep 2016 15:11:04 +0000 (09:11 -0600)]
io_u: fix overflow in 64-bit bssplit calculation

If we use a random generator that has a 64-bit output, then we'll
overflow in the calculation for what block size to select when
bssplit= is used.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'master' of https://github.com/jan--f/fio
Jens Axboe [Thu, 15 Sep 2016 13:57:38 +0000 (07:57 -0600)]
Merge branch 'master' of https://github.com/jan--f/fio

5 years agoadd simple fio.service to start fio server with systemd
Jan Fajerski [Thu, 15 Sep 2016 10:57:35 +0000 (12:57 +0200)]
add simple fio.service to start fio server with systemd

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
5 years agoclient: ignore SEND_ETA, if we can't fin a reply command
Jens Axboe [Tue, 13 Sep 2016 15:08:00 +0000 (09:08 -0600)]
client: ignore SEND_ETA, if we can't fin a reply command

This can happen if we time out a reply, but the the reply comes
in a bit later. For that case, we already decremented the ETA
count, so we should just ignore the late reply.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMakefile: fixup java path for libhdfs
Jens Axboe [Mon, 12 Sep 2016 14:10:45 +0000 (08:10 -0600)]
Makefile: fixup java path for libhdfs

Fixes: 8dc1a870973e ("The fixed CPU architecture in the Makefile will make failure on ppc64le")
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoinit: pass in right pointer to def thread options free
Jens Axboe [Mon, 12 Sep 2016 01:14:10 +0000 (19:14 -0600)]
init: pass in right pointer to def thread options free

Fixes: a609f12a1701 ("options: remove dependency of 'o' being first in 'td'")
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agosafety first!
Ben England [Thu, 8 Sep 2016 18:54:06 +0000 (14:54 -0400)]
safety first!

5 years agoadhere to fio coding standards
Ben England [Tue, 6 Sep 2016 17:12:07 +0000 (13:12 -0400)]
adhere to fio coding standards

5 years agogenerate unique pathname for each fio --client log file
Ben England [Tue, 6 Sep 2016 16:21:30 +0000 (12:21 -0400)]
generate unique pathname for each fio --client log file

5 years agoMakes use of configparser portable to older versions by:
Karl Cronburg [Tue, 6 Sep 2016 14:22:00 +0000 (10:22 -0400)]
Makes use of configparser portable to older versions by:
- relying on its' own NoOptionError exception
- using getter method instead of dictionary overriding
- and using readfp() as older version does not autodetect fp vs string types

Signed-off-by: Karl Cronburg <karl.cronburg@gmail.com>
5 years agoserver: bump version
Jens Axboe [Fri, 2 Sep 2016 17:24:59 +0000 (11:24 -0600)]
server: bump version

Previous histogram commit changes the client/server protocol,
bump the version.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'histogram-client-server' of https://github.com/cronburg/fio
Jens Axboe [Fri, 2 Sep 2016 17:24:15 +0000 (11:24 -0600)]
Merge branch 'histogram-client-server' of https://github.com/cronburg/fio

5 years agojesd219: fix alignment
Jeff Furlong [Mon, 29 Aug 2016 21:43:27 +0000 (15:43 -0600)]
jesd219: fix alignment

Previously we added the bssplit option to accommodate more advanced
blocksize lists.  However, blockalign appears to default to the smallest
element in the bssplit list.  If so, we must update the example JESD219
workload to explicitly specify blockalign=4k, to satisfy the JESD219
workload requirement ("data payloads greater than or equal to 4096 bytes
are aligned on 4k boundaries").  There is no explicit alignment
requirement for <4KB blocksizes, so at this time, there seems to be no
need to specify a separate alignment for some blocksizes in bssplit
list.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFIO-VERSION-GEN: fix dirty repo tracking
Jens Axboe [Mon, 29 Aug 2016 19:40:08 +0000 (13:40 -0600)]
FIO-VERSION-GEN: fix dirty repo tracking

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofilelock: bool conversion
Jens Axboe [Mon, 29 Aug 2016 17:44:32 +0000 (11:44 -0600)]
filelock: bool conversion

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agotrim: convert to bool
Jens Axboe [Mon, 29 Aug 2016 16:37:44 +0000 (10:37 -0600)]
trim: convert to bool

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobackend: check if we need to update rusage stats, if stat_mutex is busy
Jens Axboe [Fri, 26 Aug 2016 20:39:30 +0000 (14:39 -0600)]
backend: check if we need to update rusage stats, if stat_mutex is busy

Even with the fix to check if we need to update rusage happening
right before the stat_mutex lock, we can still deadlock. This
looks something like the below:

helper_thread                   job

lock(stat_mutex);
                                lock(stat_mutex);
down(td->rusage_sem);

And now are both effectively locked in an ABBA deadlock. The helper
thread is waiting for the job to update it's rusage, but the job
is stuck waiting for the stat_mutex.

Fix this by doing a trylock on the stat_mutex, and if it fails,
ensure that we update rusage.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoClient / server code for handling histograms. The server:
Karl Cronburg [Wed, 24 Aug 2016 20:12:20 +0000 (16:12 -0400)]
Client / server code for handling histograms. The server:

- Deals with subtracting consecutive histograms so that client
  doesn't have to recreate linked list.
- Uses existing IOLOG command code to create packets, checking
  for IO_LOG_TYPE_HIST when necessary.

And the client:

- Reconstructs the pointers to the histogram bins from the packet
  format, namely (sample_0, hist_0, sample_1, hist_1, ...) in lieu
  of the current (sample_0, sample_1, ...) format used for the (void *)
  cur_log->samples.
- Flushes histograms to file with updated pointer calculation to get
  ith sample, disabling subtraction in hist_sum() using a null
  pointer.

This does not cover plain-text transmission mode (i.e. when zlib is
not present during fio compilation).

Signed-off-by: Karl Cronburg <kcronbur@redhat.com>
5 years agofile: fio_files_done() can return bool
Jens Axboe [Fri, 26 Aug 2016 14:34:41 +0000 (08:34 -0600)]
file: fio_files_done() can return bool

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofile: fix numjobs > 1 and implied jobname as filename
Jens Axboe [Fri, 26 Aug 2016 03:00:55 +0000 (21:00 -0600)]
file: fix numjobs > 1 and implied jobname as filename

If we have a jobfile that looks like:

[global]
numjobs=4

[/dev/somedevice]

Then we fail jobs 2 and on, since we don't properly add those
files. Fix this by checking if we're generating a filename
based on the jobname.

Fixes: bcbfeefa7bce ("fio: add multi directory support")
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'epoch-histograms' of https://github.com/cronburg/fio
Jens Axboe [Thu, 25 Aug 2016 22:03:20 +0000 (16:03 -0600)]
Merge branch 'epoch-histograms' of https://github.com/cronburg/fio

5 years agoGive job file to fiologparser_hist.py so that it can auto detect
Karl Cronburg [Thu, 25 Aug 2016 21:37:21 +0000 (17:37 -0400)]
Give job file to fiologparser_hist.py so that it can auto detect
log_hist_msec. This commit also adds handling of unix epoch
timestamps by fiologparser_hist.py.

Signed-off-by: Karl Cronburg <kcronbur@redhat.com>
5 years agostat: don't trust per_unit_log() if log is NULL
Jens Axboe [Mon, 22 Aug 2016 19:23:29 +0000 (13:23 -0600)]
stat: don't trust per_unit_log() if log is NULL

We return 'false' for that case, but it's not really false. Fixes
a sigbus error on sparc, without logging.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoverify: use proper include for PATH_MAX
Jens Axboe [Sat, 20 Aug 2016 16:28:57 +0000 (10:28 -0600)]
verify: use proper include for PATH_MAX

Resolves: https://github.com/axboe/fio/issues/231
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoManual page for fiologparser_hist.py and Makefile updates to install
Karl Cronburg [Thu, 18 Aug 2016 22:56:17 +0000 (18:56 -0400)]
Manual page for fiologparser_hist.py and Makefile updates to install
them.

Signed-off-by: Karl Cronburg <kcronbur@redhat.com>
5 years agoMerge branch 'epoch' of https://github.com/cronburg/fio
Jens Axboe [Thu, 18 Aug 2016 20:38:21 +0000 (14:38 -0600)]
Merge branch 'epoch' of https://github.com/cronburg/fio

5 years agoOption for changing log files to use Unix epoch instead of being
Karl Cronburg [Thu, 18 Aug 2016 16:48:47 +0000 (12:48 -0400)]
Option for changing log files to use Unix epoch instead of being
zero-based (when fio starts) epoch.

This makes it easier to analyze the data in the context of other
benchmarking tools running at the same time as fio.

Signed-off-by: Karl Cronburg <kcronbur@redhat.com>
5 years agoparse: fix void * pointer math complaint
Jens Axboe [Thu, 18 Aug 2016 16:50:33 +0000 (10:50 -0600)]
parse: fix void * pointer math complaint

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd basic write/read-and-verify example job file
Jens Axboe [Thu, 18 Aug 2016 16:03:49 +0000 (10:03 -0600)]
Add basic write/read-and-verify example job file

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofio: use the proper enum type for the shifted IO engine flags
Jens Axboe [Thu, 18 Aug 2016 14:17:53 +0000 (08:17 -0600)]
fio: use the proper enum type for the shifted IO engine flags

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix backwards reads with --size smaller than the file size
Jens Axboe [Tue, 16 Aug 2016 21:22:17 +0000 (15:22 -0600)]
Fix backwards reads with --size smaller than the file size

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agohistogram: style and list fixups
Jens Axboe [Tue, 16 Aug 2016 20:51:41 +0000 (14:51 -0600)]
histogram: style and list fixups

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'histogram-delta' of https://github.com/cronburg/fio into histogram
Jens Axboe [Tue, 16 Aug 2016 20:47:09 +0000 (14:47 -0600)]
Merge branch 'histogram-delta' of https://github.com/cronburg/fio into histogram

5 years agoMake histogram samples non-cumulative by tracking a linked-list
Karl Cronburg [Tue, 16 Aug 2016 19:44:09 +0000 (15:44 -0400)]
Make histogram samples non-cumulative by tracking a linked-list
of the most recent histogram and differencing it when we print
to the log file(s). Linked list of pointers used to minimize
runtime impact on recording side, instead choosing to do
subtraction on the logging (when logs get printed to file) side.

This helps cleanup fiologparser_hist.py, makes the log files easier
to understand at a glance, reduces file size(s), and averts the
possibility of integer overflow in the logs (for a long enough fio
job).

5 years agoVarious cleanups
Jens Axboe [Tue, 16 Aug 2016 05:36:11 +0000 (23:36 -0600)]
Various cleanups

- Killing unused declarations

- Bool

- Making code static

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogfio: fix auto-start of backend
Jens Axboe [Tue, 16 Aug 2016 04:05:27 +0000 (22:05 -0600)]
gfio: fix auto-start of backend

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agogfio: fix link error
Jens Axboe [Mon, 15 Aug 2016 20:22:31 +0000 (14:22 -0600)]
gfio: fix link error

Fixes: a89ba4b12939 ("Fixup correct sparse warnings")
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoparse: remove dead code
Jens Axboe [Mon, 15 Aug 2016 20:22:20 +0000 (14:22 -0600)]
parse: remove dead code

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoparse: get rid of __td_var()
Jens Axboe [Mon, 15 Aug 2016 18:13:57 +0000 (12:13 -0600)]
parse: get rid of __td_var()

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFixup correct sparse warnings
Jens Axboe [Mon, 15 Aug 2016 18:03:39 +0000 (12:03 -0600)]
Fixup correct sparse warnings

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoOption updates
Jens Axboe [Mon, 15 Aug 2016 17:04:30 +0000 (11:04 -0600)]
Option updates

- Missing long name variants

- libhdfs engine, use correct spelling for 'chunk', but retain
  the old spelling as an alias.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agooptions: pass in right pointer to options free
Jens Axboe [Mon, 15 Aug 2016 15:57:09 +0000 (09:57 -0600)]
options: pass in right pointer to options free

Fixes: a609f12a1701 ("options: remove dependency of 'o' being first in 'td'")
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofio: inherit IO engine flags to 'td'
Jens Axboe [Mon, 15 Aug 2016 15:42:37 +0000 (09:42 -0600)]
fio: inherit IO engine flags to 'td'

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofio: move thread_options
Jens Axboe [Mon, 15 Aug 2016 15:39:35 +0000 (09:39 -0600)]
fio: move thread_options

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agooptions: remove dependency of 'o' being first in 'td'
Jens Axboe [Mon, 15 Aug 2016 15:31:25 +0000 (09:31 -0600)]
options: remove dependency of 'o' being first in 'td'

- We currently assume that &td->o == td, but it's unfortunate that we
  have to store 3-4K of option data in the first part of 'td'.

- Kill td_var_offset(), just use offsetof() instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoOnly enable atomic io_u flag setting/clearing if we need it
Jens Axboe [Mon, 15 Aug 2016 03:31:16 +0000 (21:31 -0600)]
Only enable atomic io_u flag setting/clearing if we need it

Make it dependent on td_async_processing(), like we do for
other cases.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: silence warning on pointer cast on 32-bit compiles
Jens Axboe [Mon, 8 Aug 2016 17:32:34 +0000 (11:32 -0600)]
iolog: silence warning on pointer cast on 32-bit compiles

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoCheck if sysfs ioscheduler entry is "none"
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:31 +0000 (02:27 +0900)]
Check if sysfs ioscheduler entry is "none"

Block devices without elevator (bio-based-dm/loop/etc) have "none"
for sysfs ioscheduler tunable, and write to this entry never fail
from the way elv_iosched_store() is implemented.

This commit checks if the entry is "none" type so as not to show
an irrelevant error message which assumes the entry has a list of
available I/O schedulers.

 # cat /sys/block/dm-0/queue/scheduler
 none
 # echo deadline > /sys/block/dm-0/queue/scheduler ; echo $?
 0
 # cat /sys/block/dm-0/queue/scheduler
 none
 # echo aaa > /sys/block/dm-0/queue/scheduler ; echo $?
 0
 # cat /sys/block/dm-0/queue/scheduler
 none

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoUse a pointer to const char* for I/O engine name (in response to aa2b823c)
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:30 +0000 (02:27 +0900)]
Use a pointer to const char* for I/O engine name (in response to aa2b823c)

aa2b823c also needed to change ioengine_ops::name size to 64,
or make it a pointer to a readonly string which seems to be better
in this case as no one is to (or should be able to) modify it.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd missing FIO_NET_CMD entry to fio_server_ops[]
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:29 +0000 (02:27 +0900)]
Add missing FIO_NET_CMD entry to fio_server_ops[]

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMake local const string array static
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:28 +0000 (02:27 +0900)]
Make local const string array static

like it is in io_ddir_name().

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoChange --output-format argument from optional to required
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:27 +0000 (02:27 +0900)]
Change --output-format argument from optional to required

Not sure if there were any reason d10983d7 chose to add if(!optarg)
conditional, but this is what other fio options (or in general) do.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoUse char* for pid_file path
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:26 +0000 (02:27 +0900)]
Use char* for pid_file path

since no function expects this to be void*.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoFix spelling error
Jens Axboe [Mon, 8 Aug 2016 16:25:31 +0000 (10:25 -0600)]
Fix spelling error

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoAdd 'unlink_each_loop' option
mrturtledev [Mon, 8 Aug 2016 15:57:14 +0000 (09:57 -0600)]
Add 'unlink_each_loop' option

If this option is set, fio will unlink the files after each loop
of a job has completed.

Original patch by Martin, various tweaks and updates from me.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoioengines: fixup td_io_unlink_file() error propagation
Jens Axboe [Mon, 8 Aug 2016 15:46:06 +0000 (09:46 -0600)]
ioengines: fixup td_io_unlink_file() error propagation

Return 0 for success, error number on error. gluster already did this,
but pmemblk did not.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: hist_sum() should return unsigned long
Jens Axboe [Mon, 8 Aug 2016 13:48:33 +0000 (07:48 -0600)]
iolog: hist_sum() should return unsigned long

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agostat: fixups to histogram logging
Jens Axboe [Sun, 7 Aug 2016 21:18:38 +0000 (15:18 -0600)]
stat: fixups to histogram logging

1) Use correct size for the 'dst' passin to __add_log_sample()
2) Various style fixups

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: style updates
Jens Axboe [Sun, 7 Aug 2016 21:11:32 +0000 (15:11 -0600)]
iolog: style updates

Fixup some slightly off coding style from the histogram logging commit.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoserver: bump protocol version
Jens Axboe [Sun, 7 Aug 2016 21:08:51 +0000 (15:08 -0600)]
server: bump protocol version

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'histograms-PR' of https://github.com/cronburg/fio
Jens Axboe [Sun, 7 Aug 2016 21:07:53 +0000 (15:07 -0600)]
Merge branch 'histograms-PR' of https://github.com/cronburg/fio

5 years agoThis commit / feature adds completion latency histogram output to fio, piggybacking
Karl Cronburg [Fri, 5 Aug 2016 18:45:00 +0000 (14:45 -0400)]
This commit / feature adds completion latency histogram output to fio, piggybacking
on the existing histograms recorded by stat.c and adding the following command
line options:

- log_hist_msec operates like log_avg_msec does, but produces files
  named *_clat_hist*
- write_hist_log has the same semantics as write_lat_log but for histograms.
- log_hist_coarseness changes how many bins are logged.

The more substantial contribution is a post-processor for digesting these log files,
located in fio/tools/hist/fiologparser_hist.py, which computes min/max/avg/percentile
statistics.

To compare the accuracy of the statistics produced by fiologparser_hist.py,
a reference implementation is located here:

https://github.com/cronburg/fio/blob/histograms/tools/hist/fiologparser_numpy.py

for computing the same statistics from the *_clat.* files fio already produces.
A comparison can be viewed here:

https://cronburg.com/fio/demo/latency.html

The reference implementation is excluded from this commit, as I'm still working
with Mark (markhpc) on getting it into the branch where he's working on fiologparser.
Also I'd eventually like to make FIO_IO_U_PLAT_GROUP_NR in stat.h
configurable (allowing for a larger maximum histogram latency value than 17
seconds), but haven't added it to keep the changes to fio minimal for now.

A more detailed description of the motivation behind this feature, and explanation
of the design process and algorithms used in the post-processor can be found here:

https://cronburg.com/fio/cloud-latency-problem-measurement/

5 years agoRevert "filesetup: ensure that we catch a file flagged for extend"
Jens Axboe [Fri, 5 Aug 2016 01:40:15 +0000 (19:40 -0600)]
Revert "filesetup: ensure that we catch a file flagged for extend"

This reverts commit bcb7260a54438ab38a5928279f7e5a3a465ed1bb.

Don't think this was a functionally sound change, let's revert
to the previous behavior of create_on_open.

5 years agotravis: don't enable rbd
Jens Axboe [Wed, 3 Aug 2016 16:18:42 +0000 (10:18 -0600)]
travis: don't enable rbd

Apparently it fails with an older version of the rbd library. Need
to figure out how to check and test for that.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agotravis: add rbd/zlib
Jens Axboe [Wed, 3 Aug 2016 16:14:10 +0000 (10:14 -0600)]
travis: add rbd/zlib

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofilesetup: ensure that we catch a file flagged for extend
Jens Axboe [Wed, 3 Aug 2016 15:53:34 +0000 (09:53 -0600)]
filesetup: ensure that we catch a file flagged for extend

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoiolog: prevent early entry from skewing entire logging run
Jens Axboe [Wed, 3 Aug 2016 15:11:56 +0000 (09:11 -0600)]
iolog: prevent early entry from skewing entire logging run

If we go out of ramp time, our wakeup trigger could cause us to enter
too early in add_log_sample(). If that happens, the time diff could be
negative (so huge unsigned). Since we carry that forward, we end up
logging every entry and not just over the average window.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agofix typo in HOWTO
Jan Fajerski [Wed, 3 Aug 2016 11:35:27 +0000 (13:35 +0200)]
fix typo in HOWTO

5 years agofix typo in HOWTO
Jan Fajerski [Wed, 3 Aug 2016 11:22:16 +0000 (13:22 +0200)]
fix typo in HOWTO

5 years agoengines/rbd: fix compile without blkin support
Jens Axboe [Tue, 2 Aug 2016 21:23:43 +0000 (15:23 -0600)]
engines/rbd: fix compile without blkin support

If we don't have rbd-blkin support, compile fails with:

engines/rbd.c:21:26: error: field ‘info’ has incomplete type
  struct blkin_trace_info info;
                          ^~~~
Makefile:313: recipe for target 'engines/rbd.o' failed

Hide 'info' behind the ifdef.

Fixes: a4c4c3460bd5 ("Add support for blkin tracing in rbd engine")
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMerge branch 'wip-traceinfo' of https://github.com/vears91/fio
Jens Axboe [Tue, 2 Aug 2016 21:05:59 +0000 (15:05 -0600)]
Merge branch 'wip-traceinfo' of https://github.com/vears91/fio

5 years agoRevert "filesetup: ensure that we align file starting offset"
Jens Axboe [Tue, 2 Aug 2016 15:07:07 +0000 (09:07 -0600)]
Revert "filesetup: ensure that we align file starting offset"

There are valid reasons to use an offset that isn't a multiple
of the block size. We previously made this change to fix an
issue with a job file, but that job file had an offset that
wasn't a multiple of the hardware block size. As long as the
offset is a multiple of the hardware block size for unbuffered
IO, then it should be fine.

This reverts commit c465cd14ca958dfdb87bb1998add722d62ae3691.

5 years agobackend: do_verify() cleanup
Jens Axboe [Mon, 1 Aug 2016 19:46:17 +0000 (13:46 -0600)]
backend: do_verify() cleanup

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agobackend: do_dry_run(): get_io_u() can return an error pointer
Jens Axboe [Fri, 29 Jul 2016 15:59:38 +0000 (09:59 -0600)]
backend: do_dry_run(): get_io_u() can return an error pointer

Don't just check for NULL.

Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoUse larger local buffer for I/O engine name
Tomohiro Kusumi [Fri, 29 Jul 2016 15:06:00 +0000 (00:06 +0900)]
Use larger local buffer for I/O engine name

16 bytes may not be large enough in the future if a new I/O engine
with a name longer than strlen(name)=15 is added.

The longest one right now seems to be "fusion-aw-sync" which is 14(+1).

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoNull terminate before (or after) strncpy(3)
Tomohiro Kusumi [Fri, 29 Jul 2016 15:05:59 +0000 (00:05 +0900)]
Null terminate before (or after) strncpy(3)

These three strncpy() calls copy at most sizeof(buffer)-1 bytes,
but buffer isn't explicitly 0 cleared, so 0 terminate the last byte.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoMake switch_ioscheduler() return 0 if FIO_HAVE_IOSCHED_SWITCH is undefined
Tomohiro Kusumi [Fri, 29 Jul 2016 15:05:58 +0000 (00:05 +0900)]
Make switch_ioscheduler() return 0 if FIO_HAVE_IOSCHED_SWITCH is undefined

Defining FIO_HAVE_IOSCHED_SWITCH currently equals saying it's Linux,
as switch_ioscheduler() only works on Linux kernel with "scheduler"
sysfs entry (though read/write to sysfs obviously compiles on others).

This commit makes the function return 0 if FIO_HAVE_IOSCHED_SWITCH
is undefined (i.e. if not Linux). This is essentially the same as
{diskutil,cgroup,blktrace}.c being compiled only on Linux.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
5 years agoUse in-place path separator "/" for Linux specific code
Tomohiro Kusumi [Fri, 29 Jul 2016 15:05:57 +0000 (00:05 +0900)]
Use in-place path separator "/" for Linux specific code

diskutil,cgroup,blktrace related code fully depend on Linux kernel,
so sprintf variants can use in-place path separator "/" instead of
FIO_OS_PATH_SEPARATOR.

In fact these Linux specific files are mix of two types within the
same file depending on who wrote them, where they were originally
using "/".

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>