fio.git
3 years agoHOWTO: modernize output examples and descriptions
Sitsofe Wheeler [Fri, 23 Jun 2017 06:01:50 +0000 (07:01 +0100)]
HOWTO: modernize output examples and descriptions

- Update output examples to reflect what is produced by the current fio
- Add hidden comments saying how to generate output for fio examples.
- Update output descriptions to match examples.
- Add missing thread lifecycle states.

3 years agoHOWTO: Reword Log File Formats and add reference
Sitsofe Wheeler [Fri, 23 Jun 2017 23:17:03 +0000 (00:17 +0100)]
HOWTO: Reword Log File Formats and add reference

- Add reference to Log File Formats in log_avg_msec
- Reword Log File Formats for improved clarity.
- In Log File Formats note that IOPS will always be 1 when using the
  default of individual I/O logging.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: reword iodepth and submit distribution text
Sitsofe Wheeler [Wed, 21 Jun 2017 06:29:54 +0000 (07:29 +0100)]
HOWTO: reword iodepth and submit distribution text

Reword the explanation text so phrasing of iodepth and submit
distributions is more similar. Also draw attention to the fact the value
represents the end-point in iodepth distributions and the start-point in
submit distributions.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: reorder client/server phrasing
Sitsofe Wheeler [Fri, 23 Jun 2017 22:34:51 +0000 (23:34 +0100)]
HOWTO: reorder client/server phrasing

Reorder sentences so that backend is in the same position as server and the
frontend is in the same position as client.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add some markup
Sitsofe Wheeler [Fri, 23 Jun 2017 22:31:52 +0000 (23:31 +0100)]
HOWTO: add some markup

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add rate example
Sitsofe Wheeler [Fri, 23 Jun 2017 22:27:23 +0000 (23:27 +0100)]
HOWTO: add rate example

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add --output-format=terse as another way to get minimal output
Sitsofe Wheeler [Fri, 23 Jun 2017 06:37:24 +0000 (07:37 +0100)]
HOWTO: add --output-format=terse as another way to get minimal output

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: reword HDFS description
Sitsofe Wheeler [Fri, 23 Jun 2017 06:36:42 +0000 (07:36 +0100)]
HOWTO: reword HDFS description

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: make filesize syntax show it can take a typed range
Sitsofe Wheeler [Fri, 23 Jun 2017 06:35:47 +0000 (07:35 +0100)]
HOWTO: make filesize syntax show it can take a typed range

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: Fix some capitalisation
Sitsofe Wheeler [Fri, 23 Jun 2017 06:35:14 +0000 (07:35 +0100)]
HOWTO: Fix some capitalisation

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: description rewording/fixes
Sitsofe Wheeler [Fri, 23 Jun 2017 06:27:31 +0000 (07:27 +0100)]
HOWTO: description rewording/fixes

Reword descriptions to make them easier to understand, fix
typos/grammar, fix formatting and reduce duplication.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: minor internal/reordering/formatting changes
Sitsofe Wheeler [Fri, 23 Jun 2017 06:08:54 +0000 (07:08 +0100)]
HOWTO: minor internal/reordering/formatting changes

3 years agoHOWTO: general consistency
Sitsofe Wheeler [Fri, 23 Jun 2017 05:53:07 +0000 (06:53 +0100)]
HOWTO: general consistency

- Prefer built in to builtin
- Link to man page when mentioning syscalls.
- Standardize on behavior (grumble) rather than behaviour.
- Prefer hostname to host name.
- Prefer filename to file name.
- Prefer timestamps to time stamps.
- Prefer bit mask to bitmask.
- Prefer zeros to zeroes.
- Reorder comparison operators so they work ;-)
- Fix up an unwanted apostrophe too.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: update command line option descriptions
Sitsofe Wheeler [Fri, 23 Jun 2017 05:36:45 +0000 (06:36 +0100)]
HOWTO: update command line option descriptions

- Make it clearer what --help will print (since page is ambiguous in
  the context of the HOWTO).
- Update the wording and syntax for --crctest so it's clear it doesn't
  always need a parameter.
- Use KiB rather than kb to refer to powers of two byte sizes and update
  the maximum number of pools.
- Make the --idle-prof description more verbose (and clear up confusion
  that might make people try and pass it literal option
  'system,percpu').

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: update directory and filename option descriptions
Sitsofe Wheeler [Thu, 22 Jun 2017 06:22:38 +0000 (07:22 +0100)]
HOWTO: update directory and filename option descriptions

- Minor grammar changes.
- Explictly note that '\' and ':' need escaping.
- Turn reference to filename_format into link.
- Split escaping into two paragraphs and add another Windows example.
- Make it clearer '-' can be a filename
- Markup stdin/stdout.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: update time specification
Sitsofe Wheeler [Thu, 22 Jun 2017 06:20:34 +0000 (07:20 +0100)]
HOWTO: update time specification

Add that time may take a suffix for day and list all time suffixes in
the same format.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: escape =
Sitsofe Wheeler [Thu, 22 Jun 2017 05:49:51 +0000 (06:49 +0100)]
HOWTO: escape =

We often don't want a space to be put before an equals sign. Workaround
Sphinx doing this by escaping the equals with a '\'.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: grammar/spelling changes
Sitsofe Wheeler [Thu, 22 Jun 2017 05:15:24 +0000 (06:15 +0100)]
HOWTO: grammar/spelling changes

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: state default time unit
Sitsofe Wheeler [Thu, 22 Jun 2017 05:55:24 +0000 (06:55 +0100)]
HOWTO: state default time unit

Where an fio option takes a time, state how the value will be
interpreted if the unit is left off.

Also remove explanation of time units in startdelay as it's clear time
can take different units already.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add defaults
Sitsofe Wheeler [Fri, 23 Jun 2017 22:57:19 +0000 (23:57 +0100)]
HOWTO: add defaults

- Update defaults.
- Linkify reference to end_fsync.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agostat: fix alignment of the iops stats
Jens Axboe [Mon, 26 Jun 2017 14:50:49 +0000 (08:50 -0600)]
stat: fix alignment of the iops stats

Fixes: 54c05828da4 ("stat: Print number of samples in bw and iops stats")
Reported-by: Elliott, Robert (Persistent Memory) <elliott@hpe.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoAndroid: fix missing sysmacros.h include
Jens Axboe [Sun, 25 Jun 2017 15:51:13 +0000 (09:51 -0600)]
Android: fix missing sysmacros.h include

Fix from Elliott Hughes and Sitsofe Wheeler.

Fixes: https://github.com/axboe/fio/issues/367
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Add iops stat and sample number information to terse format
Andreas Herrmann [Fri, 23 Jun 2017 22:27:38 +0000 (00:27 +0200)]
stat: Add iops stat and sample number information to terse format

This is inserted beyond bandwidth information. It required creation
of a new terse version (v5).

Updated man page and HOWTO accordingly. Also added hints when fields
are specific to certain terse versions.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Merge show_thread_status_terse_* functions
Andreas Herrmann [Fri, 23 Jun 2017 22:27:37 +0000 (00:27 +0200)]
stat: Merge show_thread_status_terse_* functions

Merge show_thread_status_terse_v2 and show_thread_status_terse_v3_v4
into new function show_thread_status_terse_all.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Print number of samples in bw and iops stats
Andreas Herrmann [Fri, 23 Jun 2017 22:27:36 +0000 (00:27 +0200)]
stat: Print number of samples in bw and iops stats

This adds the actual number of samples for bw and iops stats printed
in normal and json fio output. Example for normal output:

   bw (  MiB/s): min=  120, max= 3304, per=0.10%, avg=2659.34, stdev=1085.25, samples=  109
   iops        : min=102776, max=830416, avg=675639.05, stdev=280619.99, samples=   21

Note: This example was created using options
"--write_bw_log=... --log_avg_msec=100 --runtime=11s" (ie. no iops log
written). That is why number of samples differs for IOPs and BW.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Print one-line iops stat
Andreas Herrmann [Fri, 23 Jun 2017 22:27:35 +0000 (00:27 +0200)]
stat: Print one-line iops stat

This adds basic iops statistics (likewise to what we already have for
bandwidth) to normal and json fio output. Example for normal output:

   bw (  KiB/s): min=42192, max=162068, per=0.10%, avg=93236.21, stdev=22180.02
   iops : min=21090, max=81020, avg=46606.84, stdev=11088.40
    lat (usec) : 2=55.13%, 4=21.11%, 10=0.36%, 20=0.03%, 50=23.14%

Added handling for iops_stat to client/server and bump server version.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agosmalloc: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:08:01 +0000 (01:08 +0300)]
smalloc: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoverify: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:08:00 +0000 (01:08 +0300)]
verify: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoclient: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:59 +0000 (01:07 +0300)]
client: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoinit: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:58 +0000 (01:07 +0300)]
init: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

(threads is a pointer to struct thread_data)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoio_u: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:57 +0000 (01:07 +0300)]
io_u: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoserver: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:56 +0000 (01:07 +0300)]
server: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMakefile: use fmt(1) rather than tr(1) on NetBSD/etc
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:55 +0000 (01:07 +0300)]
Makefile: use fmt(1) rather than tr(1) on NetBSD/etc

e291cff1 (Use fmt -w WIDTH option instead of -WIDTH) changed the
command option to a common one to suppress an error, but it only
fixed the unknown option error without being functional on NetBSD.

This change is taken from NetBSD's pkgsrc. It may work against
other platforms that don't work with fmt(1), but only enabled for
NetBSD at the moment. FreeBSD/DragonFlyBSD/OpenBSD work with fmt(1).

(This actually works on Linux too, but the existing one should be
kept for Linux (and other platforms) provided it has been used on
various distros for years)

https://github.com/NetBSD/pkgsrc/blob/trunk/benchmarks/fio/patches/patch-Makefile

> Convert the fmt(1) command to a tr(1) one (the fmt(1) old syntax
> command is not supported on all Unix systems).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/time-test: cleanups
Jens Axboe [Fri, 23 Jun 2017 01:29:35 +0000 (19:29 -0600)]
t/time-test: cleanups

Kill dead code. It's in git history now, so can easily be resurrected.

Fixup the worst style offenses.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoiolog: ensure proper flushing of compressed logs
Jens Axboe [Thu, 22 Jun 2017 23:15:27 +0000 (17:15 -0600)]
iolog: ensure proper flushing of compressed logs

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoiolog: punt freeing of data back to original thread
Jens Axboe [Thu, 22 Jun 2017 22:20:36 +0000 (16:20 -0600)]
iolog: punt freeing of data back to original thread

This is a bit of a hack, since we can't touch the memory passed in.
But we should, at most have 1-2 entries deferred, so having an
array of 8 should suffice. Add a single-trigger warning if that
logic fails.

This should fix corruption on using log compression, when we have
to do runtime compressing.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoiolog: get work items out of shared memory pool
Jens Axboe [Thu, 22 Jun 2017 21:50:01 +0000 (15:50 -0600)]
iolog: get work items out of shared memory pool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoclient/server: bool conversion
Jens Axboe [Thu, 22 Jun 2017 20:10:23 +0000 (14:10 -0600)]
client/server: bool conversion

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoarch: tsc_reliable can be a bool
Jens Axboe [Thu, 22 Jun 2017 20:12:29 +0000 (14:12 -0600)]
arch: tsc_reliable can be a bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agocrc32c: use bool
Jens Axboe [Thu, 22 Jun 2017 20:03:14 +0000 (14:03 -0600)]
crc32c: use bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'nsec'
Jens Axboe [Thu, 22 Jun 2017 15:15:14 +0000 (09:15 -0600)]
Merge branch 'nsec'

Pull in the work from Vincent on enabling nanosecond resolution.

3 years agoFixup some style issues
Jens Axboe [Thu, 22 Jun 2017 15:15:03 +0000 (09:15 -0600)]
Fixup some style issues

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'nanosecond-2stage' of https://github.com/vincentkfu/fio into nsec
Jens Axboe [Thu, 22 Jun 2017 15:05:28 +0000 (09:05 -0600)]
Merge branch 'nanosecond-2stage' of https://github.com/vincentkfu/fio into nsec

Pull the nsec branch from Vincent.

3 years agoserver: bump server version for the change to FIO_IO_U_PLAT_GROUP_NR
Vincent Fu [Wed, 21 Jun 2017 14:42:10 +0000 (10:42 -0400)]
server: bump server version for the change to FIO_IO_U_PLAT_GROUP_NR

3 years agogettime: drop tv_valid->last_cycles and tv_valid->last_tv_valid
Vincent Fu [Tue, 20 Jun 2017 16:12:56 +0000 (12:12 -0400)]
gettime: drop tv_valid->last_cycles and tv_valid->last_tv_valid

The two variables were set on every call to fio_gettime that uses the CPU
clock. However, these values were never read anywhere. So we should drop
them to reduce the amount of work done in the fast path.

3 years agogettime: for better accuracy calculate cycles_per_msec instead of cycles_per_usec
Vincent Fu [Mon, 19 Jun 2017 19:12:36 +0000 (15:12 -0400)]
gettime: for better accuracy calculate cycles_per_msec instead of cycles_per_usec

fio (before the nsec changes) would finish about 9 sec early for a 16 hour job
when relying on the CPU clock. This is because fio calculates cycles_per_usec
to carry out the clock ticks to time conversion. cycles_per_usec only provides
4 significant digits. Changing this to cycles_per_msec provides 7 significant
digits and makes the actual job run time more closely match wall time.

3 years agonanosecond: update t/time-test.c to include experiments using seqlock for conversion
Vincent Fu [Mon, 1 May 2017 18:55:21 +0000 (14:55 -0400)]
nanosecond: update t/time-test.c to include experiments using seqlock for conversion

3 years agolib/seqlock: #include "types.h" for bool type
Vincent Fu [Thu, 27 Apr 2017 19:09:13 +0000 (15:09 -0400)]
lib/seqlock: #include "types.h" for bool type

3 years agonanosecond: add test program t/time-test for experimenting with cpu clock ticks to...
Vincent Fu [Tue, 11 Apr 2017 19:57:15 +0000 (15:57 -0400)]
nanosecond: add test program t/time-test for experimenting with cpu clock ticks to nsec conversion

3 years agonanosecond: fix up conversion of ticks to nsec by doing the conversion in 2 stages
Vincent Fu [Tue, 11 Apr 2017 17:04:08 +0000 (13:04 -0400)]
nanosecond: fix up conversion of ticks to nsec by doing the conversion in 2 stages

The ticks to nsec conversion is challenging because we care about small
differences between very large numbers. The solution used here is to do the
conversion in two stages.

The first stage carves out large chunks of time. For this patch each chunk is
approximately an hour long. The number of ticks per hour is calculated. When a
conversion is requested, an hour worth of time is accumulated each time this
number of ticks has elapsed.

The remaining ticks are handled in the second stage with the standard
multiplication and shift conversion.

This strategy deals with the multiplication-and-shift overflow problem by
limiting the scale of this operation to at most one hour. This way we can have
reasonable accuracy with this portion of the conversion. At the same time the
first stage allows the ticks to nsec conversion to be valid for long periods
of time.

3 years agonanosecond: fiologparser_hist set default --group_nr to 29 to match stat.h FIO_IO_U_P...
Vincent Fu [Wed, 22 Mar 2017 15:01:16 +0000 (11:01 -0400)]
nanosecond: fiologparser_hist set default --group_nr to 29 to match stat.h FIO_IO_U_PLAT_GROUP_NR

3 years agonanosecond: alter gfio to accommodate nanosecond timing
Vincent Fu [Tue, 21 Mar 2017 18:58:50 +0000 (14:58 -0400)]
nanosecond: alter gfio to accommodate nanosecond timing

Some of the plot labels need to be changed because they might
have values with nsec units.

3 years agonanosecond: reconcile terse output with nanosecond timing for latencies
Vincent Fu [Tue, 21 Mar 2017 15:22:08 +0000 (11:22 -0400)]
nanosecond: reconcile terse output with nanosecond timing for latencies

The terse output format is unchanged and latencies are still reported
as usec. Completion latencies are tracked with nsec precision in io_u_lat_n.
Put these latencies in the first microsecond latency bucket which is where they
would have fallen before the change to nsec timing.

3 years agonanosecond: update completion latency recording and normal, json output to use nanose...
Vincent Fu [Thu, 16 Mar 2017 16:11:46 +0000 (12:11 -0400)]
nanosecond: update completion latency recording and normal, json output to use nanoseconds

A new set of nsec bins (0-1, 2-3, 4-9, 10-19, ...) is created for tracking latencies.
The number of io_u_plat[] latency bins is increased by about 50% to accommodate the
greater resolution while retaining the same max latency value of ~9 seconds.

Example output

test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-2.20-39-g83b3
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=14.6GiB/s,w=0KiB/s][r=3832k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=17621: Thu Jun  1 15:09:38 2017
   read: IOPS=3784k, BW=14.4GiB/s (15.5GB/s)(144GiB/10000msec)
    clat (nsec): min=15, max=1685.2k, avg=26.92, stdev=433.62
     lat (nsec): min=46, max=5989.9k, avg=67.31, stdev=1659.58
    clat percentiles (nsec):
     |  1.00th=[   18],  5.00th=[   18], 10.00th=[   19], 20.00th=[   19],
     | 30.00th=[   22], 40.00th=[   22], 50.00th=[   25], 60.00th=[   26],
     | 70.00th=[   26], 80.00th=[   27], 90.00th=[   36], 95.00th=[   41],
     | 99.00th=[   46], 99.50th=[   50], 99.90th=[  104], 99.95th=[  139],
     | 99.99th=[ 7840]
   bw (  MiB/s): min=13393, max=15647, per=0.10%, avg=14784.86, stdev=558.40
    lat (nsec) : 20=20.87%, 50=78.57%, 100=0.43%, 250=0.10%, 500=0.01%
    lat (nsec) : 750=0.01%, 1000=0.01%
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%
  cpu          : usr=99.72%, sys=0.12%, ctx=321, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=37840231,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=14.4GiB/s (15.5GB/s), 14.4GiB/s-14.4GiB/s (15.5GB/s-15.5GB/s), io=144GiB (155GB), run=10000-10000msec

3 years agonanosecond: initial commit changing timeval to timespec
Vincent Fu [Tue, 14 Mar 2017 18:41:26 +0000 (14:41 -0400)]
nanosecond: initial commit changing timeval to timespec

struct timeval has usec resolution whereas struct timespec has nsec resolution.
This commit changes several dozen instances of struct timeval to struct timespec.
All of the output is unchanged.

The ticks to nsec conversion in this patch is broken but fixed in a later patch.

3 years agoEnsure that thread_stat alignment is correct
Omri Mor [Mon, 19 Jun 2017 22:41:51 +0000 (16:41 -0600)]
Ensure that thread_stat alignment is correct

clang seems to align this weirdly, and then fault when it
isn't 128-bit aligned. Align it explicitly, so it does the
right thing.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoUpdate write_hint mechanism to latest API
Jens Axboe [Mon, 19 Jun 2017 16:31:11 +0000 (10:31 -0600)]
Update write_hint mechanism to latest API

Kill the fadvise stream stuff, we never used it. Also dump the
pvsync2 options.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'dev/doc/pkg' of https://github.com/bobsaintcool/fio
Jens Axboe [Sun, 18 Jun 2017 16:32:11 +0000 (10:32 -0600)]
Merge branch 'dev/doc/pkg' of https://github.com/bobsaintcool/fio

3 years ago<README/pkg: Add reference to Arch Linux package>
Quentin Bourgeois [Sat, 17 Jun 2017 10:38:37 +0000 (12:38 +0200)]
<README/pkg: Add reference to Arch Linux package>

3 years agoFio 2.21 fio-2.21
Jens Axboe [Thu, 15 Jun 2017 18:25:03 +0000 (12:25 -0600)]
Fio 2.21

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agosync: add support for write life time hint
Jens Axboe [Wed, 14 Jun 2017 15:55:40 +0000 (09:55 -0600)]
sync: add support for write life time hint

Not documented yet, as the interface may change. This matches
what I currently have implemented on the kernel side.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agobinject: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:11 +0000 (23:02 +0300)]
binject: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosplice: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:10 +0000 (23:02 +0300)]
splice: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:09 +0000 (23:02 +0300)]
sg: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: drop redundant void* cast
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:08 +0000 (23:02 +0300)]
sg: drop redundant void* cast

No need to cast to void*.
This is the only void* cast for memcpy(3) in the entire fio.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: drop unneeded strdup from ->errdetails() handler
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:07 +0000 (23:02 +0300)]
sg: drop unneeded strdup from ->errdetails() handler

*ret as well as strdup aren't really necessary, but only make the code
complicated as shown in the previous commit. No functional change.

This commit includes reverting of the previous commit.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: add missing free(msg); in ->errdetails() handler
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:06 +0000 (23:02 +0300)]
sg: add missing free(msg); in ->errdetails() handler

msg gets malloc'd unconditionally, so we need to free msg if we're
not using msg for ret (strdup isn't really needed here to begin with).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoman: Update buffer_pattern entry in man pages
Stephen Bates [Thu, 8 Jun 2017 18:10:10 +0000 (12:10 -0600)]
man: Update buffer_pattern entry in man pages

Update the fio man page to reflect the new usage of buffer_pattern.

Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooslib/strndup: cleanup and remember to include for aux programs
Jens Axboe [Thu, 8 Jun 2017 16:07:55 +0000 (10:07 -0600)]
oslib/strndup: cleanup and remember to include for aux programs

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoconfigure: remember to initialize 'strndup' to "no"
Jens Axboe [Thu, 8 Jun 2017 15:58:40 +0000 (09:58 -0600)]
configure: remember to initialize 'strndup' to "no"

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoAdd strndup() function, if we don't have it
Jens Axboe [Thu, 8 Jun 2017 15:37:52 +0000 (09:37 -0600)]
Add strndup() function, if we don't have it

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agopattern: Add support for files in buffer_pattern argument.
Stephen Bates [Thu, 8 Jun 2017 11:46:46 +0000 (05:46 -0600)]
pattern: Add support for files in buffer_pattern argument.

It is useful to be able to initialize the buffer contents using an
input file (e.g. when testing certain pathological patterns for
compression). Add support to the buffer_pattern input argument for
reading data from a file via enclosing the filename in ``''``.

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-By: Muli Ben-Yehuda <muli@lightbitslabs.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoHOWTO: mention some details of ignore_error= option
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:49 +0000 (23:18 +0300)]
HOWTO: mention some details of ignore_error= option

ignore_error= and continue_on_error= are related in their
implementation, and easier to understand how they differ by
mentioning to it.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agouse ARRAY_SIZE() for ignore_error_nr[etype]
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:48 +0000 (23:18 +0300)]
use ARRAY_SIZE() for ignore_error_nr[etype]

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agouse enum error_type_bit for ignore_error index
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:47 +0000 (23:18 +0300)]
use enum error_type_bit for ignore_error index

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agodon't leave ignore_error_nr[etype] with 4 on blank input or error
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:46 +0000 (23:18 +0300)]
don't leave ignore_error_nr[etype] with 4 on blank input or error

This isn't a bug since td_non_fatal_error() checks errnos to ignore
by td->o.ignore_error[etype] being NULL or not, but it's still strange
to leave td->o.ignore_error_nr[etype] with 4 (or multiple of 4 on
error) when there aren't any.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofix wrong malloc size for ignore_error buffer
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:45 +0000 (23:18 +0300)]
fix wrong malloc size for ignore_error buffer

This is supposed to be allocating the initial buffer which consists
of 4 integers to store errno values to ignore.
Also explicitly zero clear the buffer.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoblktrace: ensure that dev loop doesn't truncate name
Jens Axboe [Wed, 7 Jun 2017 16:46:40 +0000 (10:46 -0600)]
blktrace: ensure that dev loop doesn't truncate name

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agodiskutil: ensure we have enough room to not write past end
Jens Axboe [Wed, 7 Jun 2017 16:45:46 +0000 (10:45 -0600)]
diskutil: ensure we have enough room to not write past end

gcc 7.1 triggers this warning:

diskutil.c: In function ‘find_block_dir’:
diskutil.c:371:22: warning: ‘__builtin___sprintf_chk’ may write a terminating nul past the end of the destination [-Wformat-overflow=]
   sprintf(full_path, "%s/%s", path, dir->d_name);
                      ^~~~~~~
In file included from /usr/include/stdio.h:936:0,
                 from diskutil.c:1:

Ensure that it's 256+1 in length.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoFix up some style
Jens Axboe [Wed, 7 Jun 2017 14:52:47 +0000 (08:52 -0600)]
Fix up some style

The offset-as-percent was a little different than what we usually
do, clean it up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agofileset: fix double addition of file offset
Jens Axboe [Wed, 7 Jun 2017 14:48:05 +0000 (08:48 -0600)]
fileset: fix double addition of file offset

The offset-as-percentage patch inadvertently screwed up the regular
offset path, fix that up.

Fixes: 89978a6b26f8 ("allow a percent value for the offset parameter")
Fixes: https://github.com/axboe/fio/issues/382
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: correct json 'io_bytes' output
Jens Axboe [Mon, 5 Jun 2017 20:05:43 +0000 (14:05 -0600)]
stat: correct json 'io_bytes' output

Since the dawn of time, this has outputted kilobytes, not bytes.
Correct this oversight, and also add a 'io_kbytes' key that shows
the old output.

Fixes: 378
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'more-minimal-info-in-docs' of https://github.com/ianchakeres/fio
Jens Axboe [Mon, 5 Jun 2017 16:54:04 +0000 (10:54 -0600)]
Merge branch 'more-minimal-info-in-docs' of https://github.com/ianchakeres/fio

3 years agoAdded information about minimal output to documentation
Ian Chakeres [Mon, 5 Jun 2017 16:02:06 +0000 (09:02 -0700)]
Added information about minimal output to documentation

Specifically, add a single line that contains a single word short-name for each of the fields included in the terse minimal output.

This information can be used in conjunction with minimal output to more easily identify which results are associated with which field.

3 years agoMerge branch 'master' of https://github.com/cbwest3/fio
Jens Axboe [Mon, 5 Jun 2017 15:36:23 +0000 (09:36 -0600)]
Merge branch 'master' of https://github.com/cbwest3/fio

3 years agoallow a percent value for the offset parameter
Brantley West [Tue, 16 May 2017 21:10:05 +0000 (17:10 -0400)]
allow a percent value for the offset parameter

3 years agoMerge branch 'size_perc' of https://github.com/sitsofe/fio
Jens Axboe [Mon, 5 Jun 2017 15:26:56 +0000 (09:26 -0600)]
Merge branch 'size_perc' of https://github.com/sitsofe/fio

3 years agofilesetup: fix size percentage calculations when using offset
Sitsofe Wheeler [Sun, 4 Jun 2017 07:22:08 +0000 (08:22 +0100)]
filesetup: fix size percentage calculations when using offset

"When an fio job uses a non-zero offset value and a non-zero size
parameter, the working set size on disk is skewed. For a 64k file, the
job provided touches the following blocks:

dd if=/dev/zero of=/tmp/fiofile bs=64k count=1
fio --stonewall --filename=/tmp/fiofile --rw=write \
    --name=firstquarter  --size=25% --offset=0   --buffer_pattern=0x11 \
    --name=secondquarter --size=25% --offset=16k --buffer_pattern=0x22 \
    --name=thirdquarter  --size=25% --offset=32k --buffer_pattern=0x33

|1111111111111111|222222222222----|33333333--------|----------------|
0                16k              32k              48k              64k

1=written by firstquarter, 2=written by secondquarter, 3=written by
thirdquarter

It seems the expected vs. actual math on working set size is
(size_param_percent * size_of_file) vs.
(size_param_percent * (size_of_file - offset))."

Fix this by always calculating the percentage of the full file's size.
Also bound the I/O to be within the existing file's size when using a
size percentage to handle the case where the addition of an offset would
otherwise force I/O to go beyond the existing file size e.g.:

dd if=/dev/zero of=/tmp/fiofile bs=64k count=1
fio --stonewall --filename=/tmp/fiofile --rw=write \
    --name=beyond --size=100% --offset=32k --buffer_pattern=0x11

In the above, we ensure we only do 32k of I/O (rather than the 64k
implied by size=100%).

Fixes: https://github.com/axboe/fio/issues/377 ("Non-zero offset and
non-zero size skew disk working set size")

Reported-by: Brantley West <cbwest@gmail.com>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agouse correct syscall name in log_err()
Tomohiro Kusumi [Fri, 2 Jun 2017 19:21:27 +0000 (22:21 +0300)]
use correct syscall name in log_err()

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agouse true/false for bool type
Tomohiro Kusumi [Fri, 2 Jun 2017 19:21:26 +0000 (22:21 +0300)]
use true/false for bool type

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agomanpage: update URL links to HOWTO/README
Tomohiro Kusumi [Fri, 26 May 2017 19:25:11 +0000 (22:25 +0300)]
manpage: update URL links to HOWTO/README

Web frontend of the official git repository seems to have been changed
after 9040e236 (manpage: Add URL links to HOWTO/README).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: mention some default option values
Tomohiro Kusumi [Fri, 26 May 2017 19:25:10 +0000 (22:25 +0300)]
verify: mention some default option values

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: add missing free(ptr);
Tomohiro Kusumi [Fri, 26 May 2017 19:25:09 +0000 (22:25 +0300)]
verify: add missing free(ptr);

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolib/output_buffer: harden buf_output_free() and kill buf_output_clear()
Jens Axboe [Wed, 24 May 2017 16:21:27 +0000 (10:21 -0600)]
lib/output_buffer: harden buf_output_free() and kill buf_output_clear()

The clear is superfluous. Kill the caller and the function.

Ensure that we leave the output buffer in a sane state, once
buf_output_free() has been called on it.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: unify the logging handlers
Jens Axboe [Wed, 24 May 2017 16:15:20 +0000 (10:15 -0600)]
log: unify the logging handlers

Just provide one handler that does the va args and buffer
printing/allocation, and make the rest of them use that.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: ensure we don't truncate the final '\0' in the log
Jens Axboe [Wed, 24 May 2017 03:51:59 +0000 (21:51 -0600)]
log: ensure we don't truncate the final '\0' in the log

Fixes: 7d64aa48 ("log: make the logging functions handle > 1024 bytes correctly")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: make the logging functions handle > 1024 bytes correctly
Jens Axboe [Wed, 24 May 2017 03:45:31 +0000 (21:45 -0600)]
log: make the logging functions handle > 1024 bytes correctly

We simply truncate the output if it's larger than our static
buffer. Make it dynamically allocated instead, and ensure that
we loop correctly to handle the full amount asked for.

This fixes a problem with truncated output. One example is
json output, with file names (for multiple files) that are
larger than 1024 bytes. With the truncated output, we would
truncate the file name and hence also miss the terminating
'"'. The latter means the json was no longer valid, either.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: make log_buf() return how much it wrote
Jens Axboe [Wed, 24 May 2017 03:43:44 +0000 (21:43 -0600)]
log: make log_buf() return how much it wrote

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMove Linux/ppc64 specific cpu_online() to os/os-linux.h
Tomohiro Kusumi [Tue, 23 May 2017 18:44:49 +0000 (21:44 +0300)]
Move Linux/ppc64 specific cpu_online() to os/os-linux.h

cpu_online() implementation for ppc64 added to os/os.h by c5dd5d97
(powerpc: fix cpus_online() to get correct max CPU number for powerpc64)
seems Linux specific, thus it should be moved to os/os-linux.h.

For example, non standard sysconf values _SC_NPROCESSORS_ONLN and
_SC_NPROCESSORS_CONF do the same thing with libc in FreeBSD, whereas
they aren't always the same with glibc. This implies c5dd5d97 is about
Linux, and wouldn't be a general solution that needs to be in os/os.h.

sysconf(3) in FreeBSD
https://github.com/freebsd/freebsd/blob/master/lib/libc/gen/sysconf.c#L588

Another possibility is AIX in addition to Linux, but below website
has an example of both _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF
having the same value under SMT.

Topic: prtconf and sysconf produce different numbers of processors
https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014250083
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoInclude sg headers in os/os-linux.h
Tomohiro Kusumi [Tue, 23 May 2017 18:44:48 +0000 (21:44 +0300)]
Include sg headers in os/os-linux.h

Since sg is Linux kernel's SCSI subsystem specific interface, it's
better to include required headers in os/os-linux.h which also defines
FIO_HAVE_SGIO, rather than conditionally including them in os/os.h.
Android has FIO_HAVE_SGIO disabled, so no need to do this for Android.

Even if another platform implements sg (compatible) ioctls within
their SCSI driver, there is no guarantee the required kernel header
will be "linux/fs.h", which then fails to compile.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>