.SH OPTIONS
.TP
.BI \-\-debug \fR=\fPtype
-Enable verbose tracing of various fio actions. May be `all' for all types
-or individual types separated by a comma (e.g. \-\-debug=file,mem will enable
+Enable verbose tracing \fItype\fR of various fio actions. May be `all' for all \fItype\fRs
+or individual types separated by a comma (e.g. `\-\-debug=file,mem' will enable
file and memory debugging). `help' will list all available tracing options.
.TP
-.BI \-\-parse-only
+.BI \-\-parse\-only
Parse options only, don't start any I/O.
.TP
.BI \-\-output \fR=\fPfilename
Write output to \fIfilename\fR.
.TP
-.BI \-\-output-format \fR=\fPformat
-Set the reporting format to \fInormal\fR, \fIterse\fR, \fIjson\fR, or
-\fIjson+\fR. Multiple formats can be selected, separate by a comma. \fIterse\fR
-is a CSV based format. \fIjson+\fR is like \fIjson\fR, except it adds a full
+.BI \-\-output\-format \fR=\fPformat
+Set the reporting \fIformat\fR to `normal', `terse', `json', or
+`json+'. Multiple formats can be selected, separate by a comma. `terse'
+is a CSV based format. `json+' is like `json', except it adds a full
dump of the latency buckets.
.TP
-.BI \-\-runtime \fR=\fPruntime
-Limit run time to \fIruntime\fR seconds.
-.TP
-.B \-\-bandwidth\-log
+.BI \-\-bandwidth\-log
Generate aggregate bandwidth logs.
.TP
-.B \-\-minimal
-Print statistics in a terse, semicolon-delimited format.
+.BI \-\-minimal
+Print statistics in a terse, semicolon\-delimited format.
.TP
-.B \-\-append-terse
-Print statistics in selected mode AND terse, semicolon-delimited format.
-Deprecated, use \-\-output-format instead to select multiple formats.
+.BI \-\-append\-terse
+Print statistics in selected mode AND terse, semicolon\-delimited format.
+\fBDeprecated\fR, use \fB\-\-output\-format\fR instead to select multiple formats.
.TP
.BI \-\-terse\-version \fR=\fPversion
-Set terse version output format (default 3, or 2, 4, 5)
+Set terse \fIversion\fR output format (default `3', or `2', `4', `5').
.TP
-.B \-\-version
+.BI \-\-version
Print version information and exit.
.TP
-.B \-\-help
+.BI \-\-help
Print a summary of the command line options and exit.
.TP
-.B \-\-cpuclock-test
+.BI \-\-cpuclock\-test
Perform test and validation of internal CPU clock.
.TP
.BI \-\-crctest \fR=\fP[test]
-Test the speed of the built-in checksumming functions. If no argument is given,
+Test the speed of the built\-in checksumming functions. If no argument is given,
all of them are tested. Alternatively, a comma separated list can be passed, in which
case the given ones are tested.
.TP
.BI \-\-cmdhelp \fR=\fPcommand
Print help information for \fIcommand\fR. May be `all' for all commands.
.TP
-.BI \-\-enghelp \fR=\fPioengine[,command]
-List all commands defined by \fIioengine\fR, or print help for \fIcommand\fR defined by \fIioengine\fR.
-If no \fIioengine\fR is given, list all available ioengines.
+.BI \-\-enghelp \fR=\fP[ioengine[,command]]
+List all commands defined by \fIioengine\fR, or print help for \fIcommand\fR
+defined by \fIioengine\fR. If no \fIioengine\fR is given, list all
+available ioengines.
.TP
.BI \-\-showcmd \fR=\fPjobfile
-Convert \fIjobfile\fR to a set of command-line options.
+Convert \fIjobfile\fR to a set of command\-line options.
.TP
.BI \-\-readonly
-Turn on safety read-only checks, preventing writes. The \-\-readonly
+Turn on safety read\-only checks, preventing writes. The \fB\-\-readonly\fR
option is an extra safety guard to prevent users from accidentally starting
a write workload when that is not desired. Fio will only write if
-`rw=write/randwrite/rw/randrw` is given. This extra safety net can be used
-as an extra precaution as \-\-readonly will also enable a write check in
+`rw=write/randwrite/rw/randrw' is given. This extra safety net can be used
+as an extra precaution as \fB\-\-readonly\fR will also enable a write check in
the I/O engine core to prevent writes due to unknown user space bug(s).
.TP
.BI \-\-eta \fR=\fPwhen
-Specifies when real-time ETA estimate should be printed. \fIwhen\fR may
+Specifies when real\-time ETA estimate should be printed. \fIwhen\fR may
be `always', `never' or `auto'.
.TP
.BI \-\-eta\-newline \fR=\fPtime
the value is interpreted in seconds.
.TP
.BI \-\-status\-interval \fR=\fPtime
-Force full status dump every \fItime\fR period passed. When the unit is omitted,
-the value is interpreted in seconds.
+Force a full status dump of cumulative (from job start) values at \fItime\fR
+intervals. This option does *not* provide per-period measurements. So
+values such as bandwidth are running averages. When the time unit is omitted,
+\fItime\fR is interpreted in seconds.
.TP
.BI \-\-section \fR=\fPname
Only run specified section \fIname\fR in job file. Multiple sections can be specified.
-The \-\-section option allows one to combine related jobs into one file.
+The \fB\-\-section\fR option allows one to combine related jobs into one file.
E.g. one job file could define light, moderate, and heavy sections. Tell
-fio to run only the "heavy" section by giving \-\-section=heavy
+fio to run only the "heavy" section by giving `\-\-section=heavy'
command line option. One can also specify the "write" operations in one
-section and "verify" operation in another section. The \-\-section option
+section and "verify" operation in another section. The \fB\-\-section\fR option
only applies to job sections. The reserved *global* section is always
parsed and used.
.TP
.BI \-\-alloc\-size \fR=\fPkb
-Set the internal smalloc pool size to \fIkb\fP in KiB. The
-\-\-alloc-size switch allows one to use a larger pool size for smalloc.
+Set the internal smalloc pool size to \fIkb\fR in KiB. The
+\fB\-\-alloc\-size\fR switch allows one to use a larger pool size for smalloc.
If running large jobs with randommap enabled, fio can run out of memory.
Smalloc is an internal allocator for shared structures from a fixed size
memory pool and can grow to 16 pools. The pool size defaults to 16MiB.
-NOTE: While running .fio_smalloc.* backing store files are visible
-in /tmp.
+NOTE: While running `.fio_smalloc.*' backing store files are visible
+in `/tmp'.
.TP
.BI \-\-warnings\-fatal
All fio parser warnings are fatal, causing fio to exit with an error.
.TP
.BI \-\-max\-jobs \fR=\fPnr
-Set the maximum number of threads/processes to support.
+Set the maximum number of threads/processes to support to \fInr\fR.
.TP
.BI \-\-server \fR=\fPargs
-Start a backend server, with \fIargs\fP specifying what to listen to. See Client/Server section.
+Start a backend server, with \fIargs\fR specifying what to listen to.
+See \fBCLIENT/SERVER\fR section.
.TP
.BI \-\-daemonize \fR=\fPpidfile
-Background a fio server, writing the pid to the given \fIpidfile\fP file.
+Background a fio server, writing the pid to the given \fIpidfile\fR file.
.TP
.BI \-\-client \fR=\fPhostname
-Instead of running the jobs locally, send and run them on the given host or set of hosts. See Client/Server section.
+Instead of running the jobs locally, send and run them on the given \fIhostname\fR
+or set of \fIhostname\fRs. See \fBCLIENT/SERVER\fR section.
.TP
-.BI \-\-remote-config \fR=\fPfile
-Tell fio server to load this local file.
+.BI \-\-remote\-config \fR=\fPfile
+Tell fio server to load this local \fIfile\fR.
.TP
.BI \-\-idle\-prof \fR=\fPoption
-Report CPU idleness. \fIoption\fP is one of the following:
+Report CPU idleness. \fIoption\fR is one of the following:
.RS
.RS
.TP
Show aggregate system idleness and unit work.
.TP
.B percpu
-As "system" but also show per CPU idleness.
+As \fBsystem\fR but also show per CPU idleness.
.RE
.RE
.TP
-.BI \-\-inflate-log \fR=\fPlog
-Inflate and output compressed log.
+.BI \-\-inflate\-log \fR=\fPlog
+Inflate and output compressed \fIlog\fR.
.TP
-.BI \-\-trigger-file \fR=\fPfile
-Execute trigger cmd when file exists.
+.BI \-\-trigger\-file \fR=\fPfile
+Execute trigger command when \fIfile\fR exists.
.TP
-.BI \-\-trigger-timeout \fR=\fPt
-Execute trigger at this time.
+.BI \-\-trigger\-timeout \fR=\fPtime
+Execute trigger at this \fItime\fR.
.TP
-.BI \-\-trigger \fR=\fPcmd
-Set this command as local trigger.
+.BI \-\-trigger \fR=\fPcommand
+Set this \fIcommand\fR as local trigger.
.TP
-.BI \-\-trigger-remote \fR=\fPcmd
-Set this command as remote trigger.
+.BI \-\-trigger\-remote \fR=\fPcommand
+Set this \fIcommand\fR as remote trigger.
.TP
-.BI \-\-aux-path \fR=\fPpath
-Use this path for fio state generated files.
+.BI \-\-aux\-path \fR=\fPpath
+Use this \fIpath\fR for fio state generated files.
.SH "JOB FILE FORMAT"
Any parameters following the options will be assumed to be job files, unless
they match a job file parameter. Multiple job files can be listed and each job
-file will be regarded as a separate group. Fio will `stonewall` execution
+file will be regarded as a separate group. Fio will \fBstonewall\fR execution
between each group.
Fio accepts one or more job files describing what it is
*global* sections if so desired. A job is only affected by a *global* section
residing above it.
-The \-\-cmdhelp option also lists all options. If used with an `option`
-argument, \-\-cmdhelp will detail the given `option`.
+The \fB\-\-cmdhelp\fR option also lists all options. If used with an \fIcommand\fR
+argument, \fB\-\-cmdhelp\fR will detail the given \fIcommand\fR.
-See the `examples/` directory in the fio source for inspiration on how to write
-job files. Note the copyright and license requirements currently apply to
-`examples/` files.
+See the `examples/' directory for inspiration on how to write job files. Note
+the copyright and license requirements currently apply to
+`examples/' files.
.SH "JOB FILE PARAMETERS"
Some parameters take an option of a given type, such as an integer or a
string. Anywhere a numeric value is required, an arithmetic expression may be
used, provided it is surrounded by parentheses. Supported operators are:
.RS
-.RS
-.TP
+.P
.B addition (+)
-.TP
-.B subtraction (-)
-.TP
+.P
+.B subtraction (\-)
+.P
.B multiplication (*)
-.TP
+.P
.B division (/)
-.TP
+.P
.B modulus (%)
-.TP
+.P
.B exponentiation (^)
.RE
-.RE
.P
For time values in expressions, units are microseconds by default. This is
different than for time values not in expressions (not enclosed in
unless otherwise specified.
.P
With `kb_base=1000', fio follows international standards for unit
-prefixes. To specify power-of-10 decimal values defined in the
+prefixes. To specify power\-of\-10 decimal values defined in the
International System of Units (SI):
.RS
.P
-Ki means kilo (K) or 1000
-.RE
-.RS
-Mi means mega (M) or 1000**2
-.RE
-.RS
-Gi means giga (G) or 1000**3
-.RE
-.RS
-Ti means tera (T) or 1000**4
-.RE
-.RS
-Pi means peta (P) or 1000**5
-.RE
+.PD 0
+K means kilo (K) or 1000
.P
-To specify power-of-2 binary values defined in IEC 80000-13:
-.RS
+M means mega (M) or 1000**2
.P
-K means kibi (Ki) or 1024
-.RE
-.RS
-M means mebi (Mi) or 1024**2
-.RE
-.RS
-G means gibi (Gi) or 1024**3
-.RE
-.RS
-T means tebi (Ti) or 1024**4
+G means giga (G) or 1000**3
+.P
+T means tera (T) or 1000**4
+.P
+P means peta (P) or 1000**5
+.PD
.RE
+.P
+To specify power\-of\-2 binary values defined in IEC 80000\-13:
.RS
-P means pebi (Pi) or 1024**5
+.P
+.PD 0
+Ki means kibi (Ki) or 1024
+.P
+Mi means mebi (Mi) or 1024**2
+.P
+Gi means gibi (Gi) or 1024**3
+.P
+Ti means tebi (Ti) or 1024**4
+.P
+Pi means pebi (Pi) or 1024**5
+.PD
.RE
.P
With `kb_base=1024' (the default), the unit prefixes are opposite
-from those specified in the SI and IEC 80000-13 standards to provide
+from those specified in the SI and IEC 80000\-13 standards to provide
compatibility with old scripts. For example, 4k means 4096.
.P
For quantities of data, an optional unit of 'B' may be included
Examples with `kb_base=1000':
.RS
.P
+.PD 0
4 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
-.RE
-.RS
+.P
1 MiB: 1048576, 1m, 1024k
-.RE
-.RS
+.P
1 MB: 1000000, 1mi, 1000ki
-.RE
-.RS
+.P
1 TiB: 1073741824, 1t, 1024m, 1048576k
-.RE
-.RS
+.P
1 TB: 1000000000, 1ti, 1000mi, 1000000ki
+.PD
.RE
.P
Examples with `kb_base=1024' (default):
.RS
.P
+.PD 0
4 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
-.RE
-.RS
+.P
1 MiB: 1048576, 1m, 1024k
-.RE
-.RS
+.P
1 MB: 1000000, 1mi, 1000ki
-.RE
-.RS
+.P
1 TiB: 1073741824, 1t, 1024m, 1048576k
-.RE
-.RS
+.P
1 TB: 1000000000, 1ti, 1000mi, 1000000ki
+.PD
.RE
.P
To specify times (units are not case sensitive):
.RS
.P
+.PD 0
D means days
-.RE
-.RS
+.P
H means hours
-.RE
-.RS
+.P
M mean minutes
-.RE
-.RS
+.P
s or sec means seconds (default)
-.RE
-.RS
+.P
ms or msec means milliseconds
-.RE
-.RS
+.P
us or usec means microseconds
+.PD
.RE
.P
If the option accepts an upper and lower range, use a colon ':' or
-minus '-' to separate such values. See `irange` parameter type.
+minus '\-' to separate such values. See \fIirange\fR parameter type.
If the lower value specified happens to be larger than the upper value
the two values are swapped.
.RE
.TP
.I irange
Integer range with suffix. Allows value range to be given, such as
-1024-4096. A colon may also be used as the separator, e.g. 1k:4k. If the
+1024\-4096. A colon may also be used as the separator, e.g. 1k:4k. If the
option allows two sets of ranges, they can be specified with a ',' or '/'
-delimiter: 1k-4k/8k-32k. Also see `int` parameter type.
+delimiter: 1k\-4k/8k\-32k. Also see \fIint\fR parameter type.
.TP
.I float_list
A list of floating point numbers, separated by a ':' character.
.TP
.BI direct \fR=\fPbool
If value is true, use non\-buffered I/O. This is usually O_DIRECT. Note that
-ZFS on Solaris doesn't support direct I/O. On Windows the synchronous
+OpenBSD and ZFS on Solaris don't support direct I/O. On Windows the synchronous
ioengines don't support direct I/O. Default: false.
.TP
.BI atomic \fR=\fPbool
.B external
Prefix to specify loading an external I/O engine object file. Append
the engine filename, e.g. `ioengine=external:/tmp/foo.o' to load
-ioengine `foo.o' in `/tmp'.
+ioengine `foo.o' in `/tmp'. The path can be either
+absolute or relative. See `engines/skeleton_external.c' in the fio source for
+details of writing an external I/O engine.
.SS "I/O engine specific parameters"
In addition, there are some parameters which are only valid when a specific
\fBioengine\fR is in use. These are used identically to normal parameters,
\fBdisable_lat\fR.
.TP
.BI clat_percentiles \fR=\fPbool
-Enable the reporting of percentiles of completion latencies.
+Enable the reporting of percentiles of completion latencies. This option is
+mutually exclusive with \fBlat_percentiles\fR.
+.TP
+.BI lat_percentiles \fR=\fPbool
+Enable the reporting of percentiles of IO latencies. This is similar to
+\fBclat_percentiles\fR, except that this includes the submission latency.
+This option is mutually exclusive with \fBclat_percentiles\fR.
.TP
.BI percentile_list \fR=\fPfloat_list
Overwrite the default list of percentiles for completion latencies and the
| 99.99th=[78119]
bw ( KiB/s): min= 532, max= 686, per=0.10%, avg=622.87, stdev=24.82, samples= 100
iops : min= 76, max= 98, avg=88.98, stdev= 3.54, samples= 100
- lat (usec) : 250=0.04%, 500=64.11%, 750=4.81%, 1000=2.79%
- lat (msec) : 2=4.16%, 4=1.84%, 10=4.90%, 20=11.33%, 50=5.37%
- lat (msec) : 100=0.65%
+ lat (usec) : 250=0.04%, 500=64.11%, 750=4.81%, 1000=2.79%
+ lat (msec) : 2=4.16%, 4=1.84%, 10=4.90%, 20=11.33%, 50=5.37%
+ lat (msec) : 100=0.65%
cpu : usr=0.27%, sys=0.18%, ctx=12072, majf=0, minf=21
IO depths : 1=85.0%, 2=13.1%, 4=1.8%, 8=0.1%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete is basically just CPU time (I/O has already been done, see slat
explanation).
.TP
+.B lat
+Total latency. Same names as slat and clat, this denotes the time from
+when fio created the I/O unit to completion of the I/O operation.
+.TP
.B bw
Bandwidth statistics based on samples. Same names as the xlat stats,
but also includes the number of samples taken (\fIsamples\fR) and an
.B iops
IOPS statistics based on samples. Same names as \fBbw\fR.
.TP
+.B lat (nsec/usec/msec)
+The distribution of I/O completion latencies. This is the time from when
+I/O leaves fio and when it gets completed. Unlike the separate
+read/write/trim sections above, the data here and in the remaining
+sections apply to all I/Os for the reporting group. 250=0.04% means that
+0.04% of the I/Os completed in under 250us. 500=64.11% means that 64.11%
+of the I/Os required 250 to 499us for completion.
+.TP
.B cpu
CPU usage. User and system time, along with the number of context
switches this thread went through, usage of system and user time, and
The number of \fBread/write/trim\fR requests issued, and how many of them were
short or dropped.
.TP
-.B IO latencies
-The distribution of I/O completion latencies. This is the time from when
-I/O leaves fio and when it gets completed. The numbers follow the same
-pattern as the I/O \fBdepths\fR, meaning that 2=1.6% means that 1.6% of the
-I/O completed within 2 msecs, 20=12.8% means that 12.8% of the I/O took
-more than 10 msecs, but less than (or equal to) 20 msecs.
+.B IO latency
+These values are for \fBlatency-target\fR and related options. When
+these options are engaged, this section describes the I/O depth required
+to meet the specified latency target.
.RE
.P
After each client has been listed, the group statistics are printed. They
.nf
terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth;read_iops;read_runtime_ms;read_slat_min;read_slat_max;read_slat_mean;read_slat_dev;read_clat_min;read_clat_max;read_clat_mean;read_clat_dev;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min;read_lat_max;read_lat_mean;read_lat_dev;read_bw_min;read_bw_max;read_bw_agg_pct;read_bw_mean;read_bw_dev;write_kb;write_bandwidth;write_iops;write_runtime_ms;write_slat_min;write_slat_max;write_slat_mean;write_slat_dev;write_clat_min;write_clat_max;write_clat_mean;write_clat_dev;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min;write_lat_max;write_lat_mean;write_lat_dev;write_bw_min;write_bw_max;write_bw_agg_pct;write_bw_mean;write_bw_dev;cpu_user;cpu_sys;cpu_csw;cpu_mjf;cpu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util
.fi
+.SH JSON OUTPUT
+The \fBjson\fR output format is intended to be both human readable and convenient
+for automated parsing. For the most part its sections mirror those of the
+\fBnormal\fR output. The \fBruntime\fR value is reported in msec and the \fBbw\fR value is
+reported in 1024 bytes per second units.
+.fi
.SH JSON+ OUTPUT
The \fBjson+\fR output format is identical to the \fBjson\fR output format except that it
adds a full dump of the completion latency bins. Each \fBbins\fR object contains a
.RS
.TP
.B Latency log
-Value is latency in usecs
+Value is latency in nsecs
.TP
.B Bandwidth log
Value is in KiB/sec