client: use temp buffer for single output flush for json/disk util
[fio.git] / fio.1
CommitLineData
523bad63 1.TH fio 1 "August 2017" "User Manual"
d60e92d1
AC
2.SH NAME
3fio \- flexible I/O tester
4.SH SYNOPSIS
5.B fio
6[\fIoptions\fR] [\fIjobfile\fR]...
7.SH DESCRIPTION
8.B fio
9is a tool that will spawn a number of threads or processes doing a
10particular type of I/O action as specified by the user.
11The typical use of fio is to write a job file matching the I/O load
12one wants to simulate.
13.SH OPTIONS
14.TP
49da1240 15.BI \-\-debug \fR=\fPtype
7db7a5a0
TK
16Enable verbose tracing \fItype\fR of various fio actions. May be `all' for all \fItype\fRs
17or individual types separated by a comma (e.g. `\-\-debug=file,mem' will enable
bdd88be3
TK
18file and memory debugging). `help' will list all available tracing options.
19.TP
7db7a5a0 20.BI \-\-parse\-only
bdd88be3 21Parse options only, don't start any I/O.
49da1240 22.TP
d60e92d1
AC
23.BI \-\-output \fR=\fPfilename
24Write output to \fIfilename\fR.
25.TP
7db7a5a0
TK
26.BI \-\-output\-format \fR=\fPformat
27Set the reporting \fIformat\fR to `normal', `terse', `json', or
28`json+'. Multiple formats can be selected, separate by a comma. `terse'
29is a CSV based format. `json+' is like `json', except it adds a full
513e37ee 30dump of the latency buckets.
e28ee21d 31.TP
7db7a5a0 32.BI \-\-bandwidth\-log
d23ae827 33Generate aggregate bandwidth logs.
d60e92d1 34.TP
7db7a5a0
TK
35.BI \-\-minimal
36Print statistics in a terse, semicolon\-delimited format.
d60e92d1 37.TP
7db7a5a0
TK
38.BI \-\-append\-terse
39Print statistics in selected mode AND terse, semicolon\-delimited format.
40\fBDeprecated\fR, use \fB\-\-output\-format\fR instead to select multiple formats.
f6a7df53 41.TP
065248bf 42.BI \-\-terse\-version \fR=\fPversion
7db7a5a0 43Set terse \fIversion\fR output format (default `3', or `2', `4', `5').
49da1240 44.TP
7db7a5a0 45.BI \-\-version
bdd88be3
TK
46Print version information and exit.
47.TP
7db7a5a0 48.BI \-\-help
bdd88be3 49Print a summary of the command line options and exit.
49da1240 50.TP
7db7a5a0 51.BI \-\-cpuclock\-test
bdd88be3 52Perform test and validation of internal CPU clock.
fec0f21c 53.TP
bdd88be3 54.BI \-\-crctest \fR=\fP[test]
7db7a5a0 55Test the speed of the built\-in checksumming functions. If no argument is given,
bdd88be3 56all of them are tested. Alternatively, a comma separated list can be passed, in which
fec0f21c
JA
57case the given ones are tested.
58.TP
49da1240 59.BI \-\-cmdhelp \fR=\fPcommand
bdd88be3 60Print help information for \fIcommand\fR. May be `all' for all commands.
49da1240 61.TP
7db7a5a0
TK
62.BI \-\-enghelp \fR=\fP[ioengine[,command]]
63List all commands defined by \fIioengine\fR, or print help for \fIcommand\fR
64defined by \fIioengine\fR. If no \fIioengine\fR is given, list all
65available ioengines.
de890a1e 66.TP
d60e92d1 67.BI \-\-showcmd \fR=\fPjobfile
7db7a5a0 68Convert \fIjobfile\fR to a set of command\-line options.
d60e92d1 69.TP
bdd88be3 70.BI \-\-readonly
4027b2a1 71Turn on safety read\-only checks, preventing writes and trims. The \fB\-\-readonly\fR
bdd88be3 72option is an extra safety guard to prevent users from accidentally starting
4027b2a1
VF
73a write or trim workload when that is not desired. Fio will only modify the
74device under test if `rw=write/randwrite/rw/randrw/trim/randtrim/trimwrite'
75is given. This safety net can be used as an extra precaution.
bdd88be3 76.TP
d60e92d1 77.BI \-\-eta \fR=\fPwhen
7db7a5a0 78Specifies when real\-time ETA estimate should be printed. \fIwhen\fR may
db37d890
JA
79be `always', `never' or `auto'. `auto' is the default, it prints ETA when
80requested if the output is a TTY. `always' disregards the output type, and
81prints ETA when requested. `never' never prints ETA.
82.TP
83.BI \-\-eta\-interval \fR=\fPtime
84By default, fio requests client ETA status roughly every second. With this
85option, the interval is configurable. Fio imposes a minimum allowed time to
86avoid flooding the console, less than 250 msec is not supported.
d60e92d1 87.TP
30b5d57f 88.BI \-\-eta\-newline \fR=\fPtime
bdd88be3
TK
89Force a new line for every \fItime\fR period passed. When the unit is omitted,
90the value is interpreted in seconds.
30b5d57f
JA
91.TP
92.BI \-\-status\-interval \fR=\fPtime
aa6cb459
VF
93Force a full status dump of cumulative (from job start) values at \fItime\fR
94intervals. This option does *not* provide per-period measurements. So
95values such as bandwidth are running averages. When the time unit is omitted,
c1f4de8a
JA
96\fItime\fR is interpreted in seconds. Note that using this option with
97`\-\-output-format=json' will yield output that technically isn't valid json,
98since the output will be collated sets of valid json. It will need to be split
99into valid sets of json after the run.
bdd88be3
TK
100.TP
101.BI \-\-section \fR=\fPname
102Only run specified section \fIname\fR in job file. Multiple sections can be specified.
7db7a5a0 103The \fB\-\-section\fR option allows one to combine related jobs into one file.
bdd88be3 104E.g. one job file could define light, moderate, and heavy sections. Tell
7db7a5a0 105fio to run only the "heavy" section by giving `\-\-section=heavy'
bdd88be3 106command line option. One can also specify the "write" operations in one
7db7a5a0 107section and "verify" operation in another section. The \fB\-\-section\fR option
bdd88be3
TK
108only applies to job sections. The reserved *global* section is always
109parsed and used.
c0a5d35e 110.TP
49da1240 111.BI \-\-alloc\-size \fR=\fPkb
7db7a5a0
TK
112Set the internal smalloc pool size to \fIkb\fR in KiB. The
113\fB\-\-alloc\-size\fR switch allows one to use a larger pool size for smalloc.
bdd88be3
TK
114If running large jobs with randommap enabled, fio can run out of memory.
115Smalloc is an internal allocator for shared structures from a fixed size
116memory pool and can grow to 16 pools. The pool size defaults to 16MiB.
7db7a5a0
TK
117NOTE: While running `.fio_smalloc.*' backing store files are visible
118in `/tmp'.
d60e92d1 119.TP
49da1240
JA
120.BI \-\-warnings\-fatal
121All fio parser warnings are fatal, causing fio to exit with an error.
9183788d 122.TP
49da1240 123.BI \-\-max\-jobs \fR=\fPnr
7db7a5a0 124Set the maximum number of threads/processes to support to \fInr\fR.
7f4811bb
RNS
125NOTE: On Linux, it may be necessary to increase the shared-memory limit
126(`/proc/sys/kernel/shmmax') if fio runs into errors while creating jobs.
d60e92d1 127.TP
49da1240 128.BI \-\-server \fR=\fPargs
7db7a5a0
TK
129Start a backend server, with \fIargs\fR specifying what to listen to.
130See \fBCLIENT/SERVER\fR section.
f57a9c59 131.TP
49da1240 132.BI \-\-daemonize \fR=\fPpidfile
7db7a5a0 133Background a fio server, writing the pid to the given \fIpidfile\fR file.
49da1240 134.TP
bdd88be3 135.BI \-\-client \fR=\fPhostname
7db7a5a0
TK
136Instead of running the jobs locally, send and run them on the given \fIhostname\fR
137or set of \fIhostname\fRs. See \fBCLIENT/SERVER\fR section.
bdd88be3 138.TP
7db7a5a0
TK
139.BI \-\-remote\-config \fR=\fPfile
140Tell fio server to load this local \fIfile\fR.
f2a2ce0e
HL
141.TP
142.BI \-\-idle\-prof \fR=\fPoption
7db7a5a0 143Report CPU idleness. \fIoption\fR is one of the following:
bdd88be3
TK
144.RS
145.RS
146.TP
147.B calibrate
148Run unit work calibration only and exit.
149.TP
150.B system
151Show aggregate system idleness and unit work.
152.TP
153.B percpu
7db7a5a0 154As \fBsystem\fR but also show per CPU idleness.
bdd88be3
TK
155.RE
156.RE
157.TP
7db7a5a0
TK
158.BI \-\-inflate\-log \fR=\fPlog
159Inflate and output compressed \fIlog\fR.
bdd88be3 160.TP
7db7a5a0
TK
161.BI \-\-trigger\-file \fR=\fPfile
162Execute trigger command when \fIfile\fR exists.
bdd88be3 163.TP
7db7a5a0
TK
164.BI \-\-trigger\-timeout \fR=\fPtime
165Execute trigger at this \fItime\fR.
bdd88be3 166.TP
7db7a5a0
TK
167.BI \-\-trigger \fR=\fPcommand
168Set this \fIcommand\fR as local trigger.
bdd88be3 169.TP
7db7a5a0
TK
170.BI \-\-trigger\-remote \fR=\fPcommand
171Set this \fIcommand\fR as remote trigger.
bdd88be3 172.TP
7db7a5a0 173.BI \-\-aux\-path \fR=\fPpath
f4401bf8
SW
174Use the directory specified by \fIpath\fP for generated state files instead
175of the current working directory.
d60e92d1 176.SH "JOB FILE FORMAT"
7a14cf18
TK
177Any parameters following the options will be assumed to be job files, unless
178they match a job file parameter. Multiple job files can be listed and each job
7db7a5a0 179file will be regarded as a separate group. Fio will \fBstonewall\fR execution
7a14cf18
TK
180between each group.
181
182Fio accepts one or more job files describing what it is
183supposed to do. The job file format is the classic ini file, where the names
184enclosed in [] brackets define the job name. You are free to use any ASCII name
185you want, except *global* which has special meaning. Following the job name is
186a sequence of zero or more parameters, one per line, that define the behavior of
187the job. If the first character in a line is a ';' or a '#', the entire line is
188discarded as a comment.
189
190A *global* section sets defaults for the jobs described in that file. A job may
191override a *global* section parameter, and a job file may even have several
192*global* sections if so desired. A job is only affected by a *global* section
193residing above it.
194
7db7a5a0
TK
195The \fB\-\-cmdhelp\fR option also lists all options. If used with an \fIcommand\fR
196argument, \fB\-\-cmdhelp\fR will detail the given \fIcommand\fR.
7a14cf18 197
7db7a5a0
TK
198See the `examples/' directory for inspiration on how to write job files. Note
199the copyright and license requirements currently apply to
200`examples/' files.
54eb4569
TK
201.SH "JOB FILE PARAMETERS"
202Some parameters take an option of a given type, such as an integer or a
203string. Anywhere a numeric value is required, an arithmetic expression may be
204used, provided it is surrounded by parentheses. Supported operators are:
d59aa780 205.RS
7db7a5a0 206.P
d59aa780 207.B addition (+)
7db7a5a0
TK
208.P
209.B subtraction (\-)
210.P
d59aa780 211.B multiplication (*)
7db7a5a0 212.P
d59aa780 213.B division (/)
7db7a5a0 214.P
d59aa780 215.B modulus (%)
7db7a5a0 216.P
d59aa780
JA
217.B exponentiation (^)
218.RE
d59aa780
JA
219.P
220For time values in expressions, units are microseconds by default. This is
221different than for time values not in expressions (not enclosed in
54eb4569
TK
222parentheses).
223.SH "PARAMETER TYPES"
224The following parameter types are used.
d60e92d1
AC
225.TP
226.I str
6b86fc18
TK
227String. A sequence of alphanumeric characters.
228.TP
229.I time
230Integer with possible time suffix. Without a unit value is interpreted as
231seconds unless otherwise specified. Accepts a suffix of 'd' for days, 'h' for
232hours, 'm' for minutes, 's' for seconds, 'ms' (or 'msec') for milliseconds and 'us'
233(or 'usec') for microseconds. For example, use 10m for 10 minutes.
d60e92d1
AC
234.TP
235.I int
6d500c2e
RE
236Integer. A whole number value, which may contain an integer prefix
237and an integer suffix.
0b43a833
TK
238.RS
239.RS
240.P
6b86fc18 241[*integer prefix*] **number** [*integer suffix*]
0b43a833
TK
242.RE
243.P
6b86fc18
TK
244The optional *integer prefix* specifies the number's base. The default
245is decimal. *0x* specifies hexadecimal.
0b43a833 246.P
6b86fc18
TK
247The optional *integer suffix* specifies the number's units, and includes an
248optional unit prefix and an optional unit. For quantities of data, the
249default unit is bytes. For quantities of time, the default unit is seconds
250unless otherwise specified.
0b43a833
TK
251.P
252With `kb_base=1000', fio follows international standards for unit
7db7a5a0 253prefixes. To specify power\-of\-10 decimal values defined in the
6b86fc18 254International System of Units (SI):
0b43a833
TK
255.RS
256.P
7db7a5a0 257.PD 0
eccce61a 258K means kilo (K) or 1000
7db7a5a0 259.P
eccce61a 260M means mega (M) or 1000**2
7db7a5a0 261.P
eccce61a 262G means giga (G) or 1000**3
7db7a5a0 263.P
eccce61a 264T means tera (T) or 1000**4
7db7a5a0 265.P
eccce61a 266P means peta (P) or 1000**5
7db7a5a0 267.PD
0b43a833
TK
268.RE
269.P
7db7a5a0 270To specify power\-of\-2 binary values defined in IEC 80000\-13:
0b43a833
TK
271.RS
272.P
7db7a5a0 273.PD 0
eccce61a 274Ki means kibi (Ki) or 1024
7db7a5a0 275.P
eccce61a 276Mi means mebi (Mi) or 1024**2
7db7a5a0 277.P
eccce61a 278Gi means gibi (Gi) or 1024**3
7db7a5a0 279.P
eccce61a 280Ti means tebi (Ti) or 1024**4
7db7a5a0 281.P
eccce61a 282Pi means pebi (Pi) or 1024**5
7db7a5a0 283.PD
0b43a833
TK
284.RE
285.P
286With `kb_base=1024' (the default), the unit prefixes are opposite
7db7a5a0 287from those specified in the SI and IEC 80000\-13 standards to provide
6b86fc18 288compatibility with old scripts. For example, 4k means 4096.
0b43a833 289.P
6b86fc18
TK
290For quantities of data, an optional unit of 'B' may be included
291(e.g., 'kB' is the same as 'k').
0b43a833 292.P
6b86fc18
TK
293The *integer suffix* is not case sensitive (e.g., m/mi mean mebi/mega,
294not milli). 'b' and 'B' both mean byte, not bit.
0b43a833
TK
295.P
296Examples with `kb_base=1000':
297.RS
298.P
7db7a5a0 299.PD 0
6d500c2e 3004 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
7db7a5a0 301.P
6d500c2e 3021 MiB: 1048576, 1m, 1024k
7db7a5a0 303.P
6d500c2e 3041 MB: 1000000, 1mi, 1000ki
7db7a5a0 305.P
6d500c2e 3061 TiB: 1073741824, 1t, 1024m, 1048576k
7db7a5a0 307.P
6d500c2e 3081 TB: 1000000000, 1ti, 1000mi, 1000000ki
7db7a5a0 309.PD
0b43a833
TK
310.RE
311.P
312Examples with `kb_base=1024' (default):
313.RS
314.P
7db7a5a0 315.PD 0
6d500c2e 3164 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
7db7a5a0 317.P
6d500c2e 3181 MiB: 1048576, 1m, 1024k
7db7a5a0 319.P
6d500c2e 3201 MB: 1000000, 1mi, 1000ki
7db7a5a0 321.P
6d500c2e 3221 TiB: 1073741824, 1t, 1024m, 1048576k
7db7a5a0 323.P
6d500c2e 3241 TB: 1000000000, 1ti, 1000mi, 1000000ki
7db7a5a0 325.PD
0b43a833
TK
326.RE
327.P
6d500c2e 328To specify times (units are not case sensitive):
0b43a833
TK
329.RS
330.P
7db7a5a0 331.PD 0
6d500c2e 332D means days
7db7a5a0 333.P
6d500c2e 334H means hours
7db7a5a0 335.P
6d500c2e 336M mean minutes
7db7a5a0 337.P
6d500c2e 338s or sec means seconds (default)
7db7a5a0 339.P
6d500c2e 340ms or msec means milliseconds
7db7a5a0 341.P
6d500c2e 342us or usec means microseconds
7db7a5a0 343.PD
0b43a833
TK
344.RE
345.P
6b86fc18 346If the option accepts an upper and lower range, use a colon ':' or
7db7a5a0 347minus '\-' to separate such values. See \fIirange\fR parameter type.
6b86fc18
TK
348If the lower value specified happens to be larger than the upper value
349the two values are swapped.
0b43a833 350.RE
d60e92d1
AC
351.TP
352.I bool
6b86fc18
TK
353Boolean. Usually parsed as an integer, however only defined for
354true and false (1 and 0).
d60e92d1
AC
355.TP
356.I irange
6b86fc18 357Integer range with suffix. Allows value range to be given, such as
7db7a5a0 3581024\-4096. A colon may also be used as the separator, e.g. 1k:4k. If the
6b86fc18 359option allows two sets of ranges, they can be specified with a ',' or '/'
7db7a5a0 360delimiter: 1k\-4k/8k\-32k. Also see \fIint\fR parameter type.
83349190
YH
361.TP
362.I float_list
6b86fc18 363A list of floating point numbers, separated by a ':' character.
523bad63 364.SH "JOB PARAMETERS"
54eb4569 365With the above in mind, here follows the complete list of fio job parameters.
523bad63 366.SS "Units"
d60e92d1 367.TP
523bad63
TK
368.BI kb_base \fR=\fPint
369Select the interpretation of unit prefixes in input parameters.
370.RS
371.RS
d60e92d1 372.TP
523bad63
TK
373.B 1000
374Inputs comply with IEC 80000\-13 and the International
375System of Units (SI). Use:
376.RS
377.P
378.PD 0
379\- power\-of\-2 values with IEC prefixes (e.g., KiB)
380.P
381\- power\-of\-10 values with SI prefixes (e.g., kB)
382.PD
383.RE
384.TP
385.B 1024
386Compatibility mode (default). To avoid breaking old scripts:
387.P
388.RS
389.PD 0
390\- power\-of\-2 values with SI prefixes
391.P
392\- power\-of\-10 values with IEC prefixes
393.PD
394.RE
395.RE
396.P
397See \fBbs\fR for more details on input parameters.
398.P
399Outputs always use correct prefixes. Most outputs include both
400side\-by\-side, like:
401.P
402.RS
403bw=2383.3kB/s (2327.4KiB/s)
404.RE
405.P
406If only one value is reported, then kb_base selects the one to use:
407.P
408.RS
409.PD 0
4101000 \-\- SI prefixes
411.P
4121024 \-\- IEC prefixes
413.PD
414.RE
415.RE
416.TP
417.BI unit_base \fR=\fPint
418Base unit for reporting. Allowed values are:
419.RS
420.RS
421.TP
422.B 0
423Use auto\-detection (default).
424.TP
425.B 8
426Byte based.
427.TP
428.B 1
429Bit based.
430.RE
431.RE
432.SS "Job description"
433.TP
434.BI name \fR=\fPstr
435ASCII name of the job. This may be used to override the name printed by fio
436for this job. Otherwise the job name is used. On the command line this
437parameter has the special purpose of also signaling the start of a new job.
9cc8cb91 438.TP
d60e92d1 439.BI description \fR=\fPstr
523bad63
TK
440Text description of the job. Doesn't do anything except dump this text
441description when this job is run. It's not parsed.
442.TP
443.BI loops \fR=\fPint
444Run the specified number of iterations of this job. Used to repeat the same
445workload a given number of times. Defaults to 1.
446.TP
447.BI numjobs \fR=\fPint
448Create the specified number of clones of this job. Each clone of job
449is spawned as an independent thread or process. May be used to setup a
450larger number of threads/processes doing the same thing. Each thread is
451reported separately; to see statistics for all clones as a whole, use
452\fBgroup_reporting\fR in conjunction with \fBnew_group\fR.
453See \fB\-\-max\-jobs\fR. Default: 1.
454.SS "Time related parameters"
455.TP
456.BI runtime \fR=\fPtime
457Tell fio to terminate processing after the specified period of time. It
458can be quite hard to determine for how long a specified job will run, so
459this parameter is handy to cap the total runtime to a given time. When
f1dd3fb1 460the unit is omitted, the value is interpreted in seconds.
523bad63
TK
461.TP
462.BI time_based
463If set, fio will run for the duration of the \fBruntime\fR specified
464even if the file(s) are completely read or written. It will simply loop over
465the same workload as many times as the \fBruntime\fR allows.
466.TP
467.BI startdelay \fR=\fPirange(int)
468Delay the start of job for the specified amount of time. Can be a single
469value or a range. When given as a range, each thread will choose a value
470randomly from within the range. Value is in seconds if a unit is omitted.
471.TP
472.BI ramp_time \fR=\fPtime
473If set, fio will run the specified workload for this amount of time before
474logging any performance numbers. Useful for letting performance settle
475before logging results, thus minimizing the runtime required for stable
476results. Note that the \fBramp_time\fR is considered lead in time for a job,
477thus it will increase the total runtime if a special timeout or
478\fBruntime\fR is specified. When the unit is omitted, the value is
479given in seconds.
480.TP
481.BI clocksource \fR=\fPstr
482Use the given clocksource as the base of timing. The supported options are:
483.RS
484.RS
485.TP
486.B gettimeofday
487\fBgettimeofday\fR\|(2)
488.TP
489.B clock_gettime
490\fBclock_gettime\fR\|(2)
491.TP
492.B cpu
493Internal CPU clock source
494.RE
495.P
496\fBcpu\fR is the preferred clocksource if it is reliable, as it is very fast (and
497fio is heavy on time calls). Fio will automatically use this clocksource if
498it's supported and considered reliable on the system it is running on,
499unless another clocksource is specifically set. For x86/x86\-64 CPUs, this
500means supporting TSC Invariant.
501.RE
502.TP
503.BI gtod_reduce \fR=\fPbool
504Enable all of the \fBgettimeofday\fR\|(2) reducing options
505(\fBdisable_clat\fR, \fBdisable_slat\fR, \fBdisable_bw_measurement\fR) plus
506reduce precision of the timeout somewhat to really shrink the
507\fBgettimeofday\fR\|(2) call count. With this option enabled, we only do
508about 0.4% of the \fBgettimeofday\fR\|(2) calls we would have done if all
509time keeping was enabled.
510.TP
511.BI gtod_cpu \fR=\fPint
512Sometimes it's cheaper to dedicate a single thread of execution to just
513getting the current time. Fio (and databases, for instance) are very
514intensive on \fBgettimeofday\fR\|(2) calls. With this option, you can set
515one CPU aside for doing nothing but logging current time to a shared memory
516location. Then the other threads/processes that run I/O workloads need only
517copy that segment, instead of entering the kernel with a
518\fBgettimeofday\fR\|(2) call. The CPU set aside for doing these time
519calls will be excluded from other uses. Fio will manually clear it from the
520CPU mask of other jobs.
521.SS "Target file/device"
d60e92d1
AC
522.TP
523.BI directory \fR=\fPstr
523bad63
TK
524Prefix \fBfilename\fRs with this directory. Used to place files in a different
525location than `./'. You can specify a number of directories by
526separating the names with a ':' character. These directories will be
527assigned equally distributed to job clones created by \fBnumjobs\fR as
528long as they are using generated filenames. If specific \fBfilename\fR(s) are
529set fio will use the first listed directory, and thereby matching the
f4401bf8
SW
530\fBfilename\fR semantic (which generates a file for each clone if not
531specified, but lets all clones use the same file if set).
523bad63
TK
532.RS
533.P
ffc90a44 534See the \fBfilename\fR option for information on how to escape ':' and '\\'
523bad63 535characters within the directory path itself.
f4401bf8
SW
536.P
537Note: To control the directory fio will use for internal state files
538use \fB\-\-aux\-path\fR.
523bad63 539.RE
d60e92d1
AC
540.TP
541.BI filename \fR=\fPstr
523bad63
TK
542Fio normally makes up a \fBfilename\fR based on the job name, thread number, and
543file number (see \fBfilename_format\fR). If you want to share files
544between threads in a job or several
545jobs with fixed file paths, specify a \fBfilename\fR for each of them to override
546the default. If the ioengine is file based, you can specify a number of files
547by separating the names with a ':' colon. So if you wanted a job to open
548`/dev/sda' and `/dev/sdb' as the two working files, you would use
549`filename=/dev/sda:/dev/sdb'. This also means that whenever this option is
550specified, \fBnrfiles\fR is ignored. The size of regular files specified
551by this option will be \fBsize\fR divided by number of files unless an
552explicit size is specified by \fBfilesize\fR.
553.RS
554.P
ffc90a44 555Each colon and backslash in the wanted path must be escaped with a '\\'
523bad63
TK
556character. For instance, if the path is `/dev/dsk/foo@3,0:c' then you
557would use `filename=/dev/dsk/foo@3,0\\:c' and if the path is
ffc90a44 558`F:\\filename' then you would use `filename=F\\:\\\\filename'.
523bad63 559.P
ffc90a44
SW
560On Windows, disk devices are accessed as `\\\\.\\PhysicalDrive0' for
561the first device, `\\\\.\\PhysicalDrive1' for the second etc.
523bad63
TK
562Note: Windows and FreeBSD prevent write access to areas
563of the disk containing in\-use data (e.g. filesystems).
564.P
565The filename `\-' is a reserved name, meaning *stdin* or *stdout*. Which
566of the two depends on the read/write direction set.
567.RE
d60e92d1 568.TP
de98bd30 569.BI filename_format \fR=\fPstr
523bad63
TK
570If sharing multiple files between jobs, it is usually necessary to have fio
571generate the exact names that you want. By default, fio will name a file
de98bd30 572based on the default file format specification of
523bad63 573`jobname.jobnumber.filenumber'. With this option, that can be
de98bd30
JA
574customized. Fio will recognize and replace the following keywords in this
575string:
576.RS
577.RS
578.TP
579.B $jobname
580The name of the worker thread or process.
581.TP
582.B $jobnum
583The incremental number of the worker thread or process.
584.TP
585.B $filenum
586The incremental number of the file for that worker thread or process.
587.RE
588.P
523bad63
TK
589To have dependent jobs share a set of files, this option can be set to have
590fio generate filenames that are shared between the two. For instance, if
591`testfiles.$filenum' is specified, file number 4 for any job will be
592named `testfiles.4'. The default of `$jobname.$jobnum.$filenum'
de98bd30 593will be used if no other format specifier is given.
645943c0
JB
594.P
595If you specify a path then the directories will be created up to the main
596directory for the file. So for example if you specify `a/b/c/$jobnum` then the
597directories a/b/c will be created before the file setup part of the job. If you
598specify \fBdirectory\fR then the path will be relative that directory, otherwise
599it is treated as the absolute path.
de98bd30 600.RE
de98bd30 601.TP
922a5be8 602.BI unique_filename \fR=\fPbool
523bad63
TK
603To avoid collisions between networked clients, fio defaults to prefixing any
604generated filenames (with a directory specified) with the source of the
605client connecting. To disable this behavior, set this option to 0.
606.TP
607.BI opendir \fR=\fPstr
608Recursively open any files below directory \fIstr\fR.
922a5be8 609.TP
3ce9dcaf 610.BI lockfile \fR=\fPstr
523bad63
TK
611Fio defaults to not locking any files before it does I/O to them. If a file
612or file descriptor is shared, fio can serialize I/O to that file to make the
613end result consistent. This is usual for emulating real workloads that share
614files. The lock modes are:
3ce9dcaf
JA
615.RS
616.RS
617.TP
618.B none
523bad63 619No locking. The default.
3ce9dcaf
JA
620.TP
621.B exclusive
523bad63 622Only one thread or process may do I/O at a time, excluding all others.
3ce9dcaf
JA
623.TP
624.B readwrite
523bad63
TK
625Read\-write locking on the file. Many readers may
626access the file at the same time, but writes get exclusive access.
3ce9dcaf 627.RE
ce594fbe 628.RE
523bad63
TK
629.TP
630.BI nrfiles \fR=\fPint
631Number of files to use for this job. Defaults to 1. The size of files
632will be \fBsize\fR divided by this unless explicit size is specified by
633\fBfilesize\fR. Files are created for each thread separately, and each
634file will have a file number within its name by default, as explained in
635\fBfilename\fR section.
636.TP
637.BI openfiles \fR=\fPint
638Number of files to keep open at the same time. Defaults to the same as
639\fBnrfiles\fR, can be set smaller to limit the number simultaneous
640opens.
641.TP
642.BI file_service_type \fR=\fPstr
643Defines how fio decides which file from a job to service next. The following
644types are defined:
645.RS
646.RS
647.TP
648.B random
649Choose a file at random.
650.TP
651.B roundrobin
652Round robin over opened files. This is the default.
653.TP
654.B sequential
655Finish one file before moving on to the next. Multiple files can
656still be open depending on \fBopenfiles\fR.
657.TP
658.B zipf
659Use a Zipf distribution to decide what file to access.
660.TP
661.B pareto
662Use a Pareto distribution to decide what file to access.
663.TP
664.B normal
665Use a Gaussian (normal) distribution to decide what file to access.
666.TP
667.B gauss
668Alias for normal.
669.RE
3ce9dcaf 670.P
523bad63
TK
671For \fBrandom\fR, \fBroundrobin\fR, and \fBsequential\fR, a postfix can be appended to
672tell fio how many I/Os to issue before switching to a new file. For example,
673specifying `file_service_type=random:8' would cause fio to issue
6748 I/Os before selecting a new file at random. For the non\-uniform
675distributions, a floating point postfix can be given to influence how the
676distribution is skewed. See \fBrandom_distribution\fR for a description
677of how that would work.
678.RE
679.TP
680.BI ioscheduler \fR=\fPstr
681Attempt to switch the device hosting the file to the specified I/O scheduler
682before running.
683.TP
684.BI create_serialize \fR=\fPbool
685If true, serialize the file creation for the jobs. This may be handy to
686avoid interleaving of data files, which may greatly depend on the filesystem
687used and even the number of processors in the system. Default: true.
688.TP
689.BI create_fsync \fR=\fPbool
690\fBfsync\fR\|(2) the data file after creation. This is the default.
691.TP
692.BI create_on_open \fR=\fPbool
693If true, don't pre\-create files but allow the job's open() to create a file
694when it's time to do I/O. Default: false \-\- pre\-create all necessary files
695when the job starts.
696.TP
697.BI create_only \fR=\fPbool
698If true, fio will only run the setup phase of the job. If files need to be
699laid out or updated on disk, only that will be done \-\- the actual job contents
700are not executed. Default: false.
701.TP
702.BI allow_file_create \fR=\fPbool
703If true, fio is permitted to create files as part of its workload. If this
704option is false, then fio will error out if
705the files it needs to use don't already exist. Default: true.
706.TP
707.BI allow_mounted_write \fR=\fPbool
708If this isn't set, fio will abort jobs that are destructive (e.g. that write)
709to what appears to be a mounted device or partition. This should help catch
710creating inadvertently destructive tests, not realizing that the test will
711destroy data on the mounted file system. Note that some platforms don't allow
712writing against a mounted device regardless of this option. Default: false.
713.TP
714.BI pre_read \fR=\fPbool
715If this is given, files will be pre\-read into memory before starting the
716given I/O operation. This will also clear the \fBinvalidate\fR flag,
717since it is pointless to pre\-read and then drop the cache. This will only
718work for I/O engines that are seek\-able, since they allow you to read the
719same data multiple times. Thus it will not work on non\-seekable I/O engines
720(e.g. network, splice). Default: false.
721.TP
722.BI unlink \fR=\fPbool
723Unlink the job files when done. Not the default, as repeated runs of that
724job would then waste time recreating the file set again and again. Default:
725false.
726.TP
727.BI unlink_each_loop \fR=\fPbool
728Unlink job files after each iteration or loop. Default: false.
729.TP
7b865a2f
BVA
730.BI zonemode \fR=\fPstr
731Accepted values are:
732.RS
733.RS
734.TP
735.B none
736The \fBzonerange\fR, \fBzonesize\fR and \fBzoneskip\fR parameters are ignored.
737.TP
738.B strided
739I/O happens in a single zone until \fBzonesize\fR bytes have been transferred.
740After that number of bytes has been transferred processing of the next zone
741starts.
742.TP
743.B zbd
744Zoned block device mode. I/O happens sequentially in each zone, even if random
745I/O has been selected. Random I/O happens across all zones instead of being
746restricted to a single zone.
747.RE
748.RE
523bad63
TK
749.TP
750.BI zonerange \fR=\fPint
7b865a2f 751Size of a single zone. See also \fBzonesize\fR and \fBzoneskip\fR.
5faddc64
BVA
752.TP
753.BI zonesize \fR=\fPint
7b865a2f
BVA
754For \fBzonemode\fR=strided, this is the number of bytes to transfer before
755skipping \fBzoneskip\fR bytes. If this parameter is smaller than
756\fBzonerange\fR then only a fraction of each zone with \fBzonerange\fR bytes
757will be accessed. If this parameter is larger than \fBzonerange\fR then each
758zone will be accessed multiple times before skipping to the next zone.
759
760For \fBzonemode\fR=zbd, this is the size of a single zone. The \fBzonerange\fR
761parameter is ignored in this mode.
523bad63
TK
762.TP
763.BI zoneskip \fR=\fPint
7b865a2f
BVA
764For \fBzonemode\fR=strided, the number of bytes to skip after \fBzonesize\fR
765bytes of data have been transferred. This parameter must be zero for
766\fBzonemode\fR=zbd.
5faddc64 767
bfbdd35b
BVA
768.TP
769.BI read_beyond_wp \fR=\fPbool
770This parameter applies to \fBzonemode=zbd\fR only.
771
772Zoned block devices are block devices that consist of multiple zones. Each
773zone has a type, e.g. conventional or sequential. A conventional zone can be
774written at any offset that is a multiple of the block size. Sequential zones
775must be written sequentially. The position at which a write must occur is
776called the write pointer. A zoned block device can be either drive
777managed, host managed or host aware. For host managed devices the host must
778ensure that writes happen sequentially. Fio recognizes host managed devices
779and serializes writes to sequential zones for these devices.
780
781If a read occurs in a sequential zone beyond the write pointer then the zoned
782block device will complete the read without reading any data from the storage
783medium. Since such reads lead to unrealistically high bandwidth and IOPS
784numbers fio only reads beyond the write pointer if explicitly told to do
785so. Default: false.
59b07544
BVA
786.TP
787.BI max_open_zones \fR=\fPint
788When running a random write test across an entire drive many more zones will be
789open than in a typical application workload. Hence this command line option
790that allows to limit the number of open zones. The number of open zones is
791defined as the number of zones to which write commands are issued.
a7c2b6fc
BVA
792.TP
793.BI zone_reset_threshold \fR=\fPfloat
794A number between zero and one that indicates the ratio of logical blocks with
795data to the total number of logical blocks in the test above which zones
796should be reset periodically.
797.TP
798.BI zone_reset_frequency \fR=\fPfloat
799A number between zero and one that indicates how often a zone reset should be
800issued if the zone reset threshold has been exceeded. A zone reset is
801submitted after each (1 / zone_reset_frequency) write requests. This and the
802previous parameter can be used to simulate garbage collection activity.
bfbdd35b 803
523bad63
TK
804.SS "I/O type"
805.TP
806.BI direct \fR=\fPbool
807If value is true, use non\-buffered I/O. This is usually O_DIRECT. Note that
8e889110 808OpenBSD and ZFS on Solaris don't support direct I/O. On Windows the synchronous
523bad63
TK
809ioengines don't support direct I/O. Default: false.
810.TP
811.BI atomic \fR=\fPbool
812If value is true, attempt to use atomic direct I/O. Atomic writes are
813guaranteed to be stable once acknowledged by the operating system. Only
814Linux supports O_ATOMIC right now.
815.TP
816.BI buffered \fR=\fPbool
817If value is true, use buffered I/O. This is the opposite of the
818\fBdirect\fR option. Defaults to true.
d60e92d1
AC
819.TP
820.BI readwrite \fR=\fPstr "\fR,\fP rw" \fR=\fPstr
523bad63 821Type of I/O pattern. Accepted values are:
d60e92d1
AC
822.RS
823.RS
824.TP
825.B read
d1429b5c 826Sequential reads.
d60e92d1
AC
827.TP
828.B write
d1429b5c 829Sequential writes.
d60e92d1 830.TP
fa769d44 831.B trim
3740cfc8 832Sequential trims (Linux block devices and SCSI character devices only).
fa769d44 833.TP
d60e92d1 834.B randread
d1429b5c 835Random reads.
d60e92d1
AC
836.TP
837.B randwrite
d1429b5c 838Random writes.
d60e92d1 839.TP
fa769d44 840.B randtrim
3740cfc8 841Random trims (Linux block devices and SCSI character devices only).
fa769d44 842.TP
523bad63
TK
843.B rw,readwrite
844Sequential mixed reads and writes.
d60e92d1 845.TP
ff6bb260 846.B randrw
523bad63 847Random mixed reads and writes.
82a90686
JA
848.TP
849.B trimwrite
523bad63
TK
850Sequential trim+write sequences. Blocks will be trimmed first,
851then the same blocks will be written to.
d60e92d1
AC
852.RE
853.P
523bad63
TK
854Fio defaults to read if the option is not specified. For the mixed I/O
855types, the default is to split them 50/50. For certain types of I/O the
856result may still be skewed a bit, since the speed may be different.
857.P
858It is possible to specify the number of I/Os to do before getting a new
859offset by appending `:<nr>' to the end of the string given. For a
860random read, it would look like `rw=randread:8' for passing in an offset
861modifier with a value of 8. If the suffix is used with a sequential I/O
862pattern, then the `<nr>' value specified will be added to the generated
863offset for each I/O turning sequential I/O into sequential I/O with holes.
864For instance, using `rw=write:4k' will skip 4k for every write. Also see
865the \fBrw_sequencer\fR option.
d60e92d1
AC
866.RE
867.TP
38dad62d 868.BI rw_sequencer \fR=\fPstr
523bad63
TK
869If an offset modifier is given by appending a number to the `rw=\fIstr\fR'
870line, then this option controls how that number modifies the I/O offset
871being generated. Accepted values are:
38dad62d
JA
872.RS
873.RS
874.TP
875.B sequential
523bad63 876Generate sequential offset.
38dad62d
JA
877.TP
878.B identical
523bad63 879Generate the same offset.
38dad62d
JA
880.RE
881.P
523bad63
TK
882\fBsequential\fR is only useful for random I/O, where fio would normally
883generate a new random offset for every I/O. If you append e.g. 8 to randread,
884you would get a new random offset for every 8 I/Os. The result would be a
885seek for only every 8 I/Os, instead of for every I/O. Use `rw=randread:8'
886to specify that. As sequential I/O is already sequential, setting
887\fBsequential\fR for that would not result in any differences. \fBidentical\fR
888behaves in a similar fashion, except it sends the same offset 8 number of
889times before generating a new offset.
38dad62d 890.RE
90fef2d1 891.TP
771e58be
JA
892.BI unified_rw_reporting \fR=\fPbool
893Fio normally reports statistics on a per data direction basis, meaning that
523bad63
TK
894reads, writes, and trims are accounted and reported separately. If this
895option is set fio sums the results and report them as "mixed" instead.
771e58be 896.TP
d60e92d1 897.BI randrepeat \fR=\fPbool
523bad63
TK
898Seed the random number generator used for random I/O patterns in a
899predictable way so the pattern is repeatable across runs. Default: true.
56e2a5fc
CE
900.TP
901.BI allrandrepeat \fR=\fPbool
902Seed all random number generators in a predictable way so results are
523bad63 903repeatable across runs. Default: false.
d60e92d1 904.TP
04778baf
JA
905.BI randseed \fR=\fPint
906Seed the random number generators based on this seed value, to be able to
907control what sequence of output is being generated. If not set, the random
908sequence depends on the \fBrandrepeat\fR setting.
909.TP
a596f047 910.BI fallocate \fR=\fPstr
523bad63
TK
911Whether pre\-allocation is performed when laying down files.
912Accepted values are:
a596f047
EG
913.RS
914.RS
915.TP
916.B none
523bad63 917Do not pre\-allocate space.
a596f047 918.TP
2c3e17be 919.B native
523bad63
TK
920Use a platform's native pre\-allocation call but fall back to
921\fBnone\fR behavior if it fails/is not implemented.
2c3e17be 922.TP
a596f047 923.B posix
523bad63 924Pre\-allocate via \fBposix_fallocate\fR\|(3).
a596f047
EG
925.TP
926.B keep
523bad63
TK
927Pre\-allocate via \fBfallocate\fR\|(2) with
928FALLOC_FL_KEEP_SIZE set.
a596f047
EG
929.TP
930.B 0
523bad63 931Backward\-compatible alias for \fBnone\fR.
a596f047
EG
932.TP
933.B 1
523bad63 934Backward\-compatible alias for \fBposix\fR.
a596f047
EG
935.RE
936.P
523bad63
TK
937May not be available on all supported platforms. \fBkeep\fR is only available
938on Linux. If using ZFS on Solaris this cannot be set to \fBposix\fR
939because ZFS doesn't support pre\-allocation. Default: \fBnative\fR if any
940pre\-allocation methods are available, \fBnone\fR if not.
a596f047 941.RE
7bc8c2cf 942.TP
ecb2083d 943.BI fadvise_hint \fR=\fPstr
c712c97a
JA
944Use \fBposix_fadvise\fR\|(2) or \fBposix_madvise\fR\|(2) to advise the kernel
945what I/O patterns are likely to be issued. Accepted values are:
ecb2083d
JA
946.RS
947.RS
948.TP
949.B 0
950Backwards compatible hint for "no hint".
951.TP
952.B 1
953Backwards compatible hint for "advise with fio workload type". This
523bad63 954uses FADV_RANDOM for a random workload, and FADV_SEQUENTIAL
ecb2083d
JA
955for a sequential workload.
956.TP
957.B sequential
523bad63 958Advise using FADV_SEQUENTIAL.
ecb2083d
JA
959.TP
960.B random
523bad63 961Advise using FADV_RANDOM.
ecb2083d
JA
962.RE
963.RE
d60e92d1 964.TP
8f4b9f24 965.BI write_hint \fR=\fPstr
523bad63
TK
966Use \fBfcntl\fR\|(2) to advise the kernel what life time to expect
967from a write. Only supported on Linux, as of version 4.13. Accepted
8f4b9f24
JA
968values are:
969.RS
970.RS
971.TP
972.B none
973No particular life time associated with this file.
974.TP
975.B short
976Data written to this file has a short life time.
977.TP
978.B medium
979Data written to this file has a medium life time.
980.TP
981.B long
982Data written to this file has a long life time.
983.TP
984.B extreme
985Data written to this file has a very long life time.
986.RE
523bad63
TK
987.P
988The values are all relative to each other, and no absolute meaning
989should be associated with them.
8f4b9f24 990.RE
37659335 991.TP
523bad63
TK
992.BI offset \fR=\fPint
993Start I/O at the provided offset in the file, given as either a fixed size in
83c8b093
JF
994bytes or a percentage. If a percentage is given, the generated offset will be
995aligned to the minimum \fBblocksize\fR or to the value of \fBoffset_align\fR if
996provided. Data before the given offset will not be touched. This
523bad63
TK
997effectively caps the file size at `real_size \- offset'. Can be combined with
998\fBsize\fR to constrain the start and end range of the I/O workload.
999A percentage can be specified by a number between 1 and 100 followed by '%',
1000for example, `offset=20%' to specify 20%.
6d500c2e 1001.TP
83c8b093
JF
1002.BI offset_align \fR=\fPint
1003If set to non-zero value, the byte offset generated by a percentage \fBoffset\fR
1004is aligned upwards to this value. Defaults to 0 meaning that a percentage
1005offset is aligned to the minimum block size.
1006.TP
523bad63
TK
1007.BI offset_increment \fR=\fPint
1008If this is provided, then the real offset becomes `\fBoffset\fR + \fBoffset_increment\fR
1009* thread_number', where the thread number is a counter that starts at 0 and
1010is incremented for each sub\-job (i.e. when \fBnumjobs\fR option is
1011specified). This option is useful if there are several jobs which are
1012intended to operate on a file in parallel disjoint segments, with even
1013spacing between the starting points.
6d500c2e 1014.TP
523bad63
TK
1015.BI number_ios \fR=\fPint
1016Fio will normally perform I/Os until it has exhausted the size of the region
1017set by \fBsize\fR, or if it exhaust the allocated time (or hits an error
1018condition). With this setting, the range/size can be set independently of
1019the number of I/Os to perform. When fio reaches this number, it will exit
1020normally and report status. Note that this does not extend the amount of I/O
1021that will be done, it will only stop fio if this condition is met before
1022other end\-of\-job criteria.
d60e92d1 1023.TP
523bad63
TK
1024.BI fsync \fR=\fPint
1025If writing to a file, issue an \fBfsync\fR\|(2) (or its equivalent) of
1026the dirty data for every number of blocks given. For example, if you give 32
1027as a parameter, fio will sync the file after every 32 writes issued. If fio is
1028using non\-buffered I/O, we may not sync the file. The exception is the sg
1029I/O engine, which synchronizes the disk cache anyway. Defaults to 0, which
1030means fio does not periodically issue and wait for a sync to complete. Also
1031see \fBend_fsync\fR and \fBfsync_on_close\fR.
6d500c2e 1032.TP
523bad63
TK
1033.BI fdatasync \fR=\fPint
1034Like \fBfsync\fR but uses \fBfdatasync\fR\|(2) to only sync data and
1035not metadata blocks. In Windows, FreeBSD, and DragonFlyBSD there is no
1036\fBfdatasync\fR\|(2) so this falls back to using \fBfsync\fR\|(2).
1037Defaults to 0, which means fio does not periodically issue and wait for a
1038data\-only sync to complete.
d60e92d1 1039.TP
523bad63
TK
1040.BI write_barrier \fR=\fPint
1041Make every N\-th write a barrier write.
901bb994 1042.TP
523bad63
TK
1043.BI sync_file_range \fR=\fPstr:int
1044Use \fBsync_file_range\fR\|(2) for every \fIint\fR number of write
1045operations. Fio will track range of writes that have happened since the last
1046\fBsync_file_range\fR\|(2) call. \fIstr\fR can currently be one or more of:
1047.RS
1048.RS
fd68418e 1049.TP
523bad63
TK
1050.B wait_before
1051SYNC_FILE_RANGE_WAIT_BEFORE
c5751c62 1052.TP
523bad63
TK
1053.B write
1054SYNC_FILE_RANGE_WRITE
c5751c62 1055.TP
523bad63
TK
1056.B wait_after
1057SYNC_FILE_RANGE_WRITE_AFTER
2fa5a241 1058.RE
523bad63
TK
1059.P
1060So if you do `sync_file_range=wait_before,write:8', fio would use
1061`SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE' for every 8
1062writes. Also see the \fBsync_file_range\fR\|(2) man page. This option is
1063Linux specific.
2fa5a241 1064.RE
ce35b1ec 1065.TP
523bad63
TK
1066.BI overwrite \fR=\fPbool
1067If true, writes to a file will always overwrite existing data. If the file
1068doesn't already exist, it will be created before the write phase begins. If
1069the file exists and is large enough for the specified write phase, nothing
1070will be done. Default: false.
5c94b008 1071.TP
523bad63
TK
1072.BI end_fsync \fR=\fPbool
1073If true, \fBfsync\fR\|(2) file contents when a write stage has completed.
1074Default: false.
d60e92d1 1075.TP
523bad63
TK
1076.BI fsync_on_close \fR=\fPbool
1077If true, fio will \fBfsync\fR\|(2) a dirty file on close. This differs
1078from \fBend_fsync\fR in that it will happen on every file close, not
1079just at the end of the job. Default: false.
d60e92d1 1080.TP
523bad63
TK
1081.BI rwmixread \fR=\fPint
1082Percentage of a mixed workload that should be reads. Default: 50.
1083.TP
1084.BI rwmixwrite \fR=\fPint
1085Percentage of a mixed workload that should be writes. If both
1086\fBrwmixread\fR and \fBrwmixwrite\fR is given and the values do not
1087add up to 100%, the latter of the two will be used to override the
1088first. This may interfere with a given rate setting, if fio is asked to
1089limit reads or writes to a certain rate. If that is the case, then the
1090distribution may be skewed. Default: 50.
1091.TP
1092.BI random_distribution \fR=\fPstr:float[,str:float][,str:float]
1093By default, fio will use a completely uniform random distribution when asked
1094to perform random I/O. Sometimes it is useful to skew the distribution in
1095specific ways, ensuring that some parts of the data is more hot than others.
1096fio includes the following distribution models:
d60e92d1
AC
1097.RS
1098.RS
1099.TP
1100.B random
523bad63 1101Uniform random distribution
8c07860d
JA
1102.TP
1103.B zipf
523bad63 1104Zipf distribution
8c07860d
JA
1105.TP
1106.B pareto
523bad63 1107Pareto distribution
8c07860d 1108.TP
dd3503d3 1109.B normal
523bad63 1110Normal (Gaussian) distribution
dd3503d3 1111.TP
523bad63
TK
1112.B zoned
1113Zoned random distribution
59466396
JA
1114.B zoned_abs
1115Zoned absolute random distribution
d60e92d1
AC
1116.RE
1117.P
523bad63
TK
1118When using a \fBzipf\fR or \fBpareto\fR distribution, an input value is also
1119needed to define the access pattern. For \fBzipf\fR, this is the `Zipf theta'.
1120For \fBpareto\fR, it's the `Pareto power'. Fio includes a test
1121program, \fBfio\-genzipf\fR, that can be used visualize what the given input
1122values will yield in terms of hit rates. If you wanted to use \fBzipf\fR with
1123a `theta' of 1.2, you would use `random_distribution=zipf:1.2' as the
1124option. If a non\-uniform model is used, fio will disable use of the random
1125map. For the \fBnormal\fR distribution, a normal (Gaussian) deviation is
1126supplied as a value between 0 and 100.
1127.P
1128For a \fBzoned\fR distribution, fio supports specifying percentages of I/O
1129access that should fall within what range of the file or device. For
1130example, given a criteria of:
d60e92d1 1131.RS
523bad63
TK
1132.P
1133.PD 0
113460% of accesses should be to the first 10%
1135.P
113630% of accesses should be to the next 20%
1137.P
11388% of accesses should be to the next 30%
1139.P
11402% of accesses should be to the next 40%
1141.PD
1142.RE
1143.P
1144we can define that through zoning of the random accesses. For the above
1145example, the user would do:
1146.RS
1147.P
1148random_distribution=zoned:60/10:30/20:8/30:2/40
1149.RE
1150.P
59466396
JA
1151A \fBzoned_abs\fR distribution works exactly like the\fBzoned\fR, except that
1152it takes absolute sizes. For example, let's say you wanted to define access
1153according to the following criteria:
1154.RS
1155.P
1156.PD 0
115760% of accesses should be to the first 20G
1158.P
115930% of accesses should be to the next 100G
1160.P
116110% of accesses should be to the next 500G
1162.PD
1163.RE
1164.P
1165we can define an absolute zoning distribution with:
1166.RS
1167.P
1168random_distribution=zoned:60/10:30/20:8/30:2/40
1169.RE
1170.P
6a16ece8
JA
1171For both \fBzoned\fR and \fBzoned_abs\fR, fio supports defining up to 256
1172separate zones.
1173.P
59466396 1174Similarly to how \fBbssplit\fR works for setting ranges and percentages
523bad63
TK
1175of block sizes. Like \fBbssplit\fR, it's possible to specify separate
1176zones for reads, writes, and trims. If just one set is given, it'll apply to
1177all of them.
1178.RE
1179.TP
1180.BI percentage_random \fR=\fPint[,int][,int]
1181For a random workload, set how big a percentage should be random. This
1182defaults to 100%, in which case the workload is fully random. It can be set
1183from anywhere from 0 to 100. Setting it to 0 would make the workload fully
1184sequential. Any setting in between will result in a random mix of sequential
1185and random I/O, at the given percentages. Comma\-separated values may be
1186specified for reads, writes, and trims as described in \fBblocksize\fR.
1187.TP
1188.BI norandommap
1189Normally fio will cover every block of the file when doing random I/O. If
1190this option is given, fio will just get a new random offset without looking
1191at past I/O history. This means that some blocks may not be read or written,
1192and that some blocks may be read/written more than once. If this option is
1193used with \fBverify\fR and multiple blocksizes (via \fBbsrange\fR),
1194only intact blocks are verified, i.e., partially\-overwritten blocks are
47e6a6e5
SW
1195ignored. With an async I/O engine and an I/O depth > 1, it is possible for
1196the same block to be overwritten, which can cause verification errors. Either
1197do not use norandommap in this case, or also use the lfsr random generator.
523bad63
TK
1198.TP
1199.BI softrandommap \fR=\fPbool
1200See \fBnorandommap\fR. If fio runs with the random block map enabled and
1201it fails to allocate the map, if this option is set it will continue without
1202a random block map. As coverage will not be as complete as with random maps,
1203this option is disabled by default.
1204.TP
1205.BI random_generator \fR=\fPstr
1206Fio supports the following engines for generating I/O offsets for random I/O:
1207.RS
1208.RS
1209.TP
1210.B tausworthe
1211Strong 2^88 cycle random number generator.
1212.TP
1213.B lfsr
1214Linear feedback shift register generator.
1215.TP
1216.B tausworthe64
1217Strong 64\-bit 2^258 cycle random number generator.
1218.RE
1219.P
1220\fBtausworthe\fR is a strong random number generator, but it requires tracking
1221on the side if we want to ensure that blocks are only read or written
1222once. \fBlfsr\fR guarantees that we never generate the same offset twice, and
1223it's also less computationally expensive. It's not a true random generator,
1224however, though for I/O purposes it's typically good enough. \fBlfsr\fR only
1225works with single block sizes, not with workloads that use multiple block
1226sizes. If used with such a workload, fio may read or write some blocks
1227multiple times. The default value is \fBtausworthe\fR, unless the required
1228space exceeds 2^32 blocks. If it does, then \fBtausworthe64\fR is
1229selected automatically.
1230.RE
1231.SS "Block size"
1232.TP
1233.BI blocksize \fR=\fPint[,int][,int] "\fR,\fB bs" \fR=\fPint[,int][,int]
1234The block size in bytes used for I/O units. Default: 4096. A single value
1235applies to reads, writes, and trims. Comma\-separated values may be
1236specified for reads, writes, and trims. A value not terminated in a comma
1237applies to subsequent types. Examples:
1238.RS
1239.RS
1240.P
1241.PD 0
1242bs=256k means 256k for reads, writes and trims.
1243.P
1244bs=8k,32k means 8k for reads, 32k for writes and trims.
1245.P
1246bs=8k,32k, means 8k for reads, 32k for writes, and default for trims.
1247.P
1248bs=,8k means default for reads, 8k for writes and trims.
1249.P
1250bs=,8k, means default for reads, 8k for writes, and default for trims.
1251.PD
1252.RE
1253.RE
1254.TP
1255.BI blocksize_range \fR=\fPirange[,irange][,irange] "\fR,\fB bsrange" \fR=\fPirange[,irange][,irange]
1256A range of block sizes in bytes for I/O units. The issued I/O unit will
1257always be a multiple of the minimum size, unless
1258\fBblocksize_unaligned\fR is set.
1259Comma\-separated ranges may be specified for reads, writes, and trims as
1260described in \fBblocksize\fR. Example:
1261.RS
1262.RS
1263.P
1264bsrange=1k\-4k,2k\-8k
1265.RE
1266.RE
1267.TP
1268.BI bssplit \fR=\fPstr[,str][,str]
1269Sometimes you want even finer grained control of the block sizes issued, not
1270just an even split between them. This option allows you to weight various
1271block sizes, so that you are able to define a specific amount of block sizes
1272issued. The format for this option is:
1273.RS
1274.RS
1275.P
1276bssplit=blocksize/percentage:blocksize/percentage
1277.RE
1278.P
1279for as many block sizes as needed. So if you want to define a workload that
1280has 50% 64k blocks, 10% 4k blocks, and 40% 32k blocks, you would write:
1281.RS
1282.P
1283bssplit=4k/10:64k/50:32k/40
1284.RE
1285.P
1286Ordering does not matter. If the percentage is left blank, fio will fill in
1287the remaining values evenly. So a bssplit option like this one:
1288.RS
1289.P
1290bssplit=4k/50:1k/:32k/
1291.RE
1292.P
1293would have 50% 4k ios, and 25% 1k and 32k ios. The percentages always add up
1294to 100, if bssplit is given a range that adds up to more, it will error out.
1295.P
1296Comma\-separated values may be specified for reads, writes, and trims as
1297described in \fBblocksize\fR.
1298.P
1299If you want a workload that has 50% 2k reads and 50% 4k reads, while having
130090% 4k writes and 10% 8k writes, you would specify:
1301.RS
1302.P
cf04b906 1303bssplit=2k/50:4k/50,4k/90:8k/10
523bad63 1304.RE
6a16ece8
JA
1305.P
1306Fio supports defining up to 64 different weights for each data direction.
523bad63
TK
1307.RE
1308.TP
1309.BI blocksize_unaligned "\fR,\fB bs_unaligned"
1310If set, fio will issue I/O units with any size within
1311\fBblocksize_range\fR, not just multiples of the minimum size. This
1312typically won't work with direct I/O, as that normally requires sector
1313alignment.
1314.TP
1315.BI bs_is_seq_rand \fR=\fPbool
1316If this option is set, fio will use the normal read,write blocksize settings
1317as sequential,random blocksize settings instead. Any random read or write
1318will use the WRITE blocksize settings, and any sequential read or write will
1319use the READ blocksize settings.
1320.TP
1321.BI blockalign \fR=\fPint[,int][,int] "\fR,\fB ba" \fR=\fPint[,int][,int]
1322Boundary to which fio will align random I/O units. Default:
1323\fBblocksize\fR. Minimum alignment is typically 512b for using direct
1324I/O, though it usually depends on the hardware block size. This option is
1325mutually exclusive with using a random map for files, so it will turn off
1326that option. Comma\-separated values may be specified for reads, writes, and
1327trims as described in \fBblocksize\fR.
1328.SS "Buffers and memory"
1329.TP
1330.BI zero_buffers
1331Initialize buffers with all zeros. Default: fill buffers with random data.
1332.TP
1333.BI refill_buffers
1334If this option is given, fio will refill the I/O buffers on every
1335submit. The default is to only fill it at init time and reuse that
1336data. Only makes sense if zero_buffers isn't specified, naturally. If data
1337verification is enabled, \fBrefill_buffers\fR is also automatically enabled.
1338.TP
1339.BI scramble_buffers \fR=\fPbool
1340If \fBrefill_buffers\fR is too costly and the target is using data
1341deduplication, then setting this option will slightly modify the I/O buffer
1342contents to defeat normal de\-dupe attempts. This is not enough to defeat
1343more clever block compression attempts, but it will stop naive dedupe of
1344blocks. Default: true.
1345.TP
1346.BI buffer_compress_percentage \fR=\fPint
72592780
SW
1347If this is set, then fio will attempt to provide I/O buffer content
1348(on WRITEs) that compresses to the specified level. Fio does this by
1349providing a mix of random data followed by fixed pattern data. The
1350fixed pattern is either zeros, or the pattern specified by
1351\fBbuffer_pattern\fR. If the \fBbuffer_pattern\fR option is used, it
1352might skew the compression ratio slightly. Setting
1353\fBbuffer_compress_percentage\fR to a value other than 100 will also
1354enable \fBrefill_buffers\fR in order to reduce the likelihood that
1355adjacent blocks are so similar that they over compress when seen
1356together. See \fBbuffer_compress_chunk\fR for how to set a finer or
1357coarser granularity of the random/fixed data regions. Defaults to unset
1358i.e., buffer data will not adhere to any compression level.
523bad63
TK
1359.TP
1360.BI buffer_compress_chunk \fR=\fPint
72592780
SW
1361This setting allows fio to manage how big the random/fixed data region
1362is when using \fBbuffer_compress_percentage\fR. When
1363\fBbuffer_compress_chunk\fR is set to some non-zero value smaller than the
1364block size, fio can repeat the random/fixed region throughout the I/O
1365buffer at the specified interval (which particularly useful when
1366bigger block sizes are used for a job). When set to 0, fio will use a
1367chunk size that matches the block size resulting in a single
1368random/fixed region within the I/O buffer. Defaults to 512. When the
1369unit is omitted, the value is interpreted in bytes.
523bad63
TK
1370.TP
1371.BI buffer_pattern \fR=\fPstr
1372If set, fio will fill the I/O buffers with this pattern or with the contents
1373of a file. If not set, the contents of I/O buffers are defined by the other
1374options related to buffer contents. The setting can be any pattern of bytes,
1375and can be prefixed with 0x for hex values. It may also be a string, where
1376the string must then be wrapped with "". Or it may also be a filename,
1377where the filename must be wrapped with '' in which case the file is
1378opened and read. Note that not all the file contents will be read if that
1379would cause the buffers to overflow. So, for example:
1380.RS
1381.RS
1382.P
1383.PD 0
1384buffer_pattern='filename'
1385.P
1386or:
1387.P
1388buffer_pattern="abcd"
1389.P
1390or:
1391.P
1392buffer_pattern=\-12
1393.P
1394or:
1395.P
1396buffer_pattern=0xdeadface
1397.PD
1398.RE
1399.P
1400Also you can combine everything together in any order:
1401.RS
1402.P
1403buffer_pattern=0xdeadface"abcd"\-12'filename'
1404.RE
1405.RE
1406.TP
1407.BI dedupe_percentage \fR=\fPint
1408If set, fio will generate this percentage of identical buffers when
1409writing. These buffers will be naturally dedupable. The contents of the
1410buffers depend on what other buffer compression settings have been set. It's
1411possible to have the individual buffers either fully compressible, or not at
72592780
SW
1412all \-\- this option only controls the distribution of unique buffers. Setting
1413this option will also enable \fBrefill_buffers\fR to prevent every buffer
1414being identical.
523bad63
TK
1415.TP
1416.BI invalidate \fR=\fPbool
1417Invalidate the buffer/page cache parts of the files to be used prior to
1418starting I/O if the platform and file type support it. Defaults to true.
1419This will be ignored if \fBpre_read\fR is also specified for the
1420same job.
1421.TP
1422.BI sync \fR=\fPbool
1423Use synchronous I/O for buffered writes. For the majority of I/O engines,
1424this means using O_SYNC. Default: false.
1425.TP
1426.BI iomem \fR=\fPstr "\fR,\fP mem" \fR=\fPstr
1427Fio can use various types of memory as the I/O unit buffer. The allowed
1428values are:
1429.RS
1430.RS
1431.TP
1432.B malloc
1433Use memory from \fBmalloc\fR\|(3) as the buffers. Default memory type.
1434.TP
1435.B shm
1436Use shared memory as the buffers. Allocated through \fBshmget\fR\|(2).
1437.TP
1438.B shmhuge
1439Same as \fBshm\fR, but use huge pages as backing.
1440.TP
1441.B mmap
1442Use \fBmmap\fR\|(2) to allocate buffers. May either be anonymous memory, or can
1443be file backed if a filename is given after the option. The format
1444is `mem=mmap:/path/to/file'.
1445.TP
1446.B mmaphuge
1447Use a memory mapped huge file as the buffer backing. Append filename
1448after mmaphuge, ala `mem=mmaphuge:/hugetlbfs/file'.
1449.TP
1450.B mmapshared
1451Same as \fBmmap\fR, but use a MMAP_SHARED mapping.
1452.TP
1453.B cudamalloc
1454Use GPU memory as the buffers for GPUDirect RDMA benchmark.
1455The \fBioengine\fR must be \fBrdma\fR.
1456.RE
1457.P
1458The area allocated is a function of the maximum allowed bs size for the job,
1459multiplied by the I/O depth given. Note that for \fBshmhuge\fR and
1460\fBmmaphuge\fR to work, the system must have free huge pages allocated. This
1461can normally be checked and set by reading/writing
1462`/proc/sys/vm/nr_hugepages' on a Linux system. Fio assumes a huge page
1463is 4MiB in size. So to calculate the number of huge pages you need for a
1464given job file, add up the I/O depth of all jobs (normally one unless
1465\fBiodepth\fR is used) and multiply by the maximum bs set. Then divide
1466that number by the huge page size. You can see the size of the huge pages in
1467`/proc/meminfo'. If no huge pages are allocated by having a non\-zero
1468number in `nr_hugepages', using \fBmmaphuge\fR or \fBshmhuge\fR will fail. Also
1469see \fBhugepage\-size\fR.
1470.P
1471\fBmmaphuge\fR also needs to have hugetlbfs mounted and the file location
1472should point there. So if it's mounted in `/huge', you would use
1473`mem=mmaphuge:/huge/somefile'.
1474.RE
1475.TP
1476.BI iomem_align \fR=\fPint "\fR,\fP mem_align" \fR=\fPint
1477This indicates the memory alignment of the I/O memory buffers. Note that
1478the given alignment is applied to the first I/O unit buffer, if using
1479\fBiodepth\fR the alignment of the following buffers are given by the
1480\fBbs\fR used. In other words, if using a \fBbs\fR that is a
1481multiple of the page sized in the system, all buffers will be aligned to
1482this value. If using a \fBbs\fR that is not page aligned, the alignment
1483of subsequent I/O memory buffers is the sum of the \fBiomem_align\fR and
1484\fBbs\fR used.
1485.TP
1486.BI hugepage\-size \fR=\fPint
1487Defines the size of a huge page. Must at least be equal to the system
1488setting, see `/proc/meminfo'. Defaults to 4MiB. Should probably
1489always be a multiple of megabytes, so using `hugepage\-size=Xm' is the
1490preferred way to set this to avoid setting a non\-pow\-2 bad value.
1491.TP
1492.BI lockmem \fR=\fPint
1493Pin the specified amount of memory with \fBmlock\fR\|(2). Can be used to
1494simulate a smaller amount of memory. The amount specified is per worker.
1495.SS "I/O size"
1496.TP
1497.BI size \fR=\fPint
1498The total size of file I/O for each thread of this job. Fio will run until
1499this many bytes has been transferred, unless runtime is limited by other options
1500(such as \fBruntime\fR, for instance, or increased/decreased by \fBio_size\fR).
1501Fio will divide this size between the available files determined by options
1502such as \fBnrfiles\fR, \fBfilename\fR, unless \fBfilesize\fR is
1503specified by the job. If the result of division happens to be 0, the size is
1504set to the physical size of the given files or devices if they exist.
1505If this option is not specified, fio will use the full size of the given
1506files or devices. If the files do not exist, size must be given. It is also
1507possible to give size as a percentage between 1 and 100. If `size=20%' is
1508given, fio will use 20% of the full size of the given files or devices.
1509Can be combined with \fBoffset\fR to constrain the start and end range
1510that I/O will be done within.
1511.TP
1512.BI io_size \fR=\fPint "\fR,\fB io_limit" \fR=\fPint
1513Normally fio operates within the region set by \fBsize\fR, which means
1514that the \fBsize\fR option sets both the region and size of I/O to be
1515performed. Sometimes that is not what you want. With this option, it is
1516possible to define just the amount of I/O that fio should do. For instance,
1517if \fBsize\fR is set to 20GiB and \fBio_size\fR is set to 5GiB, fio
1518will perform I/O within the first 20GiB but exit when 5GiB have been
1519done. The opposite is also possible \-\- if \fBsize\fR is set to 20GiB,
1520and \fBio_size\fR is set to 40GiB, then fio will do 40GiB of I/O within
1521the 0..20GiB region.
1522.TP
1523.BI filesize \fR=\fPirange(int)
1524Individual file sizes. May be a range, in which case fio will select sizes
1525for files at random within the given range and limited to \fBsize\fR in
1526total (if that is given). If not given, each created file is the same size.
1527This option overrides \fBsize\fR in terms of file size, which means
1528this value is used as a fixed size or possible range of each file.
1529.TP
1530.BI file_append \fR=\fPbool
1531Perform I/O after the end of the file. Normally fio will operate within the
1532size of a file. If this option is set, then fio will append to the file
1533instead. This has identical behavior to setting \fBoffset\fR to the size
1534of a file. This option is ignored on non\-regular files.
1535.TP
1536.BI fill_device \fR=\fPbool "\fR,\fB fill_fs" \fR=\fPbool
1537Sets size to something really large and waits for ENOSPC (no space left on
1538device) as the terminating condition. Only makes sense with sequential
1539write. For a read workload, the mount point will be filled first then I/O
1540started on the result. This option doesn't make sense if operating on a raw
1541device node, since the size of that is already known by the file system.
1542Additionally, writing beyond end\-of\-device will not return ENOSPC there.
1543.SS "I/O engine"
1544.TP
1545.BI ioengine \fR=\fPstr
1546Defines how the job issues I/O to the file. The following types are defined:
1547.RS
1548.RS
1549.TP
1550.B sync
1551Basic \fBread\fR\|(2) or \fBwrite\fR\|(2)
1552I/O. \fBlseek\fR\|(2) is used to position the I/O location.
1553See \fBfsync\fR and \fBfdatasync\fR for syncing write I/Os.
1554.TP
1555.B psync
1556Basic \fBpread\fR\|(2) or \fBpwrite\fR\|(2) I/O. Default on
1557all supported operating systems except for Windows.
1558.TP
1559.B vsync
1560Basic \fBreadv\fR\|(2) or \fBwritev\fR\|(2) I/O. Will emulate
1561queuing by coalescing adjacent I/Os into a single submission.
1562.TP
1563.B pvsync
1564Basic \fBpreadv\fR\|(2) or \fBpwritev\fR\|(2) I/O.
a46c5e01 1565.TP
2cafffbe
JA
1566.B pvsync2
1567Basic \fBpreadv2\fR\|(2) or \fBpwritev2\fR\|(2) I/O.
1568.TP
d60e92d1 1569.B libaio
523bad63
TK
1570Linux native asynchronous I/O. Note that Linux may only support
1571queued behavior with non\-buffered I/O (set `direct=1' or
1572`buffered=0').
1573This engine defines engine specific options.
d60e92d1
AC
1574.TP
1575.B posixaio
523bad63
TK
1576POSIX asynchronous I/O using \fBaio_read\fR\|(3) and
1577\fBaio_write\fR\|(3).
03e20d68
BC
1578.TP
1579.B solarisaio
1580Solaris native asynchronous I/O.
1581.TP
1582.B windowsaio
38f8c318 1583Windows native asynchronous I/O. Default on Windows.
d60e92d1
AC
1584.TP
1585.B mmap
523bad63
TK
1586File is memory mapped with \fBmmap\fR\|(2) and data copied
1587to/from using \fBmemcpy\fR\|(3).
d60e92d1
AC
1588.TP
1589.B splice
523bad63
TK
1590\fBsplice\fR\|(2) is used to transfer the data and
1591\fBvmsplice\fR\|(2) to transfer data from user space to the
1592kernel.
d60e92d1 1593.TP
d60e92d1 1594.B sg
523bad63
TK
1595SCSI generic sg v3 I/O. May either be synchronous using the SG_IO
1596ioctl, or if the target is an sg character device we use
1597\fBread\fR\|(2) and \fBwrite\fR\|(2) for asynchronous
1598I/O. Requires \fBfilename\fR option to specify either block or
3740cfc8
VF
1599character devices. This engine supports trim operations. The
1600sg engine includes engine specific options.
d60e92d1
AC
1601.TP
1602.B null
523bad63
TK
1603Doesn't transfer any data, just pretends to. This is mainly used to
1604exercise fio itself and for debugging/testing purposes.
d60e92d1
AC
1605.TP
1606.B net
523bad63
TK
1607Transfer over the network to given `host:port'. Depending on the
1608\fBprotocol\fR used, the \fBhostname\fR, \fBport\fR,
1609\fBlisten\fR and \fBfilename\fR options are used to specify
1610what sort of connection to make, while the \fBprotocol\fR option
1611determines which protocol will be used. This engine defines engine
1612specific options.
d60e92d1
AC
1613.TP
1614.B netsplice
523bad63
TK
1615Like \fBnet\fR, but uses \fBsplice\fR\|(2) and
1616\fBvmsplice\fR\|(2) to map data and send/receive.
1617This engine defines engine specific options.
d60e92d1 1618.TP
53aec0a4 1619.B cpuio
523bad63
TK
1620Doesn't transfer any data, but burns CPU cycles according to the
1621\fBcpuload\fR and \fBcpuchunks\fR options. Setting
1622\fBcpuload\fR\=85 will cause that job to do nothing but burn 85%
1623of the CPU. In case of SMP machines, use `numjobs=<nr_of_cpu>'
1624to get desired CPU usage, as the cpuload only loads a
1625single CPU at the desired rate. A job never finishes unless there is
1626at least one non\-cpuio job.
d60e92d1
AC
1627.TP
1628.B guasi
f1dd3fb1 1629The GUASI I/O engine is the Generic Userspace Asynchronous Syscall
523bad63
TK
1630Interface approach to async I/O. See \fIhttp://www.xmailserver.org/guasi\-lib.html\fR
1631for more info on GUASI.
d60e92d1 1632.TP
21b8aee8 1633.B rdma
523bad63
TK
1634The RDMA I/O engine supports both RDMA memory semantics
1635(RDMA_WRITE/RDMA_READ) and channel semantics (Send/Recv) for the
609ac152
SB
1636InfiniBand, RoCE and iWARP protocols. This engine defines engine
1637specific options.
d54fce84
DM
1638.TP
1639.B falloc
523bad63
TK
1640I/O engine that does regular fallocate to simulate data transfer as
1641fio ioengine.
1642.RS
1643.P
1644.PD 0
1645DDIR_READ does fallocate(,mode = FALLOC_FL_KEEP_SIZE,).
1646.P
1647DIR_WRITE does fallocate(,mode = 0).
1648.P
1649DDIR_TRIM does fallocate(,mode = FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE).
1650.PD
1651.RE
1652.TP
1653.B ftruncate
1654I/O engine that sends \fBftruncate\fR\|(2) operations in response
1655to write (DDIR_WRITE) events. Each ftruncate issued sets the file's
1656size to the current block offset. \fBblocksize\fR is ignored.
d54fce84
DM
1657.TP
1658.B e4defrag
523bad63
TK
1659I/O engine that does regular EXT4_IOC_MOVE_EXT ioctls to simulate
1660defragment activity in request to DDIR_WRITE event.
0d978694 1661.TP
d5f9b0ea
IF
1662.B rados
1663I/O engine supporting direct access to Ceph Reliable Autonomic Distributed
1664Object Store (RADOS) via librados. This ioengine defines engine specific
1665options.
1666.TP
0d978694 1667.B rbd
523bad63
TK
1668I/O engine supporting direct access to Ceph Rados Block Devices
1669(RBD) via librbd without the need to use the kernel rbd driver. This
1670ioengine defines engine specific options.
a7c386f4 1671.TP
c2f6a13d
LMB
1672.B http
1673I/O engine supporting GET/PUT requests over HTTP(S) with libcurl to
1674a WebDAV or S3 endpoint. This ioengine defines engine specific options.
1675
1676This engine only supports direct IO of iodepth=1; you need to scale this
1677via numjobs. blocksize defines the size of the objects to be created.
1678
1679TRIM is translated to object deletion.
1680.TP
a7c386f4 1681.B gfapi
523bad63
TK
1682Using GlusterFS libgfapi sync interface to direct access to
1683GlusterFS volumes without having to go through FUSE. This ioengine
1684defines engine specific options.
cc47f094 1685.TP
1686.B gfapi_async
523bad63
TK
1687Using GlusterFS libgfapi async interface to direct access to
1688GlusterFS volumes without having to go through FUSE. This ioengine
1689defines engine specific options.
1b10477b 1690.TP
b74e419e 1691.B libhdfs
523bad63
TK
1692Read and write through Hadoop (HDFS). The \fBfilename\fR option
1693is used to specify host,port of the hdfs name\-node to connect. This
1694engine interprets offsets a little differently. In HDFS, files once
1695created cannot be modified so random writes are not possible. To
1696imitate this the libhdfs engine expects a bunch of small files to be
1697created over HDFS and will randomly pick a file from them
1698based on the offset generated by fio backend (see the example
1699job file to create such files, use `rw=write' option). Please
1700note, it may be necessary to set environment variables to work
1701with HDFS/libhdfs properly. Each job uses its own connection to
1702HDFS.
65fa28ca
DE
1703.TP
1704.B mtd
523bad63
TK
1705Read, write and erase an MTD character device (e.g.,
1706`/dev/mtd0'). Discards are treated as erases. Depending on the
1707underlying device type, the I/O may have to go in a certain pattern,
1708e.g., on NAND, writing sequentially to erase blocks and discarding
1709before overwriting. The \fBtrimwrite\fR mode works well for this
65fa28ca 1710constraint.
5c4ef02e
JA
1711.TP
1712.B pmemblk
523bad63 1713Read and write using filesystem DAX to a file on a filesystem
363a5f65 1714mounted with DAX on a persistent memory device through the PMDK
523bad63 1715libpmemblk library.
104ee4de 1716.TP
523bad63
TK
1717.B dev\-dax
1718Read and write using device DAX to a persistent memory device (e.g.,
363a5f65 1719/dev/dax0.0) through the PMDK libpmem library.
d60e92d1 1720.TP
523bad63
TK
1721.B external
1722Prefix to specify loading an external I/O engine object file. Append
1723the engine filename, e.g. `ioengine=external:/tmp/foo.o' to load
d243fd6d
TK
1724ioengine `foo.o' in `/tmp'. The path can be either
1725absolute or relative. See `engines/skeleton_external.c' in the fio source for
1726details of writing an external I/O engine.
1216cc5a
JB
1727.TP
1728.B filecreate
b71968b1
SW
1729Simply create the files and do no I/O to them. You still need to set
1730\fBfilesize\fR so that all the accounting still occurs, but no actual I/O will be
1731done other than creating the file.
ae0db592
TI
1732.TP
1733.B libpmem
1734Read and write using mmap I/O to a file on a filesystem
363a5f65 1735mounted with DAX on a persistent memory device through the PMDK
ae0db592 1736libpmem library.
07751e10
JA
1737.TP
1738.B ime_psync
1739Synchronous read and write using DDN's Infinite Memory Engine (IME). This
1740engine is very basic and issues calls to IME whenever an IO is queued.
1741.TP
1742.B ime_psyncv
1743Synchronous read and write using DDN's Infinite Memory Engine (IME). This
1744engine uses iovecs and will try to stack as much IOs as possible (if the IOs
1745are "contiguous" and the IO depth is not exceeded) before issuing a call to IME.
1746.TP
1747.B ime_aio
1748Asynchronous read and write using DDN's Infinite Memory Engine (IME). This
1749engine will try to stack as much IOs as possible by creating requests for IME.
1750FIO will then decide when to commit these requests.
523bad63
TK
1751.SS "I/O engine specific parameters"
1752In addition, there are some parameters which are only valid when a specific
1753\fBioengine\fR is in use. These are used identically to normal parameters,
1754with the caveat that when used on the command line, they must come after the
1755\fBioengine\fR that defines them is selected.
d60e92d1 1756.TP
523bad63
TK
1757.BI (libaio)userspace_reap
1758Normally, with the libaio engine in use, fio will use the
1759\fBio_getevents\fR\|(3) system call to reap newly returned events. With
1760this flag turned on, the AIO ring will be read directly from user\-space to
1761reap events. The reaping mode is only enabled when polling for a minimum of
17620 events (e.g. when `iodepth_batch_complete=0').
3ce9dcaf 1763.TP
523bad63
TK
1764.BI (pvsync2)hipri
1765Set RWF_HIPRI on I/O, indicating to the kernel that it's of higher priority
1766than normal.
82407585 1767.TP
523bad63
TK
1768.BI (pvsync2)hipri_percentage
1769When hipri is set this determines the probability of a pvsync2 I/O being high
1770priority. The default is 100%.
d60e92d1 1771.TP
523bad63
TK
1772.BI (cpuio)cpuload \fR=\fPint
1773Attempt to use the specified percentage of CPU cycles. This is a mandatory
1774option when using cpuio I/O engine.
997b5680 1775.TP
523bad63
TK
1776.BI (cpuio)cpuchunks \fR=\fPint
1777Split the load into cycles of the given time. In microseconds.
1ad01bd1 1778.TP
523bad63
TK
1779.BI (cpuio)exit_on_io_done \fR=\fPbool
1780Detect when I/O threads are done, then exit.
d60e92d1 1781.TP
523bad63
TK
1782.BI (libhdfs)namenode \fR=\fPstr
1783The hostname or IP address of a HDFS cluster namenode to contact.
d01612f3 1784.TP
523bad63
TK
1785.BI (libhdfs)port
1786The listening port of the HFDS cluster namenode.
d60e92d1 1787.TP
523bad63
TK
1788.BI (netsplice,net)port
1789The TCP or UDP port to bind to or connect to. If this is used with
1790\fBnumjobs\fR to spawn multiple instances of the same job type, then
1791this will be the starting port number since fio will use a range of
1792ports.
d60e92d1 1793.TP
609ac152
SB
1794.BI (rdma)port
1795The port to use for RDMA-CM communication. This should be the same
1796value on the client and the server side.
1797.TP
1798.BI (netsplice,net, rdma)hostname \fR=\fPstr
1799The hostname or IP address to use for TCP, UDP or RDMA-CM based I/O.
1800If the job is a TCP listener or UDP reader, the hostname is not used
1801and must be omitted unless it is a valid UDP multicast address.
591e9e06 1802.TP
523bad63
TK
1803.BI (netsplice,net)interface \fR=\fPstr
1804The IP address of the network interface used to send or receive UDP
1805multicast.
ddf24e42 1806.TP
523bad63
TK
1807.BI (netsplice,net)ttl \fR=\fPint
1808Time\-to\-live value for outgoing UDP multicast packets. Default: 1.
d60e92d1 1809.TP
523bad63
TK
1810.BI (netsplice,net)nodelay \fR=\fPbool
1811Set TCP_NODELAY on TCP connections.
fa769d44 1812.TP
523bad63
TK
1813.BI (netsplice,net)protocol \fR=\fPstr "\fR,\fP proto" \fR=\fPstr
1814The network protocol to use. Accepted values are:
1815.RS
e76b1da4
JA
1816.RS
1817.TP
523bad63
TK
1818.B tcp
1819Transmission control protocol.
e76b1da4 1820.TP
523bad63
TK
1821.B tcpv6
1822Transmission control protocol V6.
e76b1da4 1823.TP
523bad63
TK
1824.B udp
1825User datagram protocol.
1826.TP
1827.B udpv6
1828User datagram protocol V6.
e76b1da4 1829.TP
523bad63
TK
1830.B unix
1831UNIX domain socket.
e76b1da4
JA
1832.RE
1833.P
523bad63
TK
1834When the protocol is TCP or UDP, the port must also be given, as well as the
1835hostname if the job is a TCP listener or UDP reader. For unix sockets, the
1836normal \fBfilename\fR option should be used and the port is invalid.
1837.RE
1838.TP
1839.BI (netsplice,net)listen
1840For TCP network connections, tell fio to listen for incoming connections
1841rather than initiating an outgoing connection. The \fBhostname\fR must
1842be omitted if this option is used.
1843.TP
1844.BI (netsplice,net)pingpong
1845Normally a network writer will just continue writing data, and a network
1846reader will just consume packages. If `pingpong=1' is set, a writer will
1847send its normal payload to the reader, then wait for the reader to send the
1848same payload back. This allows fio to measure network latencies. The
1849submission and completion latencies then measure local time spent sending or
1850receiving, and the completion latency measures how long it took for the
1851other end to receive and send back. For UDP multicast traffic
1852`pingpong=1' should only be set for a single reader when multiple readers
1853are listening to the same address.
1854.TP
1855.BI (netsplice,net)window_size \fR=\fPint
1856Set the desired socket buffer size for the connection.
e76b1da4 1857.TP
523bad63
TK
1858.BI (netsplice,net)mss \fR=\fPint
1859Set the TCP maximum segment size (TCP_MAXSEG).
d60e92d1 1860.TP
523bad63
TK
1861.BI (e4defrag)donorname \fR=\fPstr
1862File will be used as a block donor (swap extents between files).
d60e92d1 1863.TP
523bad63
TK
1864.BI (e4defrag)inplace \fR=\fPint
1865Configure donor file blocks allocation strategy:
1866.RS
1867.RS
d60e92d1 1868.TP
523bad63
TK
1869.B 0
1870Default. Preallocate donor's file on init.
d60e92d1 1871.TP
523bad63
TK
1872.B 1
1873Allocate space immediately inside defragment event, and free right
1874after event.
1875.RE
1876.RE
d60e92d1 1877.TP
d5f9b0ea 1878.BI (rbd,rados)clustername \fR=\fPstr
523bad63 1879Specifies the name of the Ceph cluster.
92d42d69 1880.TP
523bad63
TK
1881.BI (rbd)rbdname \fR=\fPstr
1882Specifies the name of the RBD.
92d42d69 1883.TP
d5f9b0ea
IF
1884.BI (rbd,rados)pool \fR=\fPstr
1885Specifies the name of the Ceph pool containing RBD or RADOS data.
92d42d69 1886.TP
d5f9b0ea 1887.BI (rbd,rados)clientname \fR=\fPstr
523bad63
TK
1888Specifies the username (without the 'client.' prefix) used to access the
1889Ceph cluster. If the \fBclustername\fR is specified, the \fBclientname\fR shall be
1890the full *type.id* string. If no type. prefix is given, fio will add 'client.'
1891by default.
92d42d69 1892.TP
d5f9b0ea
IF
1893.BI (rbd,rados)busy_poll \fR=\fPbool
1894Poll store instead of waiting for completion. Usually this provides better
1895throughput at cost of higher(up to 100%) CPU utilization.
1896.TP
c2f6a13d
LMB
1897.BI (http)http_host \fR=\fPstr
1898Hostname to connect to. For S3, this could be the bucket name. Default
1899is \fBlocalhost\fR
1900.TP
1901.BI (http)http_user \fR=\fPstr
1902Username for HTTP authentication.
1903.TP
1904.BI (http)http_pass \fR=\fPstr
1905Password for HTTP authentication.
1906.TP
09fd2966
LMB
1907.BI (http)https \fR=\fPstr
1908Whether to use HTTPS instead of plain HTTP. \fRon\fP enables HTTPS;
1909\fRinsecure\fP will enable HTTPS, but disable SSL peer verification (use
1910with caution!). Default is \fBoff\fR.
c2f6a13d 1911.TP
09fd2966
LMB
1912.BI (http)http_mode \fR=\fPstr
1913Which HTTP access mode to use: webdav, swift, or s3. Default is
1914\fBwebdav\fR.
c2f6a13d
LMB
1915.TP
1916.BI (http)http_s3_region \fR=\fPstr
1917The S3 region/zone to include in the request. Default is \fBus-east-1\fR.
1918.TP
1919.BI (http)http_s3_key \fR=\fPstr
1920The S3 secret key.
1921.TP
1922.BI (http)http_s3_keyid \fR=\fPstr
1923The S3 key/access id.
1924.TP
09fd2966
LMB
1925.BI (http)http_swift_auth_token \fR=\fPstr
1926The Swift auth token. See the example configuration file on how to
1927retrieve this.
1928.TP
c2f6a13d
LMB
1929.BI (http)http_verbose \fR=\fPint
1930Enable verbose requests from libcurl. Useful for debugging. 1 turns on
1931verbose logging from libcurl, 2 additionally enables HTTP IO tracing.
1932Default is \fB0\fR
1933.TP
523bad63
TK
1934.BI (mtd)skip_bad \fR=\fPbool
1935Skip operations against known bad blocks.
8116fd24 1936.TP
523bad63
TK
1937.BI (libhdfs)hdfsdirectory
1938libhdfs will create chunk in this HDFS directory.
e0a04ac1 1939.TP
523bad63
TK
1940.BI (libhdfs)chunk_size
1941The size of the chunk to use for each file.
609ac152
SB
1942.TP
1943.BI (rdma)verb \fR=\fPstr
1944The RDMA verb to use on this side of the RDMA ioengine
1945connection. Valid values are write, read, send and recv. These
1946correspond to the equivalent RDMA verbs (e.g. write = rdma_write
1947etc.). Note that this only needs to be specified on the client side of
1948the connection. See the examples folder.
1949.TP
1950.BI (rdma)bindname \fR=\fPstr
1951The name to use to bind the local RDMA-CM connection to a local RDMA
1952device. This could be a hostname or an IPv4 or IPv6 address. On the
1953server side this will be passed into the rdma_bind_addr() function and
1954on the client site it will be used in the rdma_resolve_add()
1955function. This can be useful when multiple paths exist between the
1956client and the server or in certain loopback configurations.
52b81b7c
KD
1957.TP
1958.BI (sg)readfua \fR=\fPbool
1959With readfua option set to 1, read operations include the force
1960unit access (fua) flag. Default: 0.
1961.TP
1962.BI (sg)writefua \fR=\fPbool
1963With writefua option set to 1, write operations include the force
1964unit access (fua) flag. Default: 0.
2c3a9150
VF
1965.TP
1966.BI (sg)sg_write_mode \fR=\fPstr
1967Specify the type of write commands to issue. This option can take three
1968values:
1969.RS
1970.RS
1971.TP
1972.B write (default)
1973Write opcodes are issued as usual
1974.TP
1975.B verify
1976Issue WRITE AND VERIFY commands. The BYTCHK bit is set to 0. This
1977directs the device to carry out a medium verification with no data
1978comparison. The writefua option is ignored with this selection.
1979.TP
1980.B same
1981Issue WRITE SAME commands. This transfers a single block to the device
1982and writes this same block of data to a contiguous sequence of LBAs
1983beginning at the specified offset. fio's block size parameter
1984specifies the amount of data written with each command. However, the
1985amount of data actually transferred to the device is equal to the
1986device's block (sector) size. For a device with 512 byte sectors,
1987blocksize=8k will write 16 sectors with each command. fio will still
1988generate 8k of data for each command butonly the first 512 bytes will
1989be used and transferred to the device. The writefua option is ignored
1990with this selection.
1991
523bad63
TK
1992.SS "I/O depth"
1993.TP
1994.BI iodepth \fR=\fPint
1995Number of I/O units to keep in flight against the file. Note that
1996increasing \fBiodepth\fR beyond 1 will not affect synchronous ioengines (except
1997for small degrees when \fBverify_async\fR is in use). Even async
1998engines may impose OS restrictions causing the desired depth not to be
1999achieved. This may happen on Linux when using libaio and not setting
2000`direct=1', since buffered I/O is not async on that OS. Keep an
2001eye on the I/O depth distribution in the fio output to verify that the
2002achieved depth is as expected. Default: 1.
2003.TP
2004.BI iodepth_batch_submit \fR=\fPint "\fR,\fP iodepth_batch" \fR=\fPint
2005This defines how many pieces of I/O to submit at once. It defaults to 1
2006which means that we submit each I/O as soon as it is available, but can be
2007raised to submit bigger batches of I/O at the time. If it is set to 0 the
2008\fBiodepth\fR value will be used.
2009.TP
2010.BI iodepth_batch_complete_min \fR=\fPint "\fR,\fP iodepth_batch_complete" \fR=\fPint
2011This defines how many pieces of I/O to retrieve at once. It defaults to 1
2012which means that we'll ask for a minimum of 1 I/O in the retrieval process
2013from the kernel. The I/O retrieval will go on until we hit the limit set by
2014\fBiodepth_low\fR. If this variable is set to 0, then fio will always
2015check for completed events before queuing more I/O. This helps reduce I/O
2016latency, at the cost of more retrieval system calls.
2017.TP
2018.BI iodepth_batch_complete_max \fR=\fPint
2019This defines maximum pieces of I/O to retrieve at once. This variable should
2020be used along with \fBiodepth_batch_complete_min\fR=\fIint\fR variable,
2021specifying the range of min and max amount of I/O which should be
2022retrieved. By default it is equal to \fBiodepth_batch_complete_min\fR
2023value. Example #1:
e0a04ac1 2024.RS
e0a04ac1 2025.RS
e0a04ac1 2026.P
523bad63
TK
2027.PD 0
2028iodepth_batch_complete_min=1
e0a04ac1 2029.P
523bad63
TK
2030iodepth_batch_complete_max=<iodepth>
2031.PD
e0a04ac1
JA
2032.RE
2033.P
523bad63
TK
2034which means that we will retrieve at least 1 I/O and up to the whole
2035submitted queue depth. If none of I/O has been completed yet, we will wait.
2036Example #2:
e8b1961d 2037.RS
523bad63
TK
2038.P
2039.PD 0
2040iodepth_batch_complete_min=0
2041.P
2042iodepth_batch_complete_max=<iodepth>
2043.PD
e8b1961d
JA
2044.RE
2045.P
523bad63
TK
2046which means that we can retrieve up to the whole submitted queue depth, but
2047if none of I/O has been completed yet, we will NOT wait and immediately exit
2048the system call. In this example we simply do polling.
2049.RE
e8b1961d 2050.TP
523bad63
TK
2051.BI iodepth_low \fR=\fPint
2052The low water mark indicating when to start filling the queue
2053again. Defaults to the same as \fBiodepth\fR, meaning that fio will
2054attempt to keep the queue full at all times. If \fBiodepth\fR is set to
2055e.g. 16 and \fBiodepth_low\fR is set to 4, then after fio has filled the queue of
205616 requests, it will let the depth drain down to 4 before starting to fill
2057it again.
d60e92d1 2058.TP
523bad63
TK
2059.BI serialize_overlap \fR=\fPbool
2060Serialize in-flight I/Os that might otherwise cause or suffer from data races.
2061When two or more I/Os are submitted simultaneously, there is no guarantee that
2062the I/Os will be processed or completed in the submitted order. Further, if
2063two or more of those I/Os are writes, any overlapping region between them can
2064become indeterminate/undefined on certain storage. These issues can cause
2065verification to fail erratically when at least one of the racing I/Os is
2066changing data and the overlapping region has a non-zero size. Setting
2067\fBserialize_overlap\fR tells fio to avoid provoking this behavior by explicitly
2068serializing in-flight I/Os that have a non-zero overlap. Note that setting
2069this option can reduce both performance and the \fBiodepth\fR achieved.
2070Additionally this option does not work when \fBio_submit_mode\fR is set to
2071offload. Default: false.
d60e92d1 2072.TP
523bad63
TK
2073.BI io_submit_mode \fR=\fPstr
2074This option controls how fio submits the I/O to the I/O engine. The default
2075is `inline', which means that the fio job threads submit and reap I/O
2076directly. If set to `offload', the job threads will offload I/O submission
2077to a dedicated pool of I/O threads. This requires some coordination and thus
2078has a bit of extra overhead, especially for lower queue depth I/O where it
2079can increase latencies. The benefit is that fio can manage submission rates
2080independently of the device completion rates. This avoids skewed latency
2081reporting if I/O gets backed up on the device side (the coordinated omission
2082problem).
2083.SS "I/O rate"
d60e92d1 2084.TP
523bad63
TK
2085.BI thinktime \fR=\fPtime
2086Stall the job for the specified period of time after an I/O has completed before issuing the
2087next. May be used to simulate processing being done by an application.
2088When the unit is omitted, the value is interpreted in microseconds. See
2089\fBthinktime_blocks\fR and \fBthinktime_spin\fR.
d60e92d1 2090.TP
523bad63
TK
2091.BI thinktime_spin \fR=\fPtime
2092Only valid if \fBthinktime\fR is set \- pretend to spend CPU time doing
2093something with the data received, before falling back to sleeping for the
2094rest of the period specified by \fBthinktime\fR. When the unit is
2095omitted, the value is interpreted in microseconds.
d60e92d1
AC
2096.TP
2097.BI thinktime_blocks \fR=\fPint
523bad63
TK
2098Only valid if \fBthinktime\fR is set \- control how many blocks to issue,
2099before waiting \fBthinktime\fR usecs. If not set, defaults to 1 which will make
2100fio wait \fBthinktime\fR usecs after every block. This effectively makes any
2101queue depth setting redundant, since no more than 1 I/O will be queued
2102before we have to complete it and do our \fBthinktime\fR. In other words, this
2103setting effectively caps the queue depth if the latter is larger.
d60e92d1 2104.TP
6d500c2e 2105.BI rate \fR=\fPint[,int][,int]
523bad63
TK
2106Cap the bandwidth used by this job. The number is in bytes/sec, the normal
2107suffix rules apply. Comma\-separated values may be specified for reads,
2108writes, and trims as described in \fBblocksize\fR.
2109.RS
2110.P
2111For example, using `rate=1m,500k' would limit reads to 1MiB/sec and writes to
2112500KiB/sec. Capping only reads or writes can be done with `rate=,500k' or
2113`rate=500k,' where the former will only limit writes (to 500KiB/sec) and the
2114latter will only limit reads.
2115.RE
d60e92d1 2116.TP
6d500c2e 2117.BI rate_min \fR=\fPint[,int][,int]
523bad63
TK
2118Tell fio to do whatever it can to maintain at least this bandwidth. Failing
2119to meet this requirement will cause the job to exit. Comma\-separated values
2120may be specified for reads, writes, and trims as described in
2121\fBblocksize\fR.
d60e92d1 2122.TP
6d500c2e 2123.BI rate_iops \fR=\fPint[,int][,int]
523bad63
TK
2124Cap the bandwidth to this number of IOPS. Basically the same as
2125\fBrate\fR, just specified independently of bandwidth. If the job is
2126given a block size range instead of a fixed value, the smallest block size
2127is used as the metric. Comma\-separated values may be specified for reads,
2128writes, and trims as described in \fBblocksize\fR.
d60e92d1 2129.TP
6d500c2e 2130.BI rate_iops_min \fR=\fPint[,int][,int]
523bad63
TK
2131If fio doesn't meet this rate of I/O, it will cause the job to exit.
2132Comma\-separated values may be specified for reads, writes, and trims as
2133described in \fBblocksize\fR.
d60e92d1 2134.TP
6de65959 2135.BI rate_process \fR=\fPstr
523bad63
TK
2136This option controls how fio manages rated I/O submissions. The default is
2137`linear', which submits I/O in a linear fashion with fixed delays between
2138I/Os that gets adjusted based on I/O completion rates. If this is set to
2139`poisson', fio will submit I/O based on a more real world random request
6de65959 2140flow, known as the Poisson process
523bad63 2141(\fIhttps://en.wikipedia.org/wiki/Poisson_point_process\fR). The lambda will be
5d02b083 214210^6 / IOPS for the given workload.
1a9bf814
JA
2143.TP
2144.BI rate_ignore_thinktime \fR=\fPbool
2145By default, fio will attempt to catch up to the specified rate setting, if any
2146kind of thinktime setting was used. If this option is set, then fio will
2147ignore the thinktime and continue doing IO at the specified rate, instead of
2148entering a catch-up mode after thinktime is done.
523bad63 2149.SS "I/O latency"
ff6bb260 2150.TP
523bad63 2151.BI latency_target \fR=\fPtime
3e260a46 2152If set, fio will attempt to find the max performance point that the given
523bad63
TK
2153workload will run at while maintaining a latency below this target. When
2154the unit is omitted, the value is interpreted in microseconds. See
2155\fBlatency_window\fR and \fBlatency_percentile\fR.
3e260a46 2156.TP
523bad63 2157.BI latency_window \fR=\fPtime
3e260a46 2158Used with \fBlatency_target\fR to specify the sample window that the job
523bad63
TK
2159is run at varying queue depths to test the performance. When the unit is
2160omitted, the value is interpreted in microseconds.
3e260a46
JA
2161.TP
2162.BI latency_percentile \fR=\fPfloat
523bad63
TK
2163The percentage of I/Os that must fall within the criteria specified by
2164\fBlatency_target\fR and \fBlatency_window\fR. If not set, this
2165defaults to 100.0, meaning that all I/Os must be equal or below to the value
2166set by \fBlatency_target\fR.
2167.TP
2168.BI max_latency \fR=\fPtime
2169If set, fio will exit the job with an ETIMEDOUT error if it exceeds this
2170maximum latency. When the unit is omitted, the value is interpreted in
2171microseconds.
2172.TP
2173.BI rate_cycle \fR=\fPint
2174Average bandwidth for \fBrate\fR and \fBrate_min\fR over this number
2175of milliseconds. Defaults to 1000.
2176.SS "I/O replay"
2177.TP
2178.BI write_iolog \fR=\fPstr
2179Write the issued I/O patterns to the specified file. See
2180\fBread_iolog\fR. Specify a separate file for each job, otherwise the
2181iologs will be interspersed and the file may be corrupt.
2182.TP
2183.BI read_iolog \fR=\fPstr
2184Open an iolog with the specified filename and replay the I/O patterns it
2185contains. This can be used to store a workload and replay it sometime
2186later. The iolog given may also be a blktrace binary file, which allows fio
2187to replay a workload captured by blktrace. See
2188\fBblktrace\fR\|(8) for how to capture such logging data. For blktrace
2189replay, the file needs to be turned into a blkparse binary data file first
2190(`blkparse <device> \-o /dev/null \-d file_for_fio.bin').
c70c7f58
AK
2191You can specify a number of files by separating the names with a ':' character.
2192See the \fBfilename\fR option for information on how to escape ':' and '\'
2193characters within the file names. These files will be sequentially assigned to
2194job clones created by \fBnumjobs\fR.
3e260a46 2195.TP
98e7161c
AK
2196.BI read_iolog_chunked \fR=\fPbool
2197Determines how iolog is read. If false (default) entire \fBread_iolog\fR will
2198be read at once. If selected true, input from iolog will be read gradually.
2199Useful when iolog is very large, or it is generated.
2200.TP
523bad63
TK
2201.BI replay_no_stall \fR=\fPbool
2202When replaying I/O with \fBread_iolog\fR the default behavior is to
2203attempt to respect the timestamps within the log and replay them with the
2204appropriate delay between IOPS. By setting this variable fio will not
2205respect the timestamps and attempt to replay them as fast as possible while
2206still respecting ordering. The result is the same I/O pattern to a given
2207device, but different timings.
2208.TP
6dd7fa77
JA
2209.BI replay_time_scale \fR=\fPint
2210When replaying I/O with \fBread_iolog\fR, fio will honor the original timing
2211in the trace. With this option, it's possible to scale the time. It's a
2212percentage option, if set to 50 it means run at 50% the original IO rate in
2213the trace. If set to 200, run at twice the original IO rate. Defaults to 100.
2214.TP
523bad63
TK
2215.BI replay_redirect \fR=\fPstr
2216While replaying I/O patterns using \fBread_iolog\fR the default behavior
2217is to replay the IOPS onto the major/minor device that each IOP was recorded
2218from. This is sometimes undesirable because on a different machine those
2219major/minor numbers can map to a different device. Changing hardware on the
2220same system can also result in a different major/minor mapping.
2221\fBreplay_redirect\fR causes all I/Os to be replayed onto the single specified
2222device regardless of the device it was recorded
2223from. i.e. `replay_redirect=/dev/sdc' would cause all I/O
2224in the blktrace or iolog to be replayed onto `/dev/sdc'. This means
2225multiple devices will be replayed onto a single device, if the trace
2226contains multiple devices. If you want multiple devices to be replayed
2227concurrently to multiple redirected devices you must blkparse your trace
2228into separate traces and replay them with independent fio invocations.
2229Unfortunately this also breaks the strict time ordering between multiple
2230device accesses.
2231.TP
2232.BI replay_align \fR=\fPint
2233Force alignment of I/O offsets and lengths in a trace to this power of 2
2234value.
2235.TP
2236.BI replay_scale \fR=\fPint
2237Scale sector offsets down by this factor when replaying traces.
2238.SS "Threads, processes and job synchronization"
2239.TP
38f68906
JA
2240.BI replay_skip \fR=\fPstr
2241Sometimes it's useful to skip certain IO types in a replay trace. This could
2242be, for instance, eliminating the writes in the trace. Or not replaying the
2243trims/discards, if you are redirecting to a device that doesn't support them.
2244This option takes a comma separated list of read, write, trim, sync.
2245.TP
523bad63
TK
2246.BI thread
2247Fio defaults to creating jobs by using fork, however if this option is
2248given, fio will create jobs by using POSIX Threads' function
2249\fBpthread_create\fR\|(3) to create threads instead.
2250.TP
2251.BI wait_for \fR=\fPstr
2252If set, the current job won't be started until all workers of the specified
2253waitee job are done.
2254.\" ignore blank line here from HOWTO as it looks normal without it
2255\fBwait_for\fR operates on the job name basis, so there are a few
2256limitations. First, the waitee must be defined prior to the waiter job
2257(meaning no forward references). Second, if a job is being referenced as a
2258waitee, it must have a unique name (no duplicate waitees).
2259.TP
2260.BI nice \fR=\fPint
2261Run the job with the given nice value. See man \fBnice\fR\|(2).
2262.\" ignore blank line here from HOWTO as it looks normal without it
2263On Windows, values less than \-15 set the process class to "High"; \-1 through
2264\-15 set "Above Normal"; 1 through 15 "Below Normal"; and above 15 "Idle"
2265priority class.
2266.TP
2267.BI prio \fR=\fPint
2268Set the I/O priority value of this job. Linux limits us to a positive value
2269between 0 and 7, with 0 being the highest. See man
2270\fBionice\fR\|(1). Refer to an appropriate manpage for other operating
2271systems since meaning of priority may differ.
2272.TP
2273.BI prioclass \fR=\fPint
2274Set the I/O priority class. See man \fBionice\fR\|(1).
15501535 2275.TP
d60e92d1 2276.BI cpus_allowed \fR=\fPstr
523bad63 2277Controls the same options as \fBcpumask\fR, but accepts a textual
b570e037
SW
2278specification of the permitted CPUs instead and CPUs are indexed from 0. So
2279to use CPUs 0 and 5 you would specify `cpus_allowed=0,5'. This option also
2280allows a range of CPUs to be specified \-\- say you wanted a binding to CPUs
22810, 5, and 8 to 15, you would set `cpus_allowed=0,5,8\-15'.
2282.RS
2283.P
2284On Windows, when `cpus_allowed' is unset only CPUs from fio's current
2285processor group will be used and affinity settings are inherited from the
2286system. An fio build configured to target Windows 7 makes options that set
2287CPUs processor group aware and values will set both the processor group
2288and a CPU from within that group. For example, on a system where processor
2289group 0 has 40 CPUs and processor group 1 has 32 CPUs, `cpus_allowed'
2290values between 0 and 39 will bind CPUs from processor group 0 and
2291`cpus_allowed' values between 40 and 71 will bind CPUs from processor
2292group 1. When using `cpus_allowed_policy=shared' all CPUs specified by a
2293single `cpus_allowed' option must be from the same processor group. For
2294Windows fio builds not built for Windows 7, CPUs will only be selected from
2295(and be relative to) whatever processor group fio happens to be running in
2296and CPUs from other processor groups cannot be used.
2297.RE
d60e92d1 2298.TP
c2acfbac 2299.BI cpus_allowed_policy \fR=\fPstr
523bad63
TK
2300Set the policy of how fio distributes the CPUs specified by
2301\fBcpus_allowed\fR or \fBcpumask\fR. Two policies are supported:
c2acfbac
JA
2302.RS
2303.RS
2304.TP
2305.B shared
2306All jobs will share the CPU set specified.
2307.TP
2308.B split
2309Each job will get a unique CPU from the CPU set.
2310.RE
2311.P
523bad63
TK
2312\fBshared\fR is the default behavior, if the option isn't specified. If
2313\fBsplit\fR is specified, then fio will will assign one cpu per job. If not
2314enough CPUs are given for the jobs listed, then fio will roundrobin the CPUs
2315in the set.
c2acfbac 2316.RE
c2acfbac 2317.TP
b570e037
SW
2318.BI cpumask \fR=\fPint
2319Set the CPU affinity of this job. The parameter given is a bit mask of
2320allowed CPUs the job may run on. So if you want the allowed CPUs to be 1
2321and 5, you would pass the decimal value of (1 << 1 | 1 << 5), or 34. See man
2322\fBsched_setaffinity\fR\|(2). This may not work on all supported
2323operating systems or kernel versions. This option doesn't work well for a
2324higher CPU count than what you can store in an integer mask, so it can only
2325control cpus 1\-32. For boxes with larger CPU counts, use
2326\fBcpus_allowed\fR.
2327.TP
d0b937ed 2328.BI numa_cpu_nodes \fR=\fPstr
cecbfd47 2329Set this job running on specified NUMA nodes' CPUs. The arguments allow
523bad63
TK
2330comma delimited list of cpu numbers, A\-B ranges, or `all'. Note, to enable
2331NUMA options support, fio must be built on a system with libnuma\-dev(el)
2332installed.
d0b937ed
YR
2333.TP
2334.BI numa_mem_policy \fR=\fPstr
523bad63
TK
2335Set this job's memory policy and corresponding NUMA nodes. Format of the
2336arguments:
39c7a2ca
VF
2337.RS
2338.RS
523bad63
TK
2339.P
2340<mode>[:<nodelist>]
39c7a2ca 2341.RE
523bad63 2342.P
f1dd3fb1 2343`mode' is one of the following memory policies: `default', `prefer',
523bad63
TK
2344`bind', `interleave' or `local'. For `default' and `local' memory
2345policies, no node needs to be specified. For `prefer', only one node is
2346allowed. For `bind' and `interleave' the `nodelist' may be as
2347follows: a comma delimited list of numbers, A\-B ranges, or `all'.
39c7a2ca
VF
2348.RE
2349.TP
523bad63
TK
2350.BI cgroup \fR=\fPstr
2351Add job to this control group. If it doesn't exist, it will be created. The
2352system must have a mounted cgroup blkio mount point for this to work. If
2353your system doesn't have it mounted, you can do so with:
d60e92d1
AC
2354.RS
2355.RS
d60e92d1 2356.P
523bad63
TK
2357# mount \-t cgroup \-o blkio none /cgroup
2358.RE
d60e92d1
AC
2359.RE
2360.TP
523bad63
TK
2361.BI cgroup_weight \fR=\fPint
2362Set the weight of the cgroup to this value. See the documentation that comes
2363with the kernel, allowed values are in the range of 100..1000.
d60e92d1 2364.TP
523bad63
TK
2365.BI cgroup_nodelete \fR=\fPbool
2366Normally fio will delete the cgroups it has created after the job
2367completion. To override this behavior and to leave cgroups around after the
2368job completion, set `cgroup_nodelete=1'. This can be useful if one wants
2369to inspect various cgroup files after job completion. Default: false.
c8eeb9df 2370.TP
523bad63
TK
2371.BI flow_id \fR=\fPint
2372The ID of the flow. If not specified, it defaults to being a global
2373flow. See \fBflow\fR.
d60e92d1 2374.TP
523bad63
TK
2375.BI flow \fR=\fPint
2376Weight in token\-based flow control. If this value is used, then there is
2377a 'flow counter' which is used to regulate the proportion of activity between
2378two or more jobs. Fio attempts to keep this flow counter near zero. The
2379\fBflow\fR parameter stands for how much should be added or subtracted to the
2380flow counter on each iteration of the main I/O loop. That is, if one job has
2381`flow=8' and another job has `flow=\-1', then there will be a roughly 1:8
2382ratio in how much one runs vs the other.
d60e92d1 2383.TP
523bad63
TK
2384.BI flow_watermark \fR=\fPint
2385The maximum value that the absolute value of the flow counter is allowed to
2386reach before the job must wait for a lower value of the counter.
6b7f6851 2387.TP
523bad63
TK
2388.BI flow_sleep \fR=\fPint
2389The period of time, in microseconds, to wait after the flow watermark has
2390been exceeded before retrying operations.
25460cf6 2391.TP
523bad63
TK
2392.BI stonewall "\fR,\fB wait_for_previous"
2393Wait for preceding jobs in the job file to exit, before starting this
2394one. Can be used to insert serialization points in the job file. A stone
2395wall also implies starting a new reporting group, see
2396\fBgroup_reporting\fR.
2378826d 2397.TP
523bad63
TK
2398.BI exitall
2399By default, fio will continue running all other jobs when one job finishes
2400but sometimes this is not the desired action. Setting \fBexitall\fR will
2401instead make fio terminate all other jobs when one job finishes.
e81ecca3 2402.TP
523bad63
TK
2403.BI exec_prerun \fR=\fPstr
2404Before running this job, issue the command specified through
2405\fBsystem\fR\|(3). Output is redirected in a file called `jobname.prerun.txt'.
e9f48479 2406.TP
523bad63
TK
2407.BI exec_postrun \fR=\fPstr
2408After the job completes, issue the command specified though
2409\fBsystem\fR\|(3). Output is redirected in a file called `jobname.postrun.txt'.
d60e92d1 2410.TP
523bad63
TK
2411.BI uid \fR=\fPint
2412Instead of running as the invoking user, set the user ID to this value
2413before the thread/process does any work.
39c1c323 2414.TP
523bad63
TK
2415.BI gid \fR=\fPint
2416Set group ID, see \fBuid\fR.
2417.SS "Verification"
d60e92d1 2418.TP
589e88b7 2419.BI verify_only
523bad63 2420Do not perform specified workload, only verify data still matches previous
5e4c7118 2421invocation of this workload. This option allows one to check data multiple
523bad63
TK
2422times at a later date without overwriting it. This option makes sense only
2423for workloads that write data, and does not support workloads with the
5e4c7118
JA
2424\fBtime_based\fR option set.
2425.TP
d60e92d1 2426.BI do_verify \fR=\fPbool
523bad63
TK
2427Run the verify phase after a write phase. Only valid if \fBverify\fR is
2428set. Default: true.
d60e92d1
AC
2429.TP
2430.BI verify \fR=\fPstr
523bad63
TK
2431If writing to a file, fio can verify the file contents after each iteration
2432of the job. Each verification method also implies verification of special
2433header, which is written to the beginning of each block. This header also
2434includes meta information, like offset of the block, block number, timestamp
2435when block was written, etc. \fBverify\fR can be combined with
2436\fBverify_pattern\fR option. The allowed values are:
d60e92d1
AC
2437.RS
2438.RS
2439.TP
523bad63
TK
2440.B md5
2441Use an md5 sum of the data area and store it in the header of
2442each block.
2443.TP
2444.B crc64
2445Use an experimental crc64 sum of the data area and store it in the
2446header of each block.
2447.TP
2448.B crc32c
2449Use a crc32c sum of the data area and store it in the header of
2450each block. This will automatically use hardware acceleration
2451(e.g. SSE4.2 on an x86 or CRC crypto extensions on ARM64) but will
2452fall back to software crc32c if none is found. Generally the
f1dd3fb1 2453fastest checksum fio supports when hardware accelerated.
523bad63
TK
2454.TP
2455.B crc32c\-intel
2456Synonym for crc32c.
2457.TP
2458.B crc32
2459Use a crc32 sum of the data area and store it in the header of each
2460block.
2461.TP
2462.B crc16
2463Use a crc16 sum of the data area and store it in the header of each
2464block.
2465.TP
2466.B crc7
2467Use a crc7 sum of the data area and store it in the header of each
2468block.
2469.TP
2470.B xxhash
2471Use xxhash as the checksum function. Generally the fastest software
2472checksum that fio supports.
2473.TP
2474.B sha512
2475Use sha512 as the checksum function.
2476.TP
2477.B sha256
2478Use sha256 as the checksum function.
2479.TP
2480.B sha1
2481Use optimized sha1 as the checksum function.
2482.TP
2483.B sha3\-224
2484Use optimized sha3\-224 as the checksum function.
2485.TP
2486.B sha3\-256
2487Use optimized sha3\-256 as the checksum function.
2488.TP
2489.B sha3\-384
2490Use optimized sha3\-384 as the checksum function.
2491.TP
2492.B sha3\-512
2493Use optimized sha3\-512 as the checksum function.
d60e92d1
AC
2494.TP
2495.B meta
523bad63
TK
2496This option is deprecated, since now meta information is included in
2497generic verification header and meta verification happens by
2498default. For detailed information see the description of the
2499\fBverify\fR setting. This option is kept because of
2500compatibility's sake with old configurations. Do not use it.
d60e92d1 2501.TP
59245381 2502.B pattern
523bad63
TK
2503Verify a strict pattern. Normally fio includes a header with some
2504basic information and checksumming, but if this option is set, only
2505the specific pattern set with \fBverify_pattern\fR is verified.
59245381 2506.TP
d60e92d1 2507.B null
523bad63
TK
2508Only pretend to verify. Useful for testing internals with
2509`ioengine=null', not for much else.
d60e92d1 2510.RE
523bad63
TK
2511.P
2512This option can be used for repeated burn\-in tests of a system to make sure
2513that the written data is also correctly read back. If the data direction
2514given is a read or random read, fio will assume that it should verify a
2515previously written file. If the data direction includes any form of write,
2516the verify will be of the newly written data.
47e6a6e5
SW
2517.P
2518To avoid false verification errors, do not use the norandommap option when
2519verifying data with async I/O engines and I/O depths > 1. Or use the
2520norandommap and the lfsr random generator together to avoid writing to the
2521same offset with muliple outstanding I/Os.
d60e92d1
AC
2522.RE
2523.TP
f7fa2653 2524.BI verify_offset \fR=\fPint
d60e92d1 2525Swap the verification header with data somewhere else in the block before
523bad63 2526writing. It is swapped back before verifying.
d60e92d1 2527.TP
f7fa2653 2528.BI verify_interval \fR=\fPint
523bad63
TK
2529Write the verification header at a finer granularity than the
2530\fBblocksize\fR. It will be written for chunks the size of
2531\fBverify_interval\fR. \fBblocksize\fR should divide this evenly.
d60e92d1 2532.TP
996093bb 2533.BI verify_pattern \fR=\fPstr
523bad63
TK
2534If set, fio will fill the I/O buffers with this pattern. Fio defaults to
2535filling with totally random bytes, but sometimes it's interesting to fill
2536with a known pattern for I/O verification purposes. Depending on the width
2537of the pattern, fio will fill 1/2/3/4 bytes of the buffer at the time (it can
2538be either a decimal or a hex number). The \fBverify_pattern\fR if larger than
2539a 32\-bit quantity has to be a hex number that starts with either "0x" or
2540"0X". Use with \fBverify\fR. Also, \fBverify_pattern\fR supports %o
2541format, which means that for each block offset will be written and then
2542verified back, e.g.:
2fa5a241
RP
2543.RS
2544.RS
523bad63
TK
2545.P
2546verify_pattern=%o
2fa5a241 2547.RE
523bad63 2548.P
2fa5a241 2549Or use combination of everything:
2fa5a241 2550.RS
523bad63
TK
2551.P
2552verify_pattern=0xff%o"abcd"\-12
2fa5a241
RP
2553.RE
2554.RE
996093bb 2555.TP
d60e92d1 2556.BI verify_fatal \fR=\fPbool
523bad63
TK
2557Normally fio will keep checking the entire contents before quitting on a
2558block verification failure. If this option is set, fio will exit the job on
2559the first observed failure. Default: false.
d60e92d1 2560.TP
b463e936 2561.BI verify_dump \fR=\fPbool
523bad63
TK
2562If set, dump the contents of both the original data block and the data block
2563we read off disk to files. This allows later analysis to inspect just what
2564kind of data corruption occurred. Off by default.
b463e936 2565.TP
e8462bd8 2566.BI verify_async \fR=\fPint
523bad63
TK
2567Fio will normally verify I/O inline from the submitting thread. This option
2568takes an integer describing how many async offload threads to create for I/O
2569verification instead, causing fio to offload the duty of verifying I/O
2570contents to one or more separate threads. If using this offload option, even
2571sync I/O engines can benefit from using an \fBiodepth\fR setting higher
2572than 1, as it allows them to have I/O in flight while verifies are running.
2573Defaults to 0 async threads, i.e. verification is not asynchronous.
e8462bd8
JA
2574.TP
2575.BI verify_async_cpus \fR=\fPstr
523bad63
TK
2576Tell fio to set the given CPU affinity on the async I/O verification
2577threads. See \fBcpus_allowed\fR for the format used.
e8462bd8 2578.TP
6f87418f
JA
2579.BI verify_backlog \fR=\fPint
2580Fio will normally verify the written contents of a job that utilizes verify
2581once that job has completed. In other words, everything is written then
2582everything is read back and verified. You may want to verify continually
523bad63
TK
2583instead for a variety of reasons. Fio stores the meta data associated with
2584an I/O block in memory, so for large verify workloads, quite a bit of memory
2585would be used up holding this meta data. If this option is enabled, fio will
2586write only N blocks before verifying these blocks.
6f87418f
JA
2587.TP
2588.BI verify_backlog_batch \fR=\fPint
523bad63
TK
2589Control how many blocks fio will verify if \fBverify_backlog\fR is
2590set. If not set, will default to the value of \fBverify_backlog\fR
2591(meaning the entire queue is read back and verified). If
2592\fBverify_backlog_batch\fR is less than \fBverify_backlog\fR then not all
2593blocks will be verified, if \fBverify_backlog_batch\fR is larger than
2594\fBverify_backlog\fR, some blocks will be verified more than once.
2595.TP
2596.BI verify_state_save \fR=\fPbool
2597When a job exits during the write phase of a verify workload, save its
2598current state. This allows fio to replay up until that point, if the verify
2599state is loaded for the verify read phase. The format of the filename is,
2600roughly:
2601.RS
2602.RS
2603.P
2604<type>\-<jobname>\-<jobindex>\-verify.state.
2605.RE
2606.P
2607<type> is "local" for a local run, "sock" for a client/server socket
2608connection, and "ip" (192.168.0.1, for instance) for a networked
2609client/server connection. Defaults to true.
2610.RE
2611.TP
2612.BI verify_state_load \fR=\fPbool
2613If a verify termination trigger was used, fio stores the current write state
2614of each thread. This can be used at verification time so that fio knows how
2615far it should verify. Without this information, fio will run a full
2616verification pass, according to the settings in the job file used. Default
2617false.
6f87418f 2618.TP
fa769d44
SW
2619.BI trim_percentage \fR=\fPint
2620Number of verify blocks to discard/trim.
2621.TP
2622.BI trim_verify_zero \fR=\fPbool
523bad63 2623Verify that trim/discarded blocks are returned as zeros.
fa769d44
SW
2624.TP
2625.BI trim_backlog \fR=\fPint
523bad63 2626Verify that trim/discarded blocks are returned as zeros.
fa769d44
SW
2627.TP
2628.BI trim_backlog_batch \fR=\fPint
523bad63 2629Trim this number of I/O blocks.
fa769d44
SW
2630.TP
2631.BI experimental_verify \fR=\fPbool
2632Enable experimental verification.
523bad63 2633.SS "Steady state"
fa769d44 2634.TP
523bad63
TK
2635.BI steadystate \fR=\fPstr:float "\fR,\fP ss" \fR=\fPstr:float
2636Define the criterion and limit for assessing steady state performance. The
2637first parameter designates the criterion whereas the second parameter sets
2638the threshold. When the criterion falls below the threshold for the
2639specified duration, the job will stop. For example, `iops_slope:0.1%' will
2640direct fio to terminate the job when the least squares regression slope
2641falls below 0.1% of the mean IOPS. If \fBgroup_reporting\fR is enabled
2642this will apply to all jobs in the group. Below is the list of available
2643steady state assessment criteria. All assessments are carried out using only
2644data from the rolling collection window. Threshold limits can be expressed
2645as a fixed value or as a percentage of the mean in the collection window.
2646.RS
2647.RS
d60e92d1 2648.TP
523bad63
TK
2649.B iops
2650Collect IOPS data. Stop the job if all individual IOPS measurements
2651are within the specified limit of the mean IOPS (e.g., `iops:2'
2652means that all individual IOPS values must be within 2 of the mean,
2653whereas `iops:0.2%' means that all individual IOPS values must be
2654within 0.2% of the mean IOPS to terminate the job).
d60e92d1 2655.TP
523bad63
TK
2656.B iops_slope
2657Collect IOPS data and calculate the least squares regression
2658slope. Stop the job if the slope falls below the specified limit.
d60e92d1 2659.TP
523bad63
TK
2660.B bw
2661Collect bandwidth data. Stop the job if all individual bandwidth
2662measurements are within the specified limit of the mean bandwidth.
64bbb865 2663.TP
523bad63
TK
2664.B bw_slope
2665Collect bandwidth data and calculate the least squares regression
2666slope. Stop the job if the slope falls below the specified limit.
2667.RE
2668.RE
d1c46c04 2669.TP
523bad63
TK
2670.BI steadystate_duration \fR=\fPtime "\fR,\fP ss_dur" \fR=\fPtime
2671A rolling window of this duration will be used to judge whether steady state
2672has been reached. Data will be collected once per second. The default is 0
2673which disables steady state detection. When the unit is omitted, the
2674value is interpreted in seconds.
0c63576e 2675.TP
523bad63
TK
2676.BI steadystate_ramp_time \fR=\fPtime "\fR,\fP ss_ramp" \fR=\fPtime
2677Allow the job to run for the specified duration before beginning data
2678collection for checking the steady state job termination criterion. The
2679default is 0. When the unit is omitted, the value is interpreted in seconds.
2680.SS "Measurements and reporting"
0c63576e 2681.TP
3a5db920
JA
2682.BI per_job_logs \fR=\fPbool
2683If set, this generates bw/clat/iops log with per file private filenames. If
523bad63
TK
2684not set, jobs with identical names will share the log filename. Default:
2685true.
2686.TP
2687.BI group_reporting
2688It may sometimes be interesting to display statistics for groups of jobs as
2689a whole instead of for each individual job. This is especially true if
2690\fBnumjobs\fR is used; looking at individual thread/process output
2691quickly becomes unwieldy. To see the final report per\-group instead of
2692per\-job, use \fBgroup_reporting\fR. Jobs in a file will be part of the
2693same reporting group, unless if separated by a \fBstonewall\fR, or by
2694using \fBnew_group\fR.
2695.TP
2696.BI new_group
2697Start a new reporting group. See: \fBgroup_reporting\fR. If not given,
2698all jobs in a file will be part of the same reporting group, unless
2699separated by a \fBstonewall\fR.
2700.TP
2701.BI stats \fR=\fPbool
2702By default, fio collects and shows final output results for all jobs
2703that run. If this option is set to 0, then fio will ignore it in
2704the final stat output.
3a5db920 2705.TP
836bad52 2706.BI write_bw_log \fR=\fPstr
523bad63 2707If given, write a bandwidth log for this job. Can be used to store data of
074f0817 2708the bandwidth of the jobs in their lifetime.
523bad63 2709.RS
074f0817
SW
2710.P
2711If no str argument is given, the default filename of
2712`jobname_type.x.log' is used. Even when the argument is given, fio
2713will still append the type of log. So if one specifies:
523bad63
TK
2714.RS
2715.P
074f0817 2716write_bw_log=foo
523bad63
TK
2717.RE
2718.P
074f0817
SW
2719The actual log name will be `foo_bw.x.log' where `x' is the index
2720of the job (1..N, where N is the number of jobs). If
2721\fBper_job_logs\fR is false, then the filename will not include the
2722`.x` job index.
2723.P
2724The included \fBfio_generate_plots\fR script uses gnuplot to turn these
2725text files into nice graphs. See the \fBLOG FILE FORMATS\fR section for how data is
2726structured within the file.
523bad63 2727.RE
901bb994 2728.TP
074f0817
SW
2729.BI write_lat_log \fR=\fPstr
2730Same as \fBwrite_bw_log\fR, except this option creates I/O
2731submission (e.g., `name_slat.x.log'), completion (e.g.,
2732`name_clat.x.log'), and total (e.g., `name_lat.x.log') latency
2733files instead. See \fBwrite_bw_log\fR for details about the
2734filename format and the \fBLOG FILE FORMATS\fR section for how data is structured
2735within the files.
2736.TP
1e613c9c 2737.BI write_hist_log \fR=\fPstr
074f0817
SW
2738Same as \fBwrite_bw_log\fR but writes an I/O completion latency
2739histogram file (e.g., `name_hist.x.log') instead. Note that this
2740file will be empty unless \fBlog_hist_msec\fR has also been set.
2741See \fBwrite_bw_log\fR for details about the filename format and
2742the \fBLOG FILE FORMATS\fR section for how data is structured
2743within the file.
1e613c9c 2744.TP
c8eeb9df 2745.BI write_iops_log \fR=\fPstr
074f0817 2746Same as \fBwrite_bw_log\fR, but writes an IOPS file (e.g.
15417073
SW
2747`name_iops.x.log`) instead. Because fio defaults to individual
2748I/O logging, the value entry in the IOPS log will be 1 unless windowed
2749logging (see \fBlog_avg_msec\fR) has been enabled. See
2750\fBwrite_bw_log\fR for details about the filename format and \fBLOG
2751FILE FORMATS\fR for how data is structured within the file.
c8eeb9df 2752.TP
b8bc8cba
JA
2753.BI log_avg_msec \fR=\fPint
2754By default, fio will log an entry in the iops, latency, or bw log for every
523bad63 2755I/O that completes. When writing to the disk log, that can quickly grow to a
b8bc8cba 2756very large size. Setting this option makes fio average the each log entry
e6989e10 2757over the specified period of time, reducing the resolution of the log. See
523bad63
TK
2758\fBlog_max_value\fR as well. Defaults to 0, logging all entries.
2759Also see \fBLOG FILE FORMATS\fR section.
b8bc8cba 2760.TP
1e613c9c 2761.BI log_hist_msec \fR=\fPint
523bad63
TK
2762Same as \fBlog_avg_msec\fR, but logs entries for completion latency
2763histograms. Computing latency percentiles from averages of intervals using
2764\fBlog_avg_msec\fR is inaccurate. Setting this option makes fio log
2765histogram entries over the specified period of time, reducing log sizes for
2766high IOPS devices while retaining percentile accuracy. See
074f0817
SW
2767\fBlog_hist_coarseness\fR and \fBwrite_hist_log\fR as well.
2768Defaults to 0, meaning histogram logging is disabled.
1e613c9c
KC
2769.TP
2770.BI log_hist_coarseness \fR=\fPint
523bad63
TK
2771Integer ranging from 0 to 6, defining the coarseness of the resolution of
2772the histogram logs enabled with \fBlog_hist_msec\fR. For each increment
2773in coarseness, fio outputs half as many bins. Defaults to 0, for which
2774histogram logs contain 1216 latency bins. See \fBLOG FILE FORMATS\fR section.
2775.TP
2776.BI log_max_value \fR=\fPbool
2777If \fBlog_avg_msec\fR is set, fio logs the average over that window. If
2778you instead want to log the maximum value, set this option to 1. Defaults to
27790, meaning that averaged values are logged.
1e613c9c 2780.TP
ae588852 2781.BI log_offset \fR=\fPbool
523bad63
TK
2782If this is set, the iolog options will include the byte offset for the I/O
2783entry as well as the other data values. Defaults to 0 meaning that
2784offsets are not present in logs. Also see \fBLOG FILE FORMATS\fR section.
ae588852 2785.TP
aee2ab67 2786.BI log_compression \fR=\fPint
523bad63
TK
2787If this is set, fio will compress the I/O logs as it goes, to keep the
2788memory footprint lower. When a log reaches the specified size, that chunk is
2789removed and compressed in the background. Given that I/O logs are fairly
2790highly compressible, this yields a nice memory savings for longer runs. The
2791downside is that the compression will consume some background CPU cycles, so
2792it may impact the run. This, however, is also true if the logging ends up
2793consuming most of the system memory. So pick your poison. The I/O logs are
2794saved normally at the end of a run, by decompressing the chunks and storing
2795them in the specified log file. This feature depends on the availability of
2796zlib.
aee2ab67 2797.TP
c08f9fe2 2798.BI log_compression_cpus \fR=\fPstr
523bad63
TK
2799Define the set of CPUs that are allowed to handle online log compression for
2800the I/O jobs. This can provide better isolation between performance
0cf90a62
SW
2801sensitive jobs, and background compression work. See \fBcpus_allowed\fR for
2802the format used.
c08f9fe2 2803.TP
b26317c9 2804.BI log_store_compressed \fR=\fPbool
c08f9fe2 2805If set, fio will store the log files in a compressed format. They can be
523bad63
TK
2806decompressed with fio, using the \fB\-\-inflate\-log\fR command line
2807parameter. The files will be stored with a `.fz' suffix.
b26317c9 2808.TP
3aea75b1
KC
2809.BI log_unix_epoch \fR=\fPbool
2810If set, fio will log Unix timestamps to the log files produced by enabling
523bad63 2811write_type_log for each log type, instead of the default zero\-based
3aea75b1
KC
2812timestamps.
2813.TP
66347cfa 2814.BI block_error_percentiles \fR=\fPbool
523bad63
TK
2815If set, record errors in trim block\-sized units from writes and trims and
2816output a histogram of how many trims it took to get to errors, and what kind
2817of error was encountered.
d60e92d1 2818.TP
523bad63
TK
2819.BI bwavgtime \fR=\fPint
2820Average the calculated bandwidth over the given time. Value is specified in
2821milliseconds. If the job also does bandwidth logging through
2822\fBwrite_bw_log\fR, then the minimum of this option and
2823\fBlog_avg_msec\fR will be used. Default: 500ms.
d60e92d1 2824.TP
523bad63
TK
2825.BI iopsavgtime \fR=\fPint
2826Average the calculated IOPS over the given time. Value is specified in
2827milliseconds. If the job also does IOPS logging through
2828\fBwrite_iops_log\fR, then the minimum of this option and
2829\fBlog_avg_msec\fR will be used. Default: 500ms.
d60e92d1 2830.TP
d60e92d1 2831.BI disk_util \fR=\fPbool
523bad63
TK
2832Generate disk utilization statistics, if the platform supports it.
2833Default: true.
fa769d44 2834.TP
523bad63
TK
2835.BI disable_lat \fR=\fPbool
2836Disable measurements of total latency numbers. Useful only for cutting back
2837the number of calls to \fBgettimeofday\fR\|(2), as that does impact
2838performance at really high IOPS rates. Note that to really get rid of a
2839large amount of these calls, this option must be used with
2840\fBdisable_slat\fR and \fBdisable_bw_measurement\fR as well.
9e684a49 2841.TP
523bad63
TK
2842.BI disable_clat \fR=\fPbool
2843Disable measurements of completion latency numbers. See
2844\fBdisable_lat\fR.
9e684a49 2845.TP
523bad63
TK
2846.BI disable_slat \fR=\fPbool
2847Disable measurements of submission latency numbers. See
2848\fBdisable_lat\fR.
9e684a49 2849.TP
523bad63
TK
2850.BI disable_bw_measurement \fR=\fPbool "\fR,\fP disable_bw" \fR=\fPbool
2851Disable measurements of throughput/bandwidth numbers. See
2852\fBdisable_lat\fR.
9e684a49 2853.TP
83349190 2854.BI clat_percentiles \fR=\fPbool
b599759b
JA
2855Enable the reporting of percentiles of completion latencies. This option is
2856mutually exclusive with \fBlat_percentiles\fR.
2857.TP
2858.BI lat_percentiles \fR=\fPbool
b71968b1 2859Enable the reporting of percentiles of I/O latencies. This is similar to
b599759b
JA
2860\fBclat_percentiles\fR, except that this includes the submission latency.
2861This option is mutually exclusive with \fBclat_percentiles\fR.
83349190
YH
2862.TP
2863.BI percentile_list \fR=\fPfloat_list
66347cfa 2864Overwrite the default list of percentiles for completion latencies and the
523bad63
TK
2865block error histogram. Each number is a floating number in the range
2866(0,100], and the maximum length of the list is 20. Use ':' to separate the
2867numbers, and list the numbers in ascending order. For example,
2868`\-\-percentile_list=99.5:99.9' will cause fio to report the values of
2869completion latency below which 99.5% and 99.9% of the observed latencies
2870fell, respectively.
e883cb35
JF
2871.TP
2872.BI significant_figures \fR=\fPint
c32ba107
JA
2873If using \fB\-\-output\-format\fR of `normal', set the significant figures
2874to this value. Higher values will yield more precise IOPS and throughput
2875units, while lower values will round. Requires a minimum value of 1 and a
e883cb35 2876maximum value of 10. Defaults to 4.
523bad63 2877.SS "Error handling"
e4585935 2878.TP
523bad63
TK
2879.BI exitall_on_error
2880When one job finishes in error, terminate the rest. The default is to wait
2881for each job to finish.
e4585935 2882.TP
523bad63
TK
2883.BI continue_on_error \fR=\fPstr
2884Normally fio will exit the job on the first observed failure. If this option
2885is set, fio will continue the job when there is a 'non\-fatal error' (EIO or
2886EILSEQ) until the runtime is exceeded or the I/O size specified is
2887completed. If this option is used, there are two more stats that are
2888appended, the total error count and the first error. The error field given
2889in the stats is the first error that was hit during the run.
2890The allowed values are:
2891.RS
2892.RS
046395d7 2893.TP
523bad63
TK
2894.B none
2895Exit on any I/O or verify errors.
de890a1e 2896.TP
523bad63
TK
2897.B read
2898Continue on read errors, exit on all others.
2cafffbe 2899.TP
523bad63
TK
2900.B write
2901Continue on write errors, exit on all others.
a0679ce5 2902.TP
523bad63
TK
2903.B io
2904Continue on any I/O error, exit on all others.
de890a1e 2905.TP
523bad63
TK
2906.B verify
2907Continue on verify errors, exit on all others.
de890a1e 2908.TP
523bad63
TK
2909.B all
2910Continue on all errors.
b93b6a2e 2911.TP
523bad63
TK
2912.B 0
2913Backward\-compatible alias for 'none'.
d3a623de 2914.TP
523bad63
TK
2915.B 1
2916Backward\-compatible alias for 'all'.
2917.RE
2918.RE
1d360ffb 2919.TP
523bad63
TK
2920.BI ignore_error \fR=\fPstr
2921Sometimes you want to ignore some errors during test in that case you can
2922specify error list for each error type, instead of only being able to
2923ignore the default 'non\-fatal error' using \fBcontinue_on_error\fR.
2924`ignore_error=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST' errors for
2925given error type is separated with ':'. Error may be symbol ('ENOSPC', 'ENOMEM')
2926or integer. Example:
de890a1e
SL
2927.RS
2928.RS
523bad63
TK
2929.P
2930ignore_error=EAGAIN,ENOSPC:122
2931.RE
2932.P
2933This option will ignore EAGAIN from READ, and ENOSPC and 122(EDQUOT) from
2934WRITE. This option works by overriding \fBcontinue_on_error\fR with
2935the list of errors for each error type if any.
2936.RE
de890a1e 2937.TP
523bad63
TK
2938.BI error_dump \fR=\fPbool
2939If set dump every error even if it is non fatal, true by default. If
2940disabled only fatal error will be dumped.
2941.SS "Running predefined workloads"
2942Fio includes predefined profiles that mimic the I/O workloads generated by
2943other tools.
49ccb8c1 2944.TP
523bad63
TK
2945.BI profile \fR=\fPstr
2946The predefined workload to run. Current profiles are:
2947.RS
2948.RS
de890a1e 2949.TP
523bad63
TK
2950.B tiobench
2951Threaded I/O bench (tiotest/tiobench) like workload.
49ccb8c1 2952.TP
523bad63
TK
2953.B act
2954Aerospike Certification Tool (ACT) like workload.
2955.RE
de890a1e
SL
2956.RE
2957.P
523bad63
TK
2958To view a profile's additional options use \fB\-\-cmdhelp\fR after specifying
2959the profile. For example:
2960.RS
2961.TP
2962$ fio \-\-profile=act \-\-cmdhelp
de890a1e 2963.RE
523bad63 2964.SS "Act profile options"
de890a1e 2965.TP
523bad63
TK
2966.BI device\-names \fR=\fPstr
2967Devices to use.
d54fce84 2968.TP
523bad63
TK
2969.BI load \fR=\fPint
2970ACT load multiplier. Default: 1.
7aeb1e94 2971.TP
523bad63
TK
2972.BI test\-duration\fR=\fPtime
2973How long the entire test takes to run. When the unit is omitted, the value
2974is given in seconds. Default: 24h.
1008602c 2975.TP
523bad63
TK
2976.BI threads\-per\-queue\fR=\fPint
2977Number of read I/O threads per device. Default: 8.
e5f34d95 2978.TP
523bad63
TK
2979.BI read\-req\-num\-512\-blocks\fR=\fPint
2980Number of 512B blocks to read at the time. Default: 3.
d54fce84 2981.TP
523bad63
TK
2982.BI large\-block\-op\-kbytes\fR=\fPint
2983Size of large block ops in KiB (writes). Default: 131072.
d54fce84 2984.TP
523bad63
TK
2985.BI prep
2986Set to run ACT prep phase.
2987.SS "Tiobench profile options"
6d500c2e 2988.TP
523bad63
TK
2989.BI size\fR=\fPstr
2990Size in MiB.
0d978694 2991.TP
523bad63
TK
2992.BI block\fR=\fPint
2993Block size in bytes. Default: 4096.
0d978694 2994.TP
523bad63
TK
2995.BI numruns\fR=\fPint
2996Number of runs.
0d978694 2997.TP
523bad63
TK
2998.BI dir\fR=\fPstr
2999Test directory.
65fa28ca 3000.TP
523bad63
TK
3001.BI threads\fR=\fPint
3002Number of threads.
d60e92d1 3003.SH OUTPUT
40943b9a
TK
3004Fio spits out a lot of output. While running, fio will display the status of the
3005jobs created. An example of that would be:
d60e92d1 3006.P
40943b9a
TK
3007.nf
3008 Jobs: 1 (f=1): [_(1),M(1)][24.8%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 01m:31s]
3009.fi
d1429b5c 3010.P
40943b9a
TK
3011The characters inside the first set of square brackets denote the current status of
3012each thread. The first character is the first job defined in the job file, and so
3013forth. The possible values (in typical life cycle order) are:
d60e92d1
AC
3014.RS
3015.TP
40943b9a 3016.PD 0
d60e92d1 3017.B P
40943b9a 3018Thread setup, but not started.
d60e92d1
AC
3019.TP
3020.B C
3021Thread created.
3022.TP
3023.B I
40943b9a
TK
3024Thread initialized, waiting or generating necessary data.
3025.TP
522c29f6 3026.B p
40943b9a
TK
3027Thread running pre\-reading file(s).
3028.TP
3029.B /
3030Thread is in ramp period.
d60e92d1
AC
3031.TP
3032.B R
3033Running, doing sequential reads.
3034.TP
3035.B r
3036Running, doing random reads.
3037.TP
3038.B W
3039Running, doing sequential writes.
3040.TP
3041.B w
3042Running, doing random writes.
3043.TP
3044.B M
3045Running, doing mixed sequential reads/writes.
3046.TP
3047.B m
3048Running, doing mixed random reads/writes.
3049.TP
40943b9a
TK
3050.B D
3051Running, doing sequential trims.
3052.TP
3053.B d
3054Running, doing random trims.
3055.TP
d60e92d1
AC
3056.B F
3057Running, currently waiting for \fBfsync\fR\|(2).
3058.TP
3059.B V
40943b9a
TK
3060Running, doing verification of written data.
3061.TP
3062.B f
3063Thread finishing.
d60e92d1
AC
3064.TP
3065.B E
40943b9a 3066Thread exited, not reaped by main thread yet.
d60e92d1
AC
3067.TP
3068.B \-
40943b9a
TK
3069Thread reaped.
3070.TP
3071.B X
3072Thread reaped, exited with an error.
3073.TP
3074.B K
3075Thread reaped, exited due to signal.
d1429b5c 3076.PD
40943b9a
TK
3077.RE
3078.P
3079Fio will condense the thread string as not to take up more space on the command
3080line than needed. For instance, if you have 10 readers and 10 writers running,
3081the output would look like this:
3082.P
3083.nf
3084 Jobs: 20 (f=20): [R(10),W(10)][4.0%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 57m:36s]
3085.fi
d60e92d1 3086.P
40943b9a
TK
3087Note that the status string is displayed in order, so it's possible to tell which of
3088the jobs are currently doing what. In the example above this means that jobs 1\-\-10
3089are readers and 11\-\-20 are writers.
d60e92d1 3090.P
40943b9a
TK
3091The other values are fairly self explanatory \-\- number of threads currently
3092running and doing I/O, the number of currently open files (f=), the estimated
3093completion percentage, the rate of I/O since last check (read speed listed first,
3094then write speed and optionally trim speed) in terms of bandwidth and IOPS,
3095and time to completion for the current running group. It's impossible to estimate
3096runtime of the following groups (if any).
d60e92d1 3097.P
40943b9a
TK
3098When fio is done (or interrupted by Ctrl\-C), it will show the data for
3099each thread, group of threads, and disks in that order. For each overall thread (or
3100group) the output looks like:
3101.P
3102.nf
3103 Client1: (groupid=0, jobs=1): err= 0: pid=16109: Sat Jun 24 12:07:54 2017
3104 write: IOPS=88, BW=623KiB/s (638kB/s)(30.4MiB/50032msec)
3105 slat (nsec): min=500, max=145500, avg=8318.00, stdev=4781.50
3106 clat (usec): min=170, max=78367, avg=4019.02, stdev=8293.31
3107 lat (usec): min=174, max=78375, avg=4027.34, stdev=8291.79
3108 clat percentiles (usec):
3109 | 1.00th=[ 302], 5.00th=[ 326], 10.00th=[ 343], 20.00th=[ 363],
3110 | 30.00th=[ 392], 40.00th=[ 404], 50.00th=[ 416], 60.00th=[ 445],
3111 | 70.00th=[ 816], 80.00th=[ 6718], 90.00th=[12911], 95.00th=[21627],
3112 | 99.00th=[43779], 99.50th=[51643], 99.90th=[68682], 99.95th=[72877],
3113 | 99.99th=[78119]
3114 bw ( KiB/s): min= 532, max= 686, per=0.10%, avg=622.87, stdev=24.82, samples= 100
3115 iops : min= 76, max= 98, avg=88.98, stdev= 3.54, samples= 100
d3b9694d
VF
3116 lat (usec) : 250=0.04%, 500=64.11%, 750=4.81%, 1000=2.79%
3117 lat (msec) : 2=4.16%, 4=1.84%, 10=4.90%, 20=11.33%, 50=5.37%
3118 lat (msec) : 100=0.65%
40943b9a
TK
3119 cpu : usr=0.27%, sys=0.18%, ctx=12072, majf=0, minf=21
3120 IO depths : 1=85.0%, 2=13.1%, 4=1.8%, 8=0.1%, 16=0.0%, 32=0.0%, >=64=0.0%
3121 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
3122 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
3123 issued rwt: total=0,4450,0, short=0,0,0, dropped=0,0,0
3124 latency : target=0, window=0, percentile=100.00%, depth=8
3125.fi
3126.P
3127The job name (or first job's name when using \fBgroup_reporting\fR) is printed,
3128along with the group id, count of jobs being aggregated, last error id seen (which
3129is 0 when there are no errors), pid/tid of that thread and the time the job/group
3130completed. Below are the I/O statistics for each data direction performed (showing
3131writes in the example above). In the order listed, they denote:
d60e92d1 3132.RS
d60e92d1 3133.TP
40943b9a
TK
3134.B read/write/trim
3135The string before the colon shows the I/O direction the statistics
3136are for. \fIIOPS\fR is the average I/Os performed per second. \fIBW\fR
3137is the average bandwidth rate shown as: value in power of 2 format
3138(value in power of 10 format). The last two values show: (total
3139I/O performed in power of 2 format / \fIruntime\fR of that thread).
d60e92d1
AC
3140.TP
3141.B slat
40943b9a
TK
3142Submission latency (\fImin\fR being the minimum, \fImax\fR being the
3143maximum, \fIavg\fR being the average, \fIstdev\fR being the standard
3144deviation). This is the time it took to submit the I/O. For
3145sync I/O this row is not displayed as the slat is really the
3146completion latency (since queue/complete is one operation there).
3147This value can be in nanoseconds, microseconds or milliseconds \-\-\-
3148fio will choose the most appropriate base and print that (in the
3149example above nanoseconds was the best scale). Note: in \fB\-\-minimal\fR mode
3150latencies are always expressed in microseconds.
d60e92d1
AC
3151.TP
3152.B clat
40943b9a
TK
3153Completion latency. Same names as slat, this denotes the time from
3154submission to completion of the I/O pieces. For sync I/O, clat will
3155usually be equal (or very close) to 0, as the time from submit to
3156complete is basically just CPU time (I/O has already been done, see slat
3157explanation).
d60e92d1 3158.TP
d3b9694d
VF
3159.B lat
3160Total latency. Same names as slat and clat, this denotes the time from
3161when fio created the I/O unit to completion of the I/O operation.
3162.TP
d60e92d1 3163.B bw
40943b9a
TK
3164Bandwidth statistics based on samples. Same names as the xlat stats,
3165but also includes the number of samples taken (\fIsamples\fR) and an
3166approximate percentage of total aggregate bandwidth this thread
3167received in its group (\fIper\fR). This last value is only really
3168useful if the threads in this group are on the same disk, since they
3169are then competing for disk access.
3170.TP
3171.B iops
3172IOPS statistics based on samples. Same names as \fBbw\fR.
d60e92d1 3173.TP
d3b9694d
VF
3174.B lat (nsec/usec/msec)
3175The distribution of I/O completion latencies. This is the time from when
3176I/O leaves fio and when it gets completed. Unlike the separate
3177read/write/trim sections above, the data here and in the remaining
3178sections apply to all I/Os for the reporting group. 250=0.04% means that
31790.04% of the I/Os completed in under 250us. 500=64.11% means that 64.11%
3180of the I/Os required 250 to 499us for completion.
3181.TP
d60e92d1 3182.B cpu
40943b9a
TK
3183CPU usage. User and system time, along with the number of context
3184switches this thread went through, usage of system and user time, and
3185finally the number of major and minor page faults. The CPU utilization
3186numbers are averages for the jobs in that reporting group, while the
3187context and fault counters are summed.
d60e92d1
AC
3188.TP
3189.B IO depths
40943b9a
TK
3190The distribution of I/O depths over the job lifetime. The numbers are
3191divided into powers of 2 and each entry covers depths from that value
3192up to those that are lower than the next entry \-\- e.g., 16= covers
3193depths from 16 to 31. Note that the range covered by a depth
3194distribution entry can be different to the range covered by the
3195equivalent \fBsubmit\fR/\fBcomplete\fR distribution entry.
3196.TP
3197.B IO submit
3198How many pieces of I/O were submitting in a single submit call. Each
3199entry denotes that amount and below, until the previous entry \-\- e.g.,
320016=100% means that we submitted anywhere between 9 to 16 I/Os per submit
3201call. Note that the range covered by a \fBsubmit\fR distribution entry can
3202be different to the range covered by the equivalent depth distribution
3203entry.
3204.TP
3205.B IO complete
3206Like the above \fBsubmit\fR number, but for completions instead.
3207.TP
3208.B IO issued rwt
3209The number of \fBread/write/trim\fR requests issued, and how many of them were
3210short or dropped.
d60e92d1 3211.TP
d3b9694d 3212.B IO latency
ee21ebee 3213These values are for \fBlatency_target\fR and related options. When
d3b9694d
VF
3214these options are engaged, this section describes the I/O depth required
3215to meet the specified latency target.
d60e92d1 3216.RE
d60e92d1 3217.P
40943b9a
TK
3218After each client has been listed, the group statistics are printed. They
3219will look like this:
3220.P
3221.nf
3222 Run status group 0 (all jobs):
3223 READ: bw=20.9MiB/s (21.9MB/s), 10.4MiB/s\-10.8MiB/s (10.9MB/s\-11.3MB/s), io=64.0MiB (67.1MB), run=2973\-3069msec
3224 WRITE: bw=1231KiB/s (1261kB/s), 616KiB/s\-621KiB/s (630kB/s\-636kB/s), io=64.0MiB (67.1MB), run=52747\-53223msec
3225.fi
3226.P
3227For each data direction it prints:
d60e92d1
AC
3228.RS
3229.TP
40943b9a
TK
3230.B bw
3231Aggregate bandwidth of threads in this group followed by the
3232minimum and maximum bandwidth of all the threads in this group.
3233Values outside of brackets are power\-of\-2 format and those
3234within are the equivalent value in a power\-of\-10 format.
d60e92d1 3235.TP
40943b9a
TK
3236.B io
3237Aggregate I/O performed of all threads in this group. The
3238format is the same as \fBbw\fR.
d60e92d1 3239.TP
40943b9a
TK
3240.B run
3241The smallest and longest runtimes of the threads in this group.
d60e92d1 3242.RE
d60e92d1 3243.P
40943b9a
TK
3244And finally, the disk statistics are printed. This is Linux specific.
3245They will look like this:
3246.P
3247.nf
3248 Disk stats (read/write):
3249 sda: ios=16398/16511, merge=30/162, ticks=6853/819634, in_queue=826487, util=100.00%
3250.fi
3251.P
3252Each value is printed for both reads and writes, with reads first. The
3253numbers denote:
d60e92d1
AC
3254.RS
3255.TP
3256.B ios
3257Number of I/Os performed by all groups.
3258.TP
3259.B merge
007c7be9 3260Number of merges performed by the I/O scheduler.
d60e92d1
AC
3261.TP
3262.B ticks
3263Number of ticks we kept the disk busy.
3264.TP
40943b9a 3265.B in_queue
d60e92d1
AC
3266Total time spent in the disk queue.
3267.TP
3268.B util
40943b9a
TK
3269The disk utilization. A value of 100% means we kept the disk
3270busy constantly, 50% would be a disk idling half of the time.
d60e92d1 3271.RE
8423bd11 3272.P
40943b9a
TK
3273It is also possible to get fio to dump the current output while it is running,
3274without terminating the job. To do that, send fio the USR1 signal. You can
3275also get regularly timed dumps by using the \fB\-\-status\-interval\fR
3276parameter, or by creating a file in `/tmp' named
3277`fio\-dump\-status'. If fio sees this file, it will unlink it and dump the
3278current output status.
d60e92d1 3279.SH TERSE OUTPUT
40943b9a
TK
3280For scripted usage where you typically want to generate tables or graphs of the
3281results, fio can output the results in a semicolon separated format. The format
3282is one long line of values, such as:
d60e92d1 3283.P
40943b9a
TK
3284.nf
3285 2;card0;0;0;7139336;121836;60004;1;10109;27.932460;116.933948;220;126861;3495.446807;1085.368601;226;126864;3523.635629;1089.012448;24063;99944;50.275485%;59818.274627;5540.657370;7155060;122104;60004;1;8338;29.086342;117.839068;388;128077;5032.488518;1234.785715;391;128085;5061.839412;1236.909129;23436;100928;50.287926%;59964.832030;5644.844189;14.595833%;19.394167%;123706;0;7313;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.02%;0.05%;0.16%;6.04%;40.40%;52.68%;0.64%;0.01%;0.00%;0.01%;0.00%;0.00%;0.00%;0.00%;0.00%
3286 A description of this job goes here.
3287.fi
d60e92d1 3288.P
40943b9a 3289The job description (if provided) follows on a second line.
d60e92d1 3290.P
40943b9a
TK
3291To enable terse output, use the \fB\-\-minimal\fR or
3292`\-\-output\-format=terse' command line options. The
3293first value is the version of the terse output format. If the output has to be
3294changed for some reason, this number will be incremented by 1 to signify that
3295change.
d60e92d1 3296.P
40943b9a
TK
3297Split up, the format is as follows (comments in brackets denote when a
3298field was introduced or whether it's specific to some terse version):
d60e92d1 3299.P
40943b9a
TK
3300.nf
3301 terse version, fio version [v3], jobname, groupid, error
3302.fi
525c2bfa 3303.RS
40943b9a
TK
3304.P
3305.B
3306READ status:
525c2bfa 3307.RE
40943b9a
TK
3308.P
3309.nf
3310 Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
3311 Submission latency: min, max, mean, stdev (usec)
3312 Completion latency: min, max, mean, stdev (usec)
3313 Completion latency percentiles: 20 fields (see below)
3314 Total latency: min, max, mean, stdev (usec)
3315 Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev, number of samples [v5]
3316 IOPS [v5]: min, max, mean, stdev, number of samples
3317.fi
d60e92d1 3318.RS
40943b9a
TK
3319.P
3320.B
3321WRITE status:
a2c95580 3322.RE
40943b9a
TK
3323.P
3324.nf
3325 Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
3326 Submission latency: min, max, mean, stdev (usec)
3327 Completion latency: min, max, mean, stdev (usec)
3328 Completion latency percentiles: 20 fields (see below)
3329 Total latency: min, max, mean, stdev (usec)
3330 Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev, number of samples [v5]
3331 IOPS [v5]: min, max, mean, stdev, number of samples
3332.fi
a2c95580 3333.RS
40943b9a
TK
3334.P
3335.B
3336TRIM status [all but version 3]:
d60e92d1
AC
3337.RE
3338.P
40943b9a
TK
3339.nf
3340 Fields are similar to \fBREAD/WRITE\fR status.
3341.fi
a2c95580 3342.RS
a2c95580 3343.P
40943b9a 3344.B
d1429b5c 3345CPU usage:
d60e92d1
AC
3346.RE
3347.P
40943b9a
TK
3348.nf
3349 user, system, context switches, major faults, minor faults
3350.fi
d60e92d1 3351.RS
40943b9a
TK
3352.P
3353.B
3354I/O depths:
d60e92d1
AC
3355.RE
3356.P
40943b9a
TK
3357.nf
3358 <=1, 2, 4, 8, 16, 32, >=64
3359.fi
562c2d2f 3360.RS
40943b9a
TK
3361.P
3362.B
3363I/O latencies microseconds:
562c2d2f 3364.RE
40943b9a
TK
3365.P
3366.nf
3367 <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000
3368.fi
562c2d2f 3369.RS
40943b9a
TK
3370.P
3371.B
3372I/O latencies milliseconds:
562c2d2f
DN
3373.RE
3374.P
40943b9a
TK
3375.nf
3376 <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000
3377.fi
f2f788dd 3378.RS
40943b9a
TK
3379.P
3380.B
3381Disk utilization [v3]:
f2f788dd
JA
3382.RE
3383.P
40943b9a
TK
3384.nf
3385 disk name, read ios, write ios, read merges, write merges, read ticks, write ticks, time spent in queue, disk utilization percentage
3386.fi
562c2d2f 3387.RS
d60e92d1 3388.P
40943b9a
TK
3389.B
3390Additional Info (dependent on continue_on_error, default off):
d60e92d1 3391.RE
2fc26c3d 3392.P
40943b9a
TK
3393.nf
3394 total # errors, first error code
3395.fi
2fc26c3d
IC
3396.RS
3397.P
40943b9a
TK
3398.B
3399Additional Info (dependent on description being set):
3400.RE
3401.P
2fc26c3d 3402.nf
40943b9a
TK
3403 Text description
3404.fi
3405.P
3406Completion latency percentiles can be a grouping of up to 20 sets, so for the
3407terse output fio writes all of them. Each field will look like this:
3408.P
3409.nf
3410 1.00%=6112
3411.fi
3412.P
3413which is the Xth percentile, and the `usec' latency associated with it.
3414.P
3415For \fBDisk utilization\fR, all disks used by fio are shown. So for each disk there
3416will be a disk utilization section.
3417.P
3418Below is a single line containing short names for each of the fields in the
3419minimal output v3, separated by semicolons:
3420.P
3421.nf
3422 terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth;read_iops;read_runtime_ms;read_slat_min;read_slat_max;read_slat_mean;read_slat_dev;read_clat_min;read_clat_max;read_clat_mean;read_clat_dev;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min;read_lat_max;read_lat_mean;read_lat_dev;read_bw_min;read_bw_max;read_bw_agg_pct;read_bw_mean;read_bw_dev;write_kb;write_bandwidth;write_iops;write_runtime_ms;write_slat_min;write_slat_max;write_slat_mean;write_slat_dev;write_clat_min;write_clat_max;write_clat_mean;write_clat_dev;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min;write_lat_max;write_lat_mean;write_lat_dev;write_bw_min;write_bw_max;write_bw_agg_pct;write_bw_mean;write_bw_dev;cpu_user;cpu_sys;cpu_csw;cpu_mjf;cpu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util
2fc26c3d 3423.fi
44c82dba
VF
3424.SH JSON OUTPUT
3425The \fBjson\fR output format is intended to be both human readable and convenient
3426for automated parsing. For the most part its sections mirror those of the
3427\fBnormal\fR output. The \fBruntime\fR value is reported in msec and the \fBbw\fR value is
3428reported in 1024 bytes per second units.
3429.fi
d9e557ab
VF
3430.SH JSON+ OUTPUT
3431The \fBjson+\fR output format is identical to the \fBjson\fR output format except that it
3432adds a full dump of the completion latency bins. Each \fBbins\fR object contains a
3433set of (key, value) pairs where keys are latency durations and values count how
3434many I/Os had completion latencies of the corresponding duration. For example,
3435consider:
d9e557ab 3436.RS
40943b9a 3437.P
d9e557ab
VF
3438"bins" : { "87552" : 1, "89600" : 1, "94720" : 1, "96768" : 1, "97792" : 1, "99840" : 1, "100864" : 2, "103936" : 6, "104960" : 534, "105984" : 5995, "107008" : 7529, ... }
3439.RE
40943b9a 3440.P
d9e557ab
VF
3441This data indicates that one I/O required 87,552ns to complete, two I/Os required
3442100,864ns to complete, and 7529 I/Os required 107,008ns to complete.
40943b9a 3443.P
d9e557ab 3444Also included with fio is a Python script \fBfio_jsonplus_clat2csv\fR that takes
40943b9a
TK
3445json+ output and generates CSV\-formatted latency data suitable for plotting.
3446.P
d9e557ab 3447The latency durations actually represent the midpoints of latency intervals.
40943b9a 3448For details refer to `stat.h' in the fio source.
29dbd1e5 3449.SH TRACE FILE FORMAT
40943b9a
TK
3450There are two trace file format that you can encounter. The older (v1) format is
3451unsupported since version 1.20\-rc3 (March 2008). It will still be described
29dbd1e5 3452below in case that you get an old trace and want to understand it.
29dbd1e5 3453.P
40943b9a
TK
3454In any case the trace is a simple text file with a single action per line.
3455.TP
29dbd1e5 3456.B Trace file format v1
40943b9a 3457Each line represents a single I/O action in the following format:
29dbd1e5 3458.RS
40943b9a
TK
3459.RS
3460.P
29dbd1e5 3461rw, offset, length
29dbd1e5
JA
3462.RE
3463.P
40943b9a
TK
3464where `rw=0/1' for read/write, and the `offset' and `length' entries being in bytes.
3465.P
3466This format is not supported in fio versions >= 1.20\-rc3.
3467.RE
3468.TP
29dbd1e5 3469.B Trace file format v2
40943b9a
TK
3470The second version of the trace file format was added in fio version 1.17. It
3471allows to access more then one file per trace and has a bigger set of possible
3472file actions.
29dbd1e5 3473.RS
40943b9a 3474.P
29dbd1e5 3475The first line of the trace file has to be:
40943b9a
TK
3476.RS
3477.P
3478"fio version 2 iolog"
3479.RE
3480.P
29dbd1e5 3481Following this can be lines in two different formats, which are described below.
40943b9a
TK
3482.P
3483.B
29dbd1e5 3484The file management format:
40943b9a
TK
3485.RS
3486filename action
29dbd1e5 3487.P
40943b9a 3488The `filename' is given as an absolute path. The `action' can be one of these:
29dbd1e5
JA
3489.RS
3490.TP
3491.B add
40943b9a 3492Add the given `filename' to the trace.
29dbd1e5
JA
3493.TP
3494.B open
40943b9a
TK
3495Open the file with the given `filename'. The `filename' has to have
3496been added with the \fBadd\fR action before.
29dbd1e5
JA
3497.TP
3498.B close
40943b9a
TK
3499Close the file with the given `filename'. The file has to have been
3500\fBopen\fRed before.
3501.RE
29dbd1e5 3502.RE
29dbd1e5 3503.P
40943b9a
TK
3504.B
3505The file I/O action format:
3506.RS
3507filename action offset length
29dbd1e5 3508.P
40943b9a
TK
3509The `filename' is given as an absolute path, and has to have been \fBadd\fRed and
3510\fBopen\fRed before it can be used with this format. The `offset' and `length' are
3511given in bytes. The `action' can be one of these:
29dbd1e5
JA
3512.RS
3513.TP
3514.B wait
40943b9a
TK
3515Wait for `offset' microseconds. Everything below 100 is discarded.
3516The time is relative to the previous `wait' statement.
29dbd1e5
JA
3517.TP
3518.B read
40943b9a 3519Read `length' bytes beginning from `offset'.
29dbd1e5
JA
3520.TP
3521.B write
40943b9a 3522Write `length' bytes beginning from `offset'.
29dbd1e5
JA
3523.TP
3524.B sync
40943b9a 3525\fBfsync\fR\|(2) the file.
29dbd1e5
JA
3526.TP
3527.B datasync
40943b9a 3528\fBfdatasync\fR\|(2) the file.
29dbd1e5
JA
3529.TP
3530.B trim
40943b9a
TK
3531Trim the given file from the given `offset' for `length' bytes.
3532.RE
29dbd1e5 3533.RE
29dbd1e5 3534.SH CPU IDLENESS PROFILING
40943b9a
TK
3535In some cases, we want to understand CPU overhead in a test. For example, we
3536test patches for the specific goodness of whether they reduce CPU usage.
3537Fio implements a balloon approach to create a thread per CPU that runs at idle
3538priority, meaning that it only runs when nobody else needs the cpu.
3539By measuring the amount of work completed by the thread, idleness of each CPU
3540can be derived accordingly.
3541.P
3542An unit work is defined as touching a full page of unsigned characters. Mean and
3543standard deviation of time to complete an unit work is reported in "unit work"
3544section. Options can be chosen to report detailed percpu idleness or overall
3545system idleness by aggregating percpu stats.
29dbd1e5 3546.SH VERIFICATION AND TRIGGERS
40943b9a
TK
3547Fio is usually run in one of two ways, when data verification is done. The first
3548is a normal write job of some sort with verify enabled. When the write phase has
3549completed, fio switches to reads and verifies everything it wrote. The second
3550model is running just the write phase, and then later on running the same job
3551(but with reads instead of writes) to repeat the same I/O patterns and verify
3552the contents. Both of these methods depend on the write phase being completed,
3553as fio otherwise has no idea how much data was written.
3554.P
3555With verification triggers, fio supports dumping the current write state to
3556local files. Then a subsequent read verify workload can load this state and know
3557exactly where to stop. This is useful for testing cases where power is cut to a
3558server in a managed fashion, for instance.
3559.P
29dbd1e5 3560A verification trigger consists of two things:
29dbd1e5 3561.RS
40943b9a
TK
3562.P
35631) Storing the write state of each job.
3564.P
35652) Executing a trigger command.
29dbd1e5 3566.RE
40943b9a
TK
3567.P
3568The write state is relatively small, on the order of hundreds of bytes to single
3569kilobytes. It contains information on the number of completions done, the last X
3570completions, etc.
3571.P
3572A trigger is invoked either through creation ('touch') of a specified file in
3573the system, or through a timeout setting. If fio is run with
3574`\-\-trigger\-file=/tmp/trigger\-file', then it will continually
3575check for the existence of `/tmp/trigger\-file'. When it sees this file, it
3576will fire off the trigger (thus saving state, and executing the trigger
29dbd1e5 3577command).
40943b9a
TK
3578.P
3579For client/server runs, there's both a local and remote trigger. If fio is
3580running as a server backend, it will send the job states back to the client for
3581safe storage, then execute the remote trigger, if specified. If a local trigger
3582is specified, the server will still send back the write state, but the client
3583will then execute the trigger.
29dbd1e5
JA
3584.RE
3585.P
3586.B Verification trigger example
3587.RS
40943b9a
TK
3588Let's say we want to run a powercut test on the remote Linux machine 'server'.
3589Our write workload is in `write\-test.fio'. We want to cut power to 'server' at
3590some point during the run, and we'll run this test from the safety or our local
3591machine, 'localbox'. On the server, we'll start the fio backend normally:
3592.RS
3593.P
3594server# fio \-\-server
3595.RE
3596.P
29dbd1e5 3597and on the client, we'll fire off the workload:
40943b9a
TK
3598.RS
3599.P
3600localbox$ fio \-\-client=server \-\-trigger\-file=/tmp/my\-trigger \-\-trigger\-remote="bash \-c "echo b > /proc/sysrq\-triger""
3601.RE
3602.P
3603We set `/tmp/my\-trigger' as the trigger file, and we tell fio to execute:
3604.RS
3605.P
3606echo b > /proc/sysrq\-trigger
3607.RE
3608.P
3609on the server once it has received the trigger and sent us the write state. This
3610will work, but it's not really cutting power to the server, it's merely
3611abruptly rebooting it. If we have a remote way of cutting power to the server
3612through IPMI or similar, we could do that through a local trigger command
3613instead. Let's assume we have a script that does IPMI reboot of a given hostname,
3614ipmi\-reboot. On localbox, we could then have run fio with a local trigger
3615instead:
3616.RS
3617.P
3618localbox$ fio \-\-client=server \-\-trigger\-file=/tmp/my\-trigger \-\-trigger="ipmi\-reboot server"
3619.RE
3620.P
3621For this case, fio would wait for the server to send us the write state, then
3622execute `ipmi\-reboot server' when that happened.
29dbd1e5
JA
3623.RE
3624.P
3625.B Loading verify state
3626.RS
40943b9a
TK
3627To load stored write state, a read verification job file must contain the
3628\fBverify_state_load\fR option. If that is set, fio will load the previously
29dbd1e5 3629stored state. For a local fio run this is done by loading the files directly,
40943b9a
TK
3630and on a client/server run, the server backend will ask the client to send the
3631files over and load them from there.
29dbd1e5 3632.RE
a3ae5b05 3633.SH LOG FILE FORMATS
a3ae5b05
JA
3634Fio supports a variety of log file formats, for logging latencies, bandwidth,
3635and IOPS. The logs share a common format, which looks like this:
40943b9a 3636.RS
a3ae5b05 3637.P
40943b9a
TK
3638time (msec), value, data direction, block size (bytes), offset (bytes)
3639.RE
3640.P
3641`Time' for the log entry is always in milliseconds. The `value' logged depends
3642on the type of log, it will be one of the following:
3643.RS
a3ae5b05
JA
3644.TP
3645.B Latency log
168bb587 3646Value is latency in nsecs
a3ae5b05
JA
3647.TP
3648.B Bandwidth log
6d500c2e 3649Value is in KiB/sec
a3ae5b05
JA
3650.TP
3651.B IOPS log
40943b9a
TK
3652Value is IOPS
3653.RE
a3ae5b05 3654.P
40943b9a
TK
3655`Data direction' is one of the following:
3656.RS
a3ae5b05
JA
3657.TP
3658.B 0
40943b9a 3659I/O is a READ
a3ae5b05
JA
3660.TP
3661.B 1
40943b9a 3662I/O is a WRITE
a3ae5b05
JA
3663.TP
3664.B 2