fio.1: update description for 'buffer_pattern' and 'verify_pattern'
[fio.git] / fio.1
CommitLineData
f8b8f7da 1.TH fio 1 "December 2014" "User Manual"
d60e92d1
AC
2.SH NAME
3fio \- flexible I/O tester
4.SH SYNOPSIS
5.B fio
6[\fIoptions\fR] [\fIjobfile\fR]...
7.SH DESCRIPTION
8.B fio
9is a tool that will spawn a number of threads or processes doing a
10particular type of I/O action as specified by the user.
11The typical use of fio is to write a job file matching the I/O load
12one wants to simulate.
13.SH OPTIONS
14.TP
49da1240
JA
15.BI \-\-debug \fR=\fPtype
16Enable verbose tracing of various fio actions. May be `all' for all types
17or individual types separated by a comma (eg \-\-debug=io,file). `help' will
18list all available tracing options.
19.TP
d60e92d1
AC
20.BI \-\-output \fR=\fPfilename
21Write output to \fIfilename\fR.
22.TP
e28ee21d 23.BI \-\-output-format \fR=\fPformat
52a768c1 24Set the reporting format to \fInormal\fR, \fIterse\fR, or \fIjson\fR.
e28ee21d 25.TP
b2cecdc2 26.BI \-\-runtime \fR=\fPruntime
27Limit run time to \fIruntime\fR seconds.
d60e92d1 28.TP
d60e92d1
AC
29.B \-\-bandwidth\-log
30Generate per-job bandwidth logs.
31.TP
32.B \-\-minimal
d1429b5c 33Print statistics in a terse, semicolon-delimited format.
d60e92d1 34.TP
2b8c71b0
CE
35.B \-\-append-terse
36Print statistics in selected mode AND terse, semicolon-delimited format.
37.TP
49da1240
JA
38.B \-\-version
39Display version information and exit.
40.TP
065248bf 41.BI \-\-terse\-version \fR=\fPversion
4d658652 42Set terse version output format (Current version 3, or older version 2).
49da1240
JA
43.TP
44.B \-\-help
45Display usage information and exit.
46.TP
fec0f21c
JA
47.B \-\-cpuclock-test
48Perform test and validation of internal CPU clock
49.TP
50.BI \-\-crctest[\fR=\fPtest]
51Test the speed of the builtin checksumming functions. If no argument is given,
52all of them are tested. Or a comma separated list can be passed, in which
53case the given ones are tested.
54.TP
49da1240
JA
55.BI \-\-cmdhelp \fR=\fPcommand
56Print help information for \fIcommand\fR. May be `all' for all commands.
57.TP
de890a1e
SL
58.BI \-\-enghelp \fR=\fPioengine[,command]
59List all commands defined by \fIioengine\fR, or print help for \fIcommand\fR defined by \fIioengine\fR.
60.TP
d60e92d1
AC
61.BI \-\-showcmd \fR=\fPjobfile
62Convert \fIjobfile\fR to a set of command-line options.
63.TP
d60e92d1
AC
64.BI \-\-eta \fR=\fPwhen
65Specifies when real-time ETA estimate should be printed. \fIwhen\fR may
66be one of `always', `never' or `auto'.
67.TP
30b5d57f
JA
68.BI \-\-eta\-newline \fR=\fPtime
69Force an ETA newline for every `time` period passed.
70.TP
71.BI \-\-status\-interval \fR=\fPtime
72Report full output status every `time` period passed.
73.TP
49da1240
JA
74.BI \-\-readonly
75Turn on safety read-only checks, preventing any attempted write.
76.TP
c0a5d35e 77.BI \-\-section \fR=\fPsec
cf145d90 78Only run section \fIsec\fR from job file. This option can be used multiple times to add more sections to run.
c0a5d35e 79.TP
49da1240
JA
80.BI \-\-alloc\-size \fR=\fPkb
81Set the internal smalloc pool size to \fIkb\fP kilobytes.
d60e92d1 82.TP
49da1240
JA
83.BI \-\-warnings\-fatal
84All fio parser warnings are fatal, causing fio to exit with an error.
9183788d 85.TP
49da1240 86.BI \-\-max\-jobs \fR=\fPnr
57e118a2 87Set the maximum allowed number of jobs (threads/processes) to support.
d60e92d1 88.TP
49da1240
JA
89.BI \-\-server \fR=\fPargs
90Start a backend server, with \fIargs\fP specifying what to listen to. See client/server section.
f57a9c59 91.TP
49da1240
JA
92.BI \-\-daemonize \fR=\fPpidfile
93Background a fio server, writing the pid to the given pid file.
94.TP
95.BI \-\-client \fR=\fPhost
39b5f61e 96Instead of running the jobs locally, send and run them on the given host or set of hosts. See client/server section.
f2a2ce0e
HL
97.TP
98.BI \-\-idle\-prof \fR=\fPoption
99Report cpu idleness on a system or percpu basis (\fIoption\fP=system,percpu) or run unit work calibration only (\fIoption\fP=calibrate).
d60e92d1
AC
100.SH "JOB FILE FORMAT"
101Job files are in `ini' format. They consist of one or more
102job definitions, which begin with a job name in square brackets and
103extend to the next job name. The job name can be any ASCII string
104except `global', which has a special meaning. Following the job name is
105a sequence of zero or more parameters, one per line, that define the
106behavior of the job. Any line starting with a `;' or `#' character is
d1429b5c 107considered a comment and ignored.
d9956b64
AC
108.P
109If \fIjobfile\fR is specified as `-', the job file will be read from
110standard input.
d60e92d1
AC
111.SS "Global Section"
112The global section contains default parameters for jobs specified in the
113job file. A job is only affected by global sections residing above it,
114and there may be any number of global sections. Specific job definitions
115may override any parameter set in global sections.
116.SH "JOB PARAMETERS"
117.SS Types
b470a02c
SC
118Some parameters may take arguments of a specific type.
119Anywhere a numeric value is required, an arithmetic expression may be used,
d59aa780
JA
120provided it is surrounded by parentheses. Supported operators are:
121.RS
122.RS
123.TP
124.B addition (+)
125.TP
126.B subtraction (-)
127.TP
128.B multiplication (*)
129.TP
130.B division (/)
131.TP
132.B modulus (%)
133.TP
134.B exponentiation (^)
135.RE
136.RE
137.P
138For time values in expressions, units are microseconds by default. This is
139different than for time values not in expressions (not enclosed in
140parentheses). The types used are:
d60e92d1
AC
141.TP
142.I str
143String: a sequence of alphanumeric characters.
144.TP
145.I int
d60e92d1 146SI integer: a whole number, possibly containing a suffix denoting the base unit
b09da8fa
JA
147of the value. Accepted suffixes are `k', 'M', 'G', 'T', and 'P', denoting
148kilo (1024), mega (1024^2), giga (1024^3), tera (1024^4), and peta (1024^5)
74454ce4
CE
149respectively. If prefixed with '0x', the value is assumed to be base 16
150(hexadecimal). A suffix may include a trailing 'b', for instance 'kb' is
151identical to 'k'. You can specify a base 10 value by using 'KiB', 'MiB','GiB',
152etc. This is useful for disk drives where values are often given in base 10
153values. Specifying '30GiB' will get you 30*1000^3 bytes.
154When specifying times the default suffix meaning changes, still denoting the
155base unit of the value, but accepted suffixes are 'D' (days), 'H' (hours), 'M'
0de5b26f
JA
156(minutes), 'S' Seconds, 'ms' (or msec) milli seconds, 'us' (or 'usec') micro
157seconds. Time values without a unit specify seconds.
74454ce4 158The suffixes are not case sensitive.
d60e92d1
AC
159.TP
160.I bool
161Boolean: a true or false value. `0' denotes false, `1' denotes true.
162.TP
163.I irange
164Integer range: a range of integers specified in the format
d1429b5c
AC
165\fIlower\fR:\fIupper\fR or \fIlower\fR\-\fIupper\fR. \fIlower\fR and
166\fIupper\fR may contain a suffix as described above. If an option allows two
167sets of ranges, they are separated with a `,' or `/' character. For example:
168`8\-8k/8M\-4G'.
83349190
YH
169.TP
170.I float_list
171List of floating numbers: A list of floating numbers, separated by
cecbfd47 172a ':' character.
d60e92d1
AC
173.SS "Parameter List"
174.TP
175.BI name \fR=\fPstr
d9956b64 176May be used to override the job name. On the command line, this parameter
d60e92d1
AC
177has the special purpose of signalling the start of a new job.
178.TP
179.BI description \fR=\fPstr
180Human-readable description of the job. It is printed when the job is run, but
181otherwise has no special purpose.
182.TP
183.BI directory \fR=\fPstr
184Prefix filenames with this directory. Used to place files in a location other
185than `./'.
bcbfeefa
CE
186You can specify a number of directories by separating the names with a ':'
187character. These directories will be assigned equally distributed to job clones
188creates with \fInumjobs\fR as long as they are using generated filenames.
189If specific \fIfilename(s)\fR are set fio will use the first listed directory,
190and thereby matching the \fIfilename\fR semantic which generates a file each
67445b63
JA
191clone if not specified, but let all clones use the same if set. See
192\fIfilename\fR for considerations regarding escaping certain characters on
193some platforms.
d60e92d1
AC
194.TP
195.BI filename \fR=\fPstr
196.B fio
197normally makes up a file name based on the job name, thread number, and file
d1429b5c 198number. If you want to share files between threads in a job or several jobs,
de890a1e
SL
199specify a \fIfilename\fR for each of them to override the default.
200If the I/O engine is file-based, you can specify
d1429b5c
AC
201a number of files by separating the names with a `:' character. `\-' is a
202reserved name, meaning stdin or stdout, depending on the read/write direction
67445b63
JA
203set. On Windows, disk devices are accessed as \\.\PhysicalDrive0 for the first
204device, \\.\PhysicalDrive1 for the second etc. Note: Windows and FreeBSD
205prevent write access to areas of the disk containing in-use data
206(e.g. filesystems). If the wanted filename does need to include a colon, then
4904acd5
JM
207escape that with a '\\' character. For instance, if the filename is
208"/dev/dsk/foo@3,0:c", then you would use filename="/dev/dsk/foo@3,0\\:c".
d60e92d1 209.TP
de98bd30 210.BI filename_format \fR=\fPstr
ce594fbe 211If sharing multiple files between jobs, it is usually necessary to have
de98bd30
JA
212fio generate the exact names that you want. By default, fio will name a file
213based on the default file format specification of
214\fBjobname.jobnumber.filenumber\fP. With this option, that can be
215customized. Fio will recognize and replace the following keywords in this
216string:
217.RS
218.RS
219.TP
220.B $jobname
221The name of the worker thread or process.
222.TP
223.B $jobnum
224The incremental number of the worker thread or process.
225.TP
226.B $filenum
227The incremental number of the file for that worker thread or process.
228.RE
229.P
230To have dependent jobs share a set of files, this option can be set to
231have fio generate filenames that are shared between the two. For instance,
232if \fBtestfiles.$filenum\fR is specified, file number 4 for any job will
233be named \fBtestfiles.4\fR. The default of \fB$jobname.$jobnum.$filenum\fR
234will be used if no other format specifier is given.
235.RE
236.P
237.TP
3ce9dcaf
JA
238.BI lockfile \fR=\fPstr
239Fio defaults to not locking any files before it does IO to them. If a file or
240file descriptor is shared, fio can serialize IO to that file to make the end
241result consistent. This is usual for emulating real workloads that share files.
242The lock modes are:
243.RS
244.RS
245.TP
246.B none
247No locking. This is the default.
248.TP
249.B exclusive
cf145d90 250Only one thread or process may do IO at a time, excluding all others.
3ce9dcaf
JA
251.TP
252.B readwrite
253Read-write locking on the file. Many readers may access the file at the same
254time, but writes get exclusive access.
255.RE
ce594fbe 256.RE
3ce9dcaf 257.P
d60e92d1
AC
258.BI opendir \fR=\fPstr
259Recursively open any files below directory \fIstr\fR.
260.TP
261.BI readwrite \fR=\fPstr "\fR,\fP rw" \fR=\fPstr
262Type of I/O pattern. Accepted values are:
263.RS
264.RS
265.TP
266.B read
d1429b5c 267Sequential reads.
d60e92d1
AC
268.TP
269.B write
d1429b5c 270Sequential writes.
d60e92d1 271.TP
fa769d44
SW
272.B trim
273Sequential trim (Linux block devices only).
274.TP
d60e92d1 275.B randread
d1429b5c 276Random reads.
d60e92d1
AC
277.TP
278.B randwrite
d1429b5c 279Random writes.
d60e92d1 280.TP
fa769d44
SW
281.B randtrim
282Random trim (Linux block devices only).
283.TP
10b023db 284.B rw, readwrite
d1429b5c 285Mixed sequential reads and writes.
d60e92d1
AC
286.TP
287.B randrw
d1429b5c 288Mixed random reads and writes.
82a90686
JA
289.TP
290.B trimwrite
291Trim and write mixed workload. Blocks will be trimmed first, then the same
292blocks will be written to.
d60e92d1
AC
293.RE
294.P
38dad62d
JA
295For mixed I/O, the default split is 50/50. For certain types of io the result
296may still be skewed a bit, since the speed may be different. It is possible to
3b7fa9ec 297specify a number of IO's to do before getting a new offset, this is done by
38dad62d
JA
298appending a `:\fI<nr>\fR to the end of the string given. For a random read, it
299would look like \fBrw=randread:8\fR for passing in an offset modifier with a
059b0802
JA
300value of 8. If the postfix is used with a sequential IO pattern, then the value
301specified will be added to the generated offset for each IO. For instance,
302using \fBrw=write:4k\fR will skip 4k for every write. It turns sequential IO
303into sequential IO with holes. See the \fBrw_sequencer\fR option.
d60e92d1
AC
304.RE
305.TP
38dad62d
JA
306.BI rw_sequencer \fR=\fPstr
307If an offset modifier is given by appending a number to the \fBrw=<str>\fR line,
308then this option controls how that number modifies the IO offset being
309generated. Accepted values are:
310.RS
311.RS
312.TP
313.B sequential
314Generate sequential offset
315.TP
316.B identical
317Generate the same offset
318.RE
319.P
320\fBsequential\fR is only useful for random IO, where fio would normally
321generate a new random offset for every IO. If you append eg 8 to randread, you
322would get a new random offset for every 8 IO's. The result would be a seek for
323only every 8 IO's, instead of for every IO. Use \fBrw=randread:8\fR to specify
324that. As sequential IO is already sequential, setting \fBsequential\fR for that
325would not result in any differences. \fBidentical\fR behaves in a similar
326fashion, except it sends the same offset 8 number of times before generating a
327new offset.
328.RE
329.P
330.TP
90fef2d1
JA
331.BI kb_base \fR=\fPint
332The base unit for a kilobyte. The defacto base is 2^10, 1024. Storage
333manufacturers like to use 10^3 or 1000 as a base ten unit instead, for obvious
5c9323fb 334reasons. Allowed values are 1024 or 1000, with 1024 being the default.
90fef2d1 335.TP
771e58be
JA
336.BI unified_rw_reporting \fR=\fPbool
337Fio normally reports statistics on a per data direction basis, meaning that
338read, write, and trim are accounted and reported separately. If this option is
cf145d90 339set fio sums the results and reports them as "mixed" instead.
771e58be 340.TP
d60e92d1 341.BI randrepeat \fR=\fPbool
56e2a5fc
CE
342Seed the random number generator used for random I/O patterns in a predictable
343way so the pattern is repeatable across runs. Default: true.
344.TP
345.BI allrandrepeat \fR=\fPbool
346Seed all random number generators in a predictable way so results are
347repeatable across runs. Default: false.
d60e92d1 348.TP
04778baf
JA
349.BI randseed \fR=\fPint
350Seed the random number generators based on this seed value, to be able to
351control what sequence of output is being generated. If not set, the random
352sequence depends on the \fBrandrepeat\fR setting.
353.TP
a596f047
EG
354.BI fallocate \fR=\fPstr
355Whether pre-allocation is performed when laying down files. Accepted values
356are:
357.RS
358.RS
359.TP
360.B none
361Do not pre-allocate space.
362.TP
363.B posix
ccc2b328 364Pre-allocate via \fBposix_fallocate\fR\|(3).
a596f047
EG
365.TP
366.B keep
ccc2b328 367Pre-allocate via \fBfallocate\fR\|(2) with FALLOC_FL_KEEP_SIZE set.
a596f047
EG
368.TP
369.B 0
370Backward-compatible alias for 'none'.
371.TP
372.B 1
373Backward-compatible alias for 'posix'.
374.RE
375.P
376May not be available on all supported platforms. 'keep' is only
377available on Linux. If using ZFS on Solaris this must be set to 'none'
378because ZFS doesn't support it. Default: 'posix'.
379.RE
7bc8c2cf 380.TP
d60e92d1 381.BI fadvise_hint \fR=\fPbool
cf145d90 382Use \fBposix_fadvise\fR\|(2) to advise the kernel what I/O patterns
d1429b5c 383are likely to be issued. Default: true.
d60e92d1 384.TP
37659335
JA
385.BI fadvise_stream \fR=\fPint
386Use \fBposix_fadvise\fR\|(2) to advise the kernel what stream ID the
387writes issued belong to. Only supported on Linux. Note, this option
388may change going forward.
389.TP
f7fa2653 390.BI size \fR=\fPint
d60e92d1 391Total size of I/O for this job. \fBfio\fR will run until this many bytes have
a4d3b4db
JA
392been transferred, unless limited by other options (\fBruntime\fR, for instance,
393or increased/descreased by \fBio_size\fR). Unless \fBnrfiles\fR and
394\fBfilesize\fR options are given, this amount will be divided between the
395available files for the job. If not set, fio will use the full size of the
396given files or devices. If the files do not exist, size must be given. It is
397also possible to give size as a percentage between 1 and 100. If size=20% is
398given, fio will use 20% of the full size of the given files or devices.
399.TP
400.BI io_size \fR=\fPint "\fR,\fB io_limit \fR=\fPint
77731b29
JA
401Normally fio operates within the region set by \fBsize\fR, which means that
402the \fBsize\fR option sets both the region and size of IO to be performed.
403Sometimes that is not what you want. With this option, it is possible to
404define just the amount of IO that fio should do. For instance, if \fBsize\fR
405is set to 20G and \fBio_limit\fR is set to 5G, fio will perform IO within
a4d3b4db
JA
406the first 20G but exit when 5G have been done. The opposite is also
407possible - if \fBsize\fR is set to 20G, and \fBio_size\fR is set to 40G, then
408fio will do 40G of IO within the 0..20G region.
d60e92d1 409.TP
74586c1e 410.BI fill_device \fR=\fPbool "\fR,\fB fill_fs" \fR=\fPbool
3ce9dcaf
JA
411Sets size to something really large and waits for ENOSPC (no space left on
412device) as the terminating condition. Only makes sense with sequential write.
413For a read workload, the mount point will be filled first then IO started on
4f12432e
JA
414the result. This option doesn't make sense if operating on a raw device node,
415since the size of that is already known by the file system. Additionally,
416writing beyond end-of-device will not return ENOSPC there.
3ce9dcaf 417.TP
d60e92d1
AC
418.BI filesize \fR=\fPirange
419Individual file sizes. May be a range, in which case \fBfio\fR will select sizes
d1429b5c
AC
420for files at random within the given range, limited to \fBsize\fR in total (if
421that is given). If \fBfilesize\fR is not specified, each created file is the
422same size.
d60e92d1 423.TP
bedc9dc2
JA
424.BI file_append \fR=\fPbool
425Perform IO after the end of the file. Normally fio will operate within the
426size of a file. If this option is set, then fio will append to the file
427instead. This has identical behavior to setting \fRoffset\fP to the size
0aae4ce7 428of a file. This option is ignored on non-regular files.
bedc9dc2 429.TP
f7fa2653 430.BI blocksize \fR=\fPint[,int] "\fR,\fB bs" \fR=\fPint[,int]
d9472271
JA
431Block size for I/O units. Default: 4k. Values for reads, writes, and trims
432can be specified separately in the format \fIread\fR,\fIwrite\fR,\fItrim\fR
433either of which may be empty to leave that value at its default. If a trailing
434comma isn't given, the remainder will inherit the last value set.
d60e92d1 435.TP
9183788d 436.BI blocksize_range \fR=\fPirange[,irange] "\fR,\fB bsrange" \fR=\fPirange[,irange]
d1429b5c
AC
437Specify a range of I/O block sizes. The issued I/O unit will always be a
438multiple of the minimum size, unless \fBblocksize_unaligned\fR is set. Applies
9183788d 439to both reads and writes if only one range is given, but can be specified
de8f6de9 440separately with a comma separating the values. Example: bsrange=1k-4k,2k-8k.
9183788d
JA
441Also (see \fBblocksize\fR).
442.TP
443.BI bssplit \fR=\fPstr
444This option allows even finer grained control of the block sizes issued,
445not just even splits between them. With this option, you can weight various
446block sizes for exact control of the issued IO for a job that has mixed
447block sizes. The format of the option is bssplit=blocksize/percentage,
5982a925 448optionally adding as many definitions as needed separated by a colon.
9183788d 449Example: bssplit=4k/10:64k/50:32k/40 would issue 50% 64k blocks, 10% 4k
c83cdd3e
JA
450blocks and 40% 32k blocks. \fBbssplit\fR also supports giving separate
451splits to reads and writes. The format is identical to what the
452\fBbs\fR option accepts, the read and write parts are separated with a
453comma.
d60e92d1
AC
454.TP
455.B blocksize_unaligned\fR,\fP bs_unaligned
d1429b5c
AC
456If set, any size in \fBblocksize_range\fR may be used. This typically won't
457work with direct I/O, as that normally requires sector alignment.
d60e92d1 458.TP
2b7a01d0 459.BI blockalign \fR=\fPint[,int] "\fR,\fB ba" \fR=\fPint[,int]
639ce0f3
MS
460At what boundary to align random IO offsets. Defaults to the same as 'blocksize'
461the minimum blocksize given. Minimum alignment is typically 512b
2b7a01d0
JA
462for using direct IO, though it usually depends on the hardware block size.
463This option is mutually exclusive with using a random map for files, so it
464will turn off that option.
43602667 465.TP
6aca9b3d
JA
466.BI bs_is_seq_rand \fR=\fPbool
467If this option is set, fio will use the normal read,write blocksize settings as
468sequential,random instead. Any random read or write will use the WRITE
469blocksize settings, and any sequential read or write will use the READ
470blocksize setting.
471.TP
d60e92d1 472.B zero_buffers
cf145d90 473Initialize buffers with all zeros. Default: fill buffers with random data.
d60e92d1 474.TP
901bb994
JA
475.B refill_buffers
476If this option is given, fio will refill the IO buffers on every submit. The
477default is to only fill it at init time and reuse that data. Only makes sense
478if zero_buffers isn't specified, naturally. If data verification is enabled,
479refill_buffers is also automatically enabled.
480.TP
fd68418e
JA
481.BI scramble_buffers \fR=\fPbool
482If \fBrefill_buffers\fR is too costly and the target is using data
483deduplication, then setting this option will slightly modify the IO buffer
484contents to defeat normal de-dupe attempts. This is not enough to defeat
485more clever block compression attempts, but it will stop naive dedupe
486of blocks. Default: true.
487.TP
c5751c62
JA
488.BI buffer_compress_percentage \fR=\fPint
489If this is set, then fio will attempt to provide IO buffer content (on WRITEs)
490that compress to the specified level. Fio does this by providing a mix of
d1af2894
JA
491random data and a fixed pattern. The fixed pattern is either zeroes, or the
492pattern specified by \fBbuffer_pattern\fR. If the pattern option is used, it
493might skew the compression ratio slightly. Note that this is per block size
494unit, for file/disk wide compression level that matches this setting. Note
495that this is per block size unit, for file/disk wide compression level that
496matches this setting, you'll also want to set refill_buffers.
c5751c62
JA
497.TP
498.BI buffer_compress_chunk \fR=\fPint
499See \fBbuffer_compress_percentage\fR. This setting allows fio to manage how
500big the ranges of random data and zeroed data is. Without this set, fio will
501provide \fBbuffer_compress_percentage\fR of blocksize random data, followed by
502the remaining zeroed. With this set to some chunk size smaller than the block
503size, fio can alternate random and zeroed data throughout the IO buffer.
504.TP
ce35b1ec 505.BI buffer_pattern \fR=\fPstr
cf145d90
CVB
506If set, fio will fill the IO buffers with this pattern. If not set, the contents
507of IO buffers is defined by the other options related to buffer contents. The
ce35b1ec 508setting can be any pattern of bytes, and can be prefixed with 0x for hex
02975b64 509values. It may also be a string, where the string must then be wrapped with
2fa5a241
RP
510"", e.g.:
511.RS
512.RS
513\fBbuffer_pattern\fR="abcd"
514.RS
515or
516.RE
517\fBbuffer_pattern\fR=-12
518.RS
519or
520.RE
521\fBbuffer_pattern\fR=0xdeadface
522.RE
523.LP
524Also you can combine everything together in any order:
525.LP
526.RS
527\fBbuffer_pattern\fR=0xdeadface"abcd"-12
528.RE
529.RE
ce35b1ec 530.TP
5c94b008
JA
531.BI dedupe_percentage \fR=\fPint
532If set, fio will generate this percentage of identical buffers when writing.
533These buffers will be naturally dedupable. The contents of the buffers depend
534on what other buffer compression settings have been set. It's possible to have
535the individual buffers either fully compressible, or not at all. This option
536only controls the distribution of unique buffers.
537.TP
d60e92d1
AC
538.BI nrfiles \fR=\fPint
539Number of files to use for this job. Default: 1.
540.TP
541.BI openfiles \fR=\fPint
542Number of files to keep open at the same time. Default: \fBnrfiles\fR.
543.TP
544.BI file_service_type \fR=\fPstr
545Defines how files to service are selected. The following types are defined:
546.RS
547.RS
548.TP
549.B random
5c9323fb 550Choose a file at random.
d60e92d1
AC
551.TP
552.B roundrobin
cf145d90 553Round robin over opened files (default).
5c9323fb 554.TP
6b7f6851
JA
555.B sequential
556Do each file in the set sequentially.
d60e92d1
AC
557.RE
558.P
cf145d90 559The number of I/Os to issue before switching to a new file can be specified by
d60e92d1
AC
560appending `:\fIint\fR' to the service type.
561.RE
562.TP
563.BI ioengine \fR=\fPstr
564Defines how the job issues I/O. The following types are defined:
565.RS
566.RS
567.TP
568.B sync
ccc2b328 569Basic \fBread\fR\|(2) or \fBwrite\fR\|(2) I/O. \fBfseek\fR\|(2) is used to
d60e92d1
AC
570position the I/O location.
571.TP
a31041ea 572.B psync
ccc2b328 573Basic \fBpread\fR\|(2) or \fBpwrite\fR\|(2) I/O.
a31041ea 574.TP
9183788d 575.B vsync
ccc2b328 576Basic \fBreadv\fR\|(2) or \fBwritev\fR\|(2) I/O. Will emulate queuing by
cecbfd47 577coalescing adjacent IOs into a single submission.
9183788d 578.TP
a46c5e01 579.B pvsync
ccc2b328 580Basic \fBpreadv\fR\|(2) or \fBpwritev\fR\|(2) I/O.
a46c5e01 581.TP
d60e92d1 582.B libaio
de890a1e 583Linux native asynchronous I/O. This ioengine defines engine specific options.
d60e92d1
AC
584.TP
585.B posixaio
ccc2b328 586POSIX asynchronous I/O using \fBaio_read\fR\|(3) and \fBaio_write\fR\|(3).
03e20d68
BC
587.TP
588.B solarisaio
589Solaris native asynchronous I/O.
590.TP
591.B windowsaio
592Windows native asynchronous I/O.
d60e92d1
AC
593.TP
594.B mmap
ccc2b328
SW
595File is memory mapped with \fBmmap\fR\|(2) and data copied using
596\fBmemcpy\fR\|(3).
d60e92d1
AC
597.TP
598.B splice
ccc2b328 599\fBsplice\fR\|(2) is used to transfer the data and \fBvmsplice\fR\|(2) to
d1429b5c 600transfer data from user-space to the kernel.
d60e92d1
AC
601.TP
602.B syslet-rw
603Use the syslet system calls to make regular read/write asynchronous.
604.TP
605.B sg
606SCSI generic sg v3 I/O. May be either synchronous using the SG_IO ioctl, or if
ccc2b328
SW
607the target is an sg character device, we use \fBread\fR\|(2) and
608\fBwrite\fR\|(2) for asynchronous I/O.
d60e92d1
AC
609.TP
610.B null
611Doesn't transfer any data, just pretends to. Mainly used to exercise \fBfio\fR
612itself and for debugging and testing purposes.
613.TP
614.B net
de890a1e
SL
615Transfer over the network. The protocol to be used can be defined with the
616\fBprotocol\fR parameter. Depending on the protocol, \fBfilename\fR,
617\fBhostname\fR, \fBport\fR, or \fBlisten\fR must be specified.
618This ioengine defines engine specific options.
d60e92d1
AC
619.TP
620.B netsplice
ccc2b328 621Like \fBnet\fR, but uses \fBsplice\fR\|(2) and \fBvmsplice\fR\|(2) to map data
de890a1e 622and send/receive. This ioengine defines engine specific options.
d60e92d1 623.TP
53aec0a4 624.B cpuio
d60e92d1
AC
625Doesn't transfer any data, but burns CPU cycles according to \fBcpuload\fR and
626\fBcpucycles\fR parameters.
627.TP
628.B guasi
629The GUASI I/O engine is the Generic Userspace Asynchronous Syscall Interface
cecbfd47 630approach to asynchronous I/O.
d1429b5c
AC
631.br
632See <http://www.xmailserver.org/guasi\-lib.html>.
d60e92d1 633.TP
21b8aee8 634.B rdma
85286c5c
BVA
635The RDMA I/O engine supports both RDMA memory semantics (RDMA_WRITE/RDMA_READ)
636and channel semantics (Send/Recv) for the InfiniBand, RoCE and iWARP protocols.
21b8aee8 637.TP
d60e92d1
AC
638.B external
639Loads an external I/O engine object file. Append the engine filename as
640`:\fIenginepath\fR'.
d54fce84
DM
641.TP
642.B falloc
cecbfd47 643 IO engine that does regular linux native fallocate call to simulate data
d54fce84
DM
644transfer as fio ioengine
645.br
646 DDIR_READ does fallocate(,mode = FALLOC_FL_KEEP_SIZE,)
647.br
0981fd71 648 DIR_WRITE does fallocate(,mode = 0)
d54fce84
DM
649.br
650 DDIR_TRIM does fallocate(,mode = FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE)
651.TP
652.B e4defrag
653IO engine that does regular EXT4_IOC_MOVE_EXT ioctls to simulate defragment activity
654request to DDIR_WRITE event
0d978694
DAG
655.TP
656.B rbd
657IO engine supporting direct access to Ceph Rados Block Devices (RBD) via librbd
658without the need to use the kernel rbd driver. This ioengine defines engine specific
659options.
a7c386f4 660.TP
661.B gfapi
cc47f094 662Using Glusterfs libgfapi sync interface to direct access to Glusterfs volumes without
663having to go through FUSE. This ioengine defines engine specific
664options.
665.TP
666.B gfapi_async
667Using Glusterfs libgfapi async interface to direct access to Glusterfs volumes without
a7c386f4 668having to go through FUSE. This ioengine defines engine specific
669options.
1b10477b 670.TP
b74e419e
MM
671.B libhdfs
672Read and write through Hadoop (HDFS). The \fBfilename\fR option is used to
673specify host,port of the hdfs name-node to connect. This engine interprets
674offsets a little differently. In HDFS, files once created cannot be modified.
675So random writes are not possible. To imitate this, libhdfs engine expects
676bunch of small files to be created over HDFS, and engine will randomly pick a
677file out of those files based on the offset generated by fio backend. (see the
678example job file to create such files, use rw=write option). Please note, you
679might want to set necessary environment variables to work with hdfs/libhdfs
680properly.
65fa28ca
DE
681.TP
682.B mtd
683Read, write and erase an MTD character device (e.g., /dev/mtd0). Discards are
684treated as erases. Depending on the underlying device type, the I/O may have
685to go in a certain pattern, e.g., on NAND, writing sequentially to erase blocks
686and discarding before overwriting. The writetrim mode works well for this
687constraint.
d60e92d1 688.RE
595e1734 689.P
d60e92d1
AC
690.RE
691.TP
692.BI iodepth \fR=\fPint
8489dae4
SK
693Number of I/O units to keep in flight against the file. Note that increasing
694iodepth beyond 1 will not affect synchronous ioengines (except for small
cf145d90 695degress when verify_async is in use). Even async engines may impose OS
ee72ca09
JA
696restrictions causing the desired depth not to be achieved. This may happen on
697Linux when using libaio and not setting \fBdirect\fR=1, since buffered IO is
698not async on that OS. Keep an eye on the IO depth distribution in the
699fio output to verify that the achieved depth is as expected. Default: 1.
d60e92d1
AC
700.TP
701.BI iodepth_batch \fR=\fPint
702Number of I/Os to submit at once. Default: \fBiodepth\fR.
703.TP
3ce9dcaf
JA
704.BI iodepth_batch_complete \fR=\fPint
705This defines how many pieces of IO to retrieve at once. It defaults to 1 which
706 means that we'll ask for a minimum of 1 IO in the retrieval process from the
707kernel. The IO retrieval will go on until we hit the limit set by
708\fBiodepth_low\fR. If this variable is set to 0, then fio will always check for
709completed events before queuing more IO. This helps reduce IO latency, at the
710cost of more retrieval system calls.
711.TP
d60e92d1
AC
712.BI iodepth_low \fR=\fPint
713Low watermark indicating when to start filling the queue again. Default:
714\fBiodepth\fR.
715.TP
1ad01bd1
JA
716.BI io_submit_mode \fR=\fPstr
717This option controls how fio submits the IO to the IO engine. The default is
718\fBinline\fR, which means that the fio job threads submit and reap IO directly.
719If set to \fBoffload\fR, the job threads will offload IO submission to a
720dedicated pool of IO threads. This requires some coordination and thus has a
721bit of extra overhead, especially for lower queue depth IO where it can
722increase latencies. The benefit is that fio can manage submission rates
723independently of the device completion rates. This avoids skewed latency
724reporting if IO gets back up on the device side (the coordinated omission
725problem).
726.TP
d60e92d1
AC
727.BI direct \fR=\fPbool
728If true, use non-buffered I/O (usually O_DIRECT). Default: false.
729.TP
d01612f3
CM
730.BI atomic \fR=\fPbool
731If value is true, attempt to use atomic direct IO. Atomic writes are guaranteed
732to be stable once acknowledged by the operating system. Only Linux supports
733O_ATOMIC right now.
734.TP
d60e92d1
AC
735.BI buffered \fR=\fPbool
736If true, use buffered I/O. This is the opposite of the \fBdirect\fR parameter.
737Default: true.
738.TP
f7fa2653 739.BI offset \fR=\fPint
d60e92d1
AC
740Offset in the file to start I/O. Data before the offset will not be touched.
741.TP
591e9e06
JA
742.BI offset_increment \fR=\fPint
743If this is provided, then the real offset becomes the
69bdd6ba
JH
744offset + offset_increment * thread_number, where the thread number is a
745counter that starts at 0 and is incremented for each sub-job (i.e. when
746numjobs option is specified). This option is useful if there are several jobs
747which are intended to operate on a file in parallel disjoint segments, with
748even spacing between the starting points.
591e9e06 749.TP
ddf24e42
JA
750.BI number_ios \fR=\fPint
751Fio will normally perform IOs until it has exhausted the size of the region
752set by \fBsize\fR, or if it exhaust the allocated time (or hits an error
753condition). With this setting, the range/size can be set independently of
754the number of IOs to perform. When fio reaches this number, it will exit
be3fec7d
JA
755normally and report status. Note that this does not extend the amount
756of IO that will be done, it will only stop fio if this condition is met
757before other end-of-job criteria.
ddf24e42 758.TP
d60e92d1 759.BI fsync \fR=\fPint
d1429b5c
AC
760How many I/Os to perform before issuing an \fBfsync\fR\|(2) of dirty data. If
7610, don't sync. Default: 0.
d60e92d1 762.TP
5f9099ea
JA
763.BI fdatasync \fR=\fPint
764Like \fBfsync\fR, but uses \fBfdatasync\fR\|(2) instead to only sync the
765data parts of the file. Default: 0.
766.TP
fa769d44
SW
767.BI write_barrier \fR=\fPint
768Make every Nth write a barrier write.
769.TP
e76b1da4 770.BI sync_file_range \fR=\fPstr:int
ccc2b328
SW
771Use \fBsync_file_range\fR\|(2) for every \fRval\fP number of write operations. Fio will
772track range of writes that have happened since the last \fBsync_file_range\fR\|(2) call.
e76b1da4
JA
773\fRstr\fP can currently be one or more of:
774.RS
775.TP
776.B wait_before
777SYNC_FILE_RANGE_WAIT_BEFORE
778.TP
779.B write
780SYNC_FILE_RANGE_WRITE
781.TP
782.B wait_after
783SYNC_FILE_RANGE_WRITE
784.TP
785.RE
786.P
787So if you do sync_file_range=wait_before,write:8, fio would use
788\fBSYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE\fP for every 8 writes.
ccc2b328 789Also see the \fBsync_file_range\fR\|(2) man page. This option is Linux specific.
e76b1da4 790.TP
d60e92d1 791.BI overwrite \fR=\fPbool
d1429b5c 792If writing, setup the file first and do overwrites. Default: false.
d60e92d1
AC
793.TP
794.BI end_fsync \fR=\fPbool
dbd11ead 795Sync file contents when a write stage has completed. Default: false.
d60e92d1
AC
796.TP
797.BI fsync_on_close \fR=\fPbool
798If true, sync file contents on close. This differs from \fBend_fsync\fR in that
d1429b5c 799it will happen on every close, not just at the end of the job. Default: false.
d60e92d1 800.TP
d60e92d1
AC
801.BI rwmixread \fR=\fPint
802Percentage of a mixed workload that should be reads. Default: 50.
803.TP
804.BI rwmixwrite \fR=\fPint
d1429b5c 805Percentage of a mixed workload that should be writes. If \fBrwmixread\fR and
c35dd7a6
JA
806\fBrwmixwrite\fR are given and do not sum to 100%, the latter of the two
807overrides the first. This may interfere with a given rate setting, if fio is
808asked to limit reads or writes to a certain rate. If that is the case, then
809the distribution may be skewed. Default: 50.
d60e92d1 810.TP
92d42d69
JA
811.BI random_distribution \fR=\fPstr:float
812By default, fio will use a completely uniform random distribution when asked
813to perform random IO. Sometimes it is useful to skew the distribution in
814specific ways, ensuring that some parts of the data is more hot than others.
815Fio includes the following distribution models:
816.RS
817.TP
818.B random
819Uniform random distribution
820.TP
821.B zipf
822Zipf distribution
823.TP
824.B pareto
825Pareto distribution
826.TP
827.RE
828.P
829When using a zipf or pareto distribution, an input value is also needed to
830define the access pattern. For zipf, this is the zipf theta. For pareto,
831it's the pareto power. Fio includes a test program, genzipf, that can be
832used visualize what the given input values will yield in terms of hit rates.
833If you wanted to use zipf with a theta of 1.2, you would use
834random_distribution=zipf:1.2 as the option. If a non-uniform model is used,
835fio will disable use of the random map.
836.TP
211c9b89
JA
837.BI percentage_random \fR=\fPint
838For a random workload, set how big a percentage should be random. This defaults
839to 100%, in which case the workload is fully random. It can be set from
840anywhere from 0 to 100. Setting it to 0 would make the workload fully
d9472271
JA
841sequential. It is possible to set different values for reads, writes, and
842trim. To do so, simply use a comma separated list. See \fBblocksize\fR.
211c9b89 843.TP
d60e92d1
AC
844.B norandommap
845Normally \fBfio\fR will cover every block of the file when doing random I/O. If
846this parameter is given, a new offset will be chosen without looking at past
847I/O history. This parameter is mutually exclusive with \fBverify\fR.
848.TP
744492c9 849.BI softrandommap \fR=\fPbool
3ce9dcaf
JA
850See \fBnorandommap\fR. If fio runs with the random block map enabled and it
851fails to allocate the map, if this option is set it will continue without a
852random block map. As coverage will not be as complete as with random maps, this
853option is disabled by default.
854.TP
e8b1961d
JA
855.BI random_generator \fR=\fPstr
856Fio supports the following engines for generating IO offsets for random IO:
857.RS
858.TP
859.B tausworthe
860Strong 2^88 cycle random number generator
861.TP
862.B lfsr
863Linear feedback shift register generator
864.TP
c3546b53
JA
865.B tausworthe64
866Strong 64-bit 2^258 cycle random number generator
867.TP
e8b1961d
JA
868.RE
869.P
870Tausworthe is a strong random number generator, but it requires tracking on the
871side if we want to ensure that blocks are only read or written once. LFSR
872guarantees that we never generate the same offset twice, and it's also less
873computationally expensive. It's not a true random generator, however, though
874for IO purposes it's typically good enough. LFSR only works with single block
875sizes, not with workloads that use multiple block sizes. If used with such a
876workload, fio may read or write some blocks multiple times.
877.TP
d60e92d1 878.BI nice \fR=\fPint
ccc2b328 879Run job with given nice value. See \fBnice\fR\|(2).
d60e92d1
AC
880.TP
881.BI prio \fR=\fPint
882Set I/O priority value of this job between 0 (highest) and 7 (lowest). See
ccc2b328 883\fBionice\fR\|(1).
d60e92d1
AC
884.TP
885.BI prioclass \fR=\fPint
ccc2b328 886Set I/O priority class. See \fBionice\fR\|(1).
d60e92d1
AC
887.TP
888.BI thinktime \fR=\fPint
889Stall job for given number of microseconds between issuing I/Os.
890.TP
891.BI thinktime_spin \fR=\fPint
892Pretend to spend CPU time for given number of microseconds, sleeping the rest
893of the time specified by \fBthinktime\fR. Only valid if \fBthinktime\fR is set.
894.TP
895.BI thinktime_blocks \fR=\fPint
4d01ece6
JA
896Only valid if thinktime is set - control how many blocks to issue, before
897waiting \fBthinktime\fR microseconds. If not set, defaults to 1 which will
898make fio wait \fBthinktime\fR microseconds after every block. This
899effectively makes any queue depth setting redundant, since no more than 1 IO
900will be queued before we have to complete it and do our thinktime. In other
901words, this setting effectively caps the queue depth if the latter is larger.
d60e92d1
AC
902Default: 1.
903.TP
904.BI rate \fR=\fPint
c35dd7a6
JA
905Cap bandwidth used by this job. The number is in bytes/sec, the normal postfix
906rules apply. You can use \fBrate\fR=500k to limit reads and writes to 500k each,
907or you can specify read and writes separately. Using \fBrate\fR=1m,500k would
908limit reads to 1MB/sec and writes to 500KB/sec. Capping only reads or writes
909can be done with \fBrate\fR=,500k or \fBrate\fR=500k,. The former will only
910limit writes (to 500KB/sec), the latter will only limit reads.
d60e92d1
AC
911.TP
912.BI ratemin \fR=\fPint
913Tell \fBfio\fR to do whatever it can to maintain at least the given bandwidth.
c35dd7a6
JA
914Failing to meet this requirement will cause the job to exit. The same format
915as \fBrate\fR is used for read vs write separation.
d60e92d1
AC
916.TP
917.BI rate_iops \fR=\fPint
c35dd7a6
JA
918Cap the bandwidth to this number of IOPS. Basically the same as rate, just
919specified independently of bandwidth. The same format as \fBrate\fR is used for
de8f6de9 920read vs write separation. If \fBblocksize\fR is a range, the smallest block
c35dd7a6 921size is used as the metric.
d60e92d1
AC
922.TP
923.BI rate_iops_min \fR=\fPint
c35dd7a6 924If this rate of I/O is not met, the job will exit. The same format as \fBrate\fR
de8f6de9 925is used for read vs write separation.
d60e92d1
AC
926.TP
927.BI ratecycle \fR=\fPint
928Average bandwidth for \fBrate\fR and \fBratemin\fR over this number of
929milliseconds. Default: 1000ms.
930.TP
3e260a46
JA
931.BI latency_target \fR=\fPint
932If set, fio will attempt to find the max performance point that the given
933workload will run at while maintaining a latency below this target. The
934values is given in microseconds. See \fBlatency_window\fR and
935\fBlatency_percentile\fR.
936.TP
937.BI latency_window \fR=\fPint
938Used with \fBlatency_target\fR to specify the sample window that the job
939is run at varying queue depths to test the performance. The value is given
940in microseconds.
941.TP
942.BI latency_percentile \fR=\fPfloat
943The percentage of IOs that must fall within the criteria specified by
944\fBlatency_target\fR and \fBlatency_window\fR. If not set, this defaults
945to 100.0, meaning that all IOs must be equal or below to the value set
946by \fBlatency_target\fR.
947.TP
15501535
JA
948.BI max_latency \fR=\fPint
949If set, fio will exit the job if it exceeds this maximum latency. It will exit
950with an ETIME error.
951.TP
d60e92d1
AC
952.BI cpumask \fR=\fPint
953Set CPU affinity for this job. \fIint\fR is a bitmask of allowed CPUs the job
954may run on. See \fBsched_setaffinity\fR\|(2).
955.TP
956.BI cpus_allowed \fR=\fPstr
957Same as \fBcpumask\fR, but allows a comma-delimited list of CPU numbers.
958.TP
c2acfbac
JA
959.BI cpus_allowed_policy \fR=\fPstr
960Set the policy of how fio distributes the CPUs specified by \fBcpus_allowed\fR
961or \fBcpumask\fR. Two policies are supported:
962.RS
963.RS
964.TP
965.B shared
966All jobs will share the CPU set specified.
967.TP
968.B split
969Each job will get a unique CPU from the CPU set.
970.RE
971.P
972\fBshared\fR is the default behaviour, if the option isn't specified. If
ada083cd
JA
973\fBsplit\fR is specified, then fio will assign one cpu per job. If not enough
974CPUs are given for the jobs listed, then fio will roundrobin the CPUs in
975the set.
c2acfbac
JA
976.RE
977.P
978.TP
d0b937ed 979.BI numa_cpu_nodes \fR=\fPstr
cecbfd47 980Set this job running on specified NUMA nodes' CPUs. The arguments allow
d0b937ed
YR
981comma delimited list of cpu numbers, A-B ranges, or 'all'.
982.TP
983.BI numa_mem_policy \fR=\fPstr
984Set this job's memory policy and corresponding NUMA nodes. Format of
cecbfd47 985the arguments:
d0b937ed
YR
986.RS
987.TP
988.B <mode>[:<nodelist>]
989.TP
990.B mode
991is one of the following memory policy:
992.TP
993.B default, prefer, bind, interleave, local
994.TP
995.RE
996For \fBdefault\fR and \fBlocal\fR memory policy, no \fBnodelist\fR is
997needed to be specified. For \fBprefer\fR, only one node is
998allowed. For \fBbind\fR and \fBinterleave\fR, \fBnodelist\fR allows
999comma delimited list of numbers, A-B ranges, or 'all'.
1000.TP
23ed19b0
CE
1001.BI startdelay \fR=\fPirange
1002Delay start of job for the specified number of seconds. Supports all time
1003suffixes to allow specification of hours, minutes, seconds and
bd66aa2c 1004milliseconds - seconds are the default if a unit is omitted.
23ed19b0
CE
1005Can be given as a range which causes each thread to choose randomly out of the
1006range.
d60e92d1
AC
1007.TP
1008.BI runtime \fR=\fPint
1009Terminate processing after the specified number of seconds.
1010.TP
1011.B time_based
1012If given, run for the specified \fBruntime\fR duration even if the files are
1013completely read or written. The same workload will be repeated as many times
1014as \fBruntime\fR allows.
1015.TP
901bb994
JA
1016.BI ramp_time \fR=\fPint
1017If set, fio will run the specified workload for this amount of time before
1018logging any performance numbers. Useful for letting performance settle before
1019logging results, thus minimizing the runtime required for stable results. Note
c35dd7a6
JA
1020that the \fBramp_time\fR is considered lead in time for a job, thus it will
1021increase the total runtime if a special timeout or runtime is specified.
901bb994 1022.TP
d60e92d1
AC
1023.BI invalidate \fR=\fPbool
1024Invalidate buffer-cache for the file prior to starting I/O. Default: true.
1025.TP
1026.BI sync \fR=\fPbool
1027Use synchronous I/O for buffered writes. For the majority of I/O engines,
d1429b5c 1028this means using O_SYNC. Default: false.
d60e92d1
AC
1029.TP
1030.BI iomem \fR=\fPstr "\fR,\fP mem" \fR=\fPstr
1031Allocation method for I/O unit buffer. Allowed values are:
1032.RS
1033.RS
1034.TP
1035.B malloc
ccc2b328 1036Allocate memory with \fBmalloc\fR\|(3).
d60e92d1
AC
1037.TP
1038.B shm
ccc2b328 1039Use shared memory buffers allocated through \fBshmget\fR\|(2).
d60e92d1
AC
1040.TP
1041.B shmhuge
1042Same as \fBshm\fR, but use huge pages as backing.
1043.TP
1044.B mmap
ccc2b328 1045Use \fBmmap\fR\|(2) for allocation. Uses anonymous memory unless a filename
d60e92d1
AC
1046is given after the option in the format `:\fIfile\fR'.
1047.TP
1048.B mmaphuge
1049Same as \fBmmap\fR, but use huge files as backing.
1050.RE
1051.P
1052The amount of memory allocated is the maximum allowed \fBblocksize\fR for the
1053job multiplied by \fBiodepth\fR. For \fBshmhuge\fR or \fBmmaphuge\fR to work,
1054the system must have free huge pages allocated. \fBmmaphuge\fR also needs to
2e266ba6
JA
1055have hugetlbfs mounted, and \fIfile\fR must point there. At least on Linux,
1056huge pages must be manually allocated. See \fB/proc/sys/vm/nr_hugehages\fR
1057and the documentation for that. Normally you just need to echo an appropriate
1058number, eg echoing 8 will ensure that the OS has 8 huge pages ready for
1059use.
d60e92d1
AC
1060.RE
1061.TP
d392365e 1062.BI iomem_align \fR=\fPint "\fR,\fP mem_align" \fR=\fPint
cecbfd47 1063This indicates the memory alignment of the IO memory buffers. Note that the
d529ee19
JA
1064given alignment is applied to the first IO unit buffer, if using \fBiodepth\fR
1065the alignment of the following buffers are given by the \fBbs\fR used. In
1066other words, if using a \fBbs\fR that is a multiple of the page sized in the
1067system, all buffers will be aligned to this value. If using a \fBbs\fR that
1068is not page aligned, the alignment of subsequent IO memory buffers is the
1069sum of the \fBiomem_align\fR and \fBbs\fR used.
1070.TP
f7fa2653 1071.BI hugepage\-size \fR=\fPint
d60e92d1 1072Defines the size of a huge page. Must be at least equal to the system setting.
b22989b9 1073Should be a multiple of 1MB. Default: 4MB.
d60e92d1
AC
1074.TP
1075.B exitall
1076Terminate all jobs when one finishes. Default: wait for each job to finish.
1077.TP
1078.BI bwavgtime \fR=\fPint
1079Average bandwidth calculations over the given time in milliseconds. Default:
1080500ms.
1081.TP
c8eeb9df
JA
1082.BI iopsavgtime \fR=\fPint
1083Average IOPS calculations over the given time in milliseconds. Default:
1084500ms.
1085.TP
d60e92d1 1086.BI create_serialize \fR=\fPbool
d1429b5c 1087If true, serialize file creation for the jobs. Default: true.
d60e92d1
AC
1088.TP
1089.BI create_fsync \fR=\fPbool
ccc2b328 1090\fBfsync\fR\|(2) data file after creation. Default: true.
d60e92d1 1091.TP
6b7f6851
JA
1092.BI create_on_open \fR=\fPbool
1093If true, the files are not created until they are opened for IO by the job.
1094.TP
25460cf6
JA
1095.BI create_only \fR=\fPbool
1096If true, fio will only run the setup phase of the job. If files need to be
1097laid out or updated on disk, only that will be done. The actual job contents
1098are not executed.
1099.TP
2378826d
JA
1100.BI allow_file_create \fR=\fPbool
1101If true, fio is permitted to create files as part of its workload. This is
1102the default behavior. If this option is false, then fio will error out if the
1103files it needs to use don't already exist. Default: true.
1104.TP
e81ecca3
JA
1105.BI allow_mounted_write \fR=\fPbool
1106If this isn't set, fio will abort jobs that are destructive (eg that write)
1107to what appears to be a mounted device or partition. This should help catch
1108creating inadvertently destructive tests, not realizing that the test will
1109destroy data on the mounted file system. Default: false.
1110.TP
e9f48479
JA
1111.BI pre_read \fR=\fPbool
1112If this is given, files will be pre-read into memory before starting the given
1113IO operation. This will also clear the \fR \fBinvalidate\fR flag, since it is
9c0d2241
JA
1114pointless to pre-read and then drop the cache. This will only work for IO
1115engines that are seekable, since they allow you to read the same data
1116multiple times. Thus it will not work on eg network or splice IO.
e9f48479 1117.TP
d60e92d1
AC
1118.BI unlink \fR=\fPbool
1119Unlink job files when done. Default: false.
1120.TP
1121.BI loops \fR=\fPint
1122Specifies the number of iterations (runs of the same workload) of this job.
1123Default: 1.
1124.TP
5e4c7118
JA
1125.BI verify_only \fR=\fPbool
1126Do not perform the specified workload, only verify data still matches previous
1127invocation of this workload. This option allows one to check data multiple
1128times at a later date without overwriting it. This option makes sense only for
1129workloads that write data, and does not support workloads with the
1130\fBtime_based\fR option set.
1131.TP
d60e92d1
AC
1132.BI do_verify \fR=\fPbool
1133Run the verify phase after a write phase. Only valid if \fBverify\fR is set.
1134Default: true.
1135.TP
1136.BI verify \fR=\fPstr
1137Method of verifying file contents after each iteration of the job. Allowed
1138values are:
1139.RS
1140.RS
1141.TP
844ea602 1142.B md5 crc16 crc32 crc32c crc32c-intel crc64 crc7 sha256 sha512 sha1 xxhash
0539d758
JA
1143Store appropriate checksum in the header of each block. crc32c-intel is
1144hardware accelerated SSE4.2 driven, falls back to regular crc32c if
1145not supported by the system.
d60e92d1
AC
1146.TP
1147.B meta
1148Write extra information about each I/O (timestamp, block number, etc.). The
996093bb 1149block number is verified. See \fBverify_pattern\fR as well.
d60e92d1 1150.TP
59245381
JA
1151.B pattern
1152Verify a strict pattern. Normally fio includes a header with some basic
1153information and checksumming, but if this option is set, only the
1154specific pattern set with \fBverify_pattern\fR is verified.
1155.TP
d60e92d1
AC
1156.B null
1157Pretend to verify. Used for testing internals.
1158.RE
b892dc08
JA
1159
1160This option can be used for repeated burn-in tests of a system to make sure
1161that the written data is also correctly read back. If the data direction given
1162is a read or random read, fio will assume that it should verify a previously
1163written file. If the data direction includes any form of write, the verify will
1164be of the newly written data.
d60e92d1
AC
1165.RE
1166.TP
5c9323fb 1167.BI verifysort \fR=\fPbool
d60e92d1
AC
1168If true, written verify blocks are sorted if \fBfio\fR deems it to be faster to
1169read them back in a sorted manner. Default: true.
1170.TP
fa769d44
SW
1171.BI verifysort_nr \fR=\fPint
1172Pre-load and sort verify blocks for a read workload.
1173.TP
f7fa2653 1174.BI verify_offset \fR=\fPint
d60e92d1 1175Swap the verification header with data somewhere else in the block before
d1429b5c 1176writing. It is swapped back before verifying.
d60e92d1 1177.TP
f7fa2653 1178.BI verify_interval \fR=\fPint
d60e92d1
AC
1179Write the verification header for this number of bytes, which should divide
1180\fBblocksize\fR. Default: \fBblocksize\fR.
1181.TP
996093bb
JA
1182.BI verify_pattern \fR=\fPstr
1183If set, fio will fill the io buffers with this pattern. Fio defaults to filling
1184with totally random bytes, but sometimes it's interesting to fill with a known
1185pattern for io verification purposes. Depending on the width of the pattern,
1186fio will fill 1/2/3/4 bytes of the buffer at the time(it can be either a
1187decimal or a hex number). The verify_pattern if larger than a 32-bit quantity
1188has to be a hex number that starts with either "0x" or "0X". Use with
2fa5a241
RP
1189\fBverify\fP=meta or \fBverify\fP=pattern. Also, verify_pattern supports %o
1190format, which means that for each block offset will be written and then
1191verifyied back, e.g.:
1192.RS
1193.RS
1194\fBverify_pattern\fR=%o
1195.RE
1196Or use combination of everything:
1197.LP
1198.RS
1199\fBverify_pattern\fR=0xff%o"abcd"-21
1200.RE
1201.RE
996093bb 1202.TP
d60e92d1
AC
1203.BI verify_fatal \fR=\fPbool
1204If true, exit the job on the first observed verification failure. Default:
1205false.
1206.TP
b463e936
JA
1207.BI verify_dump \fR=\fPbool
1208If set, dump the contents of both the original data block and the data block we
1209read off disk to files. This allows later analysis to inspect just what kind of
ef71e317 1210data corruption occurred. Off by default.
b463e936 1211.TP
e8462bd8
JA
1212.BI verify_async \fR=\fPint
1213Fio will normally verify IO inline from the submitting thread. This option
1214takes an integer describing how many async offload threads to create for IO
1215verification instead, causing fio to offload the duty of verifying IO contents
c85c324c
JA
1216to one or more separate threads. If using this offload option, even sync IO
1217engines can benefit from using an \fBiodepth\fR setting higher than 1, as it
1218allows them to have IO in flight while verifies are running.
e8462bd8
JA
1219.TP
1220.BI verify_async_cpus \fR=\fPstr
1221Tell fio to set the given CPU affinity on the async IO verification threads.
1222See \fBcpus_allowed\fP for the format used.
1223.TP
6f87418f
JA
1224.BI verify_backlog \fR=\fPint
1225Fio will normally verify the written contents of a job that utilizes verify
1226once that job has completed. In other words, everything is written then
1227everything is read back and verified. You may want to verify continually
1228instead for a variety of reasons. Fio stores the meta data associated with an
1229IO block in memory, so for large verify workloads, quite a bit of memory would
092f707f
DN
1230be used up holding this meta data. If this option is enabled, fio will write
1231only N blocks before verifying these blocks.
6f87418f
JA
1232.TP
1233.BI verify_backlog_batch \fR=\fPint
1234Control how many blocks fio will verify if verify_backlog is set. If not set,
1235will default to the value of \fBverify_backlog\fR (meaning the entire queue is
092f707f
DN
1236read back and verified). If \fBverify_backlog_batch\fR is less than
1237\fBverify_backlog\fR then not all blocks will be verified, if
1238\fBverify_backlog_batch\fR is larger than \fBverify_backlog\fR, some blocks
1239will be verified more than once.
6f87418f 1240.TP
fa769d44
SW
1241.BI trim_percentage \fR=\fPint
1242Number of verify blocks to discard/trim.
1243.TP
1244.BI trim_verify_zero \fR=\fPbool
1245Verify that trim/discarded blocks are returned as zeroes.
1246.TP
1247.BI trim_backlog \fR=\fPint
1248Trim after this number of blocks are written.
1249.TP
1250.BI trim_backlog_batch \fR=\fPint
1251Trim this number of IO blocks.
1252.TP
1253.BI experimental_verify \fR=\fPbool
1254Enable experimental verification.
1255.TP
ca09be4b
JA
1256.BI verify_state_save \fR=\fPbool
1257When a job exits during the write phase of a verify workload, save its
1258current state. This allows fio to replay up until that point, if the
1259verify state is loaded for the verify read phase.
1260.TP
1261.BI verify_state_load \fR=\fPbool
1262If a verify termination trigger was used, fio stores the current write
1263state of each thread. This can be used at verification time so that fio
1264knows how far it should verify. Without this information, fio will run
1265a full verification pass, according to the settings in the job file used.
1266.TP
d392365e 1267.B stonewall "\fR,\fP wait_for_previous"
5982a925 1268Wait for preceding jobs in the job file to exit before starting this one.
d60e92d1
AC
1269\fBstonewall\fR implies \fBnew_group\fR.
1270.TP
1271.B new_group
1272Start a new reporting group. If not given, all jobs in a file will be part
1273of the same reporting group, unless separated by a stonewall.
1274.TP
1275.BI numjobs \fR=\fPint
1276Number of clones (processes/threads performing the same workload) of this job.
1277Default: 1.
1278.TP
1279.B group_reporting
1280If set, display per-group reports instead of per-job when \fBnumjobs\fR is
1281specified.
1282.TP
1283.B thread
1284Use threads created with \fBpthread_create\fR\|(3) instead of processes created
1285with \fBfork\fR\|(2).
1286.TP
f7fa2653 1287.BI zonesize \fR=\fPint
d60e92d1
AC
1288Divide file into zones of the specified size in bytes. See \fBzoneskip\fR.
1289.TP
fa769d44
SW
1290.BI zonerange \fR=\fPint
1291Give size of an IO zone. See \fBzoneskip\fR.
1292.TP
f7fa2653 1293.BI zoneskip \fR=\fPint
d1429b5c 1294Skip the specified number of bytes when \fBzonesize\fR bytes of data have been
d60e92d1
AC
1295read.
1296.TP
1297.BI write_iolog \fR=\fPstr
5b42a488
SH
1298Write the issued I/O patterns to the specified file. Specify a separate file
1299for each job, otherwise the iologs will be interspersed and the file may be
1300corrupt.
d60e92d1
AC
1301.TP
1302.BI read_iolog \fR=\fPstr
1303Replay the I/O patterns contained in the specified file generated by
1304\fBwrite_iolog\fR, or may be a \fBblktrace\fR binary file.
1305.TP
64bbb865
DN
1306.BI replay_no_stall \fR=\fPint
1307While replaying I/O patterns using \fBread_iolog\fR the default behavior
1308attempts to respect timing information between I/Os. Enabling
1309\fBreplay_no_stall\fR causes I/Os to be replayed as fast as possible while
1310still respecting ordering.
1311.TP
d1c46c04
DN
1312.BI replay_redirect \fR=\fPstr
1313While replaying I/O patterns using \fBread_iolog\fR the default behavior
1314is to replay the IOPS onto the major/minor device that each IOP was recorded
1315from. Setting \fBreplay_redirect\fR causes all IOPS to be replayed onto the
1316single specified device regardless of the device it was recorded from.
1317.TP
0c63576e
JA
1318.BI replay_align \fR=\fPint
1319Force alignment of IO offsets and lengths in a trace to this power of 2 value.
1320.TP
1321.BI replay_scale \fR=\fPint
1322Scale sector offsets down by this factor when replaying traces.
1323.TP
3a5db920
JA
1324.BI per_job_logs \fR=\fPbool
1325If set, this generates bw/clat/iops log with per file private filenames. If
1326not set, jobs with identical names will share the log filename. Default: true.
1327.TP
836bad52 1328.BI write_bw_log \fR=\fPstr
901bb994
JA
1329If given, write a bandwidth log of the jobs in this job file. Can be used to
1330store data of the bandwidth of the jobs in their lifetime. The included
1331fio_generate_plots script uses gnuplot to turn these text files into nice
26b26fca 1332graphs. See \fBwrite_lat_log\fR for behaviour of given filename. For this
8ad3b3dd 1333option, the postfix is _bw.x.log, where x is the index of the job (1..N,
3a5db920
JA
1334where N is the number of jobs). If \fBper_job_logs\fR is false, then the
1335filename will not include the job index.
d60e92d1 1336.TP
836bad52 1337.BI write_lat_log \fR=\fPstr
901bb994 1338Same as \fBwrite_bw_log\fR, but writes I/O completion latencies. If no
8ad3b3dd
JA
1339filename is given with this option, the default filename of
1340"jobname_type.x.log" is used, where x is the index of the job (1..N, where
1341N is the number of jobs). Even if the filename is given, fio will still
3a5db920
JA
1342append the type of log. If \fBper_job_logs\fR is false, then the filename will
1343not include the job index.
901bb994 1344.TP
c8eeb9df
JA
1345.BI write_iops_log \fR=\fPstr
1346Same as \fBwrite_bw_log\fR, but writes IOPS. If no filename is given with this
8ad3b3dd
JA
1347option, the default filename of "jobname_type.x.log" is used, where x is the
1348index of the job (1..N, where N is the number of jobs). Even if the filename
3a5db920
JA
1349is given, fio will still append the type of log. If \fBper_job_logs\fR is false,
1350then the filename will not include the job index.
c8eeb9df 1351.TP
b8bc8cba
JA
1352.BI log_avg_msec \fR=\fPint
1353By default, fio will log an entry in the iops, latency, or bw log for every
1354IO that completes. When writing to the disk log, that can quickly grow to a
1355very large size. Setting this option makes fio average the each log entry
1356over the specified period of time, reducing the resolution of the log.
1357Defaults to 0.
1358.TP
ae588852
JA
1359.BI log_offset \fR=\fPbool
1360If this is set, the iolog options will include the byte offset for the IO
1361entry as well as the other data values.
1362.TP
aee2ab67
JA
1363.BI log_compression \fR=\fPint
1364If this is set, fio will compress the IO logs as it goes, to keep the memory
1365footprint lower. When a log reaches the specified size, that chunk is removed
1366and compressed in the background. Given that IO logs are fairly highly
1367compressible, this yields a nice memory savings for longer runs. The downside
1368is that the compression will consume some background CPU cycles, so it may
1369impact the run. This, however, is also true if the logging ends up consuming
1370most of the system memory. So pick your poison. The IO logs are saved
1371normally at the end of a run, by decompressing the chunks and storing them
1372in the specified log file. This feature depends on the availability of zlib.
1373.TP
b26317c9
JA
1374.BI log_store_compressed \fR=\fPbool
1375If set, and \fBlog\fR_compression is also set, fio will store the log files in
1376a compressed format. They can be decompressed with fio, using the
1377\fB\-\-inflate-log\fR command line parameter. The files will be stored with a
1378\fB\.fz\fR suffix.
1379.TP
66347cfa
DE
1380.BI block_error_percentiles \fR=\fPbool
1381If set, record errors in trim block-sized units from writes and trims and output
1382a histogram of how many trims it took to get to errors, and what kind of error
1383was encountered.
1384.TP
836bad52 1385.BI disable_lat \fR=\fPbool
02af0988 1386Disable measurements of total latency numbers. Useful only for cutting
ccc2b328 1387back the number of calls to \fBgettimeofday\fR\|(2), as that does impact performance at
901bb994
JA
1388really high IOPS rates. Note that to really get rid of a large amount of these
1389calls, this option must be used with disable_slat and disable_bw as well.
1390.TP
836bad52 1391.BI disable_clat \fR=\fPbool
c95f9daf 1392Disable measurements of completion latency numbers. See \fBdisable_lat\fR.
02af0988 1393.TP
836bad52 1394.BI disable_slat \fR=\fPbool
02af0988 1395Disable measurements of submission latency numbers. See \fBdisable_lat\fR.
901bb994 1396.TP
836bad52 1397.BI disable_bw_measurement \fR=\fPbool
02af0988 1398Disable measurements of throughput/bandwidth numbers. See \fBdisable_lat\fR.
d60e92d1 1399.TP
f7fa2653 1400.BI lockmem \fR=\fPint
d60e92d1 1401Pin the specified amount of memory with \fBmlock\fR\|(2). Can be used to
81c6b6cd 1402simulate a smaller amount of memory. The amount specified is per worker.
d60e92d1
AC
1403.TP
1404.BI exec_prerun \fR=\fPstr
1405Before running the job, execute the specified command with \fBsystem\fR\|(3).
ce486495
EV
1406.RS
1407Output is redirected in a file called \fBjobname.prerun.txt\fR
1408.RE
d60e92d1
AC
1409.TP
1410.BI exec_postrun \fR=\fPstr
1411Same as \fBexec_prerun\fR, but the command is executed after the job completes.
ce486495
EV
1412.RS
1413Output is redirected in a file called \fBjobname.postrun.txt\fR
1414.RE
d60e92d1
AC
1415.TP
1416.BI ioscheduler \fR=\fPstr
1417Attempt to switch the device hosting the file to the specified I/O scheduler.
1418.TP
d60e92d1 1419.BI disk_util \fR=\fPbool
d1429b5c 1420Generate disk utilization statistics if the platform supports it. Default: true.
901bb994 1421.TP
23893646
JA
1422.BI clocksource \fR=\fPstr
1423Use the given clocksource as the base of timing. The supported options are:
1424.RS
1425.TP
1426.B gettimeofday
ccc2b328 1427\fBgettimeofday\fR\|(2)
23893646
JA
1428.TP
1429.B clock_gettime
ccc2b328 1430\fBclock_gettime\fR\|(2)
23893646
JA
1431.TP
1432.B cpu
1433Internal CPU clock source
1434.TP
1435.RE
1436.P
1437\fBcpu\fR is the preferred clocksource if it is reliable, as it is very fast
1438(and fio is heavy on time calls). Fio will automatically use this clocksource
1439if it's supported and considered reliable on the system it is running on,
1440unless another clocksource is specifically set. For x86/x86-64 CPUs, this
1441means supporting TSC Invariant.
1442.TP
901bb994 1443.BI gtod_reduce \fR=\fPbool
ccc2b328 1444Enable all of the \fBgettimeofday\fR\|(2) reducing options (disable_clat, disable_slat,
901bb994 1445disable_bw) plus reduce precision of the timeout somewhat to really shrink the
ccc2b328 1446\fBgettimeofday\fR\|(2) call count. With this option enabled, we only do about 0.4% of
901bb994
JA
1447the gtod() calls we would have done if all time keeping was enabled.
1448.TP
1449.BI gtod_cpu \fR=\fPint
1450Sometimes it's cheaper to dedicate a single thread of execution to just getting
1451the current time. Fio (and databases, for instance) are very intensive on
ccc2b328 1452\fBgettimeofday\fR\|(2) calls. With this option, you can set one CPU aside for doing
901bb994
JA
1453nothing but logging current time to a shared memory location. Then the other
1454threads/processes that run IO workloads need only copy that segment, instead of
ccc2b328 1455entering the kernel with a \fBgettimeofday\fR\|(2) call. The CPU set aside for doing
901bb994
JA
1456these time calls will be excluded from other uses. Fio will manually clear it
1457from the CPU mask of other jobs.
f2bba182 1458.TP
8b28bd41
DM
1459.BI ignore_error \fR=\fPstr
1460Sometimes you want to ignore some errors during test in that case you can specify
1461error list for each error type.
1462.br
1463ignore_error=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST
1464.br
1465errors for given error type is separated with ':'.
1466Error may be symbol ('ENOSPC', 'ENOMEM') or an integer.
1467.br
1468Example: ignore_error=EAGAIN,ENOSPC:122 .
1469.br
1470This option will ignore EAGAIN from READ, and ENOSPC and 122(EDQUOT) from WRITE.
1471.TP
1472.BI error_dump \fR=\fPbool
1473If set dump every error even if it is non fatal, true by default. If disabled
1474only fatal error will be dumped
1475.TP
fa769d44
SW
1476.BI profile \fR=\fPstr
1477Select a specific builtin performance test.
1478.TP
a696fa2a
JA
1479.BI cgroup \fR=\fPstr
1480Add job to this control group. If it doesn't exist, it will be created.
6adb38a1
JA
1481The system must have a mounted cgroup blkio mount point for this to work. If
1482your system doesn't have it mounted, you can do so with:
1483
5982a925 1484# mount \-t cgroup \-o blkio none /cgroup
a696fa2a
JA
1485.TP
1486.BI cgroup_weight \fR=\fPint
1487Set the weight of the cgroup to this value. See the documentation that comes
1488with the kernel, allowed values are in the range of 100..1000.
e0b0d892 1489.TP
7de87099
VG
1490.BI cgroup_nodelete \fR=\fPbool
1491Normally fio will delete the cgroups it has created after the job completion.
1492To override this behavior and to leave cgroups around after the job completion,
1493set cgroup_nodelete=1. This can be useful if one wants to inspect various
1494cgroup files after job completion. Default: false
1495.TP
e0b0d892
JA
1496.BI uid \fR=\fPint
1497Instead of running as the invoking user, set the user ID to this value before
1498the thread/process does any work.
1499.TP
1500.BI gid \fR=\fPint
1501Set group ID, see \fBuid\fR.
83349190 1502.TP
fa769d44
SW
1503.BI unit_base \fR=\fPint
1504Base unit for reporting. Allowed values are:
1505.RS
1506.TP
1507.B 0
1508Use auto-detection (default).
1509.TP
1510.B 8
1511Byte based.
1512.TP
1513.B 1
1514Bit based.
1515.RE
1516.P
1517.TP
9e684a49
DE
1518.BI flow_id \fR=\fPint
1519The ID of the flow. If not specified, it defaults to being a global flow. See
1520\fBflow\fR.
1521.TP
1522.BI flow \fR=\fPint
1523Weight in token-based flow control. If this value is used, then there is a
1524\fBflow counter\fR which is used to regulate the proportion of activity between
1525two or more jobs. fio attempts to keep this flow counter near zero. The
1526\fBflow\fR parameter stands for how much should be added or subtracted to the
1527flow counter on each iteration of the main I/O loop. That is, if one job has
1528\fBflow=8\fR and another job has \fBflow=-1\fR, then there will be a roughly
15291:8 ratio in how much one runs vs the other.
1530.TP
1531.BI flow_watermark \fR=\fPint
1532The maximum value that the absolute value of the flow counter is allowed to
1533reach before the job must wait for a lower value of the counter.
1534.TP
1535.BI flow_sleep \fR=\fPint
1536The period of time, in microseconds, to wait after the flow watermark has been
1537exceeded before retrying operations
1538.TP
83349190
YH
1539.BI clat_percentiles \fR=\fPbool
1540Enable the reporting of percentiles of completion latencies.
1541.TP
1542.BI percentile_list \fR=\fPfloat_list
66347cfa
DE
1543Overwrite the default list of percentiles for completion latencies and the
1544block error histogram. Each number is a floating number in the range (0,100],
1545and the maximum length of the list is 20. Use ':' to separate the
3eb07285 1546numbers. For example, \-\-percentile_list=99.5:99.9 will cause fio to
83349190
YH
1547report the values of completion latency below which 99.5% and 99.9% of
1548the observed latencies fell, respectively.
de890a1e
SL
1549.SS "Ioengine Parameters List"
1550Some parameters are only valid when a specific ioengine is in use. These are
1551used identically to normal parameters, with the caveat that when used on the
cf145d90 1552command line, they must come after the ioengine.
de890a1e 1553.TP
e4585935
JA
1554.BI (cpu)cpuload \fR=\fPint
1555Attempt to use the specified percentage of CPU cycles.
1556.TP
1557.BI (cpu)cpuchunks \fR=\fPint
1558Split the load into cycles of the given time. In microseconds.
1559.TP
046395d7
JA
1560.BI (cpu)exit_on_io_done \fR=\fPbool
1561Detect when IO threads are done, then exit.
1562.TP
de890a1e
SL
1563.BI (libaio)userspace_reap
1564Normally, with the libaio engine in use, fio will use
1565the io_getevents system call to reap newly returned events.
1566With this flag turned on, the AIO ring will be read directly
1567from user-space to reap events. The reaping mode is only
1568enabled when polling for a minimum of 0 events (eg when
1569iodepth_batch_complete=0).
1570.TP
1571.BI (net,netsplice)hostname \fR=\fPstr
1572The host name or IP address to use for TCP or UDP based IO.
1573If the job is a TCP listener or UDP reader, the hostname is not
b511c9aa 1574used and must be omitted unless it is a valid UDP multicast address.
de890a1e
SL
1575.TP
1576.BI (net,netsplice)port \fR=\fPint
6315af9d
JA
1577The TCP or UDP port to bind to or connect to. If this is used with
1578\fBnumjobs\fR to spawn multiple instances of the same job type, then
1579this will be the starting port number since fio will use a range of ports.
de890a1e 1580.TP
b93b6a2e
SB
1581.BI (net,netsplice)interface \fR=\fPstr
1582The IP address of the network interface used to send or receive UDP multicast
1583packets.
1584.TP
d3a623de
SB
1585.BI (net,netsplice)ttl \fR=\fPint
1586Time-to-live value for outgoing UDP multicast packets. Default: 1
1587.TP
1d360ffb
JA
1588.BI (net,netsplice)nodelay \fR=\fPbool
1589Set TCP_NODELAY on TCP connections.
1590.TP
de890a1e
SL
1591.BI (net,netsplice)protocol \fR=\fPstr "\fR,\fP proto" \fR=\fPstr
1592The network protocol to use. Accepted values are:
1593.RS
1594.RS
1595.TP
1596.B tcp
1597Transmission control protocol
1598.TP
49ccb8c1
JA
1599.B tcpv6
1600Transmission control protocol V6
1601.TP
de890a1e 1602.B udp
f5cc3d0e 1603User datagram protocol
de890a1e 1604.TP
49ccb8c1
JA
1605.B udpv6
1606User datagram protocol V6
1607.TP
de890a1e
SL
1608.B unix
1609UNIX domain socket
1610.RE
1611.P
1612When the protocol is TCP or UDP, the port must also be given,
1613as well as the hostname if the job is a TCP listener or UDP
1614reader. For unix sockets, the normal filename option should be
1615used and the port is invalid.
1616.RE
1617.TP
1618.BI (net,netsplice)listen
1619For TCP network connections, tell fio to listen for incoming
1620connections rather than initiating an outgoing connection. The
1621hostname must be omitted if this option is used.
d54fce84 1622.TP
7aeb1e94 1623.BI (net, pingpong) \fR=\fPbool
cecbfd47 1624Normally a network writer will just continue writing data, and a network reader
cf145d90 1625will just consume packets. If pingpong=1 is set, a writer will send its normal
7aeb1e94
JA
1626payload to the reader, then wait for the reader to send the same payload back.
1627This allows fio to measure network latencies. The submission and completion
1628latencies then measure local time spent sending or receiving, and the
1629completion latency measures how long it took for the other end to receive and
b511c9aa
SB
1630send back. For UDP multicast traffic pingpong=1 should only be set for a single
1631reader when multiple readers are listening to the same address.
7aeb1e94 1632.TP
1008602c
JA
1633.BI (net, window_size) \fR=\fPint
1634Set the desired socket buffer size for the connection.
1635.TP
e5f34d95
JA
1636.BI (net, mss) \fR=\fPint
1637Set the TCP maximum segment size (TCP_MAXSEG).
1638.TP
d54fce84
DM
1639.BI (e4defrag,donorname) \fR=\fPstr
1640File will be used as a block donor (swap extents between files)
1641.TP
1642.BI (e4defrag,inplace) \fR=\fPint
1643Configure donor file block allocation strategy
1644.RS
1645.BI 0(default) :
1646Preallocate donor's file on init
1647.TP
1648.BI 1:
cecbfd47 1649allocate space immediately inside defragment event, and free right after event
d54fce84 1650.RE
0d978694
DAG
1651.TP
1652.BI (rbd)rbdname \fR=\fPstr
1653Specifies the name of the RBD.
1654.TP
1655.BI (rbd)pool \fR=\fPstr
1656Specifies the name of the Ceph pool containing the RBD.
1657.TP
1658.BI (rbd)clientname \fR=\fPstr
1659Specifies the username (without the 'client.' prefix) used to access the Ceph cluster.
65fa28ca
DE
1660.TP
1661.BI (mtd)skipbad \fR=\fPbool
1662Skip operations against known bad blocks.
d60e92d1 1663.SH OUTPUT
d1429b5c
AC
1664While running, \fBfio\fR will display the status of the created jobs. For
1665example:
d60e92d1 1666.RS
d1429b5c 1667.P
d60e92d1
AC
1668Threads: 1: [_r] [24.8% done] [ 13509/ 8334 kb/s] [eta 00h:01m:31s]
1669.RE
1670.P
d1429b5c
AC
1671The characters in the first set of brackets denote the current status of each
1672threads. The possible values are:
1673.P
1674.PD 0
d60e92d1
AC
1675.RS
1676.TP
1677.B P
1678Setup but not started.
1679.TP
1680.B C
1681Thread created.
1682.TP
1683.B I
1684Initialized, waiting.
1685.TP
1686.B R
1687Running, doing sequential reads.
1688.TP
1689.B r
1690Running, doing random reads.
1691.TP
1692.B W
1693Running, doing sequential writes.
1694.TP
1695.B w
1696Running, doing random writes.
1697.TP
1698.B M
1699Running, doing mixed sequential reads/writes.
1700.TP
1701.B m
1702Running, doing mixed random reads/writes.
1703.TP
1704.B F
1705Running, currently waiting for \fBfsync\fR\|(2).
1706.TP
1707.B V
1708Running, verifying written data.
1709.TP
1710.B E
1711Exited, not reaped by main thread.
1712.TP
1713.B \-
1714Exited, thread reaped.
1715.RE
d1429b5c 1716.PD
d60e92d1
AC
1717.P
1718The second set of brackets shows the estimated completion percentage of
1719the current group. The third set shows the read and write I/O rate,
1720respectively. Finally, the estimated run time of the job is displayed.
1721.P
1722When \fBfio\fR completes (or is interrupted by Ctrl-C), it will show data
1723for each thread, each group of threads, and each disk, in that order.
1724.P
1725Per-thread statistics first show the threads client number, group-id, and
1726error code. The remaining figures are as follows:
1727.RS
d60e92d1
AC
1728.TP
1729.B io
1730Number of megabytes of I/O performed.
1731.TP
1732.B bw
1733Average data rate (bandwidth).
1734.TP
1735.B runt
1736Threads run time.
1737.TP
1738.B slat
1739Submission latency minimum, maximum, average and standard deviation. This is
1740the time it took to submit the I/O.
1741.TP
1742.B clat
1743Completion latency minimum, maximum, average and standard deviation. This
1744is the time between submission and completion.
1745.TP
1746.B bw
1747Bandwidth minimum, maximum, percentage of aggregate bandwidth received, average
1748and standard deviation.
1749.TP
1750.B cpu
1751CPU usage statistics. Includes user and system time, number of context switches
1752this thread went through and number of major and minor page faults.
1753.TP
1754.B IO depths
1755Distribution of I/O depths. Each depth includes everything less than (or equal)
1756to it, but greater than the previous depth.
1757.TP
1758.B IO issued
1759Number of read/write requests issued, and number of short read/write requests.
1760.TP
1761.B IO latencies
1762Distribution of I/O completion latencies. The numbers follow the same pattern
1763as \fBIO depths\fR.
1764.RE
d60e92d1
AC
1765.P
1766The group statistics show:
d1429b5c 1767.PD 0
d60e92d1
AC
1768.RS
1769.TP
1770.B io
1771Number of megabytes I/O performed.
1772.TP
1773.B aggrb
1774Aggregate bandwidth of threads in the group.
1775.TP
1776.B minb
1777Minimum average bandwidth a thread saw.
1778.TP
1779.B maxb
1780Maximum average bandwidth a thread saw.
1781.TP
1782.B mint
d1429b5c 1783Shortest runtime of threads in the group.
d60e92d1
AC
1784.TP
1785.B maxt
1786Longest runtime of threads in the group.
1787.RE
d1429b5c 1788.PD
d60e92d1
AC
1789.P
1790Finally, disk statistics are printed with reads first:
d1429b5c 1791.PD 0
d60e92d1
AC
1792.RS
1793.TP
1794.B ios
1795Number of I/Os performed by all groups.
1796.TP
1797.B merge
1798Number of merges in the I/O scheduler.
1799.TP
1800.B ticks
1801Number of ticks we kept the disk busy.
1802.TP
1803.B io_queue
1804Total time spent in the disk queue.
1805.TP
1806.B util
1807Disk utilization.
1808.RE
d1429b5c 1809.PD
8423bd11
JA
1810.P
1811It is also possible to get fio to dump the current output while it is
1812running, without terminating the job. To do that, send fio the \fBUSR1\fR
1813signal.
d60e92d1 1814.SH TERSE OUTPUT
2b8c71b0
CE
1815If the \fB\-\-minimal\fR / \fB\-\-append-terse\fR options are given, the
1816results will be printed/appended in a semicolon-delimited format suitable for
1817scripted use.
1818A job description (if provided) follows on a new line. Note that the first
525c2bfa
JA
1819number in the line is the version number. If the output has to be changed
1820for some reason, this number will be incremented by 1 to signify that
1821change. The fields are:
d60e92d1
AC
1822.P
1823.RS
5e726d0a 1824.B terse version, fio version, jobname, groupid, error
d60e92d1
AC
1825.P
1826Read status:
1827.RS
312b4af2 1828.B Total I/O \fR(KB)\fP, bandwidth \fR(KB/s)\fP, IOPS, runtime \fR(ms)\fP
d60e92d1
AC
1829.P
1830Submission latency:
1831.RS
1832.B min, max, mean, standard deviation
1833.RE
1834Completion latency:
1835.RS
1836.B min, max, mean, standard deviation
1837.RE
1db92cb6
JA
1838Completion latency percentiles (20 fields):
1839.RS
1840.B Xth percentile=usec
1841.RE
525c2bfa
JA
1842Total latency:
1843.RS
1844.B min, max, mean, standard deviation
1845.RE
d60e92d1
AC
1846Bandwidth:
1847.RS
1848.B min, max, aggregate percentage of total, mean, standard deviation
1849.RE
1850.RE
1851.P
1852Write status:
1853.RS
312b4af2 1854.B Total I/O \fR(KB)\fP, bandwidth \fR(KB/s)\fP, IOPS, runtime \fR(ms)\fP
d60e92d1
AC
1855.P
1856Submission latency:
1857.RS
1858.B min, max, mean, standard deviation
1859.RE
1860Completion latency:
1861.RS
1862.B min, max, mean, standard deviation
1863.RE
1db92cb6
JA
1864Completion latency percentiles (20 fields):
1865.RS
1866.B Xth percentile=usec
1867.RE
525c2bfa
JA
1868Total latency:
1869.RS
1870.B min, max, mean, standard deviation
1871.RE
d60e92d1
AC
1872Bandwidth:
1873.RS
1874.B min, max, aggregate percentage of total, mean, standard deviation
1875.RE
1876.RE
1877.P
d1429b5c 1878CPU usage:
d60e92d1 1879.RS
bd2626f0 1880.B user, system, context switches, major page faults, minor page faults
d60e92d1
AC
1881.RE
1882.P
1883IO depth distribution:
1884.RS
1885.B <=1, 2, 4, 8, 16, 32, >=64
1886.RE
1887.P
562c2d2f 1888IO latency distribution:
d60e92d1 1889.RS
562c2d2f
DN
1890Microseconds:
1891.RS
1892.B <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000
1893.RE
1894Milliseconds:
1895.RS
1896.B <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000
1897.RE
1898.RE
1899.P
f2f788dd
JA
1900Disk utilization (1 for each disk used):
1901.RS
1902.B name, read ios, write ios, read merges, write merges, read ticks, write ticks, read in-queue time, write in-queue time, disk utilization percentage
1903.RE
1904.P
5982a925 1905Error Info (dependent on continue_on_error, default off):
562c2d2f
DN
1906.RS
1907.B total # errors, first error code
d60e92d1
AC
1908.RE
1909.P
562c2d2f 1910.B text description (if provided in config - appears on newline)
d60e92d1 1911.RE
49da1240
JA
1912.SH CLIENT / SERVER
1913Normally you would run fio as a stand-alone application on the machine
1914where the IO workload should be generated. However, it is also possible to
1915run the frontend and backend of fio separately. This makes it possible to
1916have a fio server running on the machine(s) where the IO workload should
1917be running, while controlling it from another machine.
1918
1919To start the server, you would do:
1920
1921\fBfio \-\-server=args\fR
1922
1923on that machine, where args defines what fio listens to. The arguments
811826be 1924are of the form 'type:hostname or IP:port'. 'type' is either 'ip' (or ip4)
20c67f10
MS
1925for TCP/IP v4, 'ip6' for TCP/IP v6, or 'sock' for a local unix domain
1926socket. 'hostname' is either a hostname or IP address, and 'port' is the port to
811826be 1927listen to (only valid for TCP/IP, not a local socket). Some examples:
49da1240 1928
e01e9745 19291) fio \-\-server
49da1240
JA
1930
1931 Start a fio server, listening on all interfaces on the default port (8765).
1932
e01e9745 19332) fio \-\-server=ip:hostname,4444
49da1240
JA
1934
1935 Start a fio server, listening on IP belonging to hostname and on port 4444.
1936
e01e9745 19373) fio \-\-server=ip6:::1,4444
811826be
JA
1938
1939 Start a fio server, listening on IPv6 localhost ::1 and on port 4444.
1940
e01e9745 19414) fio \-\-server=,4444
49da1240
JA
1942
1943 Start a fio server, listening on all interfaces on port 4444.
1944
e01e9745 19455) fio \-\-server=1.2.3.4
49da1240
JA
1946
1947 Start a fio server, listening on IP 1.2.3.4 on the default port.
1948
e01e9745 19496) fio \-\-server=sock:/tmp/fio.sock
49da1240
JA
1950
1951 Start a fio server, listening on the local socket /tmp/fio.sock.
1952
1953When a server is running, you can connect to it from a client. The client
1954is run with:
1955
e01e9745 1956fio \-\-local-args \-\-client=server \-\-remote-args <job file(s)>
49da1240 1957
e01e9745
MS
1958where \-\-local-args are arguments that are local to the client where it is
1959running, 'server' is the connect string, and \-\-remote-args and <job file(s)>
49da1240
JA
1960are sent to the server. The 'server' string follows the same format as it
1961does on the server side, to allow IP/hostname/socket and port strings.
1962You can connect to multiple clients as well, to do that you could run:
1963
e01e9745 1964fio \-\-client=server2 \-\-client=server2 <job file(s)>
323255cc
JA
1965
1966If the job file is located on the fio server, then you can tell the server
1967to load a local file as well. This is done by using \-\-remote-config:
1968
1969fio \-\-client=server \-\-remote-config /path/to/file.fio
1970
39b5f61e 1971Then fio will open this local (to the server) job file instead
323255cc 1972of being passed one from the client.
39b5f61e
BE
1973
1974If you have many servers (example: 100 VMs/containers), you can input a pathname
1975of a file containing host IPs/names as the parameter value for the \-\-client option.
1976For example, here is an example "host.list" file containing 2 hostnames:
1977
1978host1.your.dns.domain
1979.br
1980host2.your.dns.domain
1981
1982The fio command would then be:
1983
1984fio \-\-client=host.list <job file>
1985
1986In this mode, you cannot input server-specific parameters or job files, and all
1987servers receive the same job file.
1988
1989In order to enable fio \-\-client runs utilizing a shared filesystem from multiple hosts,
1990fio \-\-client now prepends the IP address of the server to the filename. For example,
1991if fio is using directory /mnt/nfs/fio and is writing filename fileio.tmp,
1992with a \-\-client hostfile
1993containing two hostnames h1 and h2 with IP addresses 192.168.10.120 and 192.168.10.121, then
1994fio will create two files:
1995
1996/mnt/nfs/fio/192.168.10.120.fileio.tmp
1997.br
1998/mnt/nfs/fio/192.168.10.121.fileio.tmp
1999
d60e92d1 2000.SH AUTHORS
49da1240 2001
d60e92d1 2002.B fio
aa58d252 2003was written by Jens Axboe <jens.axboe@oracle.com>,
f8b8f7da 2004now Jens Axboe <axboe@fb.com>.
d1429b5c
AC
2005.br
2006This man page was written by Aaron Carroll <aaronc@cse.unsw.edu.au> based
d60e92d1
AC
2007on documentation by Jens Axboe.
2008.SH "REPORTING BUGS"
482900c9 2009Report bugs to the \fBfio\fR mailing list <fio@vger.kernel.org>.
d1429b5c 2010See \fBREADME\fR.
d60e92d1 2011.SH "SEE ALSO"
d1429b5c
AC
2012For further documentation see \fBHOWTO\fR and \fBREADME\fR.
2013.br
2014Sample jobfiles are available in the \fBexamples\fR directory.
d60e92d1 2015