blktrace: add support for scaling and aligning replays
[fio.git] / fio.1
CommitLineData
f8b8f7da 1.TH fio 1 "December 2014" "User Manual"
d60e92d1
AC
2.SH NAME
3fio \- flexible I/O tester
4.SH SYNOPSIS
5.B fio
6[\fIoptions\fR] [\fIjobfile\fR]...
7.SH DESCRIPTION
8.B fio
9is a tool that will spawn a number of threads or processes doing a
10particular type of I/O action as specified by the user.
11The typical use of fio is to write a job file matching the I/O load
12one wants to simulate.
13.SH OPTIONS
14.TP
49da1240
JA
15.BI \-\-debug \fR=\fPtype
16Enable verbose tracing of various fio actions. May be `all' for all types
17or individual types separated by a comma (eg \-\-debug=io,file). `help' will
18list all available tracing options.
19.TP
d60e92d1
AC
20.BI \-\-output \fR=\fPfilename
21Write output to \fIfilename\fR.
22.TP
e28ee21d 23.BI \-\-output-format \fR=\fPformat
52a768c1 24Set the reporting format to \fInormal\fR, \fIterse\fR, or \fIjson\fR.
e28ee21d 25.TP
b2cecdc2 26.BI \-\-runtime \fR=\fPruntime
27Limit run time to \fIruntime\fR seconds.
d60e92d1 28.TP
d60e92d1
AC
29.B \-\-bandwidth\-log
30Generate per-job bandwidth logs.
31.TP
32.B \-\-minimal
d1429b5c 33Print statistics in a terse, semicolon-delimited format.
d60e92d1 34.TP
2b8c71b0
CE
35.B \-\-append-terse
36Print statistics in selected mode AND terse, semicolon-delimited format.
37.TP
49da1240
JA
38.B \-\-version
39Display version information and exit.
40.TP
065248bf 41.BI \-\-terse\-version \fR=\fPversion
4d658652 42Set terse version output format (Current version 3, or older version 2).
49da1240
JA
43.TP
44.B \-\-help
45Display usage information and exit.
46.TP
fec0f21c
JA
47.B \-\-cpuclock-test
48Perform test and validation of internal CPU clock
49.TP
50.BI \-\-crctest[\fR=\fPtest]
51Test the speed of the builtin checksumming functions. If no argument is given,
52all of them are tested. Or a comma separated list can be passed, in which
53case the given ones are tested.
54.TP
49da1240
JA
55.BI \-\-cmdhelp \fR=\fPcommand
56Print help information for \fIcommand\fR. May be `all' for all commands.
57.TP
de890a1e
SL
58.BI \-\-enghelp \fR=\fPioengine[,command]
59List all commands defined by \fIioengine\fR, or print help for \fIcommand\fR defined by \fIioengine\fR.
60.TP
d60e92d1
AC
61.BI \-\-showcmd \fR=\fPjobfile
62Convert \fIjobfile\fR to a set of command-line options.
63.TP
d60e92d1
AC
64.BI \-\-eta \fR=\fPwhen
65Specifies when real-time ETA estimate should be printed. \fIwhen\fR may
66be one of `always', `never' or `auto'.
67.TP
30b5d57f
JA
68.BI \-\-eta\-newline \fR=\fPtime
69Force an ETA newline for every `time` period passed.
70.TP
71.BI \-\-status\-interval \fR=\fPtime
72Report full output status every `time` period passed.
73.TP
49da1240
JA
74.BI \-\-readonly
75Turn on safety read-only checks, preventing any attempted write.
76.TP
c0a5d35e 77.BI \-\-section \fR=\fPsec
cf145d90 78Only run section \fIsec\fR from job file. This option can be used multiple times to add more sections to run.
c0a5d35e 79.TP
49da1240
JA
80.BI \-\-alloc\-size \fR=\fPkb
81Set the internal smalloc pool size to \fIkb\fP kilobytes.
d60e92d1 82.TP
49da1240
JA
83.BI \-\-warnings\-fatal
84All fio parser warnings are fatal, causing fio to exit with an error.
9183788d 85.TP
49da1240 86.BI \-\-max\-jobs \fR=\fPnr
57e118a2 87Set the maximum allowed number of jobs (threads/processes) to support.
d60e92d1 88.TP
49da1240
JA
89.BI \-\-server \fR=\fPargs
90Start a backend server, with \fIargs\fP specifying what to listen to. See client/server section.
f57a9c59 91.TP
49da1240
JA
92.BI \-\-daemonize \fR=\fPpidfile
93Background a fio server, writing the pid to the given pid file.
94.TP
95.BI \-\-client \fR=\fPhost
96Instead of running the jobs locally, send and run them on the given host.
f2a2ce0e
HL
97.TP
98.BI \-\-idle\-prof \fR=\fPoption
99Report cpu idleness on a system or percpu basis (\fIoption\fP=system,percpu) or run unit work calibration only (\fIoption\fP=calibrate).
d60e92d1
AC
100.SH "JOB FILE FORMAT"
101Job files are in `ini' format. They consist of one or more
102job definitions, which begin with a job name in square brackets and
103extend to the next job name. The job name can be any ASCII string
104except `global', which has a special meaning. Following the job name is
105a sequence of zero or more parameters, one per line, that define the
106behavior of the job. Any line starting with a `;' or `#' character is
d1429b5c 107considered a comment and ignored.
d9956b64
AC
108.P
109If \fIjobfile\fR is specified as `-', the job file will be read from
110standard input.
d60e92d1
AC
111.SS "Global Section"
112The global section contains default parameters for jobs specified in the
113job file. A job is only affected by global sections residing above it,
114and there may be any number of global sections. Specific job definitions
115may override any parameter set in global sections.
116.SH "JOB PARAMETERS"
117.SS Types
b470a02c
SC
118Some parameters may take arguments of a specific type.
119Anywhere a numeric value is required, an arithmetic expression may be used,
d59aa780
JA
120provided it is surrounded by parentheses. Supported operators are:
121.RS
122.RS
123.TP
124.B addition (+)
125.TP
126.B subtraction (-)
127.TP
128.B multiplication (*)
129.TP
130.B division (/)
131.TP
132.B modulus (%)
133.TP
134.B exponentiation (^)
135.RE
136.RE
137.P
138For time values in expressions, units are microseconds by default. This is
139different than for time values not in expressions (not enclosed in
140parentheses). The types used are:
d60e92d1
AC
141.TP
142.I str
143String: a sequence of alphanumeric characters.
144.TP
145.I int
d60e92d1 146SI integer: a whole number, possibly containing a suffix denoting the base unit
b09da8fa
JA
147of the value. Accepted suffixes are `k', 'M', 'G', 'T', and 'P', denoting
148kilo (1024), mega (1024^2), giga (1024^3), tera (1024^4), and peta (1024^5)
74454ce4
CE
149respectively. If prefixed with '0x', the value is assumed to be base 16
150(hexadecimal). A suffix may include a trailing 'b', for instance 'kb' is
151identical to 'k'. You can specify a base 10 value by using 'KiB', 'MiB','GiB',
152etc. This is useful for disk drives where values are often given in base 10
153values. Specifying '30GiB' will get you 30*1000^3 bytes.
154When specifying times the default suffix meaning changes, still denoting the
155base unit of the value, but accepted suffixes are 'D' (days), 'H' (hours), 'M'
0de5b26f
JA
156(minutes), 'S' Seconds, 'ms' (or msec) milli seconds, 'us' (or 'usec') micro
157seconds. Time values without a unit specify seconds.
74454ce4 158The suffixes are not case sensitive.
d60e92d1
AC
159.TP
160.I bool
161Boolean: a true or false value. `0' denotes false, `1' denotes true.
162.TP
163.I irange
164Integer range: a range of integers specified in the format
d1429b5c
AC
165\fIlower\fR:\fIupper\fR or \fIlower\fR\-\fIupper\fR. \fIlower\fR and
166\fIupper\fR may contain a suffix as described above. If an option allows two
167sets of ranges, they are separated with a `,' or `/' character. For example:
168`8\-8k/8M\-4G'.
83349190
YH
169.TP
170.I float_list
171List of floating numbers: A list of floating numbers, separated by
cecbfd47 172a ':' character.
d60e92d1
AC
173.SS "Parameter List"
174.TP
175.BI name \fR=\fPstr
d9956b64 176May be used to override the job name. On the command line, this parameter
d60e92d1
AC
177has the special purpose of signalling the start of a new job.
178.TP
179.BI description \fR=\fPstr
180Human-readable description of the job. It is printed when the job is run, but
181otherwise has no special purpose.
182.TP
183.BI directory \fR=\fPstr
184Prefix filenames with this directory. Used to place files in a location other
185than `./'.
bcbfeefa
CE
186You can specify a number of directories by separating the names with a ':'
187character. These directories will be assigned equally distributed to job clones
188creates with \fInumjobs\fR as long as they are using generated filenames.
189If specific \fIfilename(s)\fR are set fio will use the first listed directory,
190and thereby matching the \fIfilename\fR semantic which generates a file each
67445b63
JA
191clone if not specified, but let all clones use the same if set. See
192\fIfilename\fR for considerations regarding escaping certain characters on
193some platforms.
d60e92d1
AC
194.TP
195.BI filename \fR=\fPstr
196.B fio
197normally makes up a file name based on the job name, thread number, and file
d1429b5c 198number. If you want to share files between threads in a job or several jobs,
de890a1e
SL
199specify a \fIfilename\fR for each of them to override the default.
200If the I/O engine is file-based, you can specify
d1429b5c
AC
201a number of files by separating the names with a `:' character. `\-' is a
202reserved name, meaning stdin or stdout, depending on the read/write direction
67445b63
JA
203set. On Windows, disk devices are accessed as \\.\PhysicalDrive0 for the first
204device, \\.\PhysicalDrive1 for the second etc. Note: Windows and FreeBSD
205prevent write access to areas of the disk containing in-use data
206(e.g. filesystems). If the wanted filename does need to include a colon, then
4904acd5
JM
207escape that with a '\\' character. For instance, if the filename is
208"/dev/dsk/foo@3,0:c", then you would use filename="/dev/dsk/foo@3,0\\:c".
d60e92d1 209.TP
de98bd30 210.BI filename_format \fR=\fPstr
ce594fbe 211If sharing multiple files between jobs, it is usually necessary to have
de98bd30
JA
212fio generate the exact names that you want. By default, fio will name a file
213based on the default file format specification of
214\fBjobname.jobnumber.filenumber\fP. With this option, that can be
215customized. Fio will recognize and replace the following keywords in this
216string:
217.RS
218.RS
219.TP
220.B $jobname
221The name of the worker thread or process.
222.TP
223.B $jobnum
224The incremental number of the worker thread or process.
225.TP
226.B $filenum
227The incremental number of the file for that worker thread or process.
228.RE
229.P
230To have dependent jobs share a set of files, this option can be set to
231have fio generate filenames that are shared between the two. For instance,
232if \fBtestfiles.$filenum\fR is specified, file number 4 for any job will
233be named \fBtestfiles.4\fR. The default of \fB$jobname.$jobnum.$filenum\fR
234will be used if no other format specifier is given.
235.RE
236.P
237.TP
3ce9dcaf
JA
238.BI lockfile \fR=\fPstr
239Fio defaults to not locking any files before it does IO to them. If a file or
240file descriptor is shared, fio can serialize IO to that file to make the end
241result consistent. This is usual for emulating real workloads that share files.
242The lock modes are:
243.RS
244.RS
245.TP
246.B none
247No locking. This is the default.
248.TP
249.B exclusive
cf145d90 250Only one thread or process may do IO at a time, excluding all others.
3ce9dcaf
JA
251.TP
252.B readwrite
253Read-write locking on the file. Many readers may access the file at the same
254time, but writes get exclusive access.
255.RE
ce594fbe 256.RE
3ce9dcaf 257.P
d60e92d1
AC
258.BI opendir \fR=\fPstr
259Recursively open any files below directory \fIstr\fR.
260.TP
261.BI readwrite \fR=\fPstr "\fR,\fP rw" \fR=\fPstr
262Type of I/O pattern. Accepted values are:
263.RS
264.RS
265.TP
266.B read
d1429b5c 267Sequential reads.
d60e92d1
AC
268.TP
269.B write
d1429b5c 270Sequential writes.
d60e92d1 271.TP
fa769d44
SW
272.B trim
273Sequential trim (Linux block devices only).
274.TP
d60e92d1 275.B randread
d1429b5c 276Random reads.
d60e92d1
AC
277.TP
278.B randwrite
d1429b5c 279Random writes.
d60e92d1 280.TP
fa769d44
SW
281.B randtrim
282Random trim (Linux block devices only).
283.TP
10b023db 284.B rw, readwrite
d1429b5c 285Mixed sequential reads and writes.
d60e92d1
AC
286.TP
287.B randrw
d1429b5c 288Mixed random reads and writes.
82a90686
JA
289.TP
290.B trimwrite
291Trim and write mixed workload. Blocks will be trimmed first, then the same
292blocks will be written to.
d60e92d1
AC
293.RE
294.P
38dad62d
JA
295For mixed I/O, the default split is 50/50. For certain types of io the result
296may still be skewed a bit, since the speed may be different. It is possible to
3b7fa9ec 297specify a number of IO's to do before getting a new offset, this is done by
38dad62d
JA
298appending a `:\fI<nr>\fR to the end of the string given. For a random read, it
299would look like \fBrw=randread:8\fR for passing in an offset modifier with a
059b0802
JA
300value of 8. If the postfix is used with a sequential IO pattern, then the value
301specified will be added to the generated offset for each IO. For instance,
302using \fBrw=write:4k\fR will skip 4k for every write. It turns sequential IO
303into sequential IO with holes. See the \fBrw_sequencer\fR option.
d60e92d1
AC
304.RE
305.TP
38dad62d
JA
306.BI rw_sequencer \fR=\fPstr
307If an offset modifier is given by appending a number to the \fBrw=<str>\fR line,
308then this option controls how that number modifies the IO offset being
309generated. Accepted values are:
310.RS
311.RS
312.TP
313.B sequential
314Generate sequential offset
315.TP
316.B identical
317Generate the same offset
318.RE
319.P
320\fBsequential\fR is only useful for random IO, where fio would normally
321generate a new random offset for every IO. If you append eg 8 to randread, you
322would get a new random offset for every 8 IO's. The result would be a seek for
323only every 8 IO's, instead of for every IO. Use \fBrw=randread:8\fR to specify
324that. As sequential IO is already sequential, setting \fBsequential\fR for that
325would not result in any differences. \fBidentical\fR behaves in a similar
326fashion, except it sends the same offset 8 number of times before generating a
327new offset.
328.RE
329.P
330.TP
90fef2d1
JA
331.BI kb_base \fR=\fPint
332The base unit for a kilobyte. The defacto base is 2^10, 1024. Storage
333manufacturers like to use 10^3 or 1000 as a base ten unit instead, for obvious
5c9323fb 334reasons. Allowed values are 1024 or 1000, with 1024 being the default.
90fef2d1 335.TP
771e58be
JA
336.BI unified_rw_reporting \fR=\fPbool
337Fio normally reports statistics on a per data direction basis, meaning that
338read, write, and trim are accounted and reported separately. If this option is
cf145d90 339set fio sums the results and reports them as "mixed" instead.
771e58be 340.TP
d60e92d1 341.BI randrepeat \fR=\fPbool
56e2a5fc
CE
342Seed the random number generator used for random I/O patterns in a predictable
343way so the pattern is repeatable across runs. Default: true.
344.TP
345.BI allrandrepeat \fR=\fPbool
346Seed all random number generators in a predictable way so results are
347repeatable across runs. Default: false.
d60e92d1 348.TP
04778baf
JA
349.BI randseed \fR=\fPint
350Seed the random number generators based on this seed value, to be able to
351control what sequence of output is being generated. If not set, the random
352sequence depends on the \fBrandrepeat\fR setting.
353.TP
a596f047
EG
354.BI fallocate \fR=\fPstr
355Whether pre-allocation is performed when laying down files. Accepted values
356are:
357.RS
358.RS
359.TP
360.B none
361Do not pre-allocate space.
362.TP
363.B posix
ccc2b328 364Pre-allocate via \fBposix_fallocate\fR\|(3).
a596f047
EG
365.TP
366.B keep
ccc2b328 367Pre-allocate via \fBfallocate\fR\|(2) with FALLOC_FL_KEEP_SIZE set.
a596f047
EG
368.TP
369.B 0
370Backward-compatible alias for 'none'.
371.TP
372.B 1
373Backward-compatible alias for 'posix'.
374.RE
375.P
376May not be available on all supported platforms. 'keep' is only
377available on Linux. If using ZFS on Solaris this must be set to 'none'
378because ZFS doesn't support it. Default: 'posix'.
379.RE
7bc8c2cf 380.TP
d60e92d1 381.BI fadvise_hint \fR=\fPbool
cf145d90 382Use \fBposix_fadvise\fR\|(2) to advise the kernel what I/O patterns
d1429b5c 383are likely to be issued. Default: true.
d60e92d1 384.TP
37659335
JA
385.BI fadvise_stream \fR=\fPint
386Use \fBposix_fadvise\fR\|(2) to advise the kernel what stream ID the
387writes issued belong to. Only supported on Linux. Note, this option
388may change going forward.
389.TP
f7fa2653 390.BI size \fR=\fPint
d60e92d1 391Total size of I/O for this job. \fBfio\fR will run until this many bytes have
a4d3b4db
JA
392been transferred, unless limited by other options (\fBruntime\fR, for instance,
393or increased/descreased by \fBio_size\fR). Unless \fBnrfiles\fR and
394\fBfilesize\fR options are given, this amount will be divided between the
395available files for the job. If not set, fio will use the full size of the
396given files or devices. If the files do not exist, size must be given. It is
397also possible to give size as a percentage between 1 and 100. If size=20% is
398given, fio will use 20% of the full size of the given files or devices.
399.TP
400.BI io_size \fR=\fPint "\fR,\fB io_limit \fR=\fPint
77731b29
JA
401Normally fio operates within the region set by \fBsize\fR, which means that
402the \fBsize\fR option sets both the region and size of IO to be performed.
403Sometimes that is not what you want. With this option, it is possible to
404define just the amount of IO that fio should do. For instance, if \fBsize\fR
405is set to 20G and \fBio_limit\fR is set to 5G, fio will perform IO within
a4d3b4db
JA
406the first 20G but exit when 5G have been done. The opposite is also
407possible - if \fBsize\fR is set to 20G, and \fBio_size\fR is set to 40G, then
408fio will do 40G of IO within the 0..20G region.
d60e92d1 409.TP
74586c1e 410.BI fill_device \fR=\fPbool "\fR,\fB fill_fs" \fR=\fPbool
3ce9dcaf
JA
411Sets size to something really large and waits for ENOSPC (no space left on
412device) as the terminating condition. Only makes sense with sequential write.
413For a read workload, the mount point will be filled first then IO started on
4f12432e
JA
414the result. This option doesn't make sense if operating on a raw device node,
415since the size of that is already known by the file system. Additionally,
416writing beyond end-of-device will not return ENOSPC there.
3ce9dcaf 417.TP
d60e92d1
AC
418.BI filesize \fR=\fPirange
419Individual file sizes. May be a range, in which case \fBfio\fR will select sizes
d1429b5c
AC
420for files at random within the given range, limited to \fBsize\fR in total (if
421that is given). If \fBfilesize\fR is not specified, each created file is the
422same size.
d60e92d1 423.TP
bedc9dc2
JA
424.BI file_append \fR=\fPbool
425Perform IO after the end of the file. Normally fio will operate within the
426size of a file. If this option is set, then fio will append to the file
427instead. This has identical behavior to setting \fRoffset\fP to the size
0aae4ce7 428of a file. This option is ignored on non-regular files.
bedc9dc2 429.TP
f7fa2653 430.BI blocksize \fR=\fPint[,int] "\fR,\fB bs" \fR=\fPint[,int]
d9472271
JA
431Block size for I/O units. Default: 4k. Values for reads, writes, and trims
432can be specified separately in the format \fIread\fR,\fIwrite\fR,\fItrim\fR
433either of which may be empty to leave that value at its default. If a trailing
434comma isn't given, the remainder will inherit the last value set.
d60e92d1 435.TP
9183788d 436.BI blocksize_range \fR=\fPirange[,irange] "\fR,\fB bsrange" \fR=\fPirange[,irange]
d1429b5c
AC
437Specify a range of I/O block sizes. The issued I/O unit will always be a
438multiple of the minimum size, unless \fBblocksize_unaligned\fR is set. Applies
9183788d 439to both reads and writes if only one range is given, but can be specified
de8f6de9 440separately with a comma separating the values. Example: bsrange=1k-4k,2k-8k.
9183788d
JA
441Also (see \fBblocksize\fR).
442.TP
443.BI bssplit \fR=\fPstr
444This option allows even finer grained control of the block sizes issued,
445not just even splits between them. With this option, you can weight various
446block sizes for exact control of the issued IO for a job that has mixed
447block sizes. The format of the option is bssplit=blocksize/percentage,
5982a925 448optionally adding as many definitions as needed separated by a colon.
9183788d 449Example: bssplit=4k/10:64k/50:32k/40 would issue 50% 64k blocks, 10% 4k
c83cdd3e
JA
450blocks and 40% 32k blocks. \fBbssplit\fR also supports giving separate
451splits to reads and writes. The format is identical to what the
452\fBbs\fR option accepts, the read and write parts are separated with a
453comma.
d60e92d1
AC
454.TP
455.B blocksize_unaligned\fR,\fP bs_unaligned
d1429b5c
AC
456If set, any size in \fBblocksize_range\fR may be used. This typically won't
457work with direct I/O, as that normally requires sector alignment.
d60e92d1 458.TP
2b7a01d0 459.BI blockalign \fR=\fPint[,int] "\fR,\fB ba" \fR=\fPint[,int]
639ce0f3
MS
460At what boundary to align random IO offsets. Defaults to the same as 'blocksize'
461the minimum blocksize given. Minimum alignment is typically 512b
2b7a01d0
JA
462for using direct IO, though it usually depends on the hardware block size.
463This option is mutually exclusive with using a random map for files, so it
464will turn off that option.
43602667 465.TP
6aca9b3d
JA
466.BI bs_is_seq_rand \fR=\fPbool
467If this option is set, fio will use the normal read,write blocksize settings as
468sequential,random instead. Any random read or write will use the WRITE
469blocksize settings, and any sequential read or write will use the READ
470blocksize setting.
471.TP
d60e92d1 472.B zero_buffers
cf145d90 473Initialize buffers with all zeros. Default: fill buffers with random data.
d60e92d1 474.TP
901bb994
JA
475.B refill_buffers
476If this option is given, fio will refill the IO buffers on every submit. The
477default is to only fill it at init time and reuse that data. Only makes sense
478if zero_buffers isn't specified, naturally. If data verification is enabled,
479refill_buffers is also automatically enabled.
480.TP
fd68418e
JA
481.BI scramble_buffers \fR=\fPbool
482If \fBrefill_buffers\fR is too costly and the target is using data
483deduplication, then setting this option will slightly modify the IO buffer
484contents to defeat normal de-dupe attempts. This is not enough to defeat
485more clever block compression attempts, but it will stop naive dedupe
486of blocks. Default: true.
487.TP
c5751c62
JA
488.BI buffer_compress_percentage \fR=\fPint
489If this is set, then fio will attempt to provide IO buffer content (on WRITEs)
490that compress to the specified level. Fio does this by providing a mix of
d1af2894
JA
491random data and a fixed pattern. The fixed pattern is either zeroes, or the
492pattern specified by \fBbuffer_pattern\fR. If the pattern option is used, it
493might skew the compression ratio slightly. Note that this is per block size
494unit, for file/disk wide compression level that matches this setting. Note
495that this is per block size unit, for file/disk wide compression level that
496matches this setting, you'll also want to set refill_buffers.
c5751c62
JA
497.TP
498.BI buffer_compress_chunk \fR=\fPint
499See \fBbuffer_compress_percentage\fR. This setting allows fio to manage how
500big the ranges of random data and zeroed data is. Without this set, fio will
501provide \fBbuffer_compress_percentage\fR of blocksize random data, followed by
502the remaining zeroed. With this set to some chunk size smaller than the block
503size, fio can alternate random and zeroed data throughout the IO buffer.
504.TP
ce35b1ec 505.BI buffer_pattern \fR=\fPstr
cf145d90
CVB
506If set, fio will fill the IO buffers with this pattern. If not set, the contents
507of IO buffers is defined by the other options related to buffer contents. The
ce35b1ec 508setting can be any pattern of bytes, and can be prefixed with 0x for hex
02975b64
JA
509values. It may also be a string, where the string must then be wrapped with
510"".
ce35b1ec 511.TP
5c94b008
JA
512.BI dedupe_percentage \fR=\fPint
513If set, fio will generate this percentage of identical buffers when writing.
514These buffers will be naturally dedupable. The contents of the buffers depend
515on what other buffer compression settings have been set. It's possible to have
516the individual buffers either fully compressible, or not at all. This option
517only controls the distribution of unique buffers.
518.TP
d60e92d1
AC
519.BI nrfiles \fR=\fPint
520Number of files to use for this job. Default: 1.
521.TP
522.BI openfiles \fR=\fPint
523Number of files to keep open at the same time. Default: \fBnrfiles\fR.
524.TP
525.BI file_service_type \fR=\fPstr
526Defines how files to service are selected. The following types are defined:
527.RS
528.RS
529.TP
530.B random
5c9323fb 531Choose a file at random.
d60e92d1
AC
532.TP
533.B roundrobin
cf145d90 534Round robin over opened files (default).
5c9323fb 535.TP
6b7f6851
JA
536.B sequential
537Do each file in the set sequentially.
d60e92d1
AC
538.RE
539.P
cf145d90 540The number of I/Os to issue before switching to a new file can be specified by
d60e92d1
AC
541appending `:\fIint\fR' to the service type.
542.RE
543.TP
544.BI ioengine \fR=\fPstr
545Defines how the job issues I/O. The following types are defined:
546.RS
547.RS
548.TP
549.B sync
ccc2b328 550Basic \fBread\fR\|(2) or \fBwrite\fR\|(2) I/O. \fBfseek\fR\|(2) is used to
d60e92d1
AC
551position the I/O location.
552.TP
a31041ea 553.B psync
ccc2b328 554Basic \fBpread\fR\|(2) or \fBpwrite\fR\|(2) I/O.
a31041ea 555.TP
9183788d 556.B vsync
ccc2b328 557Basic \fBreadv\fR\|(2) or \fBwritev\fR\|(2) I/O. Will emulate queuing by
cecbfd47 558coalescing adjacent IOs into a single submission.
9183788d 559.TP
a46c5e01 560.B pvsync
ccc2b328 561Basic \fBpreadv\fR\|(2) or \fBpwritev\fR\|(2) I/O.
a46c5e01 562.TP
d60e92d1 563.B libaio
de890a1e 564Linux native asynchronous I/O. This ioengine defines engine specific options.
d60e92d1
AC
565.TP
566.B posixaio
ccc2b328 567POSIX asynchronous I/O using \fBaio_read\fR\|(3) and \fBaio_write\fR\|(3).
03e20d68
BC
568.TP
569.B solarisaio
570Solaris native asynchronous I/O.
571.TP
572.B windowsaio
573Windows native asynchronous I/O.
d60e92d1
AC
574.TP
575.B mmap
ccc2b328
SW
576File is memory mapped with \fBmmap\fR\|(2) and data copied using
577\fBmemcpy\fR\|(3).
d60e92d1
AC
578.TP
579.B splice
ccc2b328 580\fBsplice\fR\|(2) is used to transfer the data and \fBvmsplice\fR\|(2) to
d1429b5c 581transfer data from user-space to the kernel.
d60e92d1
AC
582.TP
583.B syslet-rw
584Use the syslet system calls to make regular read/write asynchronous.
585.TP
586.B sg
587SCSI generic sg v3 I/O. May be either synchronous using the SG_IO ioctl, or if
ccc2b328
SW
588the target is an sg character device, we use \fBread\fR\|(2) and
589\fBwrite\fR\|(2) for asynchronous I/O.
d60e92d1
AC
590.TP
591.B null
592Doesn't transfer any data, just pretends to. Mainly used to exercise \fBfio\fR
593itself and for debugging and testing purposes.
594.TP
595.B net
de890a1e
SL
596Transfer over the network. The protocol to be used can be defined with the
597\fBprotocol\fR parameter. Depending on the protocol, \fBfilename\fR,
598\fBhostname\fR, \fBport\fR, or \fBlisten\fR must be specified.
599This ioengine defines engine specific options.
d60e92d1
AC
600.TP
601.B netsplice
ccc2b328 602Like \fBnet\fR, but uses \fBsplice\fR\|(2) and \fBvmsplice\fR\|(2) to map data
de890a1e 603and send/receive. This ioengine defines engine specific options.
d60e92d1 604.TP
53aec0a4 605.B cpuio
d60e92d1
AC
606Doesn't transfer any data, but burns CPU cycles according to \fBcpuload\fR and
607\fBcpucycles\fR parameters.
608.TP
609.B guasi
610The GUASI I/O engine is the Generic Userspace Asynchronous Syscall Interface
cecbfd47 611approach to asynchronous I/O.
d1429b5c
AC
612.br
613See <http://www.xmailserver.org/guasi\-lib.html>.
d60e92d1 614.TP
21b8aee8 615.B rdma
85286c5c
BVA
616The RDMA I/O engine supports both RDMA memory semantics (RDMA_WRITE/RDMA_READ)
617and channel semantics (Send/Recv) for the InfiniBand, RoCE and iWARP protocols.
21b8aee8 618.TP
d60e92d1
AC
619.B external
620Loads an external I/O engine object file. Append the engine filename as
621`:\fIenginepath\fR'.
d54fce84
DM
622.TP
623.B falloc
cecbfd47 624 IO engine that does regular linux native fallocate call to simulate data
d54fce84
DM
625transfer as fio ioengine
626.br
627 DDIR_READ does fallocate(,mode = FALLOC_FL_KEEP_SIZE,)
628.br
0981fd71 629 DIR_WRITE does fallocate(,mode = 0)
d54fce84
DM
630.br
631 DDIR_TRIM does fallocate(,mode = FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE)
632.TP
633.B e4defrag
634IO engine that does regular EXT4_IOC_MOVE_EXT ioctls to simulate defragment activity
635request to DDIR_WRITE event
0d978694
DAG
636.TP
637.B rbd
638IO engine supporting direct access to Ceph Rados Block Devices (RBD) via librbd
639without the need to use the kernel rbd driver. This ioengine defines engine specific
640options.
a7c386f4 641.TP
642.B gfapi
cc47f094 643Using Glusterfs libgfapi sync interface to direct access to Glusterfs volumes without
644having to go through FUSE. This ioengine defines engine specific
645options.
646.TP
647.B gfapi_async
648Using Glusterfs libgfapi async interface to direct access to Glusterfs volumes without
a7c386f4 649having to go through FUSE. This ioengine defines engine specific
650options.
1b10477b 651.TP
b74e419e
MM
652.B libhdfs
653Read and write through Hadoop (HDFS). The \fBfilename\fR option is used to
654specify host,port of the hdfs name-node to connect. This engine interprets
655offsets a little differently. In HDFS, files once created cannot be modified.
656So random writes are not possible. To imitate this, libhdfs engine expects
657bunch of small files to be created over HDFS, and engine will randomly pick a
658file out of those files based on the offset generated by fio backend. (see the
659example job file to create such files, use rw=write option). Please note, you
660might want to set necessary environment variables to work with hdfs/libhdfs
661properly.
65fa28ca
DE
662.TP
663.B mtd
664Read, write and erase an MTD character device (e.g., /dev/mtd0). Discards are
665treated as erases. Depending on the underlying device type, the I/O may have
666to go in a certain pattern, e.g., on NAND, writing sequentially to erase blocks
667and discarding before overwriting. The writetrim mode works well for this
668constraint.
d60e92d1 669.RE
595e1734 670.P
d60e92d1
AC
671.RE
672.TP
673.BI iodepth \fR=\fPint
8489dae4
SK
674Number of I/O units to keep in flight against the file. Note that increasing
675iodepth beyond 1 will not affect synchronous ioengines (except for small
cf145d90 676degress when verify_async is in use). Even async engines may impose OS
ee72ca09
JA
677restrictions causing the desired depth not to be achieved. This may happen on
678Linux when using libaio and not setting \fBdirect\fR=1, since buffered IO is
679not async on that OS. Keep an eye on the IO depth distribution in the
680fio output to verify that the achieved depth is as expected. Default: 1.
d60e92d1
AC
681.TP
682.BI iodepth_batch \fR=\fPint
683Number of I/Os to submit at once. Default: \fBiodepth\fR.
684.TP
3ce9dcaf
JA
685.BI iodepth_batch_complete \fR=\fPint
686This defines how many pieces of IO to retrieve at once. It defaults to 1 which
687 means that we'll ask for a minimum of 1 IO in the retrieval process from the
688kernel. The IO retrieval will go on until we hit the limit set by
689\fBiodepth_low\fR. If this variable is set to 0, then fio will always check for
690completed events before queuing more IO. This helps reduce IO latency, at the
691cost of more retrieval system calls.
692.TP
d60e92d1
AC
693.BI iodepth_low \fR=\fPint
694Low watermark indicating when to start filling the queue again. Default:
695\fBiodepth\fR.
696.TP
1ad01bd1
JA
697.BI io_submit_mode \fR=\fPstr
698This option controls how fio submits the IO to the IO engine. The default is
699\fBinline\fR, which means that the fio job threads submit and reap IO directly.
700If set to \fBoffload\fR, the job threads will offload IO submission to a
701dedicated pool of IO threads. This requires some coordination and thus has a
702bit of extra overhead, especially for lower queue depth IO where it can
703increase latencies. The benefit is that fio can manage submission rates
704independently of the device completion rates. This avoids skewed latency
705reporting if IO gets back up on the device side (the coordinated omission
706problem).
707.TP
d60e92d1
AC
708.BI direct \fR=\fPbool
709If true, use non-buffered I/O (usually O_DIRECT). Default: false.
710.TP
d01612f3
CM
711.BI atomic \fR=\fPbool
712If value is true, attempt to use atomic direct IO. Atomic writes are guaranteed
713to be stable once acknowledged by the operating system. Only Linux supports
714O_ATOMIC right now.
715.TP
d60e92d1
AC
716.BI buffered \fR=\fPbool
717If true, use buffered I/O. This is the opposite of the \fBdirect\fR parameter.
718Default: true.
719.TP
f7fa2653 720.BI offset \fR=\fPint
d60e92d1
AC
721Offset in the file to start I/O. Data before the offset will not be touched.
722.TP
591e9e06
JA
723.BI offset_increment \fR=\fPint
724If this is provided, then the real offset becomes the
69bdd6ba
JH
725offset + offset_increment * thread_number, where the thread number is a
726counter that starts at 0 and is incremented for each sub-job (i.e. when
727numjobs option is specified). This option is useful if there are several jobs
728which are intended to operate on a file in parallel disjoint segments, with
729even spacing between the starting points.
591e9e06 730.TP
ddf24e42
JA
731.BI number_ios \fR=\fPint
732Fio will normally perform IOs until it has exhausted the size of the region
733set by \fBsize\fR, or if it exhaust the allocated time (or hits an error
734condition). With this setting, the range/size can be set independently of
735the number of IOs to perform. When fio reaches this number, it will exit
be3fec7d
JA
736normally and report status. Note that this does not extend the amount
737of IO that will be done, it will only stop fio if this condition is met
738before other end-of-job criteria.
ddf24e42 739.TP
d60e92d1 740.BI fsync \fR=\fPint
d1429b5c
AC
741How many I/Os to perform before issuing an \fBfsync\fR\|(2) of dirty data. If
7420, don't sync. Default: 0.
d60e92d1 743.TP
5f9099ea
JA
744.BI fdatasync \fR=\fPint
745Like \fBfsync\fR, but uses \fBfdatasync\fR\|(2) instead to only sync the
746data parts of the file. Default: 0.
747.TP
fa769d44
SW
748.BI write_barrier \fR=\fPint
749Make every Nth write a barrier write.
750.TP
e76b1da4 751.BI sync_file_range \fR=\fPstr:int
ccc2b328
SW
752Use \fBsync_file_range\fR\|(2) for every \fRval\fP number of write operations. Fio will
753track range of writes that have happened since the last \fBsync_file_range\fR\|(2) call.
e76b1da4
JA
754\fRstr\fP can currently be one or more of:
755.RS
756.TP
757.B wait_before
758SYNC_FILE_RANGE_WAIT_BEFORE
759.TP
760.B write
761SYNC_FILE_RANGE_WRITE
762.TP
763.B wait_after
764SYNC_FILE_RANGE_WRITE
765.TP
766.RE
767.P
768So if you do sync_file_range=wait_before,write:8, fio would use
769\fBSYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE\fP for every 8 writes.
ccc2b328 770Also see the \fBsync_file_range\fR\|(2) man page. This option is Linux specific.
e76b1da4 771.TP
d60e92d1 772.BI overwrite \fR=\fPbool
d1429b5c 773If writing, setup the file first and do overwrites. Default: false.
d60e92d1
AC
774.TP
775.BI end_fsync \fR=\fPbool
dbd11ead 776Sync file contents when a write stage has completed. Default: false.
d60e92d1
AC
777.TP
778.BI fsync_on_close \fR=\fPbool
779If true, sync file contents on close. This differs from \fBend_fsync\fR in that
d1429b5c 780it will happen on every close, not just at the end of the job. Default: false.
d60e92d1 781.TP
d60e92d1
AC
782.BI rwmixread \fR=\fPint
783Percentage of a mixed workload that should be reads. Default: 50.
784.TP
785.BI rwmixwrite \fR=\fPint
d1429b5c 786Percentage of a mixed workload that should be writes. If \fBrwmixread\fR and
c35dd7a6
JA
787\fBrwmixwrite\fR are given and do not sum to 100%, the latter of the two
788overrides the first. This may interfere with a given rate setting, if fio is
789asked to limit reads or writes to a certain rate. If that is the case, then
790the distribution may be skewed. Default: 50.
d60e92d1 791.TP
92d42d69
JA
792.BI random_distribution \fR=\fPstr:float
793By default, fio will use a completely uniform random distribution when asked
794to perform random IO. Sometimes it is useful to skew the distribution in
795specific ways, ensuring that some parts of the data is more hot than others.
796Fio includes the following distribution models:
797.RS
798.TP
799.B random
800Uniform random distribution
801.TP
802.B zipf
803Zipf distribution
804.TP
805.B pareto
806Pareto distribution
807.TP
808.RE
809.P
810When using a zipf or pareto distribution, an input value is also needed to
811define the access pattern. For zipf, this is the zipf theta. For pareto,
812it's the pareto power. Fio includes a test program, genzipf, that can be
813used visualize what the given input values will yield in terms of hit rates.
814If you wanted to use zipf with a theta of 1.2, you would use
815random_distribution=zipf:1.2 as the option. If a non-uniform model is used,
816fio will disable use of the random map.
817.TP
211c9b89
JA
818.BI percentage_random \fR=\fPint
819For a random workload, set how big a percentage should be random. This defaults
820to 100%, in which case the workload is fully random. It can be set from
821anywhere from 0 to 100. Setting it to 0 would make the workload fully
d9472271
JA
822sequential. It is possible to set different values for reads, writes, and
823trim. To do so, simply use a comma separated list. See \fBblocksize\fR.
211c9b89 824.TP
d60e92d1
AC
825.B norandommap
826Normally \fBfio\fR will cover every block of the file when doing random I/O. If
827this parameter is given, a new offset will be chosen without looking at past
828I/O history. This parameter is mutually exclusive with \fBverify\fR.
829.TP
744492c9 830.BI softrandommap \fR=\fPbool
3ce9dcaf
JA
831See \fBnorandommap\fR. If fio runs with the random block map enabled and it
832fails to allocate the map, if this option is set it will continue without a
833random block map. As coverage will not be as complete as with random maps, this
834option is disabled by default.
835.TP
e8b1961d
JA
836.BI random_generator \fR=\fPstr
837Fio supports the following engines for generating IO offsets for random IO:
838.RS
839.TP
840.B tausworthe
841Strong 2^88 cycle random number generator
842.TP
843.B lfsr
844Linear feedback shift register generator
845.TP
846.RE
847.P
848Tausworthe is a strong random number generator, but it requires tracking on the
849side if we want to ensure that blocks are only read or written once. LFSR
850guarantees that we never generate the same offset twice, and it's also less
851computationally expensive. It's not a true random generator, however, though
852for IO purposes it's typically good enough. LFSR only works with single block
853sizes, not with workloads that use multiple block sizes. If used with such a
854workload, fio may read or write some blocks multiple times.
855.TP
d60e92d1 856.BI nice \fR=\fPint
ccc2b328 857Run job with given nice value. See \fBnice\fR\|(2).
d60e92d1
AC
858.TP
859.BI prio \fR=\fPint
860Set I/O priority value of this job between 0 (highest) and 7 (lowest). See
ccc2b328 861\fBionice\fR\|(1).
d60e92d1
AC
862.TP
863.BI prioclass \fR=\fPint
ccc2b328 864Set I/O priority class. See \fBionice\fR\|(1).
d60e92d1
AC
865.TP
866.BI thinktime \fR=\fPint
867Stall job for given number of microseconds between issuing I/Os.
868.TP
869.BI thinktime_spin \fR=\fPint
870Pretend to spend CPU time for given number of microseconds, sleeping the rest
871of the time specified by \fBthinktime\fR. Only valid if \fBthinktime\fR is set.
872.TP
873.BI thinktime_blocks \fR=\fPint
4d01ece6
JA
874Only valid if thinktime is set - control how many blocks to issue, before
875waiting \fBthinktime\fR microseconds. If not set, defaults to 1 which will
876make fio wait \fBthinktime\fR microseconds after every block. This
877effectively makes any queue depth setting redundant, since no more than 1 IO
878will be queued before we have to complete it and do our thinktime. In other
879words, this setting effectively caps the queue depth if the latter is larger.
d60e92d1
AC
880Default: 1.
881.TP
882.BI rate \fR=\fPint
c35dd7a6
JA
883Cap bandwidth used by this job. The number is in bytes/sec, the normal postfix
884rules apply. You can use \fBrate\fR=500k to limit reads and writes to 500k each,
885or you can specify read and writes separately. Using \fBrate\fR=1m,500k would
886limit reads to 1MB/sec and writes to 500KB/sec. Capping only reads or writes
887can be done with \fBrate\fR=,500k or \fBrate\fR=500k,. The former will only
888limit writes (to 500KB/sec), the latter will only limit reads.
d60e92d1
AC
889.TP
890.BI ratemin \fR=\fPint
891Tell \fBfio\fR to do whatever it can to maintain at least the given bandwidth.
c35dd7a6
JA
892Failing to meet this requirement will cause the job to exit. The same format
893as \fBrate\fR is used for read vs write separation.
d60e92d1
AC
894.TP
895.BI rate_iops \fR=\fPint
c35dd7a6
JA
896Cap the bandwidth to this number of IOPS. Basically the same as rate, just
897specified independently of bandwidth. The same format as \fBrate\fR is used for
de8f6de9 898read vs write separation. If \fBblocksize\fR is a range, the smallest block
c35dd7a6 899size is used as the metric.
d60e92d1
AC
900.TP
901.BI rate_iops_min \fR=\fPint
c35dd7a6 902If this rate of I/O is not met, the job will exit. The same format as \fBrate\fR
de8f6de9 903is used for read vs write separation.
d60e92d1
AC
904.TP
905.BI ratecycle \fR=\fPint
906Average bandwidth for \fBrate\fR and \fBratemin\fR over this number of
907milliseconds. Default: 1000ms.
908.TP
3e260a46
JA
909.BI latency_target \fR=\fPint
910If set, fio will attempt to find the max performance point that the given
911workload will run at while maintaining a latency below this target. The
912values is given in microseconds. See \fBlatency_window\fR and
913\fBlatency_percentile\fR.
914.TP
915.BI latency_window \fR=\fPint
916Used with \fBlatency_target\fR to specify the sample window that the job
917is run at varying queue depths to test the performance. The value is given
918in microseconds.
919.TP
920.BI latency_percentile \fR=\fPfloat
921The percentage of IOs that must fall within the criteria specified by
922\fBlatency_target\fR and \fBlatency_window\fR. If not set, this defaults
923to 100.0, meaning that all IOs must be equal or below to the value set
924by \fBlatency_target\fR.
925.TP
15501535
JA
926.BI max_latency \fR=\fPint
927If set, fio will exit the job if it exceeds this maximum latency. It will exit
928with an ETIME error.
929.TP
d60e92d1
AC
930.BI cpumask \fR=\fPint
931Set CPU affinity for this job. \fIint\fR is a bitmask of allowed CPUs the job
932may run on. See \fBsched_setaffinity\fR\|(2).
933.TP
934.BI cpus_allowed \fR=\fPstr
935Same as \fBcpumask\fR, but allows a comma-delimited list of CPU numbers.
936.TP
c2acfbac
JA
937.BI cpus_allowed_policy \fR=\fPstr
938Set the policy of how fio distributes the CPUs specified by \fBcpus_allowed\fR
939or \fBcpumask\fR. Two policies are supported:
940.RS
941.RS
942.TP
943.B shared
944All jobs will share the CPU set specified.
945.TP
946.B split
947Each job will get a unique CPU from the CPU set.
948.RE
949.P
950\fBshared\fR is the default behaviour, if the option isn't specified. If
ada083cd
JA
951\fBsplit\fR is specified, then fio will assign one cpu per job. If not enough
952CPUs are given for the jobs listed, then fio will roundrobin the CPUs in
953the set.
c2acfbac
JA
954.RE
955.P
956.TP
d0b937ed 957.BI numa_cpu_nodes \fR=\fPstr
cecbfd47 958Set this job running on specified NUMA nodes' CPUs. The arguments allow
d0b937ed
YR
959comma delimited list of cpu numbers, A-B ranges, or 'all'.
960.TP
961.BI numa_mem_policy \fR=\fPstr
962Set this job's memory policy and corresponding NUMA nodes. Format of
cecbfd47 963the arguments:
d0b937ed
YR
964.RS
965.TP
966.B <mode>[:<nodelist>]
967.TP
968.B mode
969is one of the following memory policy:
970.TP
971.B default, prefer, bind, interleave, local
972.TP
973.RE
974For \fBdefault\fR and \fBlocal\fR memory policy, no \fBnodelist\fR is
975needed to be specified. For \fBprefer\fR, only one node is
976allowed. For \fBbind\fR and \fBinterleave\fR, \fBnodelist\fR allows
977comma delimited list of numbers, A-B ranges, or 'all'.
978.TP
23ed19b0
CE
979.BI startdelay \fR=\fPirange
980Delay start of job for the specified number of seconds. Supports all time
981suffixes to allow specification of hours, minutes, seconds and
bd66aa2c 982milliseconds - seconds are the default if a unit is omitted.
23ed19b0
CE
983Can be given as a range which causes each thread to choose randomly out of the
984range.
d60e92d1
AC
985.TP
986.BI runtime \fR=\fPint
987Terminate processing after the specified number of seconds.
988.TP
989.B time_based
990If given, run for the specified \fBruntime\fR duration even if the files are
991completely read or written. The same workload will be repeated as many times
992as \fBruntime\fR allows.
993.TP
901bb994
JA
994.BI ramp_time \fR=\fPint
995If set, fio will run the specified workload for this amount of time before
996logging any performance numbers. Useful for letting performance settle before
997logging results, thus minimizing the runtime required for stable results. Note
c35dd7a6
JA
998that the \fBramp_time\fR is considered lead in time for a job, thus it will
999increase the total runtime if a special timeout or runtime is specified.
901bb994 1000.TP
d60e92d1
AC
1001.BI invalidate \fR=\fPbool
1002Invalidate buffer-cache for the file prior to starting I/O. Default: true.
1003.TP
1004.BI sync \fR=\fPbool
1005Use synchronous I/O for buffered writes. For the majority of I/O engines,
d1429b5c 1006this means using O_SYNC. Default: false.
d60e92d1
AC
1007.TP
1008.BI iomem \fR=\fPstr "\fR,\fP mem" \fR=\fPstr
1009Allocation method for I/O unit buffer. Allowed values are:
1010.RS
1011.RS
1012.TP
1013.B malloc
ccc2b328 1014Allocate memory with \fBmalloc\fR\|(3).
d60e92d1
AC
1015.TP
1016.B shm
ccc2b328 1017Use shared memory buffers allocated through \fBshmget\fR\|(2).
d60e92d1
AC
1018.TP
1019.B shmhuge
1020Same as \fBshm\fR, but use huge pages as backing.
1021.TP
1022.B mmap
ccc2b328 1023Use \fBmmap\fR\|(2) for allocation. Uses anonymous memory unless a filename
d60e92d1
AC
1024is given after the option in the format `:\fIfile\fR'.
1025.TP
1026.B mmaphuge
1027Same as \fBmmap\fR, but use huge files as backing.
1028.RE
1029.P
1030The amount of memory allocated is the maximum allowed \fBblocksize\fR for the
1031job multiplied by \fBiodepth\fR. For \fBshmhuge\fR or \fBmmaphuge\fR to work,
1032the system must have free huge pages allocated. \fBmmaphuge\fR also needs to
2e266ba6
JA
1033have hugetlbfs mounted, and \fIfile\fR must point there. At least on Linux,
1034huge pages must be manually allocated. See \fB/proc/sys/vm/nr_hugehages\fR
1035and the documentation for that. Normally you just need to echo an appropriate
1036number, eg echoing 8 will ensure that the OS has 8 huge pages ready for
1037use.
d60e92d1
AC
1038.RE
1039.TP
d392365e 1040.BI iomem_align \fR=\fPint "\fR,\fP mem_align" \fR=\fPint
cecbfd47 1041This indicates the memory alignment of the IO memory buffers. Note that the
d529ee19
JA
1042given alignment is applied to the first IO unit buffer, if using \fBiodepth\fR
1043the alignment of the following buffers are given by the \fBbs\fR used. In
1044other words, if using a \fBbs\fR that is a multiple of the page sized in the
1045system, all buffers will be aligned to this value. If using a \fBbs\fR that
1046is not page aligned, the alignment of subsequent IO memory buffers is the
1047sum of the \fBiomem_align\fR and \fBbs\fR used.
1048.TP
f7fa2653 1049.BI hugepage\-size \fR=\fPint
d60e92d1 1050Defines the size of a huge page. Must be at least equal to the system setting.
b22989b9 1051Should be a multiple of 1MB. Default: 4MB.
d60e92d1
AC
1052.TP
1053.B exitall
1054Terminate all jobs when one finishes. Default: wait for each job to finish.
1055.TP
1056.BI bwavgtime \fR=\fPint
1057Average bandwidth calculations over the given time in milliseconds. Default:
1058500ms.
1059.TP
c8eeb9df
JA
1060.BI iopsavgtime \fR=\fPint
1061Average IOPS calculations over the given time in milliseconds. Default:
1062500ms.
1063.TP
d60e92d1 1064.BI create_serialize \fR=\fPbool
d1429b5c 1065If true, serialize file creation for the jobs. Default: true.
d60e92d1
AC
1066.TP
1067.BI create_fsync \fR=\fPbool
ccc2b328 1068\fBfsync\fR\|(2) data file after creation. Default: true.
d60e92d1 1069.TP
6b7f6851
JA
1070.BI create_on_open \fR=\fPbool
1071If true, the files are not created until they are opened for IO by the job.
1072.TP
25460cf6
JA
1073.BI create_only \fR=\fPbool
1074If true, fio will only run the setup phase of the job. If files need to be
1075laid out or updated on disk, only that will be done. The actual job contents
1076are not executed.
1077.TP
e9f48479
JA
1078.BI pre_read \fR=\fPbool
1079If this is given, files will be pre-read into memory before starting the given
1080IO operation. This will also clear the \fR \fBinvalidate\fR flag, since it is
9c0d2241
JA
1081pointless to pre-read and then drop the cache. This will only work for IO
1082engines that are seekable, since they allow you to read the same data
1083multiple times. Thus it will not work on eg network or splice IO.
e9f48479 1084.TP
d60e92d1
AC
1085.BI unlink \fR=\fPbool
1086Unlink job files when done. Default: false.
1087.TP
1088.BI loops \fR=\fPint
1089Specifies the number of iterations (runs of the same workload) of this job.
1090Default: 1.
1091.TP
5e4c7118
JA
1092.BI verify_only \fR=\fPbool
1093Do not perform the specified workload, only verify data still matches previous
1094invocation of this workload. This option allows one to check data multiple
1095times at a later date without overwriting it. This option makes sense only for
1096workloads that write data, and does not support workloads with the
1097\fBtime_based\fR option set.
1098.TP
d60e92d1
AC
1099.BI do_verify \fR=\fPbool
1100Run the verify phase after a write phase. Only valid if \fBverify\fR is set.
1101Default: true.
1102.TP
1103.BI verify \fR=\fPstr
1104Method of verifying file contents after each iteration of the job. Allowed
1105values are:
1106.RS
1107.RS
1108.TP
844ea602 1109.B md5 crc16 crc32 crc32c crc32c-intel crc64 crc7 sha256 sha512 sha1 xxhash
0539d758
JA
1110Store appropriate checksum in the header of each block. crc32c-intel is
1111hardware accelerated SSE4.2 driven, falls back to regular crc32c if
1112not supported by the system.
d60e92d1
AC
1113.TP
1114.B meta
1115Write extra information about each I/O (timestamp, block number, etc.). The
996093bb 1116block number is verified. See \fBverify_pattern\fR as well.
d60e92d1
AC
1117.TP
1118.B null
1119Pretend to verify. Used for testing internals.
1120.RE
b892dc08
JA
1121
1122This option can be used for repeated burn-in tests of a system to make sure
1123that the written data is also correctly read back. If the data direction given
1124is a read or random read, fio will assume that it should verify a previously
1125written file. If the data direction includes any form of write, the verify will
1126be of the newly written data.
d60e92d1
AC
1127.RE
1128.TP
5c9323fb 1129.BI verifysort \fR=\fPbool
d60e92d1
AC
1130If true, written verify blocks are sorted if \fBfio\fR deems it to be faster to
1131read them back in a sorted manner. Default: true.
1132.TP
fa769d44
SW
1133.BI verifysort_nr \fR=\fPint
1134Pre-load and sort verify blocks for a read workload.
1135.TP
f7fa2653 1136.BI verify_offset \fR=\fPint
d60e92d1 1137Swap the verification header with data somewhere else in the block before
d1429b5c 1138writing. It is swapped back before verifying.
d60e92d1 1139.TP
f7fa2653 1140.BI verify_interval \fR=\fPint
d60e92d1
AC
1141Write the verification header for this number of bytes, which should divide
1142\fBblocksize\fR. Default: \fBblocksize\fR.
1143.TP
996093bb
JA
1144.BI verify_pattern \fR=\fPstr
1145If set, fio will fill the io buffers with this pattern. Fio defaults to filling
1146with totally random bytes, but sometimes it's interesting to fill with a known
1147pattern for io verification purposes. Depending on the width of the pattern,
1148fio will fill 1/2/3/4 bytes of the buffer at the time(it can be either a
1149decimal or a hex number). The verify_pattern if larger than a 32-bit quantity
1150has to be a hex number that starts with either "0x" or "0X". Use with
1151\fBverify\fP=meta.
1152.TP
d60e92d1
AC
1153.BI verify_fatal \fR=\fPbool
1154If true, exit the job on the first observed verification failure. Default:
1155false.
1156.TP
b463e936
JA
1157.BI verify_dump \fR=\fPbool
1158If set, dump the contents of both the original data block and the data block we
1159read off disk to files. This allows later analysis to inspect just what kind of
ef71e317 1160data corruption occurred. Off by default.
b463e936 1161.TP
e8462bd8
JA
1162.BI verify_async \fR=\fPint
1163Fio will normally verify IO inline from the submitting thread. This option
1164takes an integer describing how many async offload threads to create for IO
1165verification instead, causing fio to offload the duty of verifying IO contents
c85c324c
JA
1166to one or more separate threads. If using this offload option, even sync IO
1167engines can benefit from using an \fBiodepth\fR setting higher than 1, as it
1168allows them to have IO in flight while verifies are running.
e8462bd8
JA
1169.TP
1170.BI verify_async_cpus \fR=\fPstr
1171Tell fio to set the given CPU affinity on the async IO verification threads.
1172See \fBcpus_allowed\fP for the format used.
1173.TP
6f87418f
JA
1174.BI verify_backlog \fR=\fPint
1175Fio will normally verify the written contents of a job that utilizes verify
1176once that job has completed. In other words, everything is written then
1177everything is read back and verified. You may want to verify continually
1178instead for a variety of reasons. Fio stores the meta data associated with an
1179IO block in memory, so for large verify workloads, quite a bit of memory would
092f707f
DN
1180be used up holding this meta data. If this option is enabled, fio will write
1181only N blocks before verifying these blocks.
6f87418f
JA
1182.TP
1183.BI verify_backlog_batch \fR=\fPint
1184Control how many blocks fio will verify if verify_backlog is set. If not set,
1185will default to the value of \fBverify_backlog\fR (meaning the entire queue is
092f707f
DN
1186read back and verified). If \fBverify_backlog_batch\fR is less than
1187\fBverify_backlog\fR then not all blocks will be verified, if
1188\fBverify_backlog_batch\fR is larger than \fBverify_backlog\fR, some blocks
1189will be verified more than once.
6f87418f 1190.TP
fa769d44
SW
1191.BI trim_percentage \fR=\fPint
1192Number of verify blocks to discard/trim.
1193.TP
1194.BI trim_verify_zero \fR=\fPbool
1195Verify that trim/discarded blocks are returned as zeroes.
1196.TP
1197.BI trim_backlog \fR=\fPint
1198Trim after this number of blocks are written.
1199.TP
1200.BI trim_backlog_batch \fR=\fPint
1201Trim this number of IO blocks.
1202.TP
1203.BI experimental_verify \fR=\fPbool
1204Enable experimental verification.
1205.TP
ca09be4b
JA
1206.BI verify_state_save \fR=\fPbool
1207When a job exits during the write phase of a verify workload, save its
1208current state. This allows fio to replay up until that point, if the
1209verify state is loaded for the verify read phase.
1210.TP
1211.BI verify_state_load \fR=\fPbool
1212If a verify termination trigger was used, fio stores the current write
1213state of each thread. This can be used at verification time so that fio
1214knows how far it should verify. Without this information, fio will run
1215a full verification pass, according to the settings in the job file used.
1216.TP
d392365e 1217.B stonewall "\fR,\fP wait_for_previous"
5982a925 1218Wait for preceding jobs in the job file to exit before starting this one.
d60e92d1
AC
1219\fBstonewall\fR implies \fBnew_group\fR.
1220.TP
1221.B new_group
1222Start a new reporting group. If not given, all jobs in a file will be part
1223of the same reporting group, unless separated by a stonewall.
1224.TP
1225.BI numjobs \fR=\fPint
1226Number of clones (processes/threads performing the same workload) of this job.
1227Default: 1.
1228.TP
1229.B group_reporting
1230If set, display per-group reports instead of per-job when \fBnumjobs\fR is
1231specified.
1232.TP
1233.B thread
1234Use threads created with \fBpthread_create\fR\|(3) instead of processes created
1235with \fBfork\fR\|(2).
1236.TP
f7fa2653 1237.BI zonesize \fR=\fPint
d60e92d1
AC
1238Divide file into zones of the specified size in bytes. See \fBzoneskip\fR.
1239.TP
fa769d44
SW
1240.BI zonerange \fR=\fPint
1241Give size of an IO zone. See \fBzoneskip\fR.
1242.TP
f7fa2653 1243.BI zoneskip \fR=\fPint
d1429b5c 1244Skip the specified number of bytes when \fBzonesize\fR bytes of data have been
d60e92d1
AC
1245read.
1246.TP
1247.BI write_iolog \fR=\fPstr
5b42a488
SH
1248Write the issued I/O patterns to the specified file. Specify a separate file
1249for each job, otherwise the iologs will be interspersed and the file may be
1250corrupt.
d60e92d1
AC
1251.TP
1252.BI read_iolog \fR=\fPstr
1253Replay the I/O patterns contained in the specified file generated by
1254\fBwrite_iolog\fR, or may be a \fBblktrace\fR binary file.
1255.TP
64bbb865
DN
1256.BI replay_no_stall \fR=\fPint
1257While replaying I/O patterns using \fBread_iolog\fR the default behavior
1258attempts to respect timing information between I/Os. Enabling
1259\fBreplay_no_stall\fR causes I/Os to be replayed as fast as possible while
1260still respecting ordering.
1261.TP
d1c46c04
DN
1262.BI replay_redirect \fR=\fPstr
1263While replaying I/O patterns using \fBread_iolog\fR the default behavior
1264is to replay the IOPS onto the major/minor device that each IOP was recorded
1265from. Setting \fBreplay_redirect\fR causes all IOPS to be replayed onto the
1266single specified device regardless of the device it was recorded from.
1267.TP
0c63576e
JA
1268.BI replay_align \fR=\fPint
1269Force alignment of IO offsets and lengths in a trace to this power of 2 value.
1270.TP
1271.BI replay_scale \fR=\fPint
1272Scale sector offsets down by this factor when replaying traces.
1273.TP
836bad52 1274.BI write_bw_log \fR=\fPstr
901bb994
JA
1275If given, write a bandwidth log of the jobs in this job file. Can be used to
1276store data of the bandwidth of the jobs in their lifetime. The included
1277fio_generate_plots script uses gnuplot to turn these text files into nice
26b26fca 1278graphs. See \fBwrite_lat_log\fR for behaviour of given filename. For this
8ad3b3dd
JA
1279option, the postfix is _bw.x.log, where x is the index of the job (1..N,
1280where N is the number of jobs)
d60e92d1 1281.TP
836bad52 1282.BI write_lat_log \fR=\fPstr
901bb994 1283Same as \fBwrite_bw_log\fR, but writes I/O completion latencies. If no
8ad3b3dd
JA
1284filename is given with this option, the default filename of
1285"jobname_type.x.log" is used, where x is the index of the job (1..N, where
1286N is the number of jobs). Even if the filename is given, fio will still
1287append the type of log.
901bb994 1288.TP
c8eeb9df
JA
1289.BI write_iops_log \fR=\fPstr
1290Same as \fBwrite_bw_log\fR, but writes IOPS. If no filename is given with this
8ad3b3dd
JA
1291option, the default filename of "jobname_type.x.log" is used, where x is the
1292index of the job (1..N, where N is the number of jobs). Even if the filename
1293is given, fio will still append the type of log.
c8eeb9df 1294.TP
b8bc8cba
JA
1295.BI log_avg_msec \fR=\fPint
1296By default, fio will log an entry in the iops, latency, or bw log for every
1297IO that completes. When writing to the disk log, that can quickly grow to a
1298very large size. Setting this option makes fio average the each log entry
1299over the specified period of time, reducing the resolution of the log.
1300Defaults to 0.
1301.TP
ae588852
JA
1302.BI log_offset \fR=\fPbool
1303If this is set, the iolog options will include the byte offset for the IO
1304entry as well as the other data values.
1305.TP
aee2ab67
JA
1306.BI log_compression \fR=\fPint
1307If this is set, fio will compress the IO logs as it goes, to keep the memory
1308footprint lower. When a log reaches the specified size, that chunk is removed
1309and compressed in the background. Given that IO logs are fairly highly
1310compressible, this yields a nice memory savings for longer runs. The downside
1311is that the compression will consume some background CPU cycles, so it may
1312impact the run. This, however, is also true if the logging ends up consuming
1313most of the system memory. So pick your poison. The IO logs are saved
1314normally at the end of a run, by decompressing the chunks and storing them
1315in the specified log file. This feature depends on the availability of zlib.
1316.TP
b26317c9
JA
1317.BI log_store_compressed \fR=\fPbool
1318If set, and \fBlog\fR_compression is also set, fio will store the log files in
1319a compressed format. They can be decompressed with fio, using the
1320\fB\-\-inflate-log\fR command line parameter. The files will be stored with a
1321\fB\.fz\fR suffix.
1322.TP
66347cfa
DE
1323.BI block_error_percentiles \fR=\fPbool
1324If set, record errors in trim block-sized units from writes and trims and output
1325a histogram of how many trims it took to get to errors, and what kind of error
1326was encountered.
1327.TP
836bad52 1328.BI disable_lat \fR=\fPbool
02af0988 1329Disable measurements of total latency numbers. Useful only for cutting
ccc2b328 1330back the number of calls to \fBgettimeofday\fR\|(2), as that does impact performance at
901bb994
JA
1331really high IOPS rates. Note that to really get rid of a large amount of these
1332calls, this option must be used with disable_slat and disable_bw as well.
1333.TP
836bad52 1334.BI disable_clat \fR=\fPbool
c95f9daf 1335Disable measurements of completion latency numbers. See \fBdisable_lat\fR.
02af0988 1336.TP
836bad52 1337.BI disable_slat \fR=\fPbool
02af0988 1338Disable measurements of submission latency numbers. See \fBdisable_lat\fR.
901bb994 1339.TP
836bad52 1340.BI disable_bw_measurement \fR=\fPbool
02af0988 1341Disable measurements of throughput/bandwidth numbers. See \fBdisable_lat\fR.
d60e92d1 1342.TP
f7fa2653 1343.BI lockmem \fR=\fPint
d60e92d1 1344Pin the specified amount of memory with \fBmlock\fR\|(2). Can be used to
81c6b6cd 1345simulate a smaller amount of memory. The amount specified is per worker.
d60e92d1
AC
1346.TP
1347.BI exec_prerun \fR=\fPstr
1348Before running the job, execute the specified command with \fBsystem\fR\|(3).
ce486495
EV
1349.RS
1350Output is redirected in a file called \fBjobname.prerun.txt\fR
1351.RE
d60e92d1
AC
1352.TP
1353.BI exec_postrun \fR=\fPstr
1354Same as \fBexec_prerun\fR, but the command is executed after the job completes.
ce486495
EV
1355.RS
1356Output is redirected in a file called \fBjobname.postrun.txt\fR
1357.RE
d60e92d1
AC
1358.TP
1359.BI ioscheduler \fR=\fPstr
1360Attempt to switch the device hosting the file to the specified I/O scheduler.
1361.TP
d60e92d1 1362.BI disk_util \fR=\fPbool
d1429b5c 1363Generate disk utilization statistics if the platform supports it. Default: true.
901bb994 1364.TP
23893646
JA
1365.BI clocksource \fR=\fPstr
1366Use the given clocksource as the base of timing. The supported options are:
1367.RS
1368.TP
1369.B gettimeofday
ccc2b328 1370\fBgettimeofday\fR\|(2)
23893646
JA
1371.TP
1372.B clock_gettime
ccc2b328 1373\fBclock_gettime\fR\|(2)
23893646
JA
1374.TP
1375.B cpu
1376Internal CPU clock source
1377.TP
1378.RE
1379.P
1380\fBcpu\fR is the preferred clocksource if it is reliable, as it is very fast
1381(and fio is heavy on time calls). Fio will automatically use this clocksource
1382if it's supported and considered reliable on the system it is running on,
1383unless another clocksource is specifically set. For x86/x86-64 CPUs, this
1384means supporting TSC Invariant.
1385.TP
901bb994 1386.BI gtod_reduce \fR=\fPbool
ccc2b328 1387Enable all of the \fBgettimeofday\fR\|(2) reducing options (disable_clat, disable_slat,
901bb994 1388disable_bw) plus reduce precision of the timeout somewhat to really shrink the
ccc2b328 1389\fBgettimeofday\fR\|(2) call count. With this option enabled, we only do about 0.4% of
901bb994
JA
1390the gtod() calls we would have done if all time keeping was enabled.
1391.TP
1392.BI gtod_cpu \fR=\fPint
1393Sometimes it's cheaper to dedicate a single thread of execution to just getting
1394the current time. Fio (and databases, for instance) are very intensive on
ccc2b328 1395\fBgettimeofday\fR\|(2) calls. With this option, you can set one CPU aside for doing
901bb994
JA
1396nothing but logging current time to a shared memory location. Then the other
1397threads/processes that run IO workloads need only copy that segment, instead of
ccc2b328 1398entering the kernel with a \fBgettimeofday\fR\|(2) call. The CPU set aside for doing
901bb994
JA
1399these time calls will be excluded from other uses. Fio will manually clear it
1400from the CPU mask of other jobs.
f2bba182 1401.TP
8b28bd41
DM
1402.BI ignore_error \fR=\fPstr
1403Sometimes you want to ignore some errors during test in that case you can specify
1404error list for each error type.
1405.br
1406ignore_error=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST
1407.br
1408errors for given error type is separated with ':'.
1409Error may be symbol ('ENOSPC', 'ENOMEM') or an integer.
1410.br
1411Example: ignore_error=EAGAIN,ENOSPC:122 .
1412.br
1413This option will ignore EAGAIN from READ, and ENOSPC and 122(EDQUOT) from WRITE.
1414.TP
1415.BI error_dump \fR=\fPbool
1416If set dump every error even if it is non fatal, true by default. If disabled
1417only fatal error will be dumped
1418.TP
fa769d44
SW
1419.BI profile \fR=\fPstr
1420Select a specific builtin performance test.
1421.TP
a696fa2a
JA
1422.BI cgroup \fR=\fPstr
1423Add job to this control group. If it doesn't exist, it will be created.
6adb38a1
JA
1424The system must have a mounted cgroup blkio mount point for this to work. If
1425your system doesn't have it mounted, you can do so with:
1426
5982a925 1427# mount \-t cgroup \-o blkio none /cgroup
a696fa2a
JA
1428.TP
1429.BI cgroup_weight \fR=\fPint
1430Set the weight of the cgroup to this value. See the documentation that comes
1431with the kernel, allowed values are in the range of 100..1000.
e0b0d892 1432.TP
7de87099
VG
1433.BI cgroup_nodelete \fR=\fPbool
1434Normally fio will delete the cgroups it has created after the job completion.
1435To override this behavior and to leave cgroups around after the job completion,
1436set cgroup_nodelete=1. This can be useful if one wants to inspect various
1437cgroup files after job completion. Default: false
1438.TP
e0b0d892
JA
1439.BI uid \fR=\fPint
1440Instead of running as the invoking user, set the user ID to this value before
1441the thread/process does any work.
1442.TP
1443.BI gid \fR=\fPint
1444Set group ID, see \fBuid\fR.
83349190 1445.TP
fa769d44
SW
1446.BI unit_base \fR=\fPint
1447Base unit for reporting. Allowed values are:
1448.RS
1449.TP
1450.B 0
1451Use auto-detection (default).
1452.TP
1453.B 8
1454Byte based.
1455.TP
1456.B 1
1457Bit based.
1458.RE
1459.P
1460.TP
9e684a49
DE
1461.BI flow_id \fR=\fPint
1462The ID of the flow. If not specified, it defaults to being a global flow. See
1463\fBflow\fR.
1464.TP
1465.BI flow \fR=\fPint
1466Weight in token-based flow control. If this value is used, then there is a
1467\fBflow counter\fR which is used to regulate the proportion of activity between
1468two or more jobs. fio attempts to keep this flow counter near zero. The
1469\fBflow\fR parameter stands for how much should be added or subtracted to the
1470flow counter on each iteration of the main I/O loop. That is, if one job has
1471\fBflow=8\fR and another job has \fBflow=-1\fR, then there will be a roughly
14721:8 ratio in how much one runs vs the other.
1473.TP
1474.BI flow_watermark \fR=\fPint
1475The maximum value that the absolute value of the flow counter is allowed to
1476reach before the job must wait for a lower value of the counter.
1477.TP
1478.BI flow_sleep \fR=\fPint
1479The period of time, in microseconds, to wait after the flow watermark has been
1480exceeded before retrying operations
1481.TP
83349190
YH
1482.BI clat_percentiles \fR=\fPbool
1483Enable the reporting of percentiles of completion latencies.
1484.TP
1485.BI percentile_list \fR=\fPfloat_list
66347cfa
DE
1486Overwrite the default list of percentiles for completion latencies and the
1487block error histogram. Each number is a floating number in the range (0,100],
1488and the maximum length of the list is 20. Use ':' to separate the
3eb07285 1489numbers. For example, \-\-percentile_list=99.5:99.9 will cause fio to
83349190
YH
1490report the values of completion latency below which 99.5% and 99.9% of
1491the observed latencies fell, respectively.
de890a1e
SL
1492.SS "Ioengine Parameters List"
1493Some parameters are only valid when a specific ioengine is in use. These are
1494used identically to normal parameters, with the caveat that when used on the
cf145d90 1495command line, they must come after the ioengine.
de890a1e 1496.TP
e4585935
JA
1497.BI (cpu)cpuload \fR=\fPint
1498Attempt to use the specified percentage of CPU cycles.
1499.TP
1500.BI (cpu)cpuchunks \fR=\fPint
1501Split the load into cycles of the given time. In microseconds.
1502.TP
046395d7
JA
1503.BI (cpu)exit_on_io_done \fR=\fPbool
1504Detect when IO threads are done, then exit.
1505.TP
de890a1e
SL
1506.BI (libaio)userspace_reap
1507Normally, with the libaio engine in use, fio will use
1508the io_getevents system call to reap newly returned events.
1509With this flag turned on, the AIO ring will be read directly
1510from user-space to reap events. The reaping mode is only
1511enabled when polling for a minimum of 0 events (eg when
1512iodepth_batch_complete=0).
1513.TP
1514.BI (net,netsplice)hostname \fR=\fPstr
1515The host name or IP address to use for TCP or UDP based IO.
1516If the job is a TCP listener or UDP reader, the hostname is not
b511c9aa 1517used and must be omitted unless it is a valid UDP multicast address.
de890a1e
SL
1518.TP
1519.BI (net,netsplice)port \fR=\fPint
6315af9d
JA
1520The TCP or UDP port to bind to or connect to. If this is used with
1521\fBnumjobs\fR to spawn multiple instances of the same job type, then
1522this will be the starting port number since fio will use a range of ports.
de890a1e 1523.TP
b93b6a2e
SB
1524.BI (net,netsplice)interface \fR=\fPstr
1525The IP address of the network interface used to send or receive UDP multicast
1526packets.
1527.TP
d3a623de
SB
1528.BI (net,netsplice)ttl \fR=\fPint
1529Time-to-live value for outgoing UDP multicast packets. Default: 1
1530.TP
1d360ffb
JA
1531.BI (net,netsplice)nodelay \fR=\fPbool
1532Set TCP_NODELAY on TCP connections.
1533.TP
de890a1e
SL
1534.BI (net,netsplice)protocol \fR=\fPstr "\fR,\fP proto" \fR=\fPstr
1535The network protocol to use. Accepted values are:
1536.RS
1537.RS
1538.TP
1539.B tcp
1540Transmission control protocol
1541.TP
49ccb8c1
JA
1542.B tcpv6
1543Transmission control protocol V6
1544.TP
de890a1e 1545.B udp
f5cc3d0e 1546User datagram protocol
de890a1e 1547.TP
49ccb8c1
JA
1548.B udpv6
1549User datagram protocol V6
1550.TP
de890a1e
SL
1551.B unix
1552UNIX domain socket
1553.RE
1554.P
1555When the protocol is TCP or UDP, the port must also be given,
1556as well as the hostname if the job is a TCP listener or UDP
1557reader. For unix sockets, the normal filename option should be
1558used and the port is invalid.
1559.RE
1560.TP
1561.BI (net,netsplice)listen
1562For TCP network connections, tell fio to listen for incoming
1563connections rather than initiating an outgoing connection. The
1564hostname must be omitted if this option is used.
d54fce84 1565.TP
7aeb1e94 1566.BI (net, pingpong) \fR=\fPbool
cecbfd47 1567Normally a network writer will just continue writing data, and a network reader
cf145d90 1568will just consume packets. If pingpong=1 is set, a writer will send its normal
7aeb1e94
JA
1569payload to the reader, then wait for the reader to send the same payload back.
1570This allows fio to measure network latencies. The submission and completion
1571latencies then measure local time spent sending or receiving, and the
1572completion latency measures how long it took for the other end to receive and
b511c9aa
SB
1573send back. For UDP multicast traffic pingpong=1 should only be set for a single
1574reader when multiple readers are listening to the same address.
7aeb1e94 1575.TP
1008602c
JA
1576.BI (net, window_size) \fR=\fPint
1577Set the desired socket buffer size for the connection.
1578.TP
e5f34d95
JA
1579.BI (net, mss) \fR=\fPint
1580Set the TCP maximum segment size (TCP_MAXSEG).
1581.TP
d54fce84
DM
1582.BI (e4defrag,donorname) \fR=\fPstr
1583File will be used as a block donor (swap extents between files)
1584.TP
1585.BI (e4defrag,inplace) \fR=\fPint
1586Configure donor file block allocation strategy
1587.RS
1588.BI 0(default) :
1589Preallocate donor's file on init
1590.TP
1591.BI 1:
cecbfd47 1592allocate space immediately inside defragment event, and free right after event
d54fce84 1593.RE
0d978694
DAG
1594.TP
1595.BI (rbd)rbdname \fR=\fPstr
1596Specifies the name of the RBD.
1597.TP
1598.BI (rbd)pool \fR=\fPstr
1599Specifies the name of the Ceph pool containing the RBD.
1600.TP
1601.BI (rbd)clientname \fR=\fPstr
1602Specifies the username (without the 'client.' prefix) used to access the Ceph cluster.
65fa28ca
DE
1603.TP
1604.BI (mtd)skipbad \fR=\fPbool
1605Skip operations against known bad blocks.
d60e92d1 1606.SH OUTPUT
d1429b5c
AC
1607While running, \fBfio\fR will display the status of the created jobs. For
1608example:
d60e92d1 1609.RS
d1429b5c 1610.P
d60e92d1
AC
1611Threads: 1: [_r] [24.8% done] [ 13509/ 8334 kb/s] [eta 00h:01m:31s]
1612.RE
1613.P
d1429b5c
AC
1614The characters in the first set of brackets denote the current status of each
1615threads. The possible values are:
1616.P
1617.PD 0
d60e92d1
AC
1618.RS
1619.TP
1620.B P
1621Setup but not started.
1622.TP
1623.B C
1624Thread created.
1625.TP
1626.B I
1627Initialized, waiting.
1628.TP
1629.B R
1630Running, doing sequential reads.
1631.TP
1632.B r
1633Running, doing random reads.
1634.TP
1635.B W
1636Running, doing sequential writes.
1637.TP
1638.B w
1639Running, doing random writes.
1640.TP
1641.B M
1642Running, doing mixed sequential reads/writes.
1643.TP
1644.B m
1645Running, doing mixed random reads/writes.
1646.TP
1647.B F
1648Running, currently waiting for \fBfsync\fR\|(2).
1649.TP
1650.B V
1651Running, verifying written data.
1652.TP
1653.B E
1654Exited, not reaped by main thread.
1655.TP
1656.B \-
1657Exited, thread reaped.
1658.RE
d1429b5c 1659.PD
d60e92d1
AC
1660.P
1661The second set of brackets shows the estimated completion percentage of
1662the current group. The third set shows the read and write I/O rate,
1663respectively. Finally, the estimated run time of the job is displayed.
1664.P
1665When \fBfio\fR completes (or is interrupted by Ctrl-C), it will show data
1666for each thread, each group of threads, and each disk, in that order.
1667.P
1668Per-thread statistics first show the threads client number, group-id, and
1669error code. The remaining figures are as follows:
1670.RS
d60e92d1
AC
1671.TP
1672.B io
1673Number of megabytes of I/O performed.
1674.TP
1675.B bw
1676Average data rate (bandwidth).
1677.TP
1678.B runt
1679Threads run time.
1680.TP
1681.B slat
1682Submission latency minimum, maximum, average and standard deviation. This is
1683the time it took to submit the I/O.
1684.TP
1685.B clat
1686Completion latency minimum, maximum, average and standard deviation. This
1687is the time between submission and completion.
1688.TP
1689.B bw
1690Bandwidth minimum, maximum, percentage of aggregate bandwidth received, average
1691and standard deviation.
1692.TP
1693.B cpu
1694CPU usage statistics. Includes user and system time, number of context switches
1695this thread went through and number of major and minor page faults.
1696.TP
1697.B IO depths
1698Distribution of I/O depths. Each depth includes everything less than (or equal)
1699to it, but greater than the previous depth.
1700.TP
1701.B IO issued
1702Number of read/write requests issued, and number of short read/write requests.
1703.TP
1704.B IO latencies
1705Distribution of I/O completion latencies. The numbers follow the same pattern
1706as \fBIO depths\fR.
1707.RE
d60e92d1
AC
1708.P
1709The group statistics show:
d1429b5c 1710.PD 0
d60e92d1
AC
1711.RS
1712.TP
1713.B io
1714Number of megabytes I/O performed.
1715.TP
1716.B aggrb
1717Aggregate bandwidth of threads in the group.
1718.TP
1719.B minb
1720Minimum average bandwidth a thread saw.
1721.TP
1722.B maxb
1723Maximum average bandwidth a thread saw.
1724.TP
1725.B mint
d1429b5c 1726Shortest runtime of threads in the group.
d60e92d1
AC
1727.TP
1728.B maxt
1729Longest runtime of threads in the group.
1730.RE
d1429b5c 1731.PD
d60e92d1
AC
1732.P
1733Finally, disk statistics are printed with reads first:
d1429b5c 1734.PD 0
d60e92d1
AC
1735.RS
1736.TP
1737.B ios
1738Number of I/Os performed by all groups.
1739.TP
1740.B merge
1741Number of merges in the I/O scheduler.
1742.TP
1743.B ticks
1744Number of ticks we kept the disk busy.
1745.TP
1746.B io_queue
1747Total time spent in the disk queue.
1748.TP
1749.B util
1750Disk utilization.
1751.RE
d1429b5c 1752.PD
8423bd11
JA
1753.P
1754It is also possible to get fio to dump the current output while it is
1755running, without terminating the job. To do that, send fio the \fBUSR1\fR
1756signal.
d60e92d1 1757.SH TERSE OUTPUT
2b8c71b0
CE
1758If the \fB\-\-minimal\fR / \fB\-\-append-terse\fR options are given, the
1759results will be printed/appended in a semicolon-delimited format suitable for
1760scripted use.
1761A job description (if provided) follows on a new line. Note that the first
525c2bfa
JA
1762number in the line is the version number. If the output has to be changed
1763for some reason, this number will be incremented by 1 to signify that
1764change. The fields are:
d60e92d1
AC
1765.P
1766.RS
5e726d0a 1767.B terse version, fio version, jobname, groupid, error
d60e92d1
AC
1768.P
1769Read status:
1770.RS
312b4af2 1771.B Total I/O \fR(KB)\fP, bandwidth \fR(KB/s)\fP, IOPS, runtime \fR(ms)\fP
d60e92d1
AC
1772.P
1773Submission latency:
1774.RS
1775.B min, max, mean, standard deviation
1776.RE
1777Completion latency:
1778.RS
1779.B min, max, mean, standard deviation
1780.RE
1db92cb6
JA
1781Completion latency percentiles (20 fields):
1782.RS
1783.B Xth percentile=usec
1784.RE
525c2bfa
JA
1785Total latency:
1786.RS
1787.B min, max, mean, standard deviation
1788.RE
d60e92d1
AC
1789Bandwidth:
1790.RS
1791.B min, max, aggregate percentage of total, mean, standard deviation
1792.RE
1793.RE
1794.P
1795Write status:
1796.RS
312b4af2 1797.B Total I/O \fR(KB)\fP, bandwidth \fR(KB/s)\fP, IOPS, runtime \fR(ms)\fP
d60e92d1
AC
1798.P
1799Submission latency:
1800.RS
1801.B min, max, mean, standard deviation
1802.RE
1803Completion latency:
1804.RS
1805.B min, max, mean, standard deviation
1806.RE
1db92cb6
JA
1807Completion latency percentiles (20 fields):
1808.RS
1809.B Xth percentile=usec
1810.RE
525c2bfa
JA
1811Total latency:
1812.RS
1813.B min, max, mean, standard deviation
1814.RE
d60e92d1
AC
1815Bandwidth:
1816.RS
1817.B min, max, aggregate percentage of total, mean, standard deviation
1818.RE
1819.RE
1820.P
d1429b5c 1821CPU usage:
d60e92d1 1822.RS
bd2626f0 1823.B user, system, context switches, major page faults, minor page faults
d60e92d1
AC
1824.RE
1825.P
1826IO depth distribution:
1827.RS
1828.B <=1, 2, 4, 8, 16, 32, >=64
1829.RE
1830.P
562c2d2f 1831IO latency distribution:
d60e92d1 1832.RS
562c2d2f
DN
1833Microseconds:
1834.RS
1835.B <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000
1836.RE
1837Milliseconds:
1838.RS
1839.B <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000
1840.RE
1841.RE
1842.P
f2f788dd
JA
1843Disk utilization (1 for each disk used):
1844.RS
1845.B name, read ios, write ios, read merges, write merges, read ticks, write ticks, read in-queue time, write in-queue time, disk utilization percentage
1846.RE
1847.P
5982a925 1848Error Info (dependent on continue_on_error, default off):
562c2d2f
DN
1849.RS
1850.B total # errors, first error code
d60e92d1
AC
1851.RE
1852.P
562c2d2f 1853.B text description (if provided in config - appears on newline)
d60e92d1 1854.RE
49da1240
JA
1855.SH CLIENT / SERVER
1856Normally you would run fio as a stand-alone application on the machine
1857where the IO workload should be generated. However, it is also possible to
1858run the frontend and backend of fio separately. This makes it possible to
1859have a fio server running on the machine(s) where the IO workload should
1860be running, while controlling it from another machine.
1861
1862To start the server, you would do:
1863
1864\fBfio \-\-server=args\fR
1865
1866on that machine, where args defines what fio listens to. The arguments
811826be 1867are of the form 'type:hostname or IP:port'. 'type' is either 'ip' (or ip4)
20c67f10
MS
1868for TCP/IP v4, 'ip6' for TCP/IP v6, or 'sock' for a local unix domain
1869socket. 'hostname' is either a hostname or IP address, and 'port' is the port to
811826be 1870listen to (only valid for TCP/IP, not a local socket). Some examples:
49da1240 1871
e01e9745 18721) fio \-\-server
49da1240
JA
1873
1874 Start a fio server, listening on all interfaces on the default port (8765).
1875
e01e9745 18762) fio \-\-server=ip:hostname,4444
49da1240
JA
1877
1878 Start a fio server, listening on IP belonging to hostname and on port 4444.
1879
e01e9745 18803) fio \-\-server=ip6:::1,4444
811826be
JA
1881
1882 Start a fio server, listening on IPv6 localhost ::1 and on port 4444.
1883
e01e9745 18844) fio \-\-server=,4444
49da1240
JA
1885
1886 Start a fio server, listening on all interfaces on port 4444.
1887
e01e9745 18885) fio \-\-server=1.2.3.4
49da1240
JA
1889
1890 Start a fio server, listening on IP 1.2.3.4 on the default port.
1891
e01e9745 18926) fio \-\-server=sock:/tmp/fio.sock
49da1240
JA
1893
1894 Start a fio server, listening on the local socket /tmp/fio.sock.
1895
1896When a server is running, you can connect to it from a client. The client
1897is run with:
1898
e01e9745 1899fio \-\-local-args \-\-client=server \-\-remote-args <job file(s)>
49da1240 1900
e01e9745
MS
1901where \-\-local-args are arguments that are local to the client where it is
1902running, 'server' is the connect string, and \-\-remote-args and <job file(s)>
49da1240
JA
1903are sent to the server. The 'server' string follows the same format as it
1904does on the server side, to allow IP/hostname/socket and port strings.
1905You can connect to multiple clients as well, to do that you could run:
1906
e01e9745 1907fio \-\-client=server2 \-\-client=server2 <job file(s)>
323255cc
JA
1908
1909If the job file is located on the fio server, then you can tell the server
1910to load a local file as well. This is done by using \-\-remote-config:
1911
1912fio \-\-client=server \-\-remote-config /path/to/file.fio
1913
1914Then the fio serer will open this local (to the server) job file instead
1915of being passed one from the client.
d60e92d1 1916.SH AUTHORS
49da1240 1917
d60e92d1 1918.B fio
aa58d252 1919was written by Jens Axboe <jens.axboe@oracle.com>,
f8b8f7da 1920now Jens Axboe <axboe@fb.com>.
d1429b5c
AC
1921.br
1922This man page was written by Aaron Carroll <aaronc@cse.unsw.edu.au> based
d60e92d1
AC
1923on documentation by Jens Axboe.
1924.SH "REPORTING BUGS"
482900c9 1925Report bugs to the \fBfio\fR mailing list <fio@vger.kernel.org>.
d1429b5c 1926See \fBREADME\fR.
d60e92d1 1927.SH "SEE ALSO"
d1429b5c
AC
1928For further documentation see \fBHOWTO\fR and \fBREADME\fR.
1929.br
1930Sample jobfiles are available in the \fBexamples\fR directory.
d60e92d1 1931