Fix an inverted cgroup exit check
[fio.git] / fio.1
CommitLineData
d60e92d1
AC
1.TH fio 1 "September 2007" "User Manual"
2.SH NAME
3fio \- flexible I/O tester
4.SH SYNOPSIS
5.B fio
6[\fIoptions\fR] [\fIjobfile\fR]...
7.SH DESCRIPTION
8.B fio
9is a tool that will spawn a number of threads or processes doing a
10particular type of I/O action as specified by the user.
11The typical use of fio is to write a job file matching the I/O load
12one wants to simulate.
13.SH OPTIONS
14.TP
15.BI \-\-output \fR=\fPfilename
16Write output to \fIfilename\fR.
17.TP
18.BI \-\-timeout \fR=\fPtimeout
19Limit run time to \fItimeout\fR seconds.
20.TP
21.B \-\-latency\-log
22Generate per-job latency logs.
23.TP
24.B \-\-bandwidth\-log
25Generate per-job bandwidth logs.
26.TP
27.B \-\-minimal
d1429b5c 28Print statistics in a terse, semicolon-delimited format.
d60e92d1
AC
29.TP
30.BI \-\-showcmd \fR=\fPjobfile
31Convert \fIjobfile\fR to a set of command-line options.
32.TP
33.B \-\-readonly
34Enable read-only safety checks.
35.TP
36.BI \-\-eta \fR=\fPwhen
37Specifies when real-time ETA estimate should be printed. \fIwhen\fR may
38be one of `always', `never' or `auto'.
39.TP
c0a5d35e
AC
40.BI \-\-section \fR=\fPsec
41Only run section \fIsec\fR from job file.
42.TP
d60e92d1
AC
43.BI \-\-cmdhelp \fR=\fPcommand
44Print help information for \fIcommand\fR. May be `all' for all commands.
45.TP
9183788d
JA
46.BI \-\-debug \fR=\fPtype
47Enable verbose tracing of various fio actions. May be `all' for all types
c6e13ea5 48or individual types seperated by a comma (eg \-\-debug=io,file). `help' will
9183788d
JA
49list all available tracing options.
50.TP
d60e92d1
AC
51.B \-\-help
52Display usage information and exit.
53.TP
54.B \-\-version
55Display version information and exit.
56.SH "JOB FILE FORMAT"
57Job files are in `ini' format. They consist of one or more
58job definitions, which begin with a job name in square brackets and
59extend to the next job name. The job name can be any ASCII string
60except `global', which has a special meaning. Following the job name is
61a sequence of zero or more parameters, one per line, that define the
62behavior of the job. Any line starting with a `;' or `#' character is
d1429b5c 63considered a comment and ignored.
d9956b64
AC
64.P
65If \fIjobfile\fR is specified as `-', the job file will be read from
66standard input.
d60e92d1
AC
67.SS "Global Section"
68The global section contains default parameters for jobs specified in the
69job file. A job is only affected by global sections residing above it,
70and there may be any number of global sections. Specific job definitions
71may override any parameter set in global sections.
72.SH "JOB PARAMETERS"
73.SS Types
74Some parameters may take arguments of a specific type. The types used are:
75.TP
76.I str
77String: a sequence of alphanumeric characters.
78.TP
79.I int
d60e92d1 80SI integer: a whole number, possibly containing a suffix denoting the base unit
b09da8fa
JA
81of the value. Accepted suffixes are `k', 'M', 'G', 'T', and 'P', denoting
82kilo (1024), mega (1024^2), giga (1024^3), tera (1024^4), and peta (1024^5)
83respectively. The suffix is not case sensitive. If prefixed with '0x', the
84value is assumed to be base 16 (hexadecimal).
d60e92d1
AC
85.TP
86.I bool
87Boolean: a true or false value. `0' denotes false, `1' denotes true.
88.TP
89.I irange
90Integer range: a range of integers specified in the format
d1429b5c
AC
91\fIlower\fR:\fIupper\fR or \fIlower\fR\-\fIupper\fR. \fIlower\fR and
92\fIupper\fR may contain a suffix as described above. If an option allows two
93sets of ranges, they are separated with a `,' or `/' character. For example:
94`8\-8k/8M\-4G'.
d60e92d1
AC
95.SS "Parameter List"
96.TP
97.BI name \fR=\fPstr
d9956b64 98May be used to override the job name. On the command line, this parameter
d60e92d1
AC
99has the special purpose of signalling the start of a new job.
100.TP
101.BI description \fR=\fPstr
102Human-readable description of the job. It is printed when the job is run, but
103otherwise has no special purpose.
104.TP
105.BI directory \fR=\fPstr
106Prefix filenames with this directory. Used to place files in a location other
107than `./'.
108.TP
109.BI filename \fR=\fPstr
110.B fio
111normally makes up a file name based on the job name, thread number, and file
d1429b5c
AC
112number. If you want to share files between threads in a job or several jobs,
113specify a \fIfilename\fR for each of them to override the default. If the I/O
114engine used is `net', \fIfilename\fR is the host and port to connect to in the
115format \fIhost\fR/\fIport\fR. If the I/O engine is file-based, you can specify
116a number of files by separating the names with a `:' character. `\-' is a
117reserved name, meaning stdin or stdout, depending on the read/write direction
118set.
d60e92d1 119.TP
3ce9dcaf
JA
120.BI lockfile \fR=\fPstr
121Fio defaults to not locking any files before it does IO to them. If a file or
122file descriptor is shared, fio can serialize IO to that file to make the end
123result consistent. This is usual for emulating real workloads that share files.
124The lock modes are:
125.RS
126.RS
127.TP
128.B none
129No locking. This is the default.
130.TP
131.B exclusive
132Only one thread or process may do IO at the time, excluding all others.
133.TP
134.B readwrite
135Read-write locking on the file. Many readers may access the file at the same
136time, but writes get exclusive access.
137.RE
138.P
139The option may be post-fixed with a lock batch number. If set, then each
140thread/process may do that amount of IOs to the file before giving up the lock.
141Since lock acquisition is expensive, batching the lock/unlocks will speed up IO.
142.RE
143.P
d60e92d1
AC
144.BI opendir \fR=\fPstr
145Recursively open any files below directory \fIstr\fR.
146.TP
147.BI readwrite \fR=\fPstr "\fR,\fP rw" \fR=\fPstr
148Type of I/O pattern. Accepted values are:
149.RS
150.RS
151.TP
152.B read
d1429b5c 153Sequential reads.
d60e92d1
AC
154.TP
155.B write
d1429b5c 156Sequential writes.
d60e92d1
AC
157.TP
158.B randread
d1429b5c 159Random reads.
d60e92d1
AC
160.TP
161.B randwrite
d1429b5c 162Random writes.
d60e92d1
AC
163.TP
164.B rw
d1429b5c 165Mixed sequential reads and writes.
d60e92d1
AC
166.TP
167.B randrw
d1429b5c 168Mixed random reads and writes.
d60e92d1
AC
169.RE
170.P
d1429b5c
AC
171For mixed I/O, the default split is 50/50. For random I/O, the number of I/Os
172to perform before getting a new offset can be specified by appending
173`:\fIint\fR' to the pattern type. The default is 1.
d60e92d1
AC
174.RE
175.TP
90fef2d1
JA
176.BI kb_base \fR=\fPint
177The base unit for a kilobyte. The defacto base is 2^10, 1024. Storage
178manufacturers like to use 10^3 or 1000 as a base ten unit instead, for obvious
179reasons. Allow values are 1024 or 1000, with 1024 being the default.
180.TP
d60e92d1
AC
181.BI randrepeat \fR=\fPbool
182Seed the random number generator in a predictable way so results are repeatable
d1429b5c 183across runs. Default: true.
d60e92d1
AC
184.TP
185.BI fadvise_hint \fR=\fPbool
d1429b5c
AC
186Disable use of \fIposix_fadvise\fR\|(2) to advise the kernel what I/O patterns
187are likely to be issued. Default: true.
d60e92d1 188.TP
f7fa2653 189.BI size \fR=\fPint
d60e92d1
AC
190Total size of I/O for this job. \fBfio\fR will run until this many bytes have
191been transfered, unless limited by other options (\fBruntime\fR, for instance).
192Unless \fBnr_files\fR and \fBfilesize\fR options are given, this amount will be
193divided between the available files for the job.
194.TP
3ce9dcaf
JA
195.BI fill_device \fR=\fPbool
196Sets size to something really large and waits for ENOSPC (no space left on
197device) as the terminating condition. Only makes sense with sequential write.
198For a read workload, the mount point will be filled first then IO started on
199the result.
200.TP
d60e92d1
AC
201.BI filesize \fR=\fPirange
202Individual file sizes. May be a range, in which case \fBfio\fR will select sizes
d1429b5c
AC
203for files at random within the given range, limited to \fBsize\fR in total (if
204that is given). If \fBfilesize\fR is not specified, each created file is the
205same size.
d60e92d1 206.TP
f7fa2653 207.BI blocksize \fR=\fPint[,int] "\fR,\fB bs" \fR=\fPint[,int]
d60e92d1
AC
208Block size for I/O units. Default: 4k. Values for reads and writes can be
209specified seperately in the format \fIread\fR,\fIwrite\fR, either of
210which may be empty to leave that value at its default.
211.TP
9183788d 212.BI blocksize_range \fR=\fPirange[,irange] "\fR,\fB bsrange" \fR=\fPirange[,irange]
d1429b5c
AC
213Specify a range of I/O block sizes. The issued I/O unit will always be a
214multiple of the minimum size, unless \fBblocksize_unaligned\fR is set. Applies
9183788d
JA
215to both reads and writes if only one range is given, but can be specified
216seperately with a comma seperating the values. Example: bsrange=1k-4k,2k-8k.
217Also (see \fBblocksize\fR).
218.TP
219.BI bssplit \fR=\fPstr
220This option allows even finer grained control of the block sizes issued,
221not just even splits between them. With this option, you can weight various
222block sizes for exact control of the issued IO for a job that has mixed
223block sizes. The format of the option is bssplit=blocksize/percentage,
224optionally adding as many definitions as needed seperated by a colon.
225Example: bssplit=4k/10:64k/50:32k/40 would issue 50% 64k blocks, 10% 4k
c83cdd3e
JA
226blocks and 40% 32k blocks. \fBbssplit\fR also supports giving separate
227splits to reads and writes. The format is identical to what the
228\fBbs\fR option accepts, the read and write parts are separated with a
229comma.
d60e92d1
AC
230.TP
231.B blocksize_unaligned\fR,\fP bs_unaligned
d1429b5c
AC
232If set, any size in \fBblocksize_range\fR may be used. This typically won't
233work with direct I/O, as that normally requires sector alignment.
d60e92d1 234.TP
2b7a01d0 235.BI blockalign \fR=\fPint[,int] "\fR,\fB ba" \fR=\fPint[,int]
639ce0f3
MS
236At what boundary to align random IO offsets. Defaults to the same as 'blocksize'
237the minimum blocksize given. Minimum alignment is typically 512b
2b7a01d0
JA
238for using direct IO, though it usually depends on the hardware block size.
239This option is mutually exclusive with using a random map for files, so it
240will turn off that option.
43602667 241.TP
d60e92d1
AC
242.B zero_buffers
243Initialise buffers with all zeros. Default: fill buffers with random data.
244.TP
901bb994
JA
245.B refill_buffers
246If this option is given, fio will refill the IO buffers on every submit. The
247default is to only fill it at init time and reuse that data. Only makes sense
248if zero_buffers isn't specified, naturally. If data verification is enabled,
249refill_buffers is also automatically enabled.
250.TP
d60e92d1
AC
251.BI nrfiles \fR=\fPint
252Number of files to use for this job. Default: 1.
253.TP
254.BI openfiles \fR=\fPint
255Number of files to keep open at the same time. Default: \fBnrfiles\fR.
256.TP
257.BI file_service_type \fR=\fPstr
258Defines how files to service are selected. The following types are defined:
259.RS
260.RS
261.TP
262.B random
263Choose a file at random
264.TP
265.B roundrobin
266Round robin over open files (default).
6b7f6851
JA
267.B sequential
268Do each file in the set sequentially.
d60e92d1
AC
269.RE
270.P
271The number of I/Os to issue before switching a new file can be specified by
272appending `:\fIint\fR' to the service type.
273.RE
274.TP
275.BI ioengine \fR=\fPstr
276Defines how the job issues I/O. The following types are defined:
277.RS
278.RS
279.TP
280.B sync
281Basic \fIread\fR\|(2) or \fIwrite\fR\|(2) I/O. \fIfseek\fR\|(2) is used to
282position the I/O location.
283.TP
a31041ea 284.B psync
285Basic \fIpread\fR\|(2) or \fIpwrite\fR\|(2) I/O.
286.TP
9183788d
JA
287.B vsync
288Basic \fIreadv\fR\|(2) or \fIwritev\fR\|(2) I/O. Will emulate queuing by
289coalescing adjacents IOs into a single submission.
290.TP
d60e92d1
AC
291.B libaio
292Linux native asynchronous I/O.
293.TP
294.B posixaio
295glibc POSIX asynchronous I/O using \fIaio_read\fR\|(3) and \fIaio_write\fR\|(3).
296.TP
297.B mmap
d1429b5c
AC
298File is memory mapped with \fImmap\fR\|(2) and data copied using
299\fImemcpy\fR\|(3).
d60e92d1
AC
300.TP
301.B splice
d1429b5c
AC
302\fIsplice\fR\|(2) is used to transfer the data and \fIvmsplice\fR\|(2) to
303transfer data from user-space to the kernel.
d60e92d1
AC
304.TP
305.B syslet-rw
306Use the syslet system calls to make regular read/write asynchronous.
307.TP
308.B sg
309SCSI generic sg v3 I/O. May be either synchronous using the SG_IO ioctl, or if
d1429b5c
AC
310the target is an sg character device, we use \fIread\fR\|(2) and
311\fIwrite\fR\|(2) for asynchronous I/O.
d60e92d1
AC
312.TP
313.B null
314Doesn't transfer any data, just pretends to. Mainly used to exercise \fBfio\fR
315itself and for debugging and testing purposes.
316.TP
317.B net
318Transfer over the network. \fBfilename\fR must be set appropriately to
319`\fIhost\fR/\fIport\fR' regardless of data direction. If receiving, only the
320\fIport\fR argument is used.
321.TP
322.B netsplice
323Like \fBnet\fR, but uses \fIsplice\fR\|(2) and \fIvmsplice\fR\|(2) to map data
324and send/receive.
325.TP
53aec0a4 326.B cpuio
d60e92d1
AC
327Doesn't transfer any data, but burns CPU cycles according to \fBcpuload\fR and
328\fBcpucycles\fR parameters.
329.TP
330.B guasi
331The GUASI I/O engine is the Generic Userspace Asynchronous Syscall Interface
332approach to asycnronous I/O.
d1429b5c
AC
333.br
334See <http://www.xmailserver.org/guasi\-lib.html>.
d60e92d1
AC
335.TP
336.B external
337Loads an external I/O engine object file. Append the engine filename as
338`:\fIenginepath\fR'.
339.RE
340.RE
341.TP
342.BI iodepth \fR=\fPint
343Number of I/O units to keep in flight against the file. Default: 1.
344.TP
345.BI iodepth_batch \fR=\fPint
346Number of I/Os to submit at once. Default: \fBiodepth\fR.
347.TP
3ce9dcaf
JA
348.BI iodepth_batch_complete \fR=\fPint
349This defines how many pieces of IO to retrieve at once. It defaults to 1 which
350 means that we'll ask for a minimum of 1 IO in the retrieval process from the
351kernel. The IO retrieval will go on until we hit the limit set by
352\fBiodepth_low\fR. If this variable is set to 0, then fio will always check for
353completed events before queuing more IO. This helps reduce IO latency, at the
354cost of more retrieval system calls.
355.TP
d60e92d1
AC
356.BI iodepth_low \fR=\fPint
357Low watermark indicating when to start filling the queue again. Default:
358\fBiodepth\fR.
359.TP
360.BI direct \fR=\fPbool
361If true, use non-buffered I/O (usually O_DIRECT). Default: false.
362.TP
363.BI buffered \fR=\fPbool
364If true, use buffered I/O. This is the opposite of the \fBdirect\fR parameter.
365Default: true.
366.TP
f7fa2653 367.BI offset \fR=\fPint
d60e92d1
AC
368Offset in the file to start I/O. Data before the offset will not be touched.
369.TP
370.BI fsync \fR=\fPint
d1429b5c
AC
371How many I/Os to perform before issuing an \fBfsync\fR\|(2) of dirty data. If
3720, don't sync. Default: 0.
d60e92d1 373.TP
5f9099ea
JA
374.BI fdatasync \fR=\fPint
375Like \fBfsync\fR, but uses \fBfdatasync\fR\|(2) instead to only sync the
376data parts of the file. Default: 0.
377.TP
d60e92d1 378.BI overwrite \fR=\fPbool
d1429b5c 379If writing, setup the file first and do overwrites. Default: false.
d60e92d1
AC
380.TP
381.BI end_fsync \fR=\fPbool
d1429b5c 382Sync file contents when job exits. Default: false.
d60e92d1
AC
383.TP
384.BI fsync_on_close \fR=\fPbool
385If true, sync file contents on close. This differs from \fBend_fsync\fR in that
d1429b5c 386it will happen on every close, not just at the end of the job. Default: false.
d60e92d1
AC
387.TP
388.BI rwmixcycle \fR=\fPint
389How many milliseconds before switching between reads and writes for a mixed
390workload. Default: 500ms.
391.TP
392.BI rwmixread \fR=\fPint
393Percentage of a mixed workload that should be reads. Default: 50.
394.TP
395.BI rwmixwrite \fR=\fPint
d1429b5c 396Percentage of a mixed workload that should be writes. If \fBrwmixread\fR and
c35dd7a6
JA
397\fBrwmixwrite\fR are given and do not sum to 100%, the latter of the two
398overrides the first. This may interfere with a given rate setting, if fio is
399asked to limit reads or writes to a certain rate. If that is the case, then
400the distribution may be skewed. Default: 50.
d60e92d1
AC
401.TP
402.B norandommap
403Normally \fBfio\fR will cover every block of the file when doing random I/O. If
404this parameter is given, a new offset will be chosen without looking at past
405I/O history. This parameter is mutually exclusive with \fBverify\fR.
406.TP
3ce9dcaf
JA
407.B softrandommap
408See \fBnorandommap\fR. If fio runs with the random block map enabled and it
409fails to allocate the map, if this option is set it will continue without a
410random block map. As coverage will not be as complete as with random maps, this
411option is disabled by default.
412.TP
d60e92d1
AC
413.BI nice \fR=\fPint
414Run job with given nice value. See \fInice\fR\|(2).
415.TP
416.BI prio \fR=\fPint
417Set I/O priority value of this job between 0 (highest) and 7 (lowest). See
418\fIionice\fR\|(1).
419.TP
420.BI prioclass \fR=\fPint
421Set I/O priority class. See \fIionice\fR\|(1).
422.TP
423.BI thinktime \fR=\fPint
424Stall job for given number of microseconds between issuing I/Os.
425.TP
426.BI thinktime_spin \fR=\fPint
427Pretend to spend CPU time for given number of microseconds, sleeping the rest
428of the time specified by \fBthinktime\fR. Only valid if \fBthinktime\fR is set.
429.TP
430.BI thinktime_blocks \fR=\fPint
431Number of blocks to issue before waiting \fBthinktime\fR microseconds.
432Default: 1.
433.TP
434.BI rate \fR=\fPint
c35dd7a6
JA
435Cap bandwidth used by this job. The number is in bytes/sec, the normal postfix
436rules apply. You can use \fBrate\fR=500k to limit reads and writes to 500k each,
437or you can specify read and writes separately. Using \fBrate\fR=1m,500k would
438limit reads to 1MB/sec and writes to 500KB/sec. Capping only reads or writes
439can be done with \fBrate\fR=,500k or \fBrate\fR=500k,. The former will only
440limit writes (to 500KB/sec), the latter will only limit reads.
d60e92d1
AC
441.TP
442.BI ratemin \fR=\fPint
443Tell \fBfio\fR to do whatever it can to maintain at least the given bandwidth.
c35dd7a6
JA
444Failing to meet this requirement will cause the job to exit. The same format
445as \fBrate\fR is used for read vs write separation.
d60e92d1
AC
446.TP
447.BI rate_iops \fR=\fPint
c35dd7a6
JA
448Cap the bandwidth to this number of IOPS. Basically the same as rate, just
449specified independently of bandwidth. The same format as \fBrate\fR is used for
450read vs write seperation. If \fBblocksize\fR is a range, the smallest block
451size is used as the metric.
d60e92d1
AC
452.TP
453.BI rate_iops_min \fR=\fPint
c35dd7a6
JA
454If this rate of I/O is not met, the job will exit. The same format as \fBrate\fR
455is used for read vs write seperation.
d60e92d1
AC
456.TP
457.BI ratecycle \fR=\fPint
458Average bandwidth for \fBrate\fR and \fBratemin\fR over this number of
459milliseconds. Default: 1000ms.
460.TP
461.BI cpumask \fR=\fPint
462Set CPU affinity for this job. \fIint\fR is a bitmask of allowed CPUs the job
463may run on. See \fBsched_setaffinity\fR\|(2).
464.TP
465.BI cpus_allowed \fR=\fPstr
466Same as \fBcpumask\fR, but allows a comma-delimited list of CPU numbers.
467.TP
468.BI startdelay \fR=\fPint
469Delay start of job for the specified number of seconds.
470.TP
471.BI runtime \fR=\fPint
472Terminate processing after the specified number of seconds.
473.TP
474.B time_based
475If given, run for the specified \fBruntime\fR duration even if the files are
476completely read or written. The same workload will be repeated as many times
477as \fBruntime\fR allows.
478.TP
901bb994
JA
479.BI ramp_time \fR=\fPint
480If set, fio will run the specified workload for this amount of time before
481logging any performance numbers. Useful for letting performance settle before
482logging results, thus minimizing the runtime required for stable results. Note
c35dd7a6
JA
483that the \fBramp_time\fR is considered lead in time for a job, thus it will
484increase the total runtime if a special timeout or runtime is specified.
901bb994 485.TP
d60e92d1
AC
486.BI invalidate \fR=\fPbool
487Invalidate buffer-cache for the file prior to starting I/O. Default: true.
488.TP
489.BI sync \fR=\fPbool
490Use synchronous I/O for buffered writes. For the majority of I/O engines,
d1429b5c 491this means using O_SYNC. Default: false.
d60e92d1
AC
492.TP
493.BI iomem \fR=\fPstr "\fR,\fP mem" \fR=\fPstr
494Allocation method for I/O unit buffer. Allowed values are:
495.RS
496.RS
497.TP
498.B malloc
499Allocate memory with \fImalloc\fR\|(3).
500.TP
501.B shm
502Use shared memory buffers allocated through \fIshmget\fR\|(2).
503.TP
504.B shmhuge
505Same as \fBshm\fR, but use huge pages as backing.
506.TP
507.B mmap
508Use \fImmap\fR\|(2) for allocation. Uses anonymous memory unless a filename
509is given after the option in the format `:\fIfile\fR'.
510.TP
511.B mmaphuge
512Same as \fBmmap\fR, but use huge files as backing.
513.RE
514.P
515The amount of memory allocated is the maximum allowed \fBblocksize\fR for the
516job multiplied by \fBiodepth\fR. For \fBshmhuge\fR or \fBmmaphuge\fR to work,
517the system must have free huge pages allocated. \fBmmaphuge\fR also needs to
2e266ba6
JA
518have hugetlbfs mounted, and \fIfile\fR must point there. At least on Linux,
519huge pages must be manually allocated. See \fB/proc/sys/vm/nr_hugehages\fR
520and the documentation for that. Normally you just need to echo an appropriate
521number, eg echoing 8 will ensure that the OS has 8 huge pages ready for
522use.
d60e92d1
AC
523.RE
524.TP
d529ee19
JA
525.BI iomem_align \fR=\fPint
526This indiciates the memory alignment of the IO memory buffers. Note that the
527given alignment is applied to the first IO unit buffer, if using \fBiodepth\fR
528the alignment of the following buffers are given by the \fBbs\fR used. In
529other words, if using a \fBbs\fR that is a multiple of the page sized in the
530system, all buffers will be aligned to this value. If using a \fBbs\fR that
531is not page aligned, the alignment of subsequent IO memory buffers is the
532sum of the \fBiomem_align\fR and \fBbs\fR used.
533.TP
f7fa2653 534.BI hugepage\-size \fR=\fPint
d60e92d1 535Defines the size of a huge page. Must be at least equal to the system setting.
b22989b9 536Should be a multiple of 1MB. Default: 4MB.
d60e92d1
AC
537.TP
538.B exitall
539Terminate all jobs when one finishes. Default: wait for each job to finish.
540.TP
541.BI bwavgtime \fR=\fPint
542Average bandwidth calculations over the given time in milliseconds. Default:
543500ms.
544.TP
545.BI create_serialize \fR=\fPbool
d1429b5c 546If true, serialize file creation for the jobs. Default: true.
d60e92d1
AC
547.TP
548.BI create_fsync \fR=\fPbool
549\fIfsync\fR\|(2) data file after creation. Default: true.
550.TP
6b7f6851
JA
551.BI create_on_open \fR=\fPbool
552If true, the files are not created until they are opened for IO by the job.
553.TP
e9f48479
JA
554.BI pre_read \fR=\fPbool
555If this is given, files will be pre-read into memory before starting the given
556IO operation. This will also clear the \fR \fBinvalidate\fR flag, since it is
9c0d2241
JA
557pointless to pre-read and then drop the cache. This will only work for IO
558engines that are seekable, since they allow you to read the same data
559multiple times. Thus it will not work on eg network or splice IO.
e9f48479 560.TP
d60e92d1
AC
561.BI unlink \fR=\fPbool
562Unlink job files when done. Default: false.
563.TP
564.BI loops \fR=\fPint
565Specifies the number of iterations (runs of the same workload) of this job.
566Default: 1.
567.TP
568.BI do_verify \fR=\fPbool
569Run the verify phase after a write phase. Only valid if \fBverify\fR is set.
570Default: true.
571.TP
572.BI verify \fR=\fPstr
573Method of verifying file contents after each iteration of the job. Allowed
574values are:
575.RS
576.RS
577.TP
b892dc08 578.B md5 crc16 crc32 crc32c crc32c-intel crc64 crc7 sha256 sha512 sha1
d60e92d1
AC
579Store appropriate checksum in the header of each block.
580.TP
581.B meta
582Write extra information about each I/O (timestamp, block number, etc.). The
583block number is verified.
584.TP
585.B pattern
0e92f873
RR
586Fill I/O buffers with a specific pattern that is used to verify. If the pattern
587is < 4bytes, it can either be a decimal or a hexadecimal number. If the pattern
588is > 4bytes, currently, it can only be a hexadecimal pattern starting with
589either "0x" or "0X".
d60e92d1
AC
590.TP
591.B null
592Pretend to verify. Used for testing internals.
593.RE
b892dc08
JA
594
595This option can be used for repeated burn-in tests of a system to make sure
596that the written data is also correctly read back. If the data direction given
597is a read or random read, fio will assume that it should verify a previously
598written file. If the data direction includes any form of write, the verify will
599be of the newly written data.
d60e92d1
AC
600.RE
601.TP
602.BI verify_sort \fR=\fPbool
603If true, written verify blocks are sorted if \fBfio\fR deems it to be faster to
604read them back in a sorted manner. Default: true.
605.TP
f7fa2653 606.BI verify_offset \fR=\fPint
d60e92d1 607Swap the verification header with data somewhere else in the block before
d1429b5c 608writing. It is swapped back before verifying.
d60e92d1 609.TP
f7fa2653 610.BI verify_interval \fR=\fPint
d60e92d1
AC
611Write the verification header for this number of bytes, which should divide
612\fBblocksize\fR. Default: \fBblocksize\fR.
613.TP
614.BI verify_fatal \fR=\fPbool
615If true, exit the job on the first observed verification failure. Default:
616false.
617.TP
e8462bd8
JA
618.BI verify_async \fR=\fPint
619Fio will normally verify IO inline from the submitting thread. This option
620takes an integer describing how many async offload threads to create for IO
621verification instead, causing fio to offload the duty of verifying IO contents
c85c324c
JA
622to one or more separate threads. If using this offload option, even sync IO
623engines can benefit from using an \fBiodepth\fR setting higher than 1, as it
624allows them to have IO in flight while verifies are running.
e8462bd8
JA
625.TP
626.BI verify_async_cpus \fR=\fPstr
627Tell fio to set the given CPU affinity on the async IO verification threads.
628See \fBcpus_allowed\fP for the format used.
629.TP
d60e92d1 630.B stonewall
d1429b5c 631Wait for preceeding jobs in the job file to exit before starting this one.
d60e92d1
AC
632\fBstonewall\fR implies \fBnew_group\fR.
633.TP
634.B new_group
635Start a new reporting group. If not given, all jobs in a file will be part
636of the same reporting group, unless separated by a stonewall.
637.TP
638.BI numjobs \fR=\fPint
639Number of clones (processes/threads performing the same workload) of this job.
640Default: 1.
641.TP
642.B group_reporting
643If set, display per-group reports instead of per-job when \fBnumjobs\fR is
644specified.
645.TP
646.B thread
647Use threads created with \fBpthread_create\fR\|(3) instead of processes created
648with \fBfork\fR\|(2).
649.TP
f7fa2653 650.BI zonesize \fR=\fPint
d60e92d1
AC
651Divide file into zones of the specified size in bytes. See \fBzoneskip\fR.
652.TP
f7fa2653 653.BI zoneskip \fR=\fPint
d1429b5c 654Skip the specified number of bytes when \fBzonesize\fR bytes of data have been
d60e92d1
AC
655read.
656.TP
657.BI write_iolog \fR=\fPstr
658Write the issued I/O patterns to the specified file.
659.TP
660.BI read_iolog \fR=\fPstr
661Replay the I/O patterns contained in the specified file generated by
662\fBwrite_iolog\fR, or may be a \fBblktrace\fR binary file.
663.TP
901bb994
JA
664.B write_bw_log \fR=\fPstr
665If given, write a bandwidth log of the jobs in this job file. Can be used to
666store data of the bandwidth of the jobs in their lifetime. The included
667fio_generate_plots script uses gnuplot to turn these text files into nice
668graphs. See \fBwrite_log_log\fR for behaviour of given filename. For this
669option, the postfix is _bw.log.
d60e92d1
AC
670.TP
671.B write_lat_log
901bb994
JA
672Same as \fBwrite_bw_log\fR, but writes I/O completion latencies. If no
673filename is given with this option, the default filename of "jobname_type.log"
674is used. Even if the filename is given, fio will still append the type of log.
675.TP
676.B disable_clat \fR=\fPbool
677Disable measurements of completion latency numbers. Useful only for cutting
678back the number of calls to gettimeofday, as that does impact performance at
679really high IOPS rates. Note that to really get rid of a large amount of these
680calls, this option must be used with disable_slat and disable_bw as well.
681.TP
682.B disable_slat \fR=\fPbool
683Disable measurements of submission latency numbers. See \fBdisable_clat\fR.
684.TP
685.B disable_bw_measurement \fR=\fPbool
686Disable measurements of throughput/bandwidth numbers. See \fBdisable_clat\fR.
d60e92d1 687.TP
f7fa2653 688.BI lockmem \fR=\fPint
d60e92d1
AC
689Pin the specified amount of memory with \fBmlock\fR\|(2). Can be used to
690simulate a smaller amount of memory.
691.TP
692.BI exec_prerun \fR=\fPstr
693Before running the job, execute the specified command with \fBsystem\fR\|(3).
694.TP
695.BI exec_postrun \fR=\fPstr
696Same as \fBexec_prerun\fR, but the command is executed after the job completes.
697.TP
698.BI ioscheduler \fR=\fPstr
699Attempt to switch the device hosting the file to the specified I/O scheduler.
700.TP
701.BI cpuload \fR=\fPint
702If the job is a CPU cycle-eater, attempt to use the specified percentage of
703CPU cycles.
704.TP
705.BI cpuchunks \fR=\fPint
706If the job is a CPU cycle-eater, split the load into cycles of the
707given time in milliseconds.
708.TP
709.BI disk_util \fR=\fPbool
d1429b5c 710Generate disk utilization statistics if the platform supports it. Default: true.
901bb994
JA
711.TP
712.BI gtod_reduce \fR=\fPbool
713Enable all of the gettimeofday() reducing options (disable_clat, disable_slat,
714disable_bw) plus reduce precision of the timeout somewhat to really shrink the
715gettimeofday() call count. With this option enabled, we only do about 0.4% of
716the gtod() calls we would have done if all time keeping was enabled.
717.TP
718.BI gtod_cpu \fR=\fPint
719Sometimes it's cheaper to dedicate a single thread of execution to just getting
720the current time. Fio (and databases, for instance) are very intensive on
721gettimeofday() calls. With this option, you can set one CPU aside for doing
722nothing but logging current time to a shared memory location. Then the other
723threads/processes that run IO workloads need only copy that segment, instead of
724entering the kernel with a gettimeofday() call. The CPU set aside for doing
725these time calls will be excluded from other uses. Fio will manually clear it
726from the CPU mask of other jobs.
f2bba182 727.TP
a696fa2a
JA
728.BI cgroup_root \fR=\fPstr
729Root of the mounted blkio cgroup file systems. This is a Linux
730specific IO controller. If your system doesn't have it mounted,
731you can do so with:
732
733# mount -t cgroup -o blkio none /cgroup
734
735The cgroup_root defaults to /cgroup, if mounted elsewhere please specify this
736option.
737.TP
738.BI cgroup \fR=\fPstr
739Add job to this control group. If it doesn't exist, it will be created.
740.TP
741.BI cgroup_weight \fR=\fPint
742Set the weight of the cgroup to this value. See the documentation that comes
743with the kernel, allowed values are in the range of 100..1000.
d60e92d1 744.SH OUTPUT
d1429b5c
AC
745While running, \fBfio\fR will display the status of the created jobs. For
746example:
d60e92d1 747.RS
d1429b5c 748.P
d60e92d1
AC
749Threads: 1: [_r] [24.8% done] [ 13509/ 8334 kb/s] [eta 00h:01m:31s]
750.RE
751.P
d1429b5c
AC
752The characters in the first set of brackets denote the current status of each
753threads. The possible values are:
754.P
755.PD 0
d60e92d1
AC
756.RS
757.TP
758.B P
759Setup but not started.
760.TP
761.B C
762Thread created.
763.TP
764.B I
765Initialized, waiting.
766.TP
767.B R
768Running, doing sequential reads.
769.TP
770.B r
771Running, doing random reads.
772.TP
773.B W
774Running, doing sequential writes.
775.TP
776.B w
777Running, doing random writes.
778.TP
779.B M
780Running, doing mixed sequential reads/writes.
781.TP
782.B m
783Running, doing mixed random reads/writes.
784.TP
785.B F
786Running, currently waiting for \fBfsync\fR\|(2).
787.TP
788.B V
789Running, verifying written data.
790.TP
791.B E
792Exited, not reaped by main thread.
793.TP
794.B \-
795Exited, thread reaped.
796.RE
d1429b5c 797.PD
d60e92d1
AC
798.P
799The second set of brackets shows the estimated completion percentage of
800the current group. The third set shows the read and write I/O rate,
801respectively. Finally, the estimated run time of the job is displayed.
802.P
803When \fBfio\fR completes (or is interrupted by Ctrl-C), it will show data
804for each thread, each group of threads, and each disk, in that order.
805.P
806Per-thread statistics first show the threads client number, group-id, and
807error code. The remaining figures are as follows:
808.RS
d60e92d1
AC
809.TP
810.B io
811Number of megabytes of I/O performed.
812.TP
813.B bw
814Average data rate (bandwidth).
815.TP
816.B runt
817Threads run time.
818.TP
819.B slat
820Submission latency minimum, maximum, average and standard deviation. This is
821the time it took to submit the I/O.
822.TP
823.B clat
824Completion latency minimum, maximum, average and standard deviation. This
825is the time between submission and completion.
826.TP
827.B bw
828Bandwidth minimum, maximum, percentage of aggregate bandwidth received, average
829and standard deviation.
830.TP
831.B cpu
832CPU usage statistics. Includes user and system time, number of context switches
833this thread went through and number of major and minor page faults.
834.TP
835.B IO depths
836Distribution of I/O depths. Each depth includes everything less than (or equal)
837to it, but greater than the previous depth.
838.TP
839.B IO issued
840Number of read/write requests issued, and number of short read/write requests.
841.TP
842.B IO latencies
843Distribution of I/O completion latencies. The numbers follow the same pattern
844as \fBIO depths\fR.
845.RE
d60e92d1
AC
846.P
847The group statistics show:
d1429b5c 848.PD 0
d60e92d1
AC
849.RS
850.TP
851.B io
852Number of megabytes I/O performed.
853.TP
854.B aggrb
855Aggregate bandwidth of threads in the group.
856.TP
857.B minb
858Minimum average bandwidth a thread saw.
859.TP
860.B maxb
861Maximum average bandwidth a thread saw.
862.TP
863.B mint
d1429b5c 864Shortest runtime of threads in the group.
d60e92d1
AC
865.TP
866.B maxt
867Longest runtime of threads in the group.
868.RE
d1429b5c 869.PD
d60e92d1
AC
870.P
871Finally, disk statistics are printed with reads first:
d1429b5c 872.PD 0
d60e92d1
AC
873.RS
874.TP
875.B ios
876Number of I/Os performed by all groups.
877.TP
878.B merge
879Number of merges in the I/O scheduler.
880.TP
881.B ticks
882Number of ticks we kept the disk busy.
883.TP
884.B io_queue
885Total time spent in the disk queue.
886.TP
887.B util
888Disk utilization.
889.RE
d1429b5c 890.PD
d60e92d1
AC
891.SH TERSE OUTPUT
892If the \fB\-\-minimal\fR option is given, the results will be printed in a
893semicolon-delimited format suitable for scripted use. The fields are:
894.P
895.RS
896.B jobname, groupid, error
897.P
898Read status:
899.RS
b22989b9 900.B KB I/O, bandwidth \fR(KB/s)\fP, runtime \fR(ms)\fP
d60e92d1
AC
901.P
902Submission latency:
903.RS
904.B min, max, mean, standard deviation
905.RE
906Completion latency:
907.RS
908.B min, max, mean, standard deviation
909.RE
910Bandwidth:
911.RS
912.B min, max, aggregate percentage of total, mean, standard deviation
913.RE
914.RE
915.P
916Write status:
917.RS
b22989b9 918.B KB I/O, bandwidth \fR(KB/s)\fP, runtime \fR(ms)\fP
d60e92d1
AC
919.P
920Submission latency:
921.RS
922.B min, max, mean, standard deviation
923.RE
924Completion latency:
925.RS
926.B min, max, mean, standard deviation
927.RE
928Bandwidth:
929.RS
930.B min, max, aggregate percentage of total, mean, standard deviation
931.RE
932.RE
933.P
d1429b5c 934CPU usage:
d60e92d1 935.RS
bd2626f0 936.B user, system, context switches, major page faults, minor page faults
d60e92d1
AC
937.RE
938.P
939IO depth distribution:
940.RS
941.B <=1, 2, 4, 8, 16, 32, >=64
942.RE
943.P
944IO latency distribution (ms):
945.RS
946.B <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, >=2000
947.RE
948.P
949.B text description
950.RE
951.SH AUTHORS
952.B fio
953was written by Jens Axboe <jens.axboe@oracle.com>.
d1429b5c
AC
954.br
955This man page was written by Aaron Carroll <aaronc@cse.unsw.edu.au> based
d60e92d1
AC
956on documentation by Jens Axboe.
957.SH "REPORTING BUGS"
482900c9 958Report bugs to the \fBfio\fR mailing list <fio@vger.kernel.org>.
d1429b5c 959See \fBREADME\fR.
d60e92d1 960.SH "SEE ALSO"
d1429b5c
AC
961For further documentation see \fBHOWTO\fR and \fBREADME\fR.
962.br
963Sample jobfiles are available in the \fBexamples\fR directory.
d60e92d1 964