doc: latency log unit is nsec
[fio.git] / HOWTO
CommitLineData
f80dba8d
MT
1How fio works
2-------------
3
4The first step in getting fio to simulate a desired I/O workload, is writing a
5job file describing that specific setup. A job file may contain any number of
6threads and/or files -- the typical contents of the job file is a *global*
7section defining shared parameters, and one or more job sections describing the
8jobs involved. When run, fio parses this file and sets everything up as
9described. If we break down a job from top to bottom, it contains the following
10basic parameters:
11
12`I/O type`_
13
14 Defines the I/O pattern issued to the file(s). We may only be reading
15 sequentially from this file(s), or we may be writing randomly. Or even
16 mixing reads and writes, sequentially or randomly.
17 Should we be doing buffered I/O, or direct/raw I/O?
18
19`Block size`_
20
21 In how large chunks are we issuing I/O? This may be a single value,
22 or it may describe a range of block sizes.
23
24`I/O size`_
25
26 How much data are we going to be reading/writing.
27
28`I/O engine`_
29
30 How do we issue I/O? We could be memory mapping the file, we could be
31 using regular read/write, we could be using splice, async I/O, or even
32 SG (SCSI generic sg).
33
34`I/O depth`_
35
36 If the I/O engine is async, how large a queuing depth do we want to
37 maintain?
38
39
40`Target file/device`_
41
42 How many files are we spreading the workload over.
43
44`Threads, processes and job synchronization`_
45
46 How many threads or processes should we spread this workload over.
47
48The above are the basic parameters defined for a workload, in addition there's a
49multitude of parameters that modify other aspects of how this job behaves.
50
51
52Command line options
53--------------------
54
55.. option:: --debug=type
56
f50fbdda 57 Enable verbose tracing `type` of various fio actions. May be ``all`` for all types
b034c0dd
TK
58 or individual types separated by a comma (e.g. ``--debug=file,mem`` will
59 enable file and memory debugging). Currently, additional logging is
60 available for:
f80dba8d 61
b034c0dd 62 *process*
f80dba8d 63 Dump info related to processes.
b034c0dd 64 *file*
f80dba8d 65 Dump info related to file actions.
b034c0dd 66 *io*
f80dba8d 67 Dump info related to I/O queuing.
b034c0dd 68 *mem*
f80dba8d 69 Dump info related to memory allocations.
b034c0dd 70 *blktrace*
f80dba8d 71 Dump info related to blktrace setup.
b034c0dd 72 *verify*
f80dba8d 73 Dump info related to I/O verification.
b034c0dd 74 *all*
f80dba8d 75 Enable all debug options.
b034c0dd 76 *random*
f80dba8d 77 Dump info related to random offset generation.
b034c0dd 78 *parse*
f80dba8d 79 Dump info related to option matching and parsing.
b034c0dd 80 *diskutil*
f80dba8d 81 Dump info related to disk utilization updates.
b034c0dd 82 *job:x*
f80dba8d 83 Dump info only related to job number x.
b034c0dd 84 *mutex*
f80dba8d 85 Dump info only related to mutex up/down ops.
b034c0dd 86 *profile*
f80dba8d 87 Dump info related to profile extensions.
b034c0dd 88 *time*
f80dba8d 89 Dump info related to internal time keeping.
b034c0dd 90 *net*
f80dba8d 91 Dump info related to networking connections.
b034c0dd 92 *rate*
f80dba8d 93 Dump info related to I/O rate switching.
b034c0dd 94 *compress*
f80dba8d 95 Dump info related to log compress/decompress.
b034c0dd 96 *?* or *help*
f80dba8d
MT
97 Show available debug options.
98
99.. option:: --parse-only
100
25cd4b95 101 Parse options only, don't start any I/O.
f80dba8d
MT
102
103.. option:: --output=filename
104
105 Write output to file `filename`.
106
f50fbdda 107.. option:: --output-format=format
b8f7e412 108
f50fbdda 109 Set the reporting `format` to `normal`, `terse`, `json`, or `json+`. Multiple
b8f7e412
TK
110 formats can be selected, separated by a comma. `terse` is a CSV based
111 format. `json+` is like `json`, except it adds a full dump of the latency
112 buckets.
113
114.. option:: --runtime
115 Limit run time to runtime seconds.
116
f80dba8d
MT
117.. option:: --bandwidth-log
118
119 Generate aggregate bandwidth logs.
120
121.. option:: --minimal
122
123 Print statistics in a terse, semicolon-delimited format.
124
125.. option:: --append-terse
126
b034c0dd
TK
127 Print statistics in selected mode AND terse, semicolon-delimited format.
128 **Deprecated**, use :option:`--output-format` instead to select multiple
129 formats.
f80dba8d 130
f50fbdda 131.. option:: --terse-version=version
f80dba8d 132
f50fbdda 133 Set terse `version` output format (default 3, or 2 or 4 or 5).
f80dba8d
MT
134
135.. option:: --version
136
b8f7e412 137 Print version information and exit.
f80dba8d
MT
138
139.. option:: --help
140
113f0e7c 141 Print a summary of the command line options and exit.
f80dba8d
MT
142
143.. option:: --cpuclock-test
144
145 Perform test and validation of internal CPU clock.
146
113f0e7c 147.. option:: --crctest=[test]
f80dba8d 148
b034c0dd
TK
149 Test the speed of the built-in checksumming functions. If no argument is
150 given, all of them are tested. Alternatively, a comma separated list can
151 be passed, in which case the given ones are tested.
f80dba8d
MT
152
153.. option:: --cmdhelp=command
154
155 Print help information for `command`. May be ``all`` for all commands.
156
157.. option:: --enghelp=[ioengine[,command]]
158
f50fbdda
TK
159 List all commands defined by `ioengine`, or print help for `command`
160 defined by `ioengine`. If no `ioengine` is given, list all
b034c0dd 161 available ioengines.
f80dba8d
MT
162
163.. option:: --showcmd=jobfile
164
b8f7e412 165 Convert `jobfile` to a set of command-line options.
f80dba8d
MT
166
167.. option:: --readonly
168
b034c0dd
TK
169 Turn on safety read-only checks, preventing writes. The ``--readonly``
170 option is an extra safety guard to prevent users from accidentally starting
171 a write workload when that is not desired. Fio will only write if
172 `rw=write/randwrite/rw/randrw` is given. This extra safety net can be used
173 as an extra precaution as ``--readonly`` will also enable a write check in
174 the I/O engine core to prevent writes due to unknown user space bug(s).
f80dba8d
MT
175
176.. option:: --eta=when
177
b8f7e412
TK
178 Specifies when real-time ETA estimate should be printed. `when` may be
179 `always`, `never` or `auto`.
f80dba8d
MT
180
181.. option:: --eta-newline=time
182
947e0fe0
SW
183 Force a new line for every `time` period passed. When the unit is omitted,
184 the value is interpreted in seconds.
f80dba8d
MT
185
186.. option:: --status-interval=time
187
947e0fe0
SW
188 Force full status dump every `time` period passed. When the unit is
189 omitted, the value is interpreted in seconds.
f80dba8d
MT
190
191.. option:: --section=name
192
b034c0dd
TK
193 Only run specified section `name` in job file. Multiple sections can be specified.
194 The ``--section`` option allows one to combine related jobs into one file.
195 E.g. one job file could define light, moderate, and heavy sections. Tell
196 fio to run only the "heavy" section by giving ``--section=heavy``
197 command line option. One can also specify the "write" operations in one
198 section and "verify" operation in another section. The ``--section`` option
199 only applies to job sections. The reserved *global* section is always
200 parsed and used.
f80dba8d
MT
201
202.. option:: --alloc-size=kb
203
b034c0dd
TK
204 Set the internal smalloc pool size to `kb` in KiB. The
205 ``--alloc-size`` switch allows one to use a larger pool size for smalloc.
206 If running large jobs with randommap enabled, fio can run out of memory.
207 Smalloc is an internal allocator for shared structures from a fixed size
208 memory pool and can grow to 16 pools. The pool size defaults to 16MiB.
f80dba8d 209
b034c0dd
TK
210 NOTE: While running :file:`.fio_smalloc.*` backing store files are visible
211 in :file:`/tmp`.
f80dba8d
MT
212
213.. option:: --warnings-fatal
214
b034c0dd
TK
215 All fio parser warnings are fatal, causing fio to exit with an
216 error.
f80dba8d
MT
217
218.. option:: --max-jobs=nr
219
f50fbdda 220 Set the maximum number of threads/processes to support to `nr`.
f80dba8d
MT
221
222.. option:: --server=args
223
b034c0dd
TK
224 Start a backend server, with `args` specifying what to listen to.
225 See `Client/Server`_ section.
f80dba8d
MT
226
227.. option:: --daemonize=pidfile
228
b034c0dd 229 Background a fio server, writing the pid to the given `pidfile` file.
f80dba8d
MT
230
231.. option:: --client=hostname
232
f50fbdda
TK
233 Instead of running the jobs locally, send and run them on the given `hostname`
234 or set of `hostname`s. See `Client/Server`_ section.
f80dba8d
MT
235
236.. option:: --remote-config=file
237
f50fbdda 238 Tell fio server to load this local `file`.
f80dba8d
MT
239
240.. option:: --idle-prof=option
241
b8f7e412 242 Report CPU idleness. `option` is one of the following:
113f0e7c
SW
243
244 **calibrate**
245 Run unit work calibration only and exit.
246
247 **system**
248 Show aggregate system idleness and unit work.
249
250 **percpu**
251 As **system** but also show per CPU idleness.
f80dba8d
MT
252
253.. option:: --inflate-log=log
254
f50fbdda 255 Inflate and output compressed `log`.
f80dba8d
MT
256
257.. option:: --trigger-file=file
258
f50fbdda 259 Execute trigger command when `file` exists.
f80dba8d 260
f50fbdda 261.. option:: --trigger-timeout=time
f80dba8d 262
f50fbdda 263 Execute trigger at this `time`.
f80dba8d 264
f50fbdda 265.. option:: --trigger=command
f80dba8d 266
f50fbdda 267 Set this `command` as local trigger.
f80dba8d 268
f50fbdda 269.. option:: --trigger-remote=command
f80dba8d 270
f50fbdda 271 Set this `command` as remote trigger.
f80dba8d
MT
272
273.. option:: --aux-path=path
274
f50fbdda 275 Use this `path` for fio state generated files.
f80dba8d
MT
276
277Any parameters following the options will be assumed to be job files, unless
278they match a job file parameter. Multiple job files can be listed and each job
279file will be regarded as a separate group. Fio will :option:`stonewall`
280execution between each group.
281
282
283Job file format
284---------------
285
286As previously described, fio accepts one or more job files describing what it is
287supposed to do. The job file format is the classic ini file, where the names
c60ebc45 288enclosed in [] brackets define the job name. You are free to use any ASCII name
f80dba8d
MT
289you want, except *global* which has special meaning. Following the job name is
290a sequence of zero or more parameters, one per line, that define the behavior of
291the job. If the first character in a line is a ';' or a '#', the entire line is
292discarded as a comment.
293
294A *global* section sets defaults for the jobs described in that file. A job may
295override a *global* section parameter, and a job file may even have several
296*global* sections if so desired. A job is only affected by a *global* section
297residing above it.
298
f50fbdda
TK
299The :option:`--cmdhelp` option also lists all options. If used with a `command`
300argument, :option:`--cmdhelp` will detail the given `command`.
f80dba8d
MT
301
302See the `examples/` directory for inspiration on how to write job files. Note
303the copyright and license requirements currently apply to `examples/` files.
304
305So let's look at a really simple job file that defines two processes, each
306randomly reading from a 128MiB file:
307
308.. code-block:: ini
309
310 ; -- start job file --
311 [global]
312 rw=randread
313 size=128m
314
315 [job1]
316
317 [job2]
318
319 ; -- end job file --
320
321As you can see, the job file sections themselves are empty as all the described
322parameters are shared. As no :option:`filename` option is given, fio makes up a
323`filename` for each of the jobs as it sees fit. On the command line, this job
324would look as follows::
325
326$ fio --name=global --rw=randread --size=128m --name=job1 --name=job2
327
328
329Let's look at an example that has a number of processes writing randomly to
330files:
331
332.. code-block:: ini
333
334 ; -- start job file --
335 [random-writers]
336 ioengine=libaio
337 iodepth=4
338 rw=randwrite
339 bs=32k
340 direct=0
341 size=64m
342 numjobs=4
343 ; -- end job file --
344
345Here we have no *global* section, as we only have one job defined anyway. We
346want to use async I/O here, with a depth of 4 for each file. We also increased
347the buffer size used to 32KiB and define numjobs to 4 to fork 4 identical
348jobs. The result is 4 processes each randomly writing to their own 64MiB
349file. Instead of using the above job file, you could have given the parameters
350on the command line. For this case, you would specify::
351
352$ fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4
353
354When fio is utilized as a basis of any reasonably large test suite, it might be
355desirable to share a set of standardized settings across multiple job files.
356Instead of copy/pasting such settings, any section may pull in an external
357:file:`filename.fio` file with *include filename* directive, as in the following
358example::
359
360 ; -- start job file including.fio --
361 [global]
362 filename=/tmp/test
363 filesize=1m
364 include glob-include.fio
365
366 [test]
367 rw=randread
368 bs=4k
369 time_based=1
370 runtime=10
371 include test-include.fio
372 ; -- end job file including.fio --
373
374.. code-block:: ini
375
376 ; -- start job file glob-include.fio --
377 thread=1
378 group_reporting=1
379 ; -- end job file glob-include.fio --
380
381.. code-block:: ini
382
383 ; -- start job file test-include.fio --
384 ioengine=libaio
385 iodepth=4
386 ; -- end job file test-include.fio --
387
388Settings pulled into a section apply to that section only (except *global*
389section). Include directives may be nested in that any included file may contain
390further include directive(s). Include files may not contain [] sections.
391
392
393Environment variables
394~~~~~~~~~~~~~~~~~~~~~
395
396Fio also supports environment variable expansion in job files. Any sub-string of
397the form ``${VARNAME}`` as part of an option value (in other words, on the right
398of the '='), will be expanded to the value of the environment variable called
399`VARNAME`. If no such environment variable is defined, or `VARNAME` is the
400empty string, the empty string will be substituted.
401
402As an example, let's look at a sample fio invocation and job file::
403
404$ SIZE=64m NUMJOBS=4 fio jobfile.fio
405
406.. code-block:: ini
407
408 ; -- start job file --
409 [random-writers]
410 rw=randwrite
411 size=${SIZE}
412 numjobs=${NUMJOBS}
413 ; -- end job file --
414
415This will expand to the following equivalent job file at runtime:
416
417.. code-block:: ini
418
419 ; -- start job file --
420 [random-writers]
421 rw=randwrite
422 size=64m
423 numjobs=4
424 ; -- end job file --
425
426Fio ships with a few example job files, you can also look there for inspiration.
427
428Reserved keywords
429~~~~~~~~~~~~~~~~~
430
431Additionally, fio has a set of reserved keywords that will be replaced
432internally with the appropriate value. Those keywords are:
433
434**$pagesize**
435
436 The architecture page size of the running system.
437
438**$mb_memory**
439
440 Megabytes of total memory in the system.
441
442**$ncpus**
443
444 Number of online available CPUs.
445
446These can be used on the command line or in the job file, and will be
447automatically substituted with the current system values when the job is
448run. Simple math is also supported on these keywords, so you can perform actions
449like::
450
b034c0dd 451 size=8*$mb_memory
f80dba8d
MT
452
453and get that properly expanded to 8 times the size of memory in the machine.
454
455
456Job file parameters
457-------------------
458
459This section describes in details each parameter associated with a job. Some
460parameters take an option of a given type, such as an integer or a
461string. Anywhere a numeric value is required, an arithmetic expression may be
462used, provided it is surrounded by parentheses. Supported operators are:
463
464 - addition (+)
465 - subtraction (-)
466 - multiplication (*)
467 - division (/)
468 - modulus (%)
469 - exponentiation (^)
470
471For time values in expressions, units are microseconds by default. This is
472different than for time values not in expressions (not enclosed in
473parentheses). The following types are used:
474
475
476Parameter types
477~~~~~~~~~~~~~~~
478
479**str**
b034c0dd 480 String: A sequence of alphanumeric characters.
f80dba8d
MT
481
482**time**
008d0feb
SW
483 Integer with possible time suffix. Without a unit value is interpreted as
484 seconds unless otherwise specified. Accepts a suffix of 'd' for days, 'h' for
485 hours, 'm' for minutes, 's' for seconds, 'ms' (or 'msec') for milliseconds and
486 'us' (or 'usec') for microseconds. For example, use 10m for 10 minutes.
f80dba8d
MT
487
488.. _int:
489
490**int**
491 Integer. A whole number value, which may contain an integer prefix
492 and an integer suffix:
493
b034c0dd 494 [*integer prefix*] **number** [*integer suffix*]
f80dba8d
MT
495
496 The optional *integer prefix* specifies the number's base. The default
497 is decimal. *0x* specifies hexadecimal.
498
499 The optional *integer suffix* specifies the number's units, and includes an
500 optional unit prefix and an optional unit. For quantities of data, the
947e0fe0
SW
501 default unit is bytes. For quantities of time, the default unit is seconds
502 unless otherwise specified.
f80dba8d 503
9207a0cb 504 With :option:`kb_base`\=1000, fio follows international standards for unit
f80dba8d
MT
505 prefixes. To specify power-of-10 decimal values defined in the
506 International System of Units (SI):
507
eccce61a
TK
508 * *K* -- means kilo (K) or 1000
509 * *M* -- means mega (M) or 1000**2
510 * *G* -- means giga (G) or 1000**3
511 * *T* -- means tera (T) or 1000**4
512 * *P* -- means peta (P) or 1000**5
f80dba8d
MT
513
514 To specify power-of-2 binary values defined in IEC 80000-13:
515
eccce61a
TK
516 * *Ki* -- means kibi (Ki) or 1024
517 * *Mi* -- means mebi (Mi) or 1024**2
518 * *Gi* -- means gibi (Gi) or 1024**3
519 * *Ti* -- means tebi (Ti) or 1024**4
520 * *Pi* -- means pebi (Pi) or 1024**5
f80dba8d 521
9207a0cb 522 With :option:`kb_base`\=1024 (the default), the unit prefixes are opposite
f80dba8d
MT
523 from those specified in the SI and IEC 80000-13 standards to provide
524 compatibility with old scripts. For example, 4k means 4096.
525
526 For quantities of data, an optional unit of 'B' may be included
b8f7e412 527 (e.g., 'kB' is the same as 'k').
f80dba8d
MT
528
529 The *integer suffix* is not case sensitive (e.g., m/mi mean mebi/mega,
530 not milli). 'b' and 'B' both mean byte, not bit.
531
9207a0cb 532 Examples with :option:`kb_base`\=1000:
f80dba8d
MT
533
534 * *4 KiB*: 4096, 4096b, 4096B, 4ki, 4kib, 4kiB, 4Ki, 4KiB
535 * *1 MiB*: 1048576, 1mi, 1024ki
536 * *1 MB*: 1000000, 1m, 1000k
537 * *1 TiB*: 1099511627776, 1ti, 1024gi, 1048576mi
538 * *1 TB*: 1000000000, 1t, 1000m, 1000000k
539
9207a0cb 540 Examples with :option:`kb_base`\=1024 (default):
f80dba8d
MT
541
542 * *4 KiB*: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
543 * *1 MiB*: 1048576, 1m, 1024k
544 * *1 MB*: 1000000, 1mi, 1000ki
545 * *1 TiB*: 1099511627776, 1t, 1024g, 1048576m
546 * *1 TB*: 1000000000, 1ti, 1000mi, 1000000ki
547
548 To specify times (units are not case sensitive):
549
550 * *D* -- means days
551 * *H* -- means hours
4502cb42 552 * *M* -- means minutes
f80dba8d
MT
553 * *s* -- or sec means seconds (default)
554 * *ms* -- or *msec* means milliseconds
555 * *us* -- or *usec* means microseconds
556
557 If the option accepts an upper and lower range, use a colon ':' or
558 minus '-' to separate such values. See :ref:`irange <irange>`.
4502cb42
SW
559 If the lower value specified happens to be larger than the upper value
560 the two values are swapped.
f80dba8d
MT
561
562.. _bool:
563
564**bool**
565 Boolean. Usually parsed as an integer, however only defined for
566 true and false (1 and 0).
567
568.. _irange:
569
570**irange**
571 Integer range with suffix. Allows value range to be given, such as
c60ebc45 572 1024-4096. A colon may also be used as the separator, e.g. 1k:4k. If the
f80dba8d
MT
573 option allows two sets of ranges, they can be specified with a ',' or '/'
574 delimiter: 1k-4k/8k-32k. Also see :ref:`int <int>`.
575
576**float_list**
577 A list of floating point numbers, separated by a ':' character.
578
f5c3bcf2
TK
579With the above in mind, here follows the complete list of fio job parameters.
580
f80dba8d
MT
581
582Units
583~~~~~
584
585.. option:: kb_base=int
586
587 Select the interpretation of unit prefixes in input parameters.
588
589 **1000**
590 Inputs comply with IEC 80000-13 and the International
591 System of Units (SI). Use:
592
593 - power-of-2 values with IEC prefixes (e.g., KiB)
594 - power-of-10 values with SI prefixes (e.g., kB)
595
596 **1024**
597 Compatibility mode (default). To avoid breaking old scripts:
598
599 - power-of-2 values with SI prefixes
600 - power-of-10 values with IEC prefixes
601
602 See :option:`bs` for more details on input parameters.
603
604 Outputs always use correct prefixes. Most outputs include both
605 side-by-side, like::
606
607 bw=2383.3kB/s (2327.4KiB/s)
608
609 If only one value is reported, then kb_base selects the one to use:
610
611 **1000** -- SI prefixes
612
613 **1024** -- IEC prefixes
614
615.. option:: unit_base=int
616
617 Base unit for reporting. Allowed values are:
618
619 **0**
620 Use auto-detection (default).
621 **8**
622 Byte based.
623 **1**
624 Bit based.
625
626
f80dba8d
MT
627Job description
628~~~~~~~~~~~~~~~
629
630.. option:: name=str
631
632 ASCII name of the job. This may be used to override the name printed by fio
633 for this job. Otherwise the job name is used. On the command line this
634 parameter has the special purpose of also signaling the start of a new job.
635
636.. option:: description=str
637
638 Text description of the job. Doesn't do anything except dump this text
639 description when this job is run. It's not parsed.
640
641.. option:: loops=int
642
643 Run the specified number of iterations of this job. Used to repeat the same
644 workload a given number of times. Defaults to 1.
645
646.. option:: numjobs=int
647
79591fa9
TK
648 Create the specified number of clones of this job. Each clone of job
649 is spawned as an independent thread or process. May be used to setup a
f80dba8d
MT
650 larger number of threads/processes doing the same thing. Each thread is
651 reported separately; to see statistics for all clones as a whole, use
652 :option:`group_reporting` in conjunction with :option:`new_group`.
a47b697c 653 See :option:`--max-jobs`. Default: 1.
f80dba8d
MT
654
655
656Time related parameters
657~~~~~~~~~~~~~~~~~~~~~~~
658
659.. option:: runtime=time
660
f75ede1d 661 Tell fio to terminate processing after the specified period of time. It
f80dba8d 662 can be quite hard to determine for how long a specified job will run, so
f75ede1d 663 this parameter is handy to cap the total runtime to a given time. When
947e0fe0 664 the unit is omitted, the value is intepreted in seconds.
f80dba8d
MT
665
666.. option:: time_based
667
668 If set, fio will run for the duration of the :option:`runtime` specified
669 even if the file(s) are completely read or written. It will simply loop over
670 the same workload as many times as the :option:`runtime` allows.
671
a881438b 672.. option:: startdelay=irange(time)
f80dba8d 673
947e0fe0
SW
674 Delay the start of job for the specified amount of time. Can be a single
675 value or a range. When given as a range, each thread will choose a value
676 randomly from within the range. Value is in seconds if a unit is omitted.
f80dba8d
MT
677
678.. option:: ramp_time=time
679
680 If set, fio will run the specified workload for this amount of time before
681 logging any performance numbers. Useful for letting performance settle
682 before logging results, thus minimizing the runtime required for stable
683 results. Note that the ``ramp_time`` is considered lead in time for a job,
684 thus it will increase the total runtime if a special timeout or
f75ede1d
SW
685 :option:`runtime` is specified. When the unit is omitted, the value is
686 given in seconds.
f80dba8d
MT
687
688.. option:: clocksource=str
689
690 Use the given clocksource as the base of timing. The supported options are:
691
692 **gettimeofday**
693 :manpage:`gettimeofday(2)`
694
695 **clock_gettime**
696 :manpage:`clock_gettime(2)`
697
698 **cpu**
699 Internal CPU clock source
700
701 cpu is the preferred clocksource if it is reliable, as it is very fast (and
702 fio is heavy on time calls). Fio will automatically use this clocksource if
703 it's supported and considered reliable on the system it is running on,
704 unless another clocksource is specifically set. For x86/x86-64 CPUs, this
705 means supporting TSC Invariant.
706
707.. option:: gtod_reduce=bool
708
709 Enable all of the :manpage:`gettimeofday(2)` reducing options
f75ede1d 710 (:option:`disable_clat`, :option:`disable_slat`, :option:`disable_bw_measurement`) plus
f80dba8d
MT
711 reduce precision of the timeout somewhat to really shrink the
712 :manpage:`gettimeofday(2)` call count. With this option enabled, we only do
713 about 0.4% of the :manpage:`gettimeofday(2)` calls we would have done if all
714 time keeping was enabled.
715
716.. option:: gtod_cpu=int
717
718 Sometimes it's cheaper to dedicate a single thread of execution to just
719 getting the current time. Fio (and databases, for instance) are very
720 intensive on :manpage:`gettimeofday(2)` calls. With this option, you can set
721 one CPU aside for doing nothing but logging current time to a shared memory
722 location. Then the other threads/processes that run I/O workloads need only
723 copy that segment, instead of entering the kernel with a
724 :manpage:`gettimeofday(2)` call. The CPU set aside for doing these time
725 calls will be excluded from other uses. Fio will manually clear it from the
726 CPU mask of other jobs.
727
728
729Target file/device
730~~~~~~~~~~~~~~~~~~
731
732.. option:: directory=str
733
734 Prefix filenames with this directory. Used to place files in a different
735 location than :file:`./`. You can specify a number of directories by
736 separating the names with a ':' character. These directories will be
02dd2689 737 assigned equally distributed to job clones created by :option:`numjobs` as
f80dba8d
MT
738 long as they are using generated filenames. If specific `filename(s)` are
739 set fio will use the first listed directory, and thereby matching the
740 `filename` semantic which generates a file each clone if not specified, but
741 let all clones use the same if set.
742
02dd2689
SW
743 See the :option:`filename` option for information on how to escape "``:``" and
744 "``\``" characters within the directory path itself.
f80dba8d
MT
745
746.. option:: filename=str
747
748 Fio normally makes up a `filename` based on the job name, thread number, and
02dd2689
SW
749 file number (see :option:`filename_format`). If you want to share files
750 between threads in a job or several
79591fa9
TK
751 jobs with fixed file paths, specify a `filename` for each of them to override
752 the default. If the ioengine is file based, you can specify a number of files
753 by separating the names with a ':' colon. So if you wanted a job to open
754 :file:`/dev/sda` and :file:`/dev/sdb` as the two working files, you would use
755 ``filename=/dev/sda:/dev/sdb``. This also means that whenever this option is
756 specified, :option:`nrfiles` is ignored. The size of regular files specified
02dd2689 757 by this option will be :option:`size` divided by number of files unless an
79591fa9
TK
758 explicit size is specified by :option:`filesize`.
759
02dd2689
SW
760 Each colon and backslash in the wanted path must be escaped with a ``\``
761 character. For instance, if the path is :file:`/dev/dsk/foo@3,0:c` then you
762 would use ``filename=/dev/dsk/foo@3,0\:c`` and if the path is
763 :file:`F:\\filename` then you would use ``filename=F\:\\filename``.
764
f80dba8d
MT
765 On Windows, disk devices are accessed as :file:`\\\\.\\PhysicalDrive0` for
766 the first device, :file:`\\\\.\\PhysicalDrive1` for the second etc.
767 Note: Windows and FreeBSD prevent write access to areas
02dd2689
SW
768 of the disk containing in-use data (e.g. filesystems).
769
770 The filename "`-`" is a reserved name, meaning *stdin* or *stdout*. Which
771 of the two depends on the read/write direction set.
f80dba8d
MT
772
773.. option:: filename_format=str
774
775 If sharing multiple files between jobs, it is usually necessary to have fio
776 generate the exact names that you want. By default, fio will name a file
777 based on the default file format specification of
778 :file:`jobname.jobnumber.filenumber`. With this option, that can be
779 customized. Fio will recognize and replace the following keywords in this
780 string:
781
782 **$jobname**
783 The name of the worker thread or process.
784 **$jobnum**
785 The incremental number of the worker thread or process.
786 **$filenum**
787 The incremental number of the file for that worker thread or
788 process.
789
790 To have dependent jobs share a set of files, this option can be set to have
791 fio generate filenames that are shared between the two. For instance, if
792 :file:`testfiles.$filenum` is specified, file number 4 for any job will be
793 named :file:`testfiles.4`. The default of :file:`$jobname.$jobnum.$filenum`
794 will be used if no other format specifier is given.
795
796.. option:: unique_filename=bool
797
798 To avoid collisions between networked clients, fio defaults to prefixing any
799 generated filenames (with a directory specified) with the source of the
800 client connecting. To disable this behavior, set this option to 0.
801
802.. option:: opendir=str
803
804 Recursively open any files below directory `str`.
805
806.. option:: lockfile=str
807
808 Fio defaults to not locking any files before it does I/O to them. If a file
809 or file descriptor is shared, fio can serialize I/O to that file to make the
810 end result consistent. This is usual for emulating real workloads that share
811 files. The lock modes are:
812
813 **none**
814 No locking. The default.
815 **exclusive**
816 Only one thread or process may do I/O at a time, excluding all
817 others.
818 **readwrite**
819 Read-write locking on the file. Many readers may
820 access the file at the same time, but writes get exclusive access.
821
822.. option:: nrfiles=int
823
79591fa9
TK
824 Number of files to use for this job. Defaults to 1. The size of files
825 will be :option:`size` divided by this unless explicit size is specified by
826 :option:`filesize`. Files are created for each thread separately, and each
827 file will have a file number within its name by default, as explained in
828 :option:`filename` section.
829
f80dba8d
MT
830
831.. option:: openfiles=int
832
833 Number of files to keep open at the same time. Defaults to the same as
834 :option:`nrfiles`, can be set smaller to limit the number simultaneous
835 opens.
836
837.. option:: file_service_type=str
838
839 Defines how fio decides which file from a job to service next. The following
840 types are defined:
841
842 **random**
843 Choose a file at random.
844
845 **roundrobin**
846 Round robin over opened files. This is the default.
847
848 **sequential**
849 Finish one file before moving on to the next. Multiple files can
f50fbdda 850 still be open depending on :option:`openfiles`.
f80dba8d
MT
851
852 **zipf**
c60ebc45 853 Use a *Zipf* distribution to decide what file to access.
f80dba8d
MT
854
855 **pareto**
c60ebc45 856 Use a *Pareto* distribution to decide what file to access.
f80dba8d 857
dd3503d3 858 **normal**
c60ebc45 859 Use a *Gaussian* (normal) distribution to decide what file to
f80dba8d
MT
860 access.
861
dd3503d3
SW
862 **gauss**
863 Alias for normal.
864
f80dba8d
MT
865 For *random*, *roundrobin*, and *sequential*, a postfix can be appended to
866 tell fio how many I/Os to issue before switching to a new file. For example,
867 specifying ``file_service_type=random:8`` would cause fio to issue
868 8 I/Os before selecting a new file at random. For the non-uniform
869 distributions, a floating point postfix can be given to influence how the
870 distribution is skewed. See :option:`random_distribution` for a description
871 of how that would work.
872
873.. option:: ioscheduler=str
874
875 Attempt to switch the device hosting the file to the specified I/O scheduler
876 before running.
877
878.. option:: create_serialize=bool
879
880 If true, serialize the file creation for the jobs. This may be handy to
881 avoid interleaving of data files, which may greatly depend on the filesystem
a47b697c 882 used and even the number of processors in the system. Default: true.
f80dba8d
MT
883
884.. option:: create_fsync=bool
885
22413915 886 :manpage:`fsync(2)` the data file after creation. This is the default.
f80dba8d
MT
887
888.. option:: create_on_open=bool
889
730bd7d9
SW
890 If true, don't pre-create files but allow the job's open() to create a file
891 when it's time to do I/O. Default: false -- pre-create all necessary files
892 when the job starts.
f80dba8d
MT
893
894.. option:: create_only=bool
895
896 If true, fio will only run the setup phase of the job. If files need to be
4502cb42 897 laid out or updated on disk, only that will be done -- the actual job contents
a47b697c 898 are not executed. Default: false.
f80dba8d
MT
899
900.. option:: allow_file_create=bool
901
730bd7d9
SW
902 If true, fio is permitted to create files as part of its workload. If this
903 option is false, then fio will error out if
f80dba8d
MT
904 the files it needs to use don't already exist. Default: true.
905
906.. option:: allow_mounted_write=bool
907
c60ebc45 908 If this isn't set, fio will abort jobs that are destructive (e.g. that write)
f80dba8d
MT
909 to what appears to be a mounted device or partition. This should help catch
910 creating inadvertently destructive tests, not realizing that the test will
b1db0375
TK
911 destroy data on the mounted file system. Note that some platforms don't allow
912 writing against a mounted device regardless of this option. Default: false.
f80dba8d
MT
913
914.. option:: pre_read=bool
915
916 If this is given, files will be pre-read into memory before starting the
917 given I/O operation. This will also clear the :option:`invalidate` flag,
918 since it is pointless to pre-read and then drop the cache. This will only
919 work for I/O engines that are seek-able, since they allow you to read the
a47b697c
SW
920 same data multiple times. Thus it will not work on non-seekable I/O engines
921 (e.g. network, splice). Default: false.
f80dba8d
MT
922
923.. option:: unlink=bool
924
925 Unlink the job files when done. Not the default, as repeated runs of that
a47b697c
SW
926 job would then waste time recreating the file set again and again. Default:
927 false.
f80dba8d
MT
928
929.. option:: unlink_each_loop=bool
930
a47b697c 931 Unlink job files after each iteration or loop. Default: false.
f80dba8d
MT
932
933.. option:: zonesize=int
934
935 Divide a file into zones of the specified size. See :option:`zoneskip`.
936
937.. option:: zonerange=int
938
939 Give size of an I/O zone. See :option:`zoneskip`.
940
941.. option:: zoneskip=int
942
943 Skip the specified number of bytes when :option:`zonesize` data has been
944 read. The two zone options can be used to only do I/O on zones of a file.
945
946
947I/O type
948~~~~~~~~
949
950.. option:: direct=bool
951
952 If value is true, use non-buffered I/O. This is usually O_DIRECT. Note that
953 ZFS on Solaris doesn't support direct I/O. On Windows the synchronous
954 ioengines don't support direct I/O. Default: false.
955
956.. option:: atomic=bool
957
958 If value is true, attempt to use atomic direct I/O. Atomic writes are
959 guaranteed to be stable once acknowledged by the operating system. Only
960 Linux supports O_ATOMIC right now.
961
962.. option:: buffered=bool
963
964 If value is true, use buffered I/O. This is the opposite of the
965 :option:`direct` option. Defaults to true.
966
967.. option:: readwrite=str, rw=str
968
969 Type of I/O pattern. Accepted values are:
970
971 **read**
972 Sequential reads.
973 **write**
974 Sequential writes.
975 **trim**
976 Sequential trims (Linux block devices only).
f80dba8d
MT
977 **randread**
978 Random reads.
2831be97
SW
979 **randwrite**
980 Random writes.
f80dba8d
MT
981 **randtrim**
982 Random trims (Linux block devices only).
983 **rw,readwrite**
984 Sequential mixed reads and writes.
985 **randrw**
986 Random mixed reads and writes.
987 **trimwrite**
988 Sequential trim+write sequences. Blocks will be trimmed first,
989 then the same blocks will be written to.
990
991 Fio defaults to read if the option is not specified. For the mixed I/O
992 types, the default is to split them 50/50. For certain types of I/O the
730bd7d9
SW
993 result may still be skewed a bit, since the speed may be different.
994
995 It is possible to specify the number of I/Os to do before getting a new
996 offset by appending ``:<nr>`` to the end of the string given. For a
f80dba8d
MT
997 random read, it would look like ``rw=randread:8`` for passing in an offset
998 modifier with a value of 8. If the suffix is used with a sequential I/O
730bd7d9
SW
999 pattern, then the *<nr>* value specified will be **added** to the generated
1000 offset for each I/O turning sequential I/O into sequential I/O with holes.
1001 For instance, using ``rw=write:4k`` will skip 4k for every write. Also see
1002 the :option:`rw_sequencer` option.
f80dba8d
MT
1003
1004.. option:: rw_sequencer=str
1005
1006 If an offset modifier is given by appending a number to the ``rw=<str>``
1007 line, then this option controls how that number modifies the I/O offset
1008 being generated. Accepted values are:
1009
1010 **sequential**
1011 Generate sequential offset.
1012 **identical**
1013 Generate the same offset.
1014
1015 ``sequential`` is only useful for random I/O, where fio would normally
c60ebc45 1016 generate a new random offset for every I/O. If you append e.g. 8 to randread,
007c7be9
SW
1017 you would get a new random offset for every 8 I/Os. The result would be a
1018 seek for only every 8 I/Os, instead of for every I/O. Use ``rw=randread:8``
f80dba8d
MT
1019 to specify that. As sequential I/O is already sequential, setting
1020 ``sequential`` for that would not result in any differences. ``identical``
1021 behaves in a similar fashion, except it sends the same offset 8 number of
1022 times before generating a new offset.
1023
1024.. option:: unified_rw_reporting=bool
1025
1026 Fio normally reports statistics on a per data direction basis, meaning that
1027 reads, writes, and trims are accounted and reported separately. If this
1028 option is set fio sums the results and report them as "mixed" instead.
1029
1030.. option:: randrepeat=bool
1031
1032 Seed the random number generator used for random I/O patterns in a
1033 predictable way so the pattern is repeatable across runs. Default: true.
1034
1035.. option:: allrandrepeat=bool
1036
1037 Seed all random number generators in a predictable way so results are
1038 repeatable across runs. Default: false.
1039
1040.. option:: randseed=int
1041
1042 Seed the random number generators based on this seed value, to be able to
1043 control what sequence of output is being generated. If not set, the random
1044 sequence depends on the :option:`randrepeat` setting.
1045
1046.. option:: fallocate=str
1047
1048 Whether pre-allocation is performed when laying down files.
1049 Accepted values are:
1050
1051 **none**
1052 Do not pre-allocate space.
1053
2c3e17be
SW
1054 **native**
1055 Use a platform's native pre-allocation call but fall back to
1056 **none** behavior if it fails/is not implemented.
1057
f80dba8d
MT
1058 **posix**
1059 Pre-allocate via :manpage:`posix_fallocate(3)`.
1060
1061 **keep**
1062 Pre-allocate via :manpage:`fallocate(2)` with
1063 FALLOC_FL_KEEP_SIZE set.
1064
1065 **0**
1066 Backward-compatible alias for **none**.
1067
1068 **1**
1069 Backward-compatible alias for **posix**.
1070
1071 May not be available on all supported platforms. **keep** is only available
2c3e17be
SW
1072 on Linux. If using ZFS on Solaris this cannot be set to **posix**
1073 because ZFS doesn't support pre-allocation. Default: **native** if any
1074 pre-allocation methods are available, **none** if not.
f80dba8d
MT
1075
1076.. option:: fadvise_hint=str
1077
1078 Use :manpage:`posix_fadvise(2)` to advise the kernel on what I/O patterns
1079 are likely to be issued. Accepted values are:
1080
1081 **0**
1082 Backwards-compatible hint for "no hint".
1083
1084 **1**
1085 Backwards compatible hint for "advise with fio workload type". This
1086 uses **FADV_RANDOM** for a random workload, and **FADV_SEQUENTIAL**
1087 for a sequential workload.
1088
1089 **sequential**
1090 Advise using **FADV_SEQUENTIAL**.
1091
1092 **random**
1093 Advise using **FADV_RANDOM**.
1094
8f4b9f24 1095.. option:: write_hint=str
f80dba8d 1096
8f4b9f24
JA
1097 Use :manpage:`fcntl(2)` to advise the kernel what life time to expect
1098 from a write. Only supported on Linux, as of version 4.13. Accepted
1099 values are:
1100
1101 **none**
1102 No particular life time associated with this file.
1103
1104 **short**
1105 Data written to this file has a short life time.
1106
1107 **medium**
1108 Data written to this file has a medium life time.
1109
1110 **long**
1111 Data written to this file has a long life time.
1112
1113 **extreme**
1114 Data written to this file has a very long life time.
1115
1116 The values are all relative to each other, and no absolute meaning
1117 should be associated with them.
f80dba8d
MT
1118
1119.. option:: offset=int
1120
82dbb8cb
TK
1121 Start I/O at the provided offset in the file, given as either a fixed size in
1122 bytes or a percentage. If a percentage is given, the next ``blockalign``-ed
1123 offset will be used. Data before the given offset will not be touched. This
89978a6b
BW
1124 effectively caps the file size at `real_size - offset`. Can be combined with
1125 :option:`size` to constrain the start and end range of the I/O workload.
44bb1142
TK
1126 A percentage can be specified by a number between 1 and 100 followed by '%',
1127 for example, ``offset=20%`` to specify 20%.
f80dba8d
MT
1128
1129.. option:: offset_increment=int
1130
1131 If this is provided, then the real offset becomes `offset + offset_increment
1132 * thread_number`, where the thread number is a counter that starts at 0 and
1133 is incremented for each sub-job (i.e. when :option:`numjobs` option is
1134 specified). This option is useful if there are several jobs which are
1135 intended to operate on a file in parallel disjoint segments, with even
1136 spacing between the starting points.
1137
1138.. option:: number_ios=int
1139
c60ebc45 1140 Fio will normally perform I/Os until it has exhausted the size of the region
f80dba8d
MT
1141 set by :option:`size`, or if it exhaust the allocated time (or hits an error
1142 condition). With this setting, the range/size can be set independently of
c60ebc45 1143 the number of I/Os to perform. When fio reaches this number, it will exit
f80dba8d
MT
1144 normally and report status. Note that this does not extend the amount of I/O
1145 that will be done, it will only stop fio if this condition is met before
1146 other end-of-job criteria.
1147
1148.. option:: fsync=int
1149
730bd7d9
SW
1150 If writing to a file, issue an :manpage:`fsync(2)` (or its equivalent) of
1151 the dirty data for every number of blocks given. For example, if you give 32
1152 as a parameter, fio will sync the file after every 32 writes issued. If fio is
1153 using non-buffered I/O, we may not sync the file. The exception is the sg
1154 I/O engine, which synchronizes the disk cache anyway. Defaults to 0, which
1155 means fio does not periodically issue and wait for a sync to complete. Also
1156 see :option:`end_fsync` and :option:`fsync_on_close`.
f80dba8d
MT
1157
1158.. option:: fdatasync=int
1159
1160 Like :option:`fsync` but uses :manpage:`fdatasync(2)` to only sync data and
000a5f1c 1161 not metadata blocks. In Windows, FreeBSD, and DragonFlyBSD there is no
730bd7d9
SW
1162 :manpage:`fdatasync(2)` so this falls back to using :manpage:`fsync(2)`.
1163 Defaults to 0, which means fio does not periodically issue and wait for a
1164 data-only sync to complete.
f80dba8d
MT
1165
1166.. option:: write_barrier=int
1167
2831be97 1168 Make every `N-th` write a barrier write.
f80dba8d 1169
f50fbdda 1170.. option:: sync_file_range=str:int
f80dba8d 1171
f50fbdda 1172 Use :manpage:`sync_file_range(2)` for every `int` number of write
f80dba8d
MT
1173 operations. Fio will track range of writes that have happened since the last
1174 :manpage:`sync_file_range(2)` call. `str` can currently be one or more of:
1175
1176 **wait_before**
1177 SYNC_FILE_RANGE_WAIT_BEFORE
1178 **write**
1179 SYNC_FILE_RANGE_WRITE
1180 **wait_after**
1181 SYNC_FILE_RANGE_WAIT_AFTER
1182
1183 So if you do ``sync_file_range=wait_before,write:8``, fio would use
1184 ``SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE`` for every 8
1185 writes. Also see the :manpage:`sync_file_range(2)` man page. This option is
1186 Linux specific.
1187
1188.. option:: overwrite=bool
1189
1190 If true, writes to a file will always overwrite existing data. If the file
1191 doesn't already exist, it will be created before the write phase begins. If
1192 the file exists and is large enough for the specified write phase, nothing
a47b697c 1193 will be done. Default: false.
f80dba8d
MT
1194
1195.. option:: end_fsync=bool
1196
a47b697c
SW
1197 If true, :manpage:`fsync(2)` file contents when a write stage has completed.
1198 Default: false.
f80dba8d
MT
1199
1200.. option:: fsync_on_close=bool
1201
1202 If true, fio will :manpage:`fsync(2)` a dirty file on close. This differs
a47b697c
SW
1203 from :option:`end_fsync` in that it will happen on every file close, not
1204 just at the end of the job. Default: false.
f80dba8d
MT
1205
1206.. option:: rwmixread=int
1207
1208 Percentage of a mixed workload that should be reads. Default: 50.
1209
1210.. option:: rwmixwrite=int
1211
1212 Percentage of a mixed workload that should be writes. If both
1213 :option:`rwmixread` and :option:`rwmixwrite` is given and the values do not
1214 add up to 100%, the latter of the two will be used to override the
1215 first. This may interfere with a given rate setting, if fio is asked to
1216 limit reads or writes to a certain rate. If that is the case, then the
1217 distribution may be skewed. Default: 50.
1218
1219.. option:: random_distribution=str:float[,str:float][,str:float]
1220
1221 By default, fio will use a completely uniform random distribution when asked
1222 to perform random I/O. Sometimes it is useful to skew the distribution in
1223 specific ways, ensuring that some parts of the data is more hot than others.
1224 fio includes the following distribution models:
1225
1226 **random**
1227 Uniform random distribution
1228
1229 **zipf**
1230 Zipf distribution
1231
1232 **pareto**
1233 Pareto distribution
1234
b2f4b559 1235 **normal**
c60ebc45 1236 Normal (Gaussian) distribution
f80dba8d
MT
1237
1238 **zoned**
1239 Zoned random distribution
1240
1241 When using a **zipf** or **pareto** distribution, an input value is also
f50fbdda 1242 needed to define the access pattern. For **zipf**, this is the `Zipf
c60ebc45 1243 theta`. For **pareto**, it's the `Pareto power`. Fio includes a test
f50fbdda 1244 program, :command:`fio-genzipf`, that can be used visualize what the given input
f80dba8d
MT
1245 values will yield in terms of hit rates. If you wanted to use **zipf** with
1246 a `theta` of 1.2, you would use ``random_distribution=zipf:1.2`` as the
1247 option. If a non-uniform model is used, fio will disable use of the random
b2f4b559
SW
1248 map. For the **normal** distribution, a normal (Gaussian) deviation is
1249 supplied as a value between 0 and 100.
f80dba8d
MT
1250
1251 For a **zoned** distribution, fio supports specifying percentages of I/O
1252 access that should fall within what range of the file or device. For
1253 example, given a criteria of:
1254
f50fbdda
TK
1255 * 60% of accesses should be to the first 10%
1256 * 30% of accesses should be to the next 20%
1257 * 8% of accesses should be to the next 30%
1258 * 2% of accesses should be to the next 40%
f80dba8d
MT
1259
1260 we can define that through zoning of the random accesses. For the above
1261 example, the user would do::
1262
1263 random_distribution=zoned:60/10:30/20:8/30:2/40
1264
1265 similarly to how :option:`bssplit` works for setting ranges and percentages
1266 of block sizes. Like :option:`bssplit`, it's possible to specify separate
1267 zones for reads, writes, and trims. If just one set is given, it'll apply to
1268 all of them.
1269
1270.. option:: percentage_random=int[,int][,int]
1271
1272 For a random workload, set how big a percentage should be random. This
1273 defaults to 100%, in which case the workload is fully random. It can be set
1274 from anywhere from 0 to 100. Setting it to 0 would make the workload fully
1275 sequential. Any setting in between will result in a random mix of sequential
1276 and random I/O, at the given percentages. Comma-separated values may be
1277 specified for reads, writes, and trims as described in :option:`blocksize`.
1278
1279.. option:: norandommap
1280
1281 Normally fio will cover every block of the file when doing random I/O. If
1282 this option is given, fio will just get a new random offset without looking
1283 at past I/O history. This means that some blocks may not be read or written,
1284 and that some blocks may be read/written more than once. If this option is
1285 used with :option:`verify` and multiple blocksizes (via :option:`bsrange`),
1286 only intact blocks are verified, i.e., partially-overwritten blocks are
1287 ignored.
1288
1289.. option:: softrandommap=bool
1290
1291 See :option:`norandommap`. If fio runs with the random block map enabled and
1292 it fails to allocate the map, if this option is set it will continue without
1293 a random block map. As coverage will not be as complete as with random maps,
1294 this option is disabled by default.
1295
1296.. option:: random_generator=str
1297
f50fbdda 1298 Fio supports the following engines for generating I/O offsets for random I/O:
f80dba8d
MT
1299
1300 **tausworthe**
f50fbdda 1301 Strong 2^88 cycle random number generator.
f80dba8d 1302 **lfsr**
f50fbdda 1303 Linear feedback shift register generator.
f80dba8d 1304 **tausworthe64**
f50fbdda 1305 Strong 64-bit 2^258 cycle random number generator.
f80dba8d
MT
1306
1307 **tausworthe** is a strong random number generator, but it requires tracking
1308 on the side if we want to ensure that blocks are only read or written
f50fbdda 1309 once. **lfsr** guarantees that we never generate the same offset twice, and
f80dba8d 1310 it's also less computationally expensive. It's not a true random generator,
f50fbdda 1311 however, though for I/O purposes it's typically good enough. **lfsr** only
f80dba8d
MT
1312 works with single block sizes, not with workloads that use multiple block
1313 sizes. If used with such a workload, fio may read or write some blocks
1314 multiple times. The default value is **tausworthe**, unless the required
1315 space exceeds 2^32 blocks. If it does, then **tausworthe64** is
1316 selected automatically.
1317
1318
1319Block size
1320~~~~~~~~~~
1321
1322.. option:: blocksize=int[,int][,int], bs=int[,int][,int]
1323
1324 The block size in bytes used for I/O units. Default: 4096. A single value
1325 applies to reads, writes, and trims. Comma-separated values may be
1326 specified for reads, writes, and trims. A value not terminated in a comma
1327 applies to subsequent types.
1328
1329 Examples:
1330
1331 **bs=256k**
1332 means 256k for reads, writes and trims.
1333
1334 **bs=8k,32k**
1335 means 8k for reads, 32k for writes and trims.
1336
1337 **bs=8k,32k,**
1338 means 8k for reads, 32k for writes, and default for trims.
1339
1340 **bs=,8k**
1341 means default for reads, 8k for writes and trims.
1342
1343 **bs=,8k,**
b443ae44 1344 means default for reads, 8k for writes, and default for trims.
f80dba8d
MT
1345
1346.. option:: blocksize_range=irange[,irange][,irange], bsrange=irange[,irange][,irange]
1347
1348 A range of block sizes in bytes for I/O units. The issued I/O unit will
1349 always be a multiple of the minimum size, unless
1350 :option:`blocksize_unaligned` is set.
1351
1352 Comma-separated ranges may be specified for reads, writes, and trims as
1353 described in :option:`blocksize`.
1354
1355 Example: ``bsrange=1k-4k,2k-8k``.
1356
1357.. option:: bssplit=str[,str][,str]
1358
1359 Sometimes you want even finer grained control of the block sizes issued, not
1360 just an even split between them. This option allows you to weight various
1361 block sizes, so that you are able to define a specific amount of block sizes
1362 issued. The format for this option is::
1363
1364 bssplit=blocksize/percentage:blocksize/percentage
1365
1366 for as many block sizes as needed. So if you want to define a workload that
1367 has 50% 64k blocks, 10% 4k blocks, and 40% 32k blocks, you would write::
1368
1369 bssplit=4k/10:64k/50:32k/40
1370
1371 Ordering does not matter. If the percentage is left blank, fio will fill in
1372 the remaining values evenly. So a bssplit option like this one::
1373
1374 bssplit=4k/50:1k/:32k/
1375
1376 would have 50% 4k ios, and 25% 1k and 32k ios. The percentages always add up
1377 to 100, if bssplit is given a range that adds up to more, it will error out.
1378
1379 Comma-separated values may be specified for reads, writes, and trims as
1380 described in :option:`blocksize`.
1381
1382 If you want a workload that has 50% 2k reads and 50% 4k reads, while having
1383 90% 4k writes and 10% 8k writes, you would specify::
1384
1385 bssplit=2k/50:4k/50,4k/90,8k/10
1386
1387.. option:: blocksize_unaligned, bs_unaligned
1388
1389 If set, fio will issue I/O units with any size within
1390 :option:`blocksize_range`, not just multiples of the minimum size. This
1391 typically won't work with direct I/O, as that normally requires sector
1392 alignment.
1393
589e88b7 1394.. option:: bs_is_seq_rand=bool
f80dba8d
MT
1395
1396 If this option is set, fio will use the normal read,write blocksize settings
1397 as sequential,random blocksize settings instead. Any random read or write
1398 will use the WRITE blocksize settings, and any sequential read or write will
1399 use the READ blocksize settings.
1400
1401.. option:: blockalign=int[,int][,int], ba=int[,int][,int]
1402
1403 Boundary to which fio will align random I/O units. Default:
1404 :option:`blocksize`. Minimum alignment is typically 512b for using direct
1405 I/O, though it usually depends on the hardware block size. This option is
1406 mutually exclusive with using a random map for files, so it will turn off
1407 that option. Comma-separated values may be specified for reads, writes, and
1408 trims as described in :option:`blocksize`.
1409
1410
1411Buffers and memory
1412~~~~~~~~~~~~~~~~~~
1413
1414.. option:: zero_buffers
1415
1416 Initialize buffers with all zeros. Default: fill buffers with random data.
1417
1418.. option:: refill_buffers
1419
1420 If this option is given, fio will refill the I/O buffers on every
1421 submit. The default is to only fill it at init time and reuse that
1422 data. Only makes sense if zero_buffers isn't specified, naturally. If data
1423 verification is enabled, `refill_buffers` is also automatically enabled.
1424
1425.. option:: scramble_buffers=bool
1426
1427 If :option:`refill_buffers` is too costly and the target is using data
1428 deduplication, then setting this option will slightly modify the I/O buffer
1429 contents to defeat normal de-dupe attempts. This is not enough to defeat
1430 more clever block compression attempts, but it will stop naive dedupe of
1431 blocks. Default: true.
1432
1433.. option:: buffer_compress_percentage=int
1434
1435 If this is set, then fio will attempt to provide I/O buffer content (on
730bd7d9 1436 WRITEs) that compresses to the specified level. Fio does this by providing a
22413915 1437 mix of random data and a fixed pattern. The fixed pattern is either zeros,
f80dba8d
MT
1438 or the pattern specified by :option:`buffer_pattern`. If the pattern option
1439 is used, it might skew the compression ratio slightly. Note that this is per
1440 block size unit, for file/disk wide compression level that matches this
1441 setting, you'll also want to set :option:`refill_buffers`.
1442
1443.. option:: buffer_compress_chunk=int
1444
1445 See :option:`buffer_compress_percentage`. This setting allows fio to manage
1446 how big the ranges of random data and zeroed data is. Without this set, fio
1447 will provide :option:`buffer_compress_percentage` of blocksize random data,
1448 followed by the remaining zeroed. With this set to some chunk size smaller
1449 than the block size, fio can alternate random and zeroed data throughout the
1450 I/O buffer.
1451
1452.. option:: buffer_pattern=str
1453
a1554f65
SB
1454 If set, fio will fill the I/O buffers with this pattern or with the contents
1455 of a file. If not set, the contents of I/O buffers are defined by the other
1456 options related to buffer contents. The setting can be any pattern of bytes,
1457 and can be prefixed with 0x for hex values. It may also be a string, where
1458 the string must then be wrapped with ``""``. Or it may also be a filename,
1459 where the filename must be wrapped with ``''`` in which case the file is
1460 opened and read. Note that not all the file contents will be read if that
1461 would cause the buffers to overflow. So, for example::
1462
1463 buffer_pattern='filename'
1464
1465 or::
f80dba8d
MT
1466
1467 buffer_pattern="abcd"
1468
1469 or::
1470
1471 buffer_pattern=-12
1472
1473 or::
1474
1475 buffer_pattern=0xdeadface
1476
1477 Also you can combine everything together in any order::
1478
a1554f65 1479 buffer_pattern=0xdeadface"abcd"-12'filename'
f80dba8d
MT
1480
1481.. option:: dedupe_percentage=int
1482
1483 If set, fio will generate this percentage of identical buffers when
1484 writing. These buffers will be naturally dedupable. The contents of the
1485 buffers depend on what other buffer compression settings have been set. It's
1486 possible to have the individual buffers either fully compressible, or not at
1487 all. This option only controls the distribution of unique buffers.
1488
1489.. option:: invalidate=bool
1490
730bd7d9
SW
1491 Invalidate the buffer/page cache parts of the files to be used prior to
1492 starting I/O if the platform and file type support it. Defaults to true.
21c1b29e
TK
1493 This will be ignored if :option:`pre_read` is also specified for the
1494 same job.
f80dba8d
MT
1495
1496.. option:: sync=bool
1497
1498 Use synchronous I/O for buffered writes. For the majority of I/O engines,
1499 this means using O_SYNC. Default: false.
1500
1501.. option:: iomem=str, mem=str
1502
1503 Fio can use various types of memory as the I/O unit buffer. The allowed
1504 values are:
1505
1506 **malloc**
1507 Use memory from :manpage:`malloc(3)` as the buffers. Default memory
1508 type.
1509
1510 **shm**
1511 Use shared memory as the buffers. Allocated through
1512 :manpage:`shmget(2)`.
1513
1514 **shmhuge**
1515 Same as shm, but use huge pages as backing.
1516
1517 **mmap**
22413915 1518 Use :manpage:`mmap(2)` to allocate buffers. May either be anonymous memory, or can
f80dba8d
MT
1519 be file backed if a filename is given after the option. The format
1520 is `mem=mmap:/path/to/file`.
1521
1522 **mmaphuge**
1523 Use a memory mapped huge file as the buffer backing. Append filename
1524 after mmaphuge, ala `mem=mmaphuge:/hugetlbfs/file`.
1525
1526 **mmapshared**
1527 Same as mmap, but use a MMAP_SHARED mapping.
1528
03553853
YR
1529 **cudamalloc**
1530 Use GPU memory as the buffers for GPUDirect RDMA benchmark.
f50fbdda 1531 The :option:`ioengine` must be `rdma`.
03553853 1532
f80dba8d
MT
1533 The area allocated is a function of the maximum allowed bs size for the job,
1534 multiplied by the I/O depth given. Note that for **shmhuge** and
1535 **mmaphuge** to work, the system must have free huge pages allocated. This
1536 can normally be checked and set by reading/writing
1537 :file:`/proc/sys/vm/nr_hugepages` on a Linux system. Fio assumes a huge page
1538 is 4MiB in size. So to calculate the number of huge pages you need for a
1539 given job file, add up the I/O depth of all jobs (normally one unless
1540 :option:`iodepth` is used) and multiply by the maximum bs set. Then divide
1541 that number by the huge page size. You can see the size of the huge pages in
1542 :file:`/proc/meminfo`. If no huge pages are allocated by having a non-zero
1543 number in `nr_hugepages`, using **mmaphuge** or **shmhuge** will fail. Also
1544 see :option:`hugepage-size`.
1545
1546 **mmaphuge** also needs to have hugetlbfs mounted and the file location
1547 should point there. So if it's mounted in :file:`/huge`, you would use
1548 `mem=mmaphuge:/huge/somefile`.
1549
f50fbdda 1550.. option:: iomem_align=int, mem_align=int
f80dba8d
MT
1551
1552 This indicates the memory alignment of the I/O memory buffers. Note that
1553 the given alignment is applied to the first I/O unit buffer, if using
1554 :option:`iodepth` the alignment of the following buffers are given by the
1555 :option:`bs` used. In other words, if using a :option:`bs` that is a
1556 multiple of the page sized in the system, all buffers will be aligned to
1557 this value. If using a :option:`bs` that is not page aligned, the alignment
1558 of subsequent I/O memory buffers is the sum of the :option:`iomem_align` and
1559 :option:`bs` used.
1560
1561.. option:: hugepage-size=int
1562
1563 Defines the size of a huge page. Must at least be equal to the system
1564 setting, see :file:`/proc/meminfo`. Defaults to 4MiB. Should probably
1565 always be a multiple of megabytes, so using ``hugepage-size=Xm`` is the
1566 preferred way to set this to avoid setting a non-pow-2 bad value.
1567
1568.. option:: lockmem=int
1569
1570 Pin the specified amount of memory with :manpage:`mlock(2)`. Can be used to
1571 simulate a smaller amount of memory. The amount specified is per worker.
1572
1573
1574I/O size
1575~~~~~~~~
1576
1577.. option:: size=int
1578
79591fa9
TK
1579 The total size of file I/O for each thread of this job. Fio will run until
1580 this many bytes has been transferred, unless runtime is limited by other options
1581 (such as :option:`runtime`, for instance, or increased/decreased by :option:`io_size`).
1582 Fio will divide this size between the available files determined by options
1583 such as :option:`nrfiles`, :option:`filename`, unless :option:`filesize` is
1584 specified by the job. If the result of division happens to be 0, the size is
c4aa2d08 1585 set to the physical size of the given files or devices if they exist.
79591fa9 1586 If this option is not specified, fio will use the full size of the given
f80dba8d
MT
1587 files or devices. If the files do not exist, size must be given. It is also
1588 possible to give size as a percentage between 1 and 100. If ``size=20%`` is
1589 given, fio will use 20% of the full size of the given files or devices.
9d25d068
SW
1590 Can be combined with :option:`offset` to constrain the start and end range
1591 that I/O will be done within.
f80dba8d
MT
1592
1593.. option:: io_size=int, io_limit=int
1594
1595 Normally fio operates within the region set by :option:`size`, which means
1596 that the :option:`size` option sets both the region and size of I/O to be
1597 performed. Sometimes that is not what you want. With this option, it is
1598 possible to define just the amount of I/O that fio should do. For instance,
1599 if :option:`size` is set to 20GiB and :option:`io_size` is set to 5GiB, fio
1600 will perform I/O within the first 20GiB but exit when 5GiB have been
1601 done. The opposite is also possible -- if :option:`size` is set to 20GiB,
1602 and :option:`io_size` is set to 40GiB, then fio will do 40GiB of I/O within
1603 the 0..20GiB region.
1604
7fdd97ca 1605.. option:: filesize=irange(int)
f80dba8d
MT
1606
1607 Individual file sizes. May be a range, in which case fio will select sizes
1608 for files at random within the given range and limited to :option:`size` in
1609 total (if that is given). If not given, each created file is the same size.
79591fa9
TK
1610 This option overrides :option:`size` in terms of file size, which means
1611 this value is used as a fixed size or possible range of each file.
f80dba8d
MT
1612
1613.. option:: file_append=bool
1614
1615 Perform I/O after the end of the file. Normally fio will operate within the
1616 size of a file. If this option is set, then fio will append to the file
1617 instead. This has identical behavior to setting :option:`offset` to the size
1618 of a file. This option is ignored on non-regular files.
1619
1620.. option:: fill_device=bool, fill_fs=bool
1621
1622 Sets size to something really large and waits for ENOSPC (no space left on
1623 device) as the terminating condition. Only makes sense with sequential
1624 write. For a read workload, the mount point will be filled first then I/O
1625 started on the result. This option doesn't make sense if operating on a raw
1626 device node, since the size of that is already known by the file system.
1627 Additionally, writing beyond end-of-device will not return ENOSPC there.
1628
1629
1630I/O engine
1631~~~~~~~~~~
1632
1633.. option:: ioengine=str
1634
1635 Defines how the job issues I/O to the file. The following types are defined:
1636
1637 **sync**
1638 Basic :manpage:`read(2)` or :manpage:`write(2)`
1639 I/O. :manpage:`lseek(2)` is used to position the I/O location.
54227e6b 1640 See :option:`fsync` and :option:`fdatasync` for syncing write I/Os.
f80dba8d
MT
1641
1642 **psync**
1643 Basic :manpage:`pread(2)` or :manpage:`pwrite(2)` I/O. Default on
1644 all supported operating systems except for Windows.
1645
1646 **vsync**
1647 Basic :manpage:`readv(2)` or :manpage:`writev(2)` I/O. Will emulate
c60ebc45 1648 queuing by coalescing adjacent I/Os into a single submission.
f80dba8d
MT
1649
1650 **pvsync**
1651 Basic :manpage:`preadv(2)` or :manpage:`pwritev(2)` I/O.
1652
1653 **pvsync2**
1654 Basic :manpage:`preadv2(2)` or :manpage:`pwritev2(2)` I/O.
1655
1656 **libaio**
1657 Linux native asynchronous I/O. Note that Linux may only support
22413915 1658 queued behavior with non-buffered I/O (set ``direct=1`` or
f80dba8d
MT
1659 ``buffered=0``).
1660 This engine defines engine specific options.
1661
1662 **posixaio**
1663 POSIX asynchronous I/O using :manpage:`aio_read(3)` and
1664 :manpage:`aio_write(3)`.
1665
1666 **solarisaio**
1667 Solaris native asynchronous I/O.
1668
1669 **windowsaio**
1670 Windows native asynchronous I/O. Default on Windows.
1671
1672 **mmap**
1673 File is memory mapped with :manpage:`mmap(2)` and data copied
1674 to/from using :manpage:`memcpy(3)`.
1675
1676 **splice**
1677 :manpage:`splice(2)` is used to transfer the data and
1678 :manpage:`vmsplice(2)` to transfer data from user space to the
1679 kernel.
1680
1681 **sg**
1682 SCSI generic sg v3 I/O. May either be synchronous using the SG_IO
1683 ioctl, or if the target is an sg character device we use
1684 :manpage:`read(2)` and :manpage:`write(2)` for asynchronous
f50fbdda
TK
1685 I/O. Requires :option:`filename` option to specify either block or
1686 character devices.
f80dba8d
MT
1687
1688 **null**
1689 Doesn't transfer any data, just pretends to. This is mainly used to
1690 exercise fio itself and for debugging/testing purposes.
1691
1692 **net**
1693 Transfer over the network to given ``host:port``. Depending on the
1694 :option:`protocol` used, the :option:`hostname`, :option:`port`,
1695 :option:`listen` and :option:`filename` options are used to specify
1696 what sort of connection to make, while the :option:`protocol` option
1697 determines which protocol will be used. This engine defines engine
1698 specific options.
1699
1700 **netsplice**
1701 Like **net**, but uses :manpage:`splice(2)` and
1702 :manpage:`vmsplice(2)` to map data and send/receive.
1703 This engine defines engine specific options.
1704
1705 **cpuio**
1706 Doesn't transfer any data, but burns CPU cycles according to the
1707 :option:`cpuload` and :option:`cpuchunks` options. Setting
9207a0cb 1708 :option:`cpuload`\=85 will cause that job to do nothing but burn 85%
f50fbdda
TK
1709 of the CPU. In case of SMP machines, use :option:`numjobs`=<nr_of_cpu>
1710 to get desired CPU usage, as the cpuload only loads a
f80dba8d
MT
1711 single CPU at the desired rate. A job never finishes unless there is
1712 at least one non-cpuio job.
1713
1714 **guasi**
1715 The GUASI I/O engine is the Generic Userspace Asyncronous Syscall
1716 Interface approach to async I/O. See
1717
1718 http://www.xmailserver.org/guasi-lib.html
1719
1720 for more info on GUASI.
1721
1722 **rdma**
1723 The RDMA I/O engine supports both RDMA memory semantics
1724 (RDMA_WRITE/RDMA_READ) and channel semantics (Send/Recv) for the
1725 InfiniBand, RoCE and iWARP protocols.
1726
1727 **falloc**
1728 I/O engine that does regular fallocate to simulate data transfer as
1729 fio ioengine.
1730
1731 DDIR_READ
1732 does fallocate(,mode = FALLOC_FL_KEEP_SIZE,).
1733
1734 DDIR_WRITE
1735 does fallocate(,mode = 0).
1736
1737 DDIR_TRIM
1738 does fallocate(,mode = FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE).
1739
761cd093
SW
1740 **ftruncate**
1741 I/O engine that sends :manpage:`ftruncate(2)` operations in response
1742 to write (DDIR_WRITE) events. Each ftruncate issued sets the file's
f50fbdda 1743 size to the current block offset. :option:`blocksize` is ignored.
761cd093 1744
f80dba8d
MT
1745 **e4defrag**
1746 I/O engine that does regular EXT4_IOC_MOVE_EXT ioctls to simulate
1747 defragment activity in request to DDIR_WRITE event.
1748
1749 **rbd**
1750 I/O engine supporting direct access to Ceph Rados Block Devices
1751 (RBD) via librbd without the need to use the kernel rbd driver. This
1752 ioengine defines engine specific options.
1753
1754 **gfapi**
ac8ca2af
SW
1755 Using GlusterFS libgfapi sync interface to direct access to
1756 GlusterFS volumes without having to go through FUSE. This ioengine
f80dba8d
MT
1757 defines engine specific options.
1758
1759 **gfapi_async**
ac8ca2af
SW
1760 Using GlusterFS libgfapi async interface to direct access to
1761 GlusterFS volumes without having to go through FUSE. This ioengine
f80dba8d
MT
1762 defines engine specific options.
1763
1764 **libhdfs**
f50fbdda 1765 Read and write through Hadoop (HDFS). The :option:`filename` option
f80dba8d
MT
1766 is used to specify host,port of the hdfs name-node to connect. This
1767 engine interprets offsets a little differently. In HDFS, files once
e25c0c91
SW
1768 created cannot be modified so random writes are not possible. To
1769 imitate this the libhdfs engine expects a bunch of small files to be
1770 created over HDFS and will randomly pick a file from them
1771 based on the offset generated by fio backend (see the example
f80dba8d 1772 job file to create such files, use ``rw=write`` option). Please
e25c0c91
SW
1773 note, it may be necessary to set environment variables to work
1774 with HDFS/libhdfs properly. Each job uses its own connection to
f80dba8d
MT
1775 HDFS.
1776
1777 **mtd**
1778 Read, write and erase an MTD character device (e.g.,
1779 :file:`/dev/mtd0`). Discards are treated as erases. Depending on the
1780 underlying device type, the I/O may have to go in a certain pattern,
1781 e.g., on NAND, writing sequentially to erase blocks and discarding
c298ee71 1782 before overwriting. The `trimwrite` mode works well for this
f80dba8d
MT
1783 constraint.
1784
1785 **pmemblk**
1786 Read and write using filesystem DAX to a file on a filesystem
1787 mounted with DAX on a persistent memory device through the NVML
1788 libpmemblk library.
1789
1790 **dev-dax**
1791 Read and write using device DAX to a persistent memory device (e.g.,
1792 /dev/dax0.0) through the NVML libpmem library.
1793
1794 **external**
1795 Prefix to specify loading an external I/O engine object file. Append
c60ebc45 1796 the engine filename, e.g. ``ioengine=external:/tmp/foo.o`` to load
f80dba8d
MT
1797 ioengine :file:`foo.o` in :file:`/tmp`.
1798
1799
1800I/O engine specific parameters
1801~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1802
1803In addition, there are some parameters which are only valid when a specific
f50fbdda
TK
1804:option:`ioengine` is in use. These are used identically to normal parameters,
1805with the caveat that when used on the command line, they must come after the
f80dba8d
MT
1806:option:`ioengine` that defines them is selected.
1807
1808.. option:: userspace_reap : [libaio]
1809
1810 Normally, with the libaio engine in use, fio will use the
1811 :manpage:`io_getevents(2)` system call to reap newly returned events. With
1812 this flag turned on, the AIO ring will be read directly from user-space to
1813 reap events. The reaping mode is only enabled when polling for a minimum of
c60ebc45 1814 0 events (e.g. when :option:`iodepth_batch_complete` `=0`).
f80dba8d 1815
9d25d068 1816.. option:: hipri : [pvsync2]
f80dba8d
MT
1817
1818 Set RWF_HIPRI on I/O, indicating to the kernel that it's of higher priority
1819 than normal.
1820
a0679ce5
SB
1821.. option:: hipri_percentage : [pvsync2]
1822
f50fbdda 1823 When hipri is set this determines the probability of a pvsync2 I/O being high
a0679ce5
SB
1824 priority. The default is 100%.
1825
f80dba8d
MT
1826.. option:: cpuload=int : [cpuio]
1827
da19cdb4
TK
1828 Attempt to use the specified percentage of CPU cycles. This is a mandatory
1829 option when using cpuio I/O engine.
f80dba8d
MT
1830
1831.. option:: cpuchunks=int : [cpuio]
1832
1833 Split the load into cycles of the given time. In microseconds.
1834
1835.. option:: exit_on_io_done=bool : [cpuio]
1836
1837 Detect when I/O threads are done, then exit.
1838
f80dba8d
MT
1839.. option:: namenode=str : [libhdfs]
1840
22413915 1841 The hostname or IP address of a HDFS cluster namenode to contact.
f80dba8d
MT
1842
1843.. option:: port=int
1844
f50fbdda
TK
1845 [libhdfs]
1846
1847 The listening port of the HFDS cluster namenode.
1848
f80dba8d
MT
1849 [netsplice], [net]
1850
1851 The TCP or UDP port to bind to or connect to. If this is used with
1852 :option:`numjobs` to spawn multiple instances of the same job type, then
1853 this will be the starting port number since fio will use a range of
1854 ports.
1855
f50fbdda 1856.. option:: hostname=str : [netsplice] [net]
f80dba8d 1857
f50fbdda
TK
1858 The hostname or IP address to use for TCP or UDP based I/O. If the job is
1859 a TCP listener or UDP reader, the hostname is not used and must be omitted
1860 unless it is a valid UDP multicast address.
f80dba8d
MT
1861
1862.. option:: interface=str : [netsplice] [net]
1863
1864 The IP address of the network interface used to send or receive UDP
1865 multicast.
1866
1867.. option:: ttl=int : [netsplice] [net]
1868
1869 Time-to-live value for outgoing UDP multicast packets. Default: 1.
1870
1871.. option:: nodelay=bool : [netsplice] [net]
1872
1873 Set TCP_NODELAY on TCP connections.
1874
f50fbdda 1875.. option:: protocol=str, proto=str : [netsplice] [net]
f80dba8d
MT
1876
1877 The network protocol to use. Accepted values are:
1878
1879 **tcp**
1880 Transmission control protocol.
1881 **tcpv6**
1882 Transmission control protocol V6.
1883 **udp**
1884 User datagram protocol.
1885 **udpv6**
1886 User datagram protocol V6.
1887 **unix**
1888 UNIX domain socket.
1889
1890 When the protocol is TCP or UDP, the port must also be given, as well as the
1891 hostname if the job is a TCP listener or UDP reader. For unix sockets, the
f50fbdda 1892 normal :option:`filename` option should be used and the port is invalid.
f80dba8d 1893
e9184ec1 1894.. option:: listen : [netsplice] [net]
f80dba8d
MT
1895
1896 For TCP network connections, tell fio to listen for incoming connections
1897 rather than initiating an outgoing connection. The :option:`hostname` must
1898 be omitted if this option is used.
1899
e9184ec1 1900.. option:: pingpong : [netsplice] [net]
f80dba8d
MT
1901
1902 Normally a network writer will just continue writing data, and a network
1903 reader will just consume packages. If ``pingpong=1`` is set, a writer will
1904 send its normal payload to the reader, then wait for the reader to send the
1905 same payload back. This allows fio to measure network latencies. The
1906 submission and completion latencies then measure local time spent sending or
1907 receiving, and the completion latency measures how long it took for the
1908 other end to receive and send back. For UDP multicast traffic
1909 ``pingpong=1`` should only be set for a single reader when multiple readers
1910 are listening to the same address.
1911
e9184ec1 1912.. option:: window_size : [netsplice] [net]
f80dba8d
MT
1913
1914 Set the desired socket buffer size for the connection.
1915
e9184ec1 1916.. option:: mss : [netsplice] [net]
f80dba8d
MT
1917
1918 Set the TCP maximum segment size (TCP_MAXSEG).
1919
1920.. option:: donorname=str : [e4defrag]
1921
730bd7d9 1922 File will be used as a block donor (swap extents between files).
f80dba8d
MT
1923
1924.. option:: inplace=int : [e4defrag]
1925
1926 Configure donor file blocks allocation strategy:
1927
1928 **0**
1929 Default. Preallocate donor's file on init.
1930 **1**
2b455dbf 1931 Allocate space immediately inside defragment event, and free right
f80dba8d
MT
1932 after event.
1933
1934.. option:: clustername=str : [rbd]
1935
1936 Specifies the name of the Ceph cluster.
1937
1938.. option:: rbdname=str : [rbd]
1939
1940 Specifies the name of the RBD.
1941
1942.. option:: pool=str : [rbd]
1943
1944 Specifies the name of the Ceph pool containing RBD.
1945
1946.. option:: clientname=str : [rbd]
1947
1948 Specifies the username (without the 'client.' prefix) used to access the
1949 Ceph cluster. If the *clustername* is specified, the *clientname* shall be
1950 the full *type.id* string. If no type. prefix is given, fio will add
1951 'client.' by default.
1952
1953.. option:: skip_bad=bool : [mtd]
1954
1955 Skip operations against known bad blocks.
1956
1957.. option:: hdfsdirectory : [libhdfs]
1958
1959 libhdfs will create chunk in this HDFS directory.
1960
1961.. option:: chunk_size : [libhdfs]
1962
2b455dbf 1963 The size of the chunk to use for each file.
f80dba8d
MT
1964
1965
1966I/O depth
1967~~~~~~~~~
1968
1969.. option:: iodepth=int
1970
1971 Number of I/O units to keep in flight against the file. Note that
1972 increasing *iodepth* beyond 1 will not affect synchronous ioengines (except
c60ebc45 1973 for small degrees when :option:`verify_async` is in use). Even async
f80dba8d
MT
1974 engines may impose OS restrictions causing the desired depth not to be
1975 achieved. This may happen on Linux when using libaio and not setting
9207a0cb 1976 :option:`direct`\=1, since buffered I/O is not async on that OS. Keep an
f80dba8d
MT
1977 eye on the I/O depth distribution in the fio output to verify that the
1978 achieved depth is as expected. Default: 1.
1979
1980.. option:: iodepth_batch_submit=int, iodepth_batch=int
1981
1982 This defines how many pieces of I/O to submit at once. It defaults to 1
1983 which means that we submit each I/O as soon as it is available, but can be
1984 raised to submit bigger batches of I/O at the time. If it is set to 0 the
1985 :option:`iodepth` value will be used.
1986
1987.. option:: iodepth_batch_complete_min=int, iodepth_batch_complete=int
1988
1989 This defines how many pieces of I/O to retrieve at once. It defaults to 1
1990 which means that we'll ask for a minimum of 1 I/O in the retrieval process
1991 from the kernel. The I/O retrieval will go on until we hit the limit set by
1992 :option:`iodepth_low`. If this variable is set to 0, then fio will always
1993 check for completed events before queuing more I/O. This helps reduce I/O
1994 latency, at the cost of more retrieval system calls.
1995
1996.. option:: iodepth_batch_complete_max=int
1997
1998 This defines maximum pieces of I/O to retrieve at once. This variable should
9207a0cb 1999 be used along with :option:`iodepth_batch_complete_min`\=int variable,
f80dba8d 2000 specifying the range of min and max amount of I/O which should be
730bd7d9 2001 retrieved. By default it is equal to the :option:`iodepth_batch_complete_min`
f80dba8d
MT
2002 value.
2003
2004 Example #1::
2005
2006 iodepth_batch_complete_min=1
2007 iodepth_batch_complete_max=<iodepth>
2008
2009 which means that we will retrieve at least 1 I/O and up to the whole
2010 submitted queue depth. If none of I/O has been completed yet, we will wait.
2011
2012 Example #2::
2013
2014 iodepth_batch_complete_min=0
2015 iodepth_batch_complete_max=<iodepth>
2016
2017 which means that we can retrieve up to the whole submitted queue depth, but
2018 if none of I/O has been completed yet, we will NOT wait and immediately exit
2019 the system call. In this example we simply do polling.
2020
2021.. option:: iodepth_low=int
2022
2023 The low water mark indicating when to start filling the queue
2024 again. Defaults to the same as :option:`iodepth`, meaning that fio will
2025 attempt to keep the queue full at all times. If :option:`iodepth` is set to
c60ebc45 2026 e.g. 16 and *iodepth_low* is set to 4, then after fio has filled the queue of
f80dba8d
MT
2027 16 requests, it will let the depth drain down to 4 before starting to fill
2028 it again.
2029
997b5680
SW
2030.. option:: serialize_overlap=bool
2031
2032 Serialize in-flight I/Os that might otherwise cause or suffer from data races.
2033 When two or more I/Os are submitted simultaneously, there is no guarantee that
2034 the I/Os will be processed or completed in the submitted order. Further, if
2035 two or more of those I/Os are writes, any overlapping region between them can
2036 become indeterminate/undefined on certain storage. These issues can cause
2037 verification to fail erratically when at least one of the racing I/Os is
2038 changing data and the overlapping region has a non-zero size. Setting
2039 ``serialize_overlap`` tells fio to avoid provoking this behavior by explicitly
2040 serializing in-flight I/Os that have a non-zero overlap. Note that setting
2041 this option can reduce both performance and the `:option:iodepth` achieved.
2042 Additionally this option does not work when :option:`io_submit_mode` is set to
2043 offload. Default: false.
2044
f80dba8d
MT
2045.. option:: io_submit_mode=str
2046
2047 This option controls how fio submits the I/O to the I/O engine. The default
2048 is `inline`, which means that the fio job threads submit and reap I/O
2049 directly. If set to `offload`, the job threads will offload I/O submission
2050 to a dedicated pool of I/O threads. This requires some coordination and thus
2051 has a bit of extra overhead, especially for lower queue depth I/O where it
2052 can increase latencies. The benefit is that fio can manage submission rates
2053 independently of the device completion rates. This avoids skewed latency
730bd7d9 2054 reporting if I/O gets backed up on the device side (the coordinated omission
f80dba8d
MT
2055 problem).
2056
2057
2058I/O rate
2059~~~~~~~~
2060
a881438b 2061.. option:: thinktime=time
f80dba8d 2062
f75ede1d
SW
2063 Stall the job for the specified period of time after an I/O has completed before issuing the
2064 next. May be used to simulate processing being done by an application.
947e0fe0 2065 When the unit is omitted, the value is interpreted in microseconds. See
f80dba8d
MT
2066 :option:`thinktime_blocks` and :option:`thinktime_spin`.
2067
a881438b 2068.. option:: thinktime_spin=time
f80dba8d
MT
2069
2070 Only valid if :option:`thinktime` is set - pretend to spend CPU time doing
2071 something with the data received, before falling back to sleeping for the
f75ede1d 2072 rest of the period specified by :option:`thinktime`. When the unit is
947e0fe0 2073 omitted, the value is interpreted in microseconds.
f80dba8d
MT
2074
2075.. option:: thinktime_blocks=int
2076
2077 Only valid if :option:`thinktime` is set - control how many blocks to issue,
f50fbdda
TK
2078 before waiting :option:`thinktime` usecs. If not set, defaults to 1 which will make
2079 fio wait :option:`thinktime` usecs after every block. This effectively makes any
f80dba8d 2080 queue depth setting redundant, since no more than 1 I/O will be queued
f50fbdda 2081 before we have to complete it and do our :option:`thinktime`. In other words, this
f80dba8d 2082 setting effectively caps the queue depth if the latter is larger.
71bfa161 2083
f80dba8d 2084.. option:: rate=int[,int][,int]
71bfa161 2085
f80dba8d
MT
2086 Cap the bandwidth used by this job. The number is in bytes/sec, the normal
2087 suffix rules apply. Comma-separated values may be specified for reads,
2088 writes, and trims as described in :option:`blocksize`.
71bfa161 2089
b25b3464
SW
2090 For example, using `rate=1m,500k` would limit reads to 1MiB/sec and writes to
2091 500KiB/sec. Capping only reads or writes can be done with `rate=,500k` or
2092 `rate=500k,` where the former will only limit writes (to 500KiB/sec) and the
2093 latter will only limit reads.
2094
f80dba8d 2095.. option:: rate_min=int[,int][,int]
71bfa161 2096
f80dba8d
MT
2097 Tell fio to do whatever it can to maintain at least this bandwidth. Failing
2098 to meet this requirement will cause the job to exit. Comma-separated values
2099 may be specified for reads, writes, and trims as described in
2100 :option:`blocksize`.
71bfa161 2101
f80dba8d 2102.. option:: rate_iops=int[,int][,int]
71bfa161 2103
f80dba8d
MT
2104 Cap the bandwidth to this number of IOPS. Basically the same as
2105 :option:`rate`, just specified independently of bandwidth. If the job is
2106 given a block size range instead of a fixed value, the smallest block size
2107 is used as the metric. Comma-separated values may be specified for reads,
2108 writes, and trims as described in :option:`blocksize`.
71bfa161 2109
f80dba8d 2110.. option:: rate_iops_min=int[,int][,int]
71bfa161 2111
f80dba8d
MT
2112 If fio doesn't meet this rate of I/O, it will cause the job to exit.
2113 Comma-separated values may be specified for reads, writes, and trims as
2114 described in :option:`blocksize`.
71bfa161 2115
f80dba8d 2116.. option:: rate_process=str
66c098b8 2117
f80dba8d
MT
2118 This option controls how fio manages rated I/O submissions. The default is
2119 `linear`, which submits I/O in a linear fashion with fixed delays between
c60ebc45 2120 I/Os that gets adjusted based on I/O completion rates. If this is set to
f80dba8d
MT
2121 `poisson`, fio will submit I/O based on a more real world random request
2122 flow, known as the Poisson process
2123 (https://en.wikipedia.org/wiki/Poisson_point_process). The lambda will be
2124 10^6 / IOPS for the given workload.
71bfa161
JA
2125
2126
f80dba8d
MT
2127I/O latency
2128~~~~~~~~~~~
71bfa161 2129
a881438b 2130.. option:: latency_target=time
71bfa161 2131
f80dba8d 2132 If set, fio will attempt to find the max performance point that the given
f75ede1d 2133 workload will run at while maintaining a latency below this target. When
947e0fe0 2134 the unit is omitted, the value is interpreted in microseconds. See
f75ede1d 2135 :option:`latency_window` and :option:`latency_percentile`.
71bfa161 2136
a881438b 2137.. option:: latency_window=time
71bfa161 2138
f80dba8d 2139 Used with :option:`latency_target` to specify the sample window that the job
f75ede1d 2140 is run at varying queue depths to test the performance. When the unit is
947e0fe0 2141 omitted, the value is interpreted in microseconds.
b4692828 2142
f80dba8d 2143.. option:: latency_percentile=float
71bfa161 2144
c60ebc45 2145 The percentage of I/Os that must fall within the criteria specified by
f80dba8d 2146 :option:`latency_target` and :option:`latency_window`. If not set, this
c60ebc45 2147 defaults to 100.0, meaning that all I/Os must be equal or below to the value
f80dba8d 2148 set by :option:`latency_target`.
71bfa161 2149
a881438b 2150.. option:: max_latency=time
71bfa161 2151
f75ede1d 2152 If set, fio will exit the job with an ETIMEDOUT error if it exceeds this
947e0fe0 2153 maximum latency. When the unit is omitted, the value is interpreted in
f75ede1d 2154 microseconds.
71bfa161 2155
f80dba8d 2156.. option:: rate_cycle=int
71bfa161 2157
f80dba8d 2158 Average bandwidth for :option:`rate` and :option:`rate_min` over this number
a47b697c 2159 of milliseconds. Defaults to 1000.
71bfa161 2160
71bfa161 2161
f80dba8d
MT
2162I/O replay
2163~~~~~~~~~~
71bfa161 2164
f80dba8d 2165.. option:: write_iolog=str
c2b1e753 2166
f80dba8d
MT
2167 Write the issued I/O patterns to the specified file. See
2168 :option:`read_iolog`. Specify a separate file for each job, otherwise the
2169 iologs will be interspersed and the file may be corrupt.
c2b1e753 2170
f80dba8d 2171.. option:: read_iolog=str
71bfa161 2172
22413915 2173 Open an iolog with the specified filename and replay the I/O patterns it
f80dba8d
MT
2174 contains. This can be used to store a workload and replay it sometime
2175 later. The iolog given may also be a blktrace binary file, which allows fio
2176 to replay a workload captured by :command:`blktrace`. See
2177 :manpage:`blktrace(8)` for how to capture such logging data. For blktrace
2178 replay, the file needs to be turned into a blkparse binary data file first
2179 (``blkparse <device> -o /dev/null -d file_for_fio.bin``).
71bfa161 2180
589e88b7 2181.. option:: replay_no_stall=bool
71bfa161 2182
f80dba8d 2183 When replaying I/O with :option:`read_iolog` the default behavior is to
22413915 2184 attempt to respect the timestamps within the log and replay them with the
f80dba8d
MT
2185 appropriate delay between IOPS. By setting this variable fio will not
2186 respect the timestamps and attempt to replay them as fast as possible while
2187 still respecting ordering. The result is the same I/O pattern to a given
2188 device, but different timings.
71bfa161 2189
f80dba8d 2190.. option:: replay_redirect=str
b4692828 2191
f80dba8d
MT
2192 While replaying I/O patterns using :option:`read_iolog` the default behavior
2193 is to replay the IOPS onto the major/minor device that each IOP was recorded
2194 from. This is sometimes undesirable because on a different machine those
2195 major/minor numbers can map to a different device. Changing hardware on the
2196 same system can also result in a different major/minor mapping.
730bd7d9 2197 ``replay_redirect`` causes all I/Os to be replayed onto the single specified
f80dba8d 2198 device regardless of the device it was recorded
9207a0cb 2199 from. i.e. :option:`replay_redirect`\= :file:`/dev/sdc` would cause all I/O
f80dba8d
MT
2200 in the blktrace or iolog to be replayed onto :file:`/dev/sdc`. This means
2201 multiple devices will be replayed onto a single device, if the trace
2202 contains multiple devices. If you want multiple devices to be replayed
2203 concurrently to multiple redirected devices you must blkparse your trace
2204 into separate traces and replay them with independent fio invocations.
2205 Unfortunately this also breaks the strict time ordering between multiple
2206 device accesses.
71bfa161 2207
f80dba8d 2208.. option:: replay_align=int
74929ac2 2209
f80dba8d
MT
2210 Force alignment of I/O offsets and lengths in a trace to this power of 2
2211 value.
3c54bc46 2212
f80dba8d 2213.. option:: replay_scale=int
3c54bc46 2214
f80dba8d 2215 Scale sector offsets down by this factor when replaying traces.
3c54bc46 2216
3c54bc46 2217
f80dba8d
MT
2218Threads, processes and job synchronization
2219~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3c54bc46 2220
f80dba8d 2221.. option:: thread
3c54bc46 2222
730bd7d9
SW
2223 Fio defaults to creating jobs by using fork, however if this option is
2224 given, fio will create jobs by using POSIX Threads' function
2225 :manpage:`pthread_create(3)` to create threads instead.
71bfa161 2226
f80dba8d 2227.. option:: wait_for=str
74929ac2 2228
730bd7d9
SW
2229 If set, the current job won't be started until all workers of the specified
2230 waitee job are done.
74929ac2 2231
f80dba8d
MT
2232 ``wait_for`` operates on the job name basis, so there are a few
2233 limitations. First, the waitee must be defined prior to the waiter job
2234 (meaning no forward references). Second, if a job is being referenced as a
2235 waitee, it must have a unique name (no duplicate waitees).
74929ac2 2236
f80dba8d 2237.. option:: nice=int
892a6ffc 2238
f80dba8d 2239 Run the job with the given nice value. See man :manpage:`nice(2)`.
892a6ffc 2240
f80dba8d
MT
2241 On Windows, values less than -15 set the process class to "High"; -1 through
2242 -15 set "Above Normal"; 1 through 15 "Below Normal"; and above 15 "Idle"
2243 priority class.
74929ac2 2244
f80dba8d 2245.. option:: prio=int
71bfa161 2246
f80dba8d
MT
2247 Set the I/O priority value of this job. Linux limits us to a positive value
2248 between 0 and 7, with 0 being the highest. See man
2249 :manpage:`ionice(1)`. Refer to an appropriate manpage for other operating
2250 systems since meaning of priority may differ.
71bfa161 2251
f80dba8d 2252.. option:: prioclass=int
d59aa780 2253
f80dba8d 2254 Set the I/O priority class. See man :manpage:`ionice(1)`.
d59aa780 2255
f80dba8d 2256.. option:: cpumask=int
71bfa161 2257
22413915
SW
2258 Set the CPU affinity of this job. The parameter given is a bit mask of
2259 allowed CPUs the job may run on. So if you want the allowed CPUs to be 1
f80dba8d
MT
2260 and 5, you would pass the decimal value of (1 << 1 | 1 << 5), or 34. See man
2261 :manpage:`sched_setaffinity(2)`. This may not work on all supported
2262 operating systems or kernel versions. This option doesn't work well for a
2263 higher CPU count than what you can store in an integer mask, so it can only
2264 control cpus 1-32. For boxes with larger CPU counts, use
2265 :option:`cpus_allowed`.
6d500c2e 2266
f80dba8d 2267.. option:: cpus_allowed=str
6d500c2e 2268
730bd7d9
SW
2269 Controls the same options as :option:`cpumask`, but accepts a textual
2270 specification of the permitted CPUs instead. So to use CPUs 1 and 5 you
2271 would specify ``cpus_allowed=1,5``. This option also allows a range of CPUs
2272 to be specified -- say you wanted a binding to CPUs 1, 5, and 8 to 15, you
2273 would set ``cpus_allowed=1,5,8-15``.
6d500c2e 2274
f80dba8d 2275.. option:: cpus_allowed_policy=str
6d500c2e 2276
f80dba8d 2277 Set the policy of how fio distributes the CPUs specified by
730bd7d9 2278 :option:`cpus_allowed` or :option:`cpumask`. Two policies are supported:
6d500c2e 2279
f80dba8d
MT
2280 **shared**
2281 All jobs will share the CPU set specified.
2282 **split**
2283 Each job will get a unique CPU from the CPU set.
6d500c2e 2284
22413915 2285 **shared** is the default behavior, if the option isn't specified. If
f80dba8d
MT
2286 **split** is specified, then fio will will assign one cpu per job. If not
2287 enough CPUs are given for the jobs listed, then fio will roundrobin the CPUs
2288 in the set.
6d500c2e 2289
f80dba8d 2290.. option:: numa_cpu_nodes=str
6d500c2e 2291
f80dba8d
MT
2292 Set this job running on specified NUMA nodes' CPUs. The arguments allow
2293 comma delimited list of cpu numbers, A-B ranges, or `all`. Note, to enable
ac8ca2af 2294 NUMA options support, fio must be built on a system with libnuma-dev(el)
f80dba8d 2295 installed.
61b9861d 2296
f80dba8d 2297.. option:: numa_mem_policy=str
61b9861d 2298
f80dba8d
MT
2299 Set this job's memory policy and corresponding NUMA nodes. Format of the
2300 arguments::
5c94b008 2301
f80dba8d 2302 <mode>[:<nodelist>]
ce35b1ec 2303
730bd7d9
SW
2304 ``mode`` is one of the following memory poicies: ``default``, ``prefer``,
2305 ``bind``, ``interleave`` or ``local``. For ``default`` and ``local`` memory
2306 policies, no node needs to be specified. For ``prefer``, only one node is
2307 allowed. For ``bind`` and ``interleave`` the ``nodelist`` may be as
2308 follows: a comma delimited list of numbers, A-B ranges, or `all`.
71bfa161 2309
f80dba8d 2310.. option:: cgroup=str
390b1537 2311
f80dba8d
MT
2312 Add job to this control group. If it doesn't exist, it will be created. The
2313 system must have a mounted cgroup blkio mount point for this to work. If
2314 your system doesn't have it mounted, you can do so with::
5af1c6f3 2315
f80dba8d 2316 # mount -t cgroup -o blkio none /cgroup
5af1c6f3 2317
f80dba8d 2318.. option:: cgroup_weight=int
5af1c6f3 2319
f80dba8d
MT
2320 Set the weight of the cgroup to this value. See the documentation that comes
2321 with the kernel, allowed values are in the range of 100..1000.
a086c257 2322
f80dba8d 2323.. option:: cgroup_nodelete=bool
8c07860d 2324
f80dba8d
MT
2325 Normally fio will delete the cgroups it has created after the job
2326 completion. To override this behavior and to leave cgroups around after the
2327 job completion, set ``cgroup_nodelete=1``. This can be useful if one wants
2328 to inspect various cgroup files after job completion. Default: false.
8c07860d 2329
f80dba8d 2330.. option:: flow_id=int
8c07860d 2331
f80dba8d
MT
2332 The ID of the flow. If not specified, it defaults to being a global
2333 flow. See :option:`flow`.
1907dbc6 2334
f80dba8d 2335.. option:: flow=int
71bfa161 2336
f80dba8d
MT
2337 Weight in token-based flow control. If this value is used, then there is a
2338 'flow counter' which is used to regulate the proportion of activity between
2339 two or more jobs. Fio attempts to keep this flow counter near zero. The
2340 ``flow`` parameter stands for how much should be added or subtracted to the
2341 flow counter on each iteration of the main I/O loop. That is, if one job has
2342 ``flow=8`` and another job has ``flow=-1``, then there will be a roughly 1:8
2343 ratio in how much one runs vs the other.
71bfa161 2344
f80dba8d 2345.. option:: flow_watermark=int
a31041ea 2346
f80dba8d
MT
2347 The maximum value that the absolute value of the flow counter is allowed to
2348 reach before the job must wait for a lower value of the counter.
82407585 2349
f80dba8d 2350.. option:: flow_sleep=int
82407585 2351
f80dba8d
MT
2352 The period of time, in microseconds, to wait after the flow watermark has
2353 been exceeded before retrying operations.
82407585 2354
f80dba8d 2355.. option:: stonewall, wait_for_previous
82407585 2356
f80dba8d
MT
2357 Wait for preceding jobs in the job file to exit, before starting this
2358 one. Can be used to insert serialization points in the job file. A stone
2359 wall also implies starting a new reporting group, see
2360 :option:`group_reporting`.
2361
2362.. option:: exitall
2363
730bd7d9
SW
2364 By default, fio will continue running all other jobs when one job finishes
2365 but sometimes this is not the desired action. Setting ``exitall`` will
2366 instead make fio terminate all other jobs when one job finishes.
f80dba8d
MT
2367
2368.. option:: exec_prerun=str
2369
2370 Before running this job, issue the command specified through
2371 :manpage:`system(3)`. Output is redirected in a file called
2372 :file:`jobname.prerun.txt`.
2373
2374.. option:: exec_postrun=str
2375
2376 After the job completes, issue the command specified though
2377 :manpage:`system(3)`. Output is redirected in a file called
2378 :file:`jobname.postrun.txt`.
2379
2380.. option:: uid=int
2381
2382 Instead of running as the invoking user, set the user ID to this value
2383 before the thread/process does any work.
2384
2385.. option:: gid=int
2386
2387 Set group ID, see :option:`uid`.
2388
2389
2390Verification
2391~~~~~~~~~~~~
2392
2393.. option:: verify_only
2394
2395 Do not perform specified workload, only verify data still matches previous
2396 invocation of this workload. This option allows one to check data multiple
2397 times at a later date without overwriting it. This option makes sense only
2398 for workloads that write data, and does not support workloads with the
2399 :option:`time_based` option set.
2400
2401.. option:: do_verify=bool
2402
2403 Run the verify phase after a write phase. Only valid if :option:`verify` is
2404 set. Default: true.
2405
2406.. option:: verify=str
2407
2408 If writing to a file, fio can verify the file contents after each iteration
2409 of the job. Each verification method also implies verification of special
2410 header, which is written to the beginning of each block. This header also
2411 includes meta information, like offset of the block, block number, timestamp
2412 when block was written, etc. :option:`verify` can be combined with
2413 :option:`verify_pattern` option. The allowed values are:
2414
2415 **md5**
2416 Use an md5 sum of the data area and store it in the header of
2417 each block.
2418
2419 **crc64**
2420 Use an experimental crc64 sum of the data area and store it in the
2421 header of each block.
2422
2423 **crc32c**
a5896300
SW
2424 Use a crc32c sum of the data area and store it in the header of
2425 each block. This will automatically use hardware acceleration
2426 (e.g. SSE4.2 on an x86 or CRC crypto extensions on ARM64) but will
2427 fall back to software crc32c if none is found. Generally the
2428 fatest checksum fio supports when hardware accelerated.
f80dba8d
MT
2429
2430 **crc32c-intel**
a5896300 2431 Synonym for crc32c.
f80dba8d
MT
2432
2433 **crc32**
2434 Use a crc32 sum of the data area and store it in the header of each
2435 block.
2436
2437 **crc16**
2438 Use a crc16 sum of the data area and store it in the header of each
2439 block.
2440
2441 **crc7**
2442 Use a crc7 sum of the data area and store it in the header of each
2443 block.
2444
2445 **xxhash**
2446 Use xxhash as the checksum function. Generally the fastest software
2447 checksum that fio supports.
2448
2449 **sha512**
2450 Use sha512 as the checksum function.
2451
2452 **sha256**
2453 Use sha256 as the checksum function.
2454
2455 **sha1**
2456 Use optimized sha1 as the checksum function.
82407585 2457
ae3a5acc
JA
2458 **sha3-224**
2459 Use optimized sha3-224 as the checksum function.
2460
2461 **sha3-256**
2462 Use optimized sha3-256 as the checksum function.
2463
2464 **sha3-384**
2465 Use optimized sha3-384 as the checksum function.
2466
2467 **sha3-512**
2468 Use optimized sha3-512 as the checksum function.
2469
f80dba8d
MT
2470 **meta**
2471 This option is deprecated, since now meta information is included in
2472 generic verification header and meta verification happens by
2473 default. For detailed information see the description of the
2474 :option:`verify` setting. This option is kept because of
2475 compatibility's sake with old configurations. Do not use it.
2476
2477 **pattern**
2478 Verify a strict pattern. Normally fio includes a header with some
2479 basic information and checksumming, but if this option is set, only
2480 the specific pattern set with :option:`verify_pattern` is verified.
2481
2482 **null**
2483 Only pretend to verify. Useful for testing internals with
9207a0cb 2484 :option:`ioengine`\=null, not for much else.
f80dba8d
MT
2485
2486 This option can be used for repeated burn-in tests of a system to make sure
2487 that the written data is also correctly read back. If the data direction
2488 given is a read or random read, fio will assume that it should verify a
2489 previously written file. If the data direction includes any form of write,
2490 the verify will be of the newly written data.
2491
2492.. option:: verifysort=bool
2493
2494 If true, fio will sort written verify blocks when it deems it faster to read
2495 them back in a sorted manner. This is often the case when overwriting an
2496 existing file, since the blocks are already laid out in the file system. You
2497 can ignore this option unless doing huge amounts of really fast I/O where
2498 the red-black tree sorting CPU time becomes significant. Default: true.
2499
2500.. option:: verifysort_nr=int
2501
2b455dbf 2502 Pre-load and sort verify blocks for a read workload.
f80dba8d
MT
2503
2504.. option:: verify_offset=int
2505
2506 Swap the verification header with data somewhere else in the block before
2507 writing. It is swapped back before verifying.
2508
2509.. option:: verify_interval=int
2510
2511 Write the verification header at a finer granularity than the
2512 :option:`blocksize`. It will be written for chunks the size of
2513 ``verify_interval``. :option:`blocksize` should divide this evenly.
2514
2515.. option:: verify_pattern=str
2516
2517 If set, fio will fill the I/O buffers with this pattern. Fio defaults to
2518 filling with totally random bytes, but sometimes it's interesting to fill
2519 with a known pattern for I/O verification purposes. Depending on the width
730bd7d9 2520 of the pattern, fio will fill 1/2/3/4 bytes of the buffer at the time (it can
f80dba8d
MT
2521 be either a decimal or a hex number). The ``verify_pattern`` if larger than
2522 a 32-bit quantity has to be a hex number that starts with either "0x" or
2523 "0X". Use with :option:`verify`. Also, ``verify_pattern`` supports %o
2524 format, which means that for each block offset will be written and then
2525 verified back, e.g.::
61b9861d
RP
2526
2527 verify_pattern=%o
2528
f80dba8d
MT
2529 Or use combination of everything::
2530
61b9861d 2531 verify_pattern=0xff%o"abcd"-12
e28218f3 2532
f80dba8d
MT
2533.. option:: verify_fatal=bool
2534
2535 Normally fio will keep checking the entire contents before quitting on a
2536 block verification failure. If this option is set, fio will exit the job on
2537 the first observed failure. Default: false.
2538
2539.. option:: verify_dump=bool
2540
2541 If set, dump the contents of both the original data block and the data block
2542 we read off disk to files. This allows later analysis to inspect just what
2543 kind of data corruption occurred. Off by default.
2544
2545.. option:: verify_async=int
2546
2547 Fio will normally verify I/O inline from the submitting thread. This option
2548 takes an integer describing how many async offload threads to create for I/O
2549 verification instead, causing fio to offload the duty of verifying I/O
2550 contents to one or more separate threads. If using this offload option, even
2551 sync I/O engines can benefit from using an :option:`iodepth` setting higher
2552 than 1, as it allows them to have I/O in flight while verifies are running.
d7e6ea1c 2553 Defaults to 0 async threads, i.e. verification is not asynchronous.
f80dba8d
MT
2554
2555.. option:: verify_async_cpus=str
2556
2557 Tell fio to set the given CPU affinity on the async I/O verification
2558 threads. See :option:`cpus_allowed` for the format used.
2559
2560.. option:: verify_backlog=int
2561
2562 Fio will normally verify the written contents of a job that utilizes verify
2563 once that job has completed. In other words, everything is written then
2564 everything is read back and verified. You may want to verify continually
2565 instead for a variety of reasons. Fio stores the meta data associated with
2566 an I/O block in memory, so for large verify workloads, quite a bit of memory
2567 would be used up holding this meta data. If this option is enabled, fio will
2568 write only N blocks before verifying these blocks.
2569
2570.. option:: verify_backlog_batch=int
2571
2572 Control how many blocks fio will verify if :option:`verify_backlog` is
2573 set. If not set, will default to the value of :option:`verify_backlog`
2574 (meaning the entire queue is read back and verified). If
2575 ``verify_backlog_batch`` is less than :option:`verify_backlog` then not all
2576 blocks will be verified, if ``verify_backlog_batch`` is larger than
2577 :option:`verify_backlog`, some blocks will be verified more than once.
2578
2579.. option:: verify_state_save=bool
2580
2581 When a job exits during the write phase of a verify workload, save its
2582 current state. This allows fio to replay up until that point, if the verify
2583 state is loaded for the verify read phase. The format of the filename is,
2584 roughly::
2585
f50fbdda 2586 <type>-<jobname>-<jobindex>-verify.state.
f80dba8d
MT
2587
2588 <type> is "local" for a local run, "sock" for a client/server socket
2589 connection, and "ip" (192.168.0.1, for instance) for a networked
d7e6ea1c 2590 client/server connection. Defaults to true.
f80dba8d
MT
2591
2592.. option:: verify_state_load=bool
2593
2594 If a verify termination trigger was used, fio stores the current write state
2595 of each thread. This can be used at verification time so that fio knows how
2596 far it should verify. Without this information, fio will run a full
a47b697c
SW
2597 verification pass, according to the settings in the job file used. Default
2598 false.
f80dba8d
MT
2599
2600.. option:: trim_percentage=int
2601
2602 Number of verify blocks to discard/trim.
2603
2604.. option:: trim_verify_zero=bool
2605
22413915 2606 Verify that trim/discarded blocks are returned as zeros.
f80dba8d
MT
2607
2608.. option:: trim_backlog=int
2609
5cfd1e9a 2610 Trim after this number of blocks are written.
f80dba8d
MT
2611
2612.. option:: trim_backlog_batch=int
2613
2614 Trim this number of I/O blocks.
2615
2616.. option:: experimental_verify=bool
2617
2618 Enable experimental verification.
2619
f80dba8d
MT
2620Steady state
2621~~~~~~~~~~~~
2622
2623.. option:: steadystate=str:float, ss=str:float
2624
2625 Define the criterion and limit for assessing steady state performance. The
2626 first parameter designates the criterion whereas the second parameter sets
2627 the threshold. When the criterion falls below the threshold for the
2628 specified duration, the job will stop. For example, `iops_slope:0.1%` will
2629 direct fio to terminate the job when the least squares regression slope
2630 falls below 0.1% of the mean IOPS. If :option:`group_reporting` is enabled
2631 this will apply to all jobs in the group. Below is the list of available
2632 steady state assessment criteria. All assessments are carried out using only
2633 data from the rolling collection window. Threshold limits can be expressed
2634 as a fixed value or as a percentage of the mean in the collection window.
2635
2636 **iops**
2637 Collect IOPS data. Stop the job if all individual IOPS measurements
2638 are within the specified limit of the mean IOPS (e.g., ``iops:2``
2639 means that all individual IOPS values must be within 2 of the mean,
2640 whereas ``iops:0.2%`` means that all individual IOPS values must be
2641 within 0.2% of the mean IOPS to terminate the job).
2642
2643 **iops_slope**
2644 Collect IOPS data and calculate the least squares regression
2645 slope. Stop the job if the slope falls below the specified limit.
2646
2647 **bw**
2648 Collect bandwidth data. Stop the job if all individual bandwidth
2649 measurements are within the specified limit of the mean bandwidth.
2650
2651 **bw_slope**
2652 Collect bandwidth data and calculate the least squares regression
2653 slope. Stop the job if the slope falls below the specified limit.
2654
2655.. option:: steadystate_duration=time, ss_dur=time
2656
2657 A rolling window of this duration will be used to judge whether steady state
2658 has been reached. Data will be collected once per second. The default is 0
f75ede1d 2659 which disables steady state detection. When the unit is omitted, the
947e0fe0 2660 value is interpreted in seconds.
f80dba8d
MT
2661
2662.. option:: steadystate_ramp_time=time, ss_ramp=time
2663
2664 Allow the job to run for the specified duration before beginning data
2665 collection for checking the steady state job termination criterion. The
947e0fe0 2666 default is 0. When the unit is omitted, the value is interpreted in seconds.
f80dba8d
MT
2667
2668
2669Measurements and reporting
2670~~~~~~~~~~~~~~~~~~~~~~~~~~
2671
2672.. option:: per_job_logs=bool
2673
2674 If set, this generates bw/clat/iops log with per file private filenames. If
2675 not set, jobs with identical names will share the log filename. Default:
2676 true.
2677
2678.. option:: group_reporting
2679
2680 It may sometimes be interesting to display statistics for groups of jobs as
2681 a whole instead of for each individual job. This is especially true if
2682 :option:`numjobs` is used; looking at individual thread/process output
2683 quickly becomes unwieldy. To see the final report per-group instead of
2684 per-job, use :option:`group_reporting`. Jobs in a file will be part of the
2685 same reporting group, unless if separated by a :option:`stonewall`, or by
2686 using :option:`new_group`.
2687
2688.. option:: new_group
2689
2690 Start a new reporting group. See: :option:`group_reporting`. If not given,
2691 all jobs in a file will be part of the same reporting group, unless
2692 separated by a :option:`stonewall`.
2693
589e88b7 2694.. option:: stats=bool
8243be59
JA
2695
2696 By default, fio collects and shows final output results for all jobs
2697 that run. If this option is set to 0, then fio will ignore it in
2698 the final stat output.
2699
f80dba8d
MT
2700.. option:: write_bw_log=str
2701
2702 If given, write a bandwidth log for this job. Can be used to store data of
2703 the bandwidth of the jobs in their lifetime. The included
2704 :command:`fio_generate_plots` script uses :command:`gnuplot` to turn these
22413915 2705 text files into nice graphs. See :option:`write_lat_log` for behavior of
f80dba8d
MT
2706 given filename. For this option, the postfix is :file:`_bw.x.log`, where `x`
2707 is the index of the job (`1..N`, where `N` is the number of jobs). If
2708 :option:`per_job_logs` is false, then the filename will not include the job
2709 index. See `Log File Formats`_.
2710
2711.. option:: write_lat_log=str
2712
2713 Same as :option:`write_bw_log`, except that this option stores I/O
2714 submission, completion, and total latencies instead. If no filename is given
2715 with this option, the default filename of :file:`jobname_type.log` is
2716 used. Even if the filename is given, fio will still append the type of
2717 log. So if one specifies::
e3cedca7
JA
2718
2719 write_lat_log=foo
2720
f80dba8d 2721 The actual log names will be :file:`foo_slat.x.log`, :file:`foo_clat.x.log`,
f50fbdda
TK
2722 and :file:`foo_lat.x.log`, where `x` is the index of the job (`1..N`, where `N`
2723 is the number of jobs). This helps :command:`fio_generate_plots` find the
f80dba8d
MT
2724 logs automatically. If :option:`per_job_logs` is false, then the filename
2725 will not include the job index. See `Log File Formats`_.
be4ecfdf 2726
f80dba8d 2727.. option:: write_hist_log=str
06842027 2728
f80dba8d
MT
2729 Same as :option:`write_lat_log`, but writes I/O completion latency
2730 histograms. If no filename is given with this option, the default filename
2731 of :file:`jobname_clat_hist.x.log` is used, where `x` is the index of the
f50fbdda 2732 job (`1..N`, where `N` is the number of jobs). Even if the filename is given,
f80dba8d
MT
2733 fio will still append the type of log. If :option:`per_job_logs` is false,
2734 then the filename will not include the job index. See `Log File Formats`_.
06842027 2735
f80dba8d 2736.. option:: write_iops_log=str
06842027 2737
f80dba8d
MT
2738 Same as :option:`write_bw_log`, but writes IOPS. If no filename is given
2739 with this option, the default filename of :file:`jobname_type.x.log` is
f50fbdda 2740 used, where `x` is the index of the job (`1..N`, where `N` is the number of
f80dba8d
MT
2741 jobs). Even if the filename is given, fio will still append the type of
2742 log. If :option:`per_job_logs` is false, then the filename will not include
2743 the job index. See `Log File Formats`_.
06842027 2744
f80dba8d 2745.. option:: log_avg_msec=int
06842027 2746
f80dba8d
MT
2747 By default, fio will log an entry in the iops, latency, or bw log for every
2748 I/O that completes. When writing to the disk log, that can quickly grow to a
2749 very large size. Setting this option makes fio average the each log entry
2750 over the specified period of time, reducing the resolution of the log. See
2751 :option:`log_max_value` as well. Defaults to 0, logging all entries.
6fc82095 2752 Also see `Log File Formats`_.
06842027 2753
f80dba8d 2754.. option:: log_hist_msec=int
06842027 2755
f80dba8d
MT
2756 Same as :option:`log_avg_msec`, but logs entries for completion latency
2757 histograms. Computing latency percentiles from averages of intervals using
c60ebc45 2758 :option:`log_avg_msec` is inaccurate. Setting this option makes fio log
f80dba8d
MT
2759 histogram entries over the specified period of time, reducing log sizes for
2760 high IOPS devices while retaining percentile accuracy. See
2761 :option:`log_hist_coarseness` as well. Defaults to 0, meaning histogram
2762 logging is disabled.
06842027 2763
f80dba8d 2764.. option:: log_hist_coarseness=int
06842027 2765
f80dba8d
MT
2766 Integer ranging from 0 to 6, defining the coarseness of the resolution of
2767 the histogram logs enabled with :option:`log_hist_msec`. For each increment
2768 in coarseness, fio outputs half as many bins. Defaults to 0, for which
2769 histogram logs contain 1216 latency bins. See `Log File Formats`_.
8b28bd41 2770
f80dba8d 2771.. option:: log_max_value=bool
66c098b8 2772
f80dba8d
MT
2773 If :option:`log_avg_msec` is set, fio logs the average over that window. If
2774 you instead want to log the maximum value, set this option to 1. Defaults to
2775 0, meaning that averaged values are logged.
a696fa2a 2776
589e88b7 2777.. option:: log_offset=bool
a696fa2a 2778
f80dba8d 2779 If this is set, the iolog options will include the byte offset for the I/O
5a83478f
SW
2780 entry as well as the other data values. Defaults to 0 meaning that
2781 offsets are not present in logs. Also see `Log File Formats`_.
71bfa161 2782
f80dba8d 2783.. option:: log_compression=int
7de87099 2784
f80dba8d
MT
2785 If this is set, fio will compress the I/O logs as it goes, to keep the
2786 memory footprint lower. When a log reaches the specified size, that chunk is
2787 removed and compressed in the background. Given that I/O logs are fairly
2788 highly compressible, this yields a nice memory savings for longer runs. The
2789 downside is that the compression will consume some background CPU cycles, so
2790 it may impact the run. This, however, is also true if the logging ends up
2791 consuming most of the system memory. So pick your poison. The I/O logs are
2792 saved normally at the end of a run, by decompressing the chunks and storing
2793 them in the specified log file. This feature depends on the availability of
2794 zlib.
e0b0d892 2795
f80dba8d 2796.. option:: log_compression_cpus=str
e0b0d892 2797
f80dba8d
MT
2798 Define the set of CPUs that are allowed to handle online log compression for
2799 the I/O jobs. This can provide better isolation between performance
2800 sensitive jobs, and background compression work.
9e684a49 2801
f80dba8d 2802.. option:: log_store_compressed=bool
9e684a49 2803
f80dba8d
MT
2804 If set, fio will store the log files in a compressed format. They can be
2805 decompressed with fio, using the :option:`--inflate-log` command line
2806 parameter. The files will be stored with a :file:`.fz` suffix.
9e684a49 2807
f80dba8d 2808.. option:: log_unix_epoch=bool
9e684a49 2809
f80dba8d
MT
2810 If set, fio will log Unix timestamps to the log files produced by enabling
2811 write_type_log for each log type, instead of the default zero-based
2812 timestamps.
2813
2814.. option:: block_error_percentiles=bool
2815
2816 If set, record errors in trim block-sized units from writes and trims and
2817 output a histogram of how many trims it took to get to errors, and what kind
2818 of error was encountered.
2819
2820.. option:: bwavgtime=int
2821
2822 Average the calculated bandwidth over the given time. Value is specified in
2823 milliseconds. If the job also does bandwidth logging through
2824 :option:`write_bw_log`, then the minimum of this option and
2825 :option:`log_avg_msec` will be used. Default: 500ms.
2826
2827.. option:: iopsavgtime=int
2828
2829 Average the calculated IOPS over the given time. Value is specified in
2830 milliseconds. If the job also does IOPS logging through
2831 :option:`write_iops_log`, then the minimum of this option and
2832 :option:`log_avg_msec` will be used. Default: 500ms.
2833
2834.. option:: disk_util=bool
2835
2836 Generate disk utilization statistics, if the platform supports it.
2837 Default: true.
2838
2839.. option:: disable_lat=bool
2840
2841 Disable measurements of total latency numbers. Useful only for cutting back
2842 the number of calls to :manpage:`gettimeofday(2)`, as that does impact
2843 performance at really high IOPS rates. Note that to really get rid of a
2844 large amount of these calls, this option must be used with
f75ede1d 2845 :option:`disable_slat` and :option:`disable_bw_measurement` as well.
f80dba8d
MT
2846
2847.. option:: disable_clat=bool
2848
2849 Disable measurements of completion latency numbers. See
2850 :option:`disable_lat`.
2851
2852.. option:: disable_slat=bool
2853
2854 Disable measurements of submission latency numbers. See
f50fbdda 2855 :option:`disable_lat`.
f80dba8d 2856
f75ede1d 2857.. option:: disable_bw_measurement=bool, disable_bw=bool
f80dba8d
MT
2858
2859 Disable measurements of throughput/bandwidth numbers. See
2860 :option:`disable_lat`.
2861
2862.. option:: clat_percentiles=bool
2863
2864 Enable the reporting of percentiles of completion latencies.
2865
2866.. option:: percentile_list=float_list
2867
2868 Overwrite the default list of percentiles for completion latencies and the
2869 block error histogram. Each number is a floating number in the range
2870 (0,100], and the maximum length of the list is 20. Use ``:`` to separate the
2871 numbers, and list the numbers in ascending order. For example,
2872 ``--percentile_list=99.5:99.9`` will cause fio to report the values of
2873 completion latency below which 99.5% and 99.9% of the observed latencies
2874 fell, respectively.
2875
2876
2877Error handling
2878~~~~~~~~~~~~~~
2879
2880.. option:: exitall_on_error
2881
2882 When one job finishes in error, terminate the rest. The default is to wait
2883 for each job to finish.
2884
2885.. option:: continue_on_error=str
2886
2887 Normally fio will exit the job on the first observed failure. If this option
2888 is set, fio will continue the job when there is a 'non-fatal error' (EIO or
2889 EILSEQ) until the runtime is exceeded or the I/O size specified is
2890 completed. If this option is used, there are two more stats that are
2891 appended, the total error count and the first error. The error field given
2892 in the stats is the first error that was hit during the run.
2893
2894 The allowed values are:
2895
2896 **none**
2897 Exit on any I/O or verify errors.
2898
2899 **read**
2900 Continue on read errors, exit on all others.
2901
2902 **write**
2903 Continue on write errors, exit on all others.
2904
2905 **io**
2906 Continue on any I/O error, exit on all others.
2907
2908 **verify**
2909 Continue on verify errors, exit on all others.
2910
2911 **all**
2912 Continue on all errors.
2913
2914 **0**
2915 Backward-compatible alias for 'none'.
2916
2917 **1**
2918 Backward-compatible alias for 'all'.
2919
2920.. option:: ignore_error=str
2921
2922 Sometimes you want to ignore some errors during test in that case you can
a35ef7cb
TK
2923 specify error list for each error type, instead of only being able to
2924 ignore the default 'non-fatal error' using :option:`continue_on_error`.
f80dba8d
MT
2925 ``ignore_error=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST`` errors for
2926 given error type is separated with ':'. Error may be symbol ('ENOSPC',
2927 'ENOMEM') or integer. Example::
2928
2929 ignore_error=EAGAIN,ENOSPC:122
2930
2931 This option will ignore EAGAIN from READ, and ENOSPC and 122(EDQUOT) from
a35ef7cb
TK
2932 WRITE. This option works by overriding :option:`continue_on_error` with
2933 the list of errors for each error type if any.
f80dba8d
MT
2934
2935.. option:: error_dump=bool
2936
2937 If set dump every error even if it is non fatal, true by default. If
2938 disabled only fatal error will be dumped.
2939
f75ede1d
SW
2940Running predefined workloads
2941----------------------------
2942
2943Fio includes predefined profiles that mimic the I/O workloads generated by
2944other tools.
2945
2946.. option:: profile=str
2947
2948 The predefined workload to run. Current profiles are:
2949
2950 **tiobench**
2951 Threaded I/O bench (tiotest/tiobench) like workload.
2952
2953 **act**
2954 Aerospike Certification Tool (ACT) like workload.
2955
2956To view a profile's additional options use :option:`--cmdhelp` after specifying
2957the profile. For example::
2958
f50fbdda 2959 $ fio --profile=act --cmdhelp
f75ede1d
SW
2960
2961Act profile options
2962~~~~~~~~~~~~~~~~~~~
2963
2964.. option:: device-names=str
2965 :noindex:
2966
2967 Devices to use.
2968
2969.. option:: load=int
2970 :noindex:
2971
2972 ACT load multiplier. Default: 1.
2973
2974.. option:: test-duration=time
2975 :noindex:
2976
947e0fe0
SW
2977 How long the entire test takes to run. When the unit is omitted, the value
2978 is given in seconds. Default: 24h.
f75ede1d
SW
2979
2980.. option:: threads-per-queue=int
2981 :noindex:
2982
f50fbdda 2983 Number of read I/O threads per device. Default: 8.
f75ede1d
SW
2984
2985.. option:: read-req-num-512-blocks=int
2986 :noindex:
2987
2988 Number of 512B blocks to read at the time. Default: 3.
2989
2990.. option:: large-block-op-kbytes=int
2991 :noindex:
2992
2993 Size of large block ops in KiB (writes). Default: 131072.
2994
2995.. option:: prep
2996 :noindex:
2997
2998 Set to run ACT prep phase.
2999
3000Tiobench profile options
3001~~~~~~~~~~~~~~~~~~~~~~~~
3002
3003.. option:: size=str
3004 :noindex:
3005
f50fbdda 3006 Size in MiB.
f75ede1d
SW
3007
3008.. option:: block=int
3009 :noindex:
3010
3011 Block size in bytes. Default: 4096.
3012
3013.. option:: numruns=int
3014 :noindex:
3015
3016 Number of runs.
3017
3018.. option:: dir=str
3019 :noindex:
3020
3021 Test directory.
3022
3023.. option:: threads=int
3024 :noindex:
3025
3026 Number of threads.
f80dba8d
MT
3027
3028Interpreting the output
3029-----------------------
3030
36214730
SW
3031..
3032 Example output was based on the following:
3033 TZ=UTC fio --iodepth=8 --ioengine=null --size=100M --time_based \
3034 --rate=1256k --bs=14K --name=quick --runtime=1s --name=mixed \
3035 --runtime=2m --rw=rw
3036
f80dba8d
MT
3037Fio spits out a lot of output. While running, fio will display the status of the
3038jobs created. An example of that would be::
3039
9d25d068 3040 Jobs: 1 (f=1): [_(1),M(1)][24.8%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 01m:31s]
f80dba8d 3041
36214730
SW
3042The characters inside the first set of square brackets denote the current status of
3043each thread. The first character is the first job defined in the job file, and so
3044forth. The possible values (in typical life cycle order) are:
f80dba8d
MT
3045
3046+------+-----+-----------------------------------------------------------+
3047| Idle | Run | |
3048+======+=====+===========================================================+
3049| P | | Thread setup, but not started. |
3050+------+-----+-----------------------------------------------------------+
3051| C | | Thread created. |
3052+------+-----+-----------------------------------------------------------+
3053| I | | Thread initialized, waiting or generating necessary data. |
3054+------+-----+-----------------------------------------------------------+
3055| | p | Thread running pre-reading file(s). |
3056+------+-----+-----------------------------------------------------------+
36214730
SW
3057| | / | Thread is in ramp period. |
3058+------+-----+-----------------------------------------------------------+
f80dba8d
MT
3059| | R | Running, doing sequential reads. |
3060+------+-----+-----------------------------------------------------------+
3061| | r | Running, doing random reads. |
3062+------+-----+-----------------------------------------------------------+
3063| | W | Running, doing sequential writes. |
3064+------+-----+-----------------------------------------------------------+
3065| | w | Running, doing random writes. |
3066+------+-----+-----------------------------------------------------------+
3067| | M | Running, doing mixed sequential reads/writes. |
3068+------+-----+-----------------------------------------------------------+
3069| | m | Running, doing mixed random reads/writes. |
3070+------+-----+-----------------------------------------------------------+
36214730
SW
3071| | D | Running, doing sequential trims. |
3072+------+-----+-----------------------------------------------------------+
3073| | d | Running, doing random trims. |
3074+------+-----+-----------------------------------------------------------+
3075| | F | Running, currently waiting for :manpage:`fsync(2)`. |
f80dba8d
MT
3076+------+-----+-----------------------------------------------------------+
3077| | V | Running, doing verification of written data. |
3078+------+-----+-----------------------------------------------------------+
36214730
SW
3079| f | | Thread finishing. |
3080+------+-----+-----------------------------------------------------------+
f80dba8d
MT
3081| E | | Thread exited, not reaped by main thread yet. |
3082+------+-----+-----------------------------------------------------------+
36214730 3083| _ | | Thread reaped. |
f80dba8d
MT
3084+------+-----+-----------------------------------------------------------+
3085| X | | Thread reaped, exited with an error. |
3086+------+-----+-----------------------------------------------------------+
3087| K | | Thread reaped, exited due to signal. |
3088+------+-----+-----------------------------------------------------------+
3089
36214730
SW
3090..
3091 Example output was based on the following:
3092 TZ=UTC fio --iodepth=8 --ioengine=null --size=100M --runtime=58m \
3093 --time_based --rate=2512k --bs=256K --numjobs=10 \
3094 --name=readers --rw=read --name=writers --rw=write
3095
f80dba8d 3096Fio will condense the thread string as not to take up more space on the command
36214730 3097line than needed. For instance, if you have 10 readers and 10 writers running,
f80dba8d
MT
3098the output would look like this::
3099
9d25d068 3100 Jobs: 20 (f=20): [R(10),W(10)][4.0%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 57m:36s]
f80dba8d 3101
36214730
SW
3102Note that the status string is displayed in order, so it's possible to tell which of
3103the jobs are currently doing what. In the example above this means that jobs 1--10
3104are readers and 11--20 are writers.
f80dba8d
MT
3105
3106The other values are fairly self explanatory -- number of threads currently
36214730
SW
3107running and doing I/O, the number of currently open files (f=), the estimated
3108completion percentage, the rate of I/O since last check (read speed listed first,
f50fbdda
TK
3109then write speed and optionally trim speed) in terms of bandwidth and IOPS,
3110and time to completion for the current running group. It's impossible to estimate
3111runtime of the following groups (if any).
36214730
SW
3112
3113..
3114 Example output was based on the following:
3115 TZ=UTC fio --iodepth=16 --ioengine=posixaio --filename=/tmp/fiofile \
3116 --direct=1 --size=100M --time_based --runtime=50s --rate_iops=89 \
3117 --bs=7K --name=Client1 --rw=write
3118
3119When fio is done (or interrupted by :kbd:`Ctrl-C`), it will show the data for
3120each thread, group of threads, and disks in that order. For each overall thread (or
3121group) the output looks like::
3122
3123 Client1: (groupid=0, jobs=1): err= 0: pid=16109: Sat Jun 24 12:07:54 2017
3124 write: IOPS=88, BW=623KiB/s (638kB/s)(30.4MiB/50032msec)
3125 slat (nsec): min=500, max=145500, avg=8318.00, stdev=4781.50
3126 clat (usec): min=170, max=78367, avg=4019.02, stdev=8293.31
3127 lat (usec): min=174, max=78375, avg=4027.34, stdev=8291.79
3128 clat percentiles (usec):
3129 | 1.00th=[ 302], 5.00th=[ 326], 10.00th=[ 343], 20.00th=[ 363],
3130 | 30.00th=[ 392], 40.00th=[ 404], 50.00th=[ 416], 60.00th=[ 445],
3131 | 70.00th=[ 816], 80.00th=[ 6718], 90.00th=[12911], 95.00th=[21627],
3132 | 99.00th=[43779], 99.50th=[51643], 99.90th=[68682], 99.95th=[72877],
3133 | 99.99th=[78119]
3134 bw ( KiB/s): min= 532, max= 686, per=0.10%, avg=622.87, stdev=24.82, samples= 100
3135 iops : min= 76, max= 98, avg=88.98, stdev= 3.54, samples= 100
29092211
VF
3136 lat (usec) : 250=0.04%, 500=64.11%, 750=4.81%, 1000=2.79%
3137 lat (msec) : 2=4.16%, 4=1.84%, 10=4.90%, 20=11.33%, 50=5.37%
3138 lat (msec) : 100=0.65%
36214730
SW
3139 cpu : usr=0.27%, sys=0.18%, ctx=12072, majf=0, minf=21
3140 IO depths : 1=85.0%, 2=13.1%, 4=1.8%, 8=0.1%, 16=0.0%, 32=0.0%, >=64=0.0%
3141 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
3142 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
3143 issued rwt: total=0,4450,0, short=0,0,0, dropped=0,0,0
3144 latency : target=0, window=0, percentile=100.00%, depth=8
3145
3146The job name (or first job's name when using :option:`group_reporting`) is printed,
3147along with the group id, count of jobs being aggregated, last error id seen (which
3148is 0 when there are no errors), pid/tid of that thread and the time the job/group
3149completed. Below are the I/O statistics for each data direction performed (showing
3150writes in the example above). In the order listed, they denote:
3151
3152**read/write/trim**
3153 The string before the colon shows the I/O direction the statistics
3154 are for. **IOPS** is the average I/Os performed per second. **BW**
3155 is the average bandwidth rate shown as: value in power of 2 format
3156 (value in power of 10 format). The last two values show: (**total
3157 I/O performed** in power of 2 format / **runtime** of that thread).
f80dba8d
MT
3158
3159**slat**
36214730
SW
3160 Submission latency (**min** being the minimum, **max** being the
3161 maximum, **avg** being the average, **stdev** being the standard
3162 deviation). This is the time it took to submit the I/O. For
3163 sync I/O this row is not displayed as the slat is really the
3164 completion latency (since queue/complete is one operation there).
3165 This value can be in nanoseconds, microseconds or milliseconds ---
3166 fio will choose the most appropriate base and print that (in the
3167 example above nanoseconds was the best scale). Note: in :option:`--minimal` mode
0d237712 3168 latencies are always expressed in microseconds.
f80dba8d
MT
3169
3170**clat**
3171 Completion latency. Same names as slat, this denotes the time from
3172 submission to completion of the I/O pieces. For sync I/O, clat will
3173 usually be equal (or very close) to 0, as the time from submit to
3174 complete is basically just CPU time (I/O has already been done, see slat
3175 explanation).
3176
29092211
VF
3177**lat**
3178 Total latency. Same names as slat and clat, this denotes the time from
3179 when fio created the I/O unit to completion of the I/O operation.
3180
f80dba8d 3181**bw**
36214730
SW
3182 Bandwidth statistics based on samples. Same names as the xlat stats,
3183 but also includes the number of samples taken (**samples**) and an
3184 approximate percentage of total aggregate bandwidth this thread
3185 received in its group (**per**). This last value is only really
3186 useful if the threads in this group are on the same disk, since they
3187 are then competing for disk access.
3188
3189**iops**
3190 IOPS statistics based on samples. Same names as bw.
f80dba8d 3191
29092211
VF
3192**lat (nsec/usec/msec)**
3193 The distribution of I/O completion latencies. This is the time from when
3194 I/O leaves fio and when it gets completed. Unlike the separate
3195 read/write/trim sections above, the data here and in the remaining
3196 sections apply to all I/Os for the reporting group. 250=0.04% means that
3197 0.04% of the I/Os completed in under 250us. 500=64.11% means that 64.11%
3198 of the I/Os required 250 to 499us for completion.
3199
f80dba8d
MT
3200**cpu**
3201 CPU usage. User and system time, along with the number of context
3202 switches this thread went through, usage of system and user time, and
3203 finally the number of major and minor page faults. The CPU utilization
3204 numbers are averages for the jobs in that reporting group, while the
23a8e176 3205 context and fault counters are summed.
f80dba8d
MT
3206
3207**IO depths**
a2140525
SW
3208 The distribution of I/O depths over the job lifetime. The numbers are
3209 divided into powers of 2 and each entry covers depths from that value
3210 up to those that are lower than the next entry -- e.g., 16= covers
3211 depths from 16 to 31. Note that the range covered by a depth
3212 distribution entry can be different to the range covered by the
3213 equivalent submit/complete distribution entry.
f80dba8d
MT
3214
3215**IO submit**
3216 How many pieces of I/O were submitting in a single submit call. Each
c60ebc45 3217 entry denotes that amount and below, until the previous entry -- e.g.,
a2140525
SW
3218 16=100% means that we submitted anywhere between 9 to 16 I/Os per submit
3219 call. Note that the range covered by a submit distribution entry can
3220 be different to the range covered by the equivalent depth distribution
3221 entry.
f80dba8d
MT
3222
3223**IO complete**
3224 Like the above submit number, but for completions instead.
3225
36214730
SW
3226**IO issued rwt**
3227 The number of read/write/trim requests issued, and how many of them were
3228 short or dropped.
f80dba8d 3229
29092211
VF
3230**IO latency**
3231 These values are for `--latency-target` and related options. When
3232 these options are engaged, this section describes the I/O depth required
3233 to meet the specified latency target.
71bfa161 3234
36214730
SW
3235..
3236 Example output was based on the following:
3237 TZ=UTC fio --ioengine=null --iodepth=2 --size=100M --numjobs=2 \
3238 --rate_process=poisson --io_limit=32M --name=read --bs=128k \
3239 --rate=11M --name=write --rw=write --bs=2k --rate=700k
3240
71bfa161 3241After each client has been listed, the group statistics are printed. They
f80dba8d 3242will look like this::
71bfa161 3243
f80dba8d 3244 Run status group 0 (all jobs):
36214730
SW
3245 READ: bw=20.9MiB/s (21.9MB/s), 10.4MiB/s-10.8MiB/s (10.9MB/s-11.3MB/s), io=64.0MiB (67.1MB), run=2973-3069msec
3246 WRITE: bw=1231KiB/s (1261kB/s), 616KiB/s-621KiB/s (630kB/s-636kB/s), io=64.0MiB (67.1MB), run=52747-53223msec
71bfa161 3247
36214730 3248For each data direction it prints:
71bfa161 3249
36214730
SW
3250**bw**
3251 Aggregate bandwidth of threads in this group followed by the
3252 minimum and maximum bandwidth of all the threads in this group.
3253 Values outside of brackets are power-of-2 format and those
3254 within are the equivalent value in a power-of-10 format.
f80dba8d 3255**io**
36214730
SW
3256 Aggregate I/O performed of all threads in this group. The
3257 format is the same as bw.
3258**run**
3259 The smallest and longest runtimes of the threads in this group.
71bfa161 3260
f50fbdda 3261And finally, the disk statistics are printed. This is Linux specific. They will look like this::
71bfa161 3262
f80dba8d
MT
3263 Disk stats (read/write):
3264 sda: ios=16398/16511, merge=30/162, ticks=6853/819634, in_queue=826487, util=100.00%
71bfa161
JA
3265
3266Each value is printed for both reads and writes, with reads first. The
3267numbers denote:
3268
f80dba8d 3269**ios**
c60ebc45 3270 Number of I/Os performed by all groups.
f80dba8d 3271**merge**
007c7be9 3272 Number of merges performed by the I/O scheduler.
f80dba8d
MT
3273**ticks**
3274 Number of ticks we kept the disk busy.
36214730 3275**in_queue**
f80dba8d
MT
3276 Total time spent in the disk queue.
3277**util**
3278 The disk utilization. A value of 100% means we kept the disk
71bfa161
JA
3279 busy constantly, 50% would be a disk idling half of the time.
3280
f80dba8d
MT
3281It is also possible to get fio to dump the current output while it is running,
3282without terminating the job. To do that, send fio the **USR1** signal. You can
3283also get regularly timed dumps by using the :option:`--status-interval`
3284parameter, or by creating a file in :file:`/tmp` named
3285:file:`fio-dump-status`. If fio sees this file, it will unlink it and dump the
3286current output status.
8423bd11 3287
71bfa161 3288
f80dba8d
MT
3289Terse output
3290------------
71bfa161 3291
f80dba8d
MT
3292For scripted usage where you typically want to generate tables or graphs of the
3293results, fio can output the results in a semicolon separated format. The format
3294is one long line of values, such as::
71bfa161 3295
f80dba8d
MT
3296 2;card0;0;0;7139336;121836;60004;1;10109;27.932460;116.933948;220;126861;3495.446807;1085.368601;226;126864;3523.635629;1089.012448;24063;99944;50.275485%;59818.274627;5540.657370;7155060;122104;60004;1;8338;29.086342;117.839068;388;128077;5032.488518;1234.785715;391;128085;5061.839412;1236.909129;23436;100928;50.287926%;59964.832030;5644.844189;14.595833%;19.394167%;123706;0;7313;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.02%;0.05%;0.16%;6.04%;40.40%;52.68%;0.64%;0.01%;0.00%;0.01%;0.00%;0.00%;0.00%;0.00%;0.00%
3297 A description of this job goes here.
562c2d2f
DN
3298
3299The job description (if provided) follows on a second line.
71bfa161 3300
a7f77fa6
SW
3301To enable terse output, use the :option:`--minimal` or
3302:option:`--output-format`\=terse command line options. The
f80dba8d
MT
3303first value is the version of the terse output format. If the output has to be
3304changed for some reason, this number will be incremented by 1 to signify that
3305change.
6820cb3b 3306
a2c95580 3307Split up, the format is as follows (comments in brackets denote when a
007c7be9 3308field was introduced or whether it's specific to some terse version):
71bfa161 3309
f80dba8d
MT
3310 ::
3311
f50fbdda 3312 terse version, fio version [v3], jobname, groupid, error
f80dba8d
MT
3313
3314 READ status::
3315
3316 Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
3317 Submission latency: min, max, mean, stdev (usec)
3318 Completion latency: min, max, mean, stdev (usec)
3319 Completion latency percentiles: 20 fields (see below)
3320 Total latency: min, max, mean, stdev (usec)
f50fbdda
TK
3321 Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev, number of samples [v5]
3322 IOPS [v5]: min, max, mean, stdev, number of samples
f80dba8d
MT
3323
3324 WRITE status:
3325
3326 ::
3327
3328 Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
3329 Submission latency: min, max, mean, stdev (usec)
247823cc 3330 Completion latency: min, max, mean, stdev (usec)
f80dba8d
MT
3331 Completion latency percentiles: 20 fields (see below)
3332 Total latency: min, max, mean, stdev (usec)
f50fbdda
TK
3333 Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev, number of samples [v5]
3334 IOPS [v5]: min, max, mean, stdev, number of samples
a2c95580
AH
3335
3336 TRIM status [all but version 3]:
3337
f50fbdda 3338 Fields are similar to READ/WRITE status.
f80dba8d
MT
3339
3340 CPU usage::
3341
3342 user, system, context switches, major faults, minor faults
3343
3344 I/O depths::
3345
3346 <=1, 2, 4, 8, 16, 32, >=64
3347
3348 I/O latencies microseconds::
3349
3350 <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000
3351
3352 I/O latencies milliseconds::
3353
3354 <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000
3355
a2c95580 3356 Disk utilization [v3]::
f80dba8d 3357
f50fbdda
TK
3358 disk name, read ios, write ios, read merges, write merges, read ticks, write ticks,
3359 time spent in queue, disk utilization percentage
f80dba8d
MT
3360
3361 Additional Info (dependent on continue_on_error, default off)::
3362
3363 total # errors, first error code
3364
3365 Additional Info (dependent on description being set)::
3366
3367 Text description
3368
3369Completion latency percentiles can be a grouping of up to 20 sets, so for the
3370terse output fio writes all of them. Each field will look like this::
1db92cb6 3371
f50fbdda 3372 1.00%=6112
1db92cb6 3373
f80dba8d 3374which is the Xth percentile, and the `usec` latency associated with it.
1db92cb6 3375
f50fbdda 3376For `Disk utilization`, all disks used by fio are shown. So for each disk there
f80dba8d 3377will be a disk utilization section.
f2f788dd 3378
2fc26c3d 3379Below is a single line containing short names for each of the fields in the
2831be97 3380minimal output v3, separated by semicolons::
2fc26c3d 3381
f50fbdda 3382 terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth;read_iops;read_runtime_ms;read_slat_min;read_slat_max;read_slat_mean;read_slat_dev;read_clat_min;read_clat_max;read_clat_mean;read_clat_dev;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min;read_lat_max;read_lat_mean;read_lat_dev;read_bw_min;read_bw_max;read_bw_agg_pct;read_bw_mean;read_bw_dev;write_kb;write_bandwidth;write_iops;write_runtime_ms;write_slat_min;write_slat_max;write_slat_mean;write_slat_dev;write_clat_min;write_clat_max;write_clat_mean;write_clat_dev;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min;write_lat_max;write_lat_mean;write_lat_dev;write_bw_min;write_bw_max;write_bw_agg_pct;write_bw_mean;write_bw_dev;cpu_user;cpu_sys;cpu_csw;cpu_mjf;cpu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util
2fc26c3d 3383
25c8b9d7 3384
d29c4a91
VF
3385JSON+ output
3386------------
3387
3388The `json+` output format is identical to the `json` output format except that it
3389adds a full dump of the completion latency bins. Each `bins` object contains a
3390set of (key, value) pairs where keys are latency durations and values count how
3391many I/Os had completion latencies of the corresponding duration. For example,
3392consider:
3393
3394 "bins" : { "87552" : 1, "89600" : 1, "94720" : 1, "96768" : 1, "97792" : 1, "99840" : 1, "100864" : 2, "103936" : 6, "104960" : 534, "105984" : 5995, "107008" : 7529, ... }
3395
3396This data indicates that one I/O required 87,552ns to complete, two I/Os required
3397100,864ns to complete, and 7529 I/Os required 107,008ns to complete.
3398
3399Also included with fio is a Python script `fio_jsonplus_clat2csv` that takes
3400json+ output and generates CSV-formatted latency data suitable for plotting.
3401
3402The latency durations actually represent the midpoints of latency intervals.
f50fbdda 3403For details refer to :file:`stat.h`.
d29c4a91
VF
3404
3405
f80dba8d
MT
3406Trace file format
3407-----------------
3408
3409There are two trace file format that you can encounter. The older (v1) format is
3410unsupported since version 1.20-rc3 (March 2008). It will still be described
25c8b9d7
PD
3411below in case that you get an old trace and want to understand it.
3412
3413In any case the trace is a simple text file with a single action per line.
3414
3415
f80dba8d
MT
3416Trace file format v1
3417~~~~~~~~~~~~~~~~~~~~
3418
3419Each line represents a single I/O action in the following format::
3420
3421 rw, offset, length
25c8b9d7 3422
f50fbdda 3423where `rw=0/1` for read/write, and the `offset` and `length` entries being in bytes.
25c8b9d7 3424
22413915 3425This format is not supported in fio versions >= 1.20-rc3.
25c8b9d7 3426
25c8b9d7 3427
f80dba8d
MT
3428Trace file format v2
3429~~~~~~~~~~~~~~~~~~~~
25c8b9d7 3430
f80dba8d
MT
3431The second version of the trace file format was added in fio version 1.17. It
3432allows to access more then one file per trace and has a bigger set of possible
3433file actions.
25c8b9d7 3434
f80dba8d 3435The first line of the trace file has to be::
25c8b9d7 3436
f80dba8d 3437 fio version 2 iolog
25c8b9d7
PD
3438
3439Following this can be lines in two different formats, which are described below.
3440
f80dba8d 3441The file management format::
25c8b9d7 3442
f80dba8d 3443 filename action
25c8b9d7 3444
f50fbdda 3445The `filename` is given as an absolute path. The `action` can be one of these:
25c8b9d7 3446
f80dba8d 3447**add**
f50fbdda 3448 Add the given `filename` to the trace.
f80dba8d 3449**open**
f50fbdda 3450 Open the file with the given `filename`. The `filename` has to have
f80dba8d
MT
3451 been added with the **add** action before.
3452**close**
f50fbdda 3453 Close the file with the given `filename`. The file has to have been
f80dba8d
MT
3454 opened before.
3455
3456
3457The file I/O action format::
3458
3459 filename action offset length
3460
3461The `filename` is given as an absolute path, and has to have been added and
3462opened before it can be used with this format. The `offset` and `length` are
3463given in bytes. The `action` can be one of these:
3464
3465**wait**
3466 Wait for `offset` microseconds. Everything below 100 is discarded.
3467 The time is relative to the previous `wait` statement.
3468**read**
3469 Read `length` bytes beginning from `offset`.
3470**write**
3471 Write `length` bytes beginning from `offset`.
3472**sync**
3473 :manpage:`fsync(2)` the file.
3474**datasync**
3475 :manpage:`fdatasync(2)` the file.
3476**trim**
3477 Trim the given file from the given `offset` for `length` bytes.
3478
3479CPU idleness profiling
3480----------------------
3481
3482In some cases, we want to understand CPU overhead in a test. For example, we
3483test patches for the specific goodness of whether they reduce CPU usage.
3484Fio implements a balloon approach to create a thread per CPU that runs at idle
3485priority, meaning that it only runs when nobody else needs the cpu.
3486By measuring the amount of work completed by the thread, idleness of each CPU
3487can be derived accordingly.
3488
3489An unit work is defined as touching a full page of unsigned characters. Mean and
3490standard deviation of time to complete an unit work is reported in "unit work"
3491section. Options can be chosen to report detailed percpu idleness or overall
3492system idleness by aggregating percpu stats.
3493
3494
3495Verification and triggers
3496-------------------------
3497
3498Fio is usually run in one of two ways, when data verification is done. The first
3499is a normal write job of some sort with verify enabled. When the write phase has
3500completed, fio switches to reads and verifies everything it wrote. The second
3501model is running just the write phase, and then later on running the same job
3502(but with reads instead of writes) to repeat the same I/O patterns and verify
3503the contents. Both of these methods depend on the write phase being completed,
3504as fio otherwise has no idea how much data was written.
3505
3506With verification triggers, fio supports dumping the current write state to
3507local files. Then a subsequent read verify workload can load this state and know
3508exactly where to stop. This is useful for testing cases where power is cut to a
3509server in a managed fashion, for instance.
99b9a85a
JA
3510
3511A verification trigger consists of two things:
3512
f80dba8d
MT
35131) Storing the write state of each job.
35142) Executing a trigger command.
99b9a85a 3515
f80dba8d
MT
3516The write state is relatively small, on the order of hundreds of bytes to single
3517kilobytes. It contains information on the number of completions done, the last X
3518completions, etc.
99b9a85a 3519
f80dba8d
MT
3520A trigger is invoked either through creation ('touch') of a specified file in
3521the system, or through a timeout setting. If fio is run with
9207a0cb 3522:option:`--trigger-file`\= :file:`/tmp/trigger-file`, then it will continually
f80dba8d
MT
3523check for the existence of :file:`/tmp/trigger-file`. When it sees this file, it
3524will fire off the trigger (thus saving state, and executing the trigger
99b9a85a
JA
3525command).
3526
f80dba8d
MT
3527For client/server runs, there's both a local and remote trigger. If fio is
3528running as a server backend, it will send the job states back to the client for
3529safe storage, then execute the remote trigger, if specified. If a local trigger
3530is specified, the server will still send back the write state, but the client
3531will then execute the trigger.
99b9a85a 3532
f80dba8d
MT
3533Verification trigger example
3534~~~~~~~~~~~~~~~~~~~~~~~~~~~~
99b9a85a 3535
f50fbdda
TK
3536Let's say we want to run a powercut test on the remote Linux machine 'server'.
3537Our write workload is in :file:`write-test.fio`. We want to cut power to 'server' at
f80dba8d
MT
3538some point during the run, and we'll run this test from the safety or our local
3539machine, 'localbox'. On the server, we'll start the fio backend normally::
99b9a85a 3540
f80dba8d 3541 server# fio --server
99b9a85a 3542
f80dba8d 3543and on the client, we'll fire off the workload::
99b9a85a 3544
f80dba8d 3545 localbox$ fio --client=server --trigger-file=/tmp/my-trigger --trigger-remote="bash -c \"echo b > /proc/sysrq-triger\""
99b9a85a 3546
f80dba8d 3547We set :file:`/tmp/my-trigger` as the trigger file, and we tell fio to execute::
99b9a85a 3548
f80dba8d 3549 echo b > /proc/sysrq-trigger
99b9a85a 3550
f80dba8d
MT
3551on the server once it has received the trigger and sent us the write state. This
3552will work, but it's not **really** cutting power to the server, it's merely
3553abruptly rebooting it. If we have a remote way of cutting power to the server
3554through IPMI or similar, we could do that through a local trigger command
4502cb42 3555instead. Let's assume we have a script that does IPMI reboot of a given hostname,
f80dba8d
MT
3556ipmi-reboot. On localbox, we could then have run fio with a local trigger
3557instead::
99b9a85a 3558
f80dba8d 3559 localbox$ fio --client=server --trigger-file=/tmp/my-trigger --trigger="ipmi-reboot server"
99b9a85a 3560
f80dba8d
MT
3561For this case, fio would wait for the server to send us the write state, then
3562execute ``ipmi-reboot server`` when that happened.
3563
3564Loading verify state
3565~~~~~~~~~~~~~~~~~~~~
3566
4502cb42 3567To load stored write state, a read verification job file must contain the
f80dba8d 3568:option:`verify_state_load` option. If that is set, fio will load the previously
99b9a85a 3569stored state. For a local fio run this is done by loading the files directly,
f80dba8d
MT
3570and on a client/server run, the server backend will ask the client to send the
3571files over and load them from there.
a3ae5b05
JA
3572
3573
f80dba8d
MT
3574Log File Formats
3575----------------
a3ae5b05
JA
3576
3577Fio supports a variety of log file formats, for logging latencies, bandwidth,
3578and IOPS. The logs share a common format, which looks like this:
3579
5a83478f
SW
3580 *time* (`msec`), *value*, *data direction*, *block size* (`bytes`),
3581 *offset* (`bytes`)
a3ae5b05 3582
5a83478f 3583*Time* for the log entry is always in milliseconds. The *value* logged depends
a3ae5b05
JA
3584on the type of log, it will be one of the following:
3585
f80dba8d 3586 **Latency log**
168bb587 3587 Value is latency in nsecs
f80dba8d
MT
3588 **Bandwidth log**
3589 Value is in KiB/sec
3590 **IOPS log**
3591 Value is IOPS
3592
3593*Data direction* is one of the following:
3594
3595 **0**
3596 I/O is a READ
3597 **1**
3598 I/O is a WRITE
3599 **2**
3600 I/O is a TRIM
3601
5a83478f
SW
3602The entry's *block size* is always in bytes. The *offset* is the offset, in bytes,
3603from the start of the file, for that particular I/O. The logging of the offset can be
3604toggled with :option:`log_offset`.
f80dba8d 3605
6fc82095 3606Fio defaults to logging every individual I/O. When IOPS are logged for individual
5a83478f 3607I/Os the *value* entry will always be 1. If windowed logging is enabled through
6fc82095
SW
3608:option:`log_avg_msec`, fio logs the average values over the specified period of time.
3609If windowed logging is enabled and :option:`log_max_value` is set, then fio logs
5a83478f
SW
3610maximum values in that window instead of averages. Since *data direction*, *block
3611size* and *offset* are per-I/O values, if windowed logging is enabled they
3612aren't applicable and will be 0.
f80dba8d 3613
b8f7e412 3614Client/Server
f80dba8d
MT
3615-------------
3616
3617Normally fio is invoked as a stand-alone application on the machine where the
6cf30ac0
SW
3618I/O workload should be generated. However, the backend and frontend of fio can
3619be run separately i.e., the fio server can generate an I/O workload on the "Device
3620Under Test" while being controlled by a client on another machine.
f80dba8d
MT
3621
3622Start the server on the machine which has access to the storage DUT::
3623
f50fbdda 3624 $ fio --server=args
f80dba8d 3625
dbb257bb 3626where `args` defines what fio listens to. The arguments are of the form
f80dba8d
MT
3627``type,hostname`` or ``IP,port``. *type* is either ``ip`` (or ip4) for TCP/IP
3628v4, ``ip6`` for TCP/IP v6, or ``sock`` for a local unix domain socket.
3629*hostname* is either a hostname or IP address, and *port* is the port to listen
3630to (only valid for TCP/IP, not a local socket). Some examples:
3631
36321) ``fio --server``
3633
3634 Start a fio server, listening on all interfaces on the default port (8765).
3635
36362) ``fio --server=ip:hostname,4444``
3637
3638 Start a fio server, listening on IP belonging to hostname and on port 4444.
3639
36403) ``fio --server=ip6:::1,4444``
3641
3642 Start a fio server, listening on IPv6 localhost ::1 and on port 4444.
3643
36444) ``fio --server=,4444``
3645
3646 Start a fio server, listening on all interfaces on port 4444.
3647
36485) ``fio --server=1.2.3.4``
3649
3650 Start a fio server, listening on IP 1.2.3.4 on the default port.
3651
36526) ``fio --server=sock:/tmp/fio.sock``
3653
dbb257bb 3654 Start a fio server, listening on the local socket :file:`/tmp/fio.sock`.
f80dba8d
MT
3655
3656Once a server is running, a "client" can connect to the fio server with::
3657
3658 fio <local-args> --client=<server> <remote-args> <job file(s)>
3659
3660where `local-args` are arguments for the client where it is running, `server`
3661is the connect string, and `remote-args` and `job file(s)` are sent to the
3662server. The `server` string follows the same format as it does on the server
3663side, to allow IP/hostname/socket and port strings.
3664
3665Fio can connect to multiple servers this way::
3666
3667 fio --client=<server1> <job file(s)> --client=<server2> <job file(s)>
3668
3669If the job file is located on the fio server, then you can tell the server to
3670load a local file as well. This is done by using :option:`--remote-config` ::
3671
3672 fio --client=server --remote-config /path/to/file.fio
3673
3674Then fio will open this local (to the server) job file instead of being passed
3675one from the client.
3676
3677If you have many servers (example: 100 VMs/containers), you can input a pathname
3678of a file containing host IPs/names as the parameter value for the
3679:option:`--client` option. For example, here is an example :file:`host.list`
3680file containing 2 hostnames::
3681
3682 host1.your.dns.domain
3683 host2.your.dns.domain
3684
3685The fio command would then be::
a3ae5b05 3686
f80dba8d 3687 fio --client=host.list <job file(s)>
a3ae5b05 3688
f80dba8d
MT
3689In this mode, you cannot input server-specific parameters or job files -- all
3690servers receive the same job file.
a3ae5b05 3691
f80dba8d
MT
3692In order to let ``fio --client`` runs use a shared filesystem from multiple
3693hosts, ``fio --client`` now prepends the IP address of the server to the
4502cb42 3694filename. For example, if fio is using the directory :file:`/mnt/nfs/fio` and is
f80dba8d
MT
3695writing filename :file:`fileio.tmp`, with a :option:`--client` `hostfile`
3696containing two hostnames ``h1`` and ``h2`` with IP addresses 192.168.10.120 and
3697192.168.10.121, then fio will create two files::
a3ae5b05 3698
f80dba8d
MT
3699 /mnt/nfs/fio/192.168.10.120.fileio.tmp
3700 /mnt/nfs/fio/192.168.10.121.fileio.tmp