doc: minor consistency and spelling changes
[fio.git] / HOWTO
CommitLineData
f80dba8d
MT
1How fio works
2-------------
3
4The first step in getting fio to simulate a desired I/O workload, is writing a
5job file describing that specific setup. A job file may contain any number of
6threads and/or files -- the typical contents of the job file is a *global*
7section defining shared parameters, and one or more job sections describing the
8jobs involved. When run, fio parses this file and sets everything up as
9described. If we break down a job from top to bottom, it contains the following
10basic parameters:
11
12`I/O type`_
13
14 Defines the I/O pattern issued to the file(s). We may only be reading
15 sequentially from this file(s), or we may be writing randomly. Or even
16 mixing reads and writes, sequentially or randomly.
17 Should we be doing buffered I/O, or direct/raw I/O?
18
19`Block size`_
20
21 In how large chunks are we issuing I/O? This may be a single value,
22 or it may describe a range of block sizes.
23
24`I/O size`_
25
26 How much data are we going to be reading/writing.
27
28`I/O engine`_
29
30 How do we issue I/O? We could be memory mapping the file, we could be
31 using regular read/write, we could be using splice, async I/O, or even
32 SG (SCSI generic sg).
33
34`I/O depth`_
35
36 If the I/O engine is async, how large a queuing depth do we want to
37 maintain?
38
39
40`Target file/device`_
41
42 How many files are we spreading the workload over.
43
44`Threads, processes and job synchronization`_
45
46 How many threads or processes should we spread this workload over.
47
48The above are the basic parameters defined for a workload, in addition there's a
49multitude of parameters that modify other aspects of how this job behaves.
50
51
52Command line options
53--------------------
54
55.. option:: --debug=type
56
57 Enable verbose tracing of various fio actions. May be ``all`` for all types
c60ebc45 58 or individual types separated by a comma (e.g. ``--debug=file,mem`` will
f80dba8d
MT
59 enable file and memory debugging). Currently, additional logging is
60 available for:
61
62 *process*
63 Dump info related to processes.
64 *file*
65 Dump info related to file actions.
66 *io*
67 Dump info related to I/O queuing.
68 *mem*
69 Dump info related to memory allocations.
70 *blktrace*
71 Dump info related to blktrace setup.
72 *verify*
73 Dump info related to I/O verification.
74 *all*
75 Enable all debug options.
76 *random*
77 Dump info related to random offset generation.
78 *parse*
79 Dump info related to option matching and parsing.
80 *diskutil*
81 Dump info related to disk utilization updates.
82 *job:x*
83 Dump info only related to job number x.
84 *mutex*
85 Dump info only related to mutex up/down ops.
86 *profile*
87 Dump info related to profile extensions.
88 *time*
89 Dump info related to internal time keeping.
90 *net*
91 Dump info related to networking connections.
92 *rate*
93 Dump info related to I/O rate switching.
94 *compress*
95 Dump info related to log compress/decompress.
96 *?* or *help*
97 Show available debug options.
98
99.. option:: --parse-only
100
101 Parse options only, don\'t start any I/O.
102
103.. option:: --output=filename
104
105 Write output to file `filename`.
106
107.. option:: --bandwidth-log
108
109 Generate aggregate bandwidth logs.
110
111.. option:: --minimal
112
113 Print statistics in a terse, semicolon-delimited format.
114
115.. option:: --append-terse
116
117 Print statistics in selected mode AND terse, semicolon-delimited format.
118 **deprecated**, use :option:`--output-format` instead to select multiple
119 formats.
120
121.. option:: --output-format=type
122
123 Set the reporting format to `normal`, `terse`, `json`, or `json+`. Multiple
124 formats can be selected, separate by a comma. `terse` is a CSV based
125 format. `json+` is like `json`, except it adds a full dump of the latency
126 buckets.
127
128.. option:: --terse-version=type
129
130 Set terse version output format (default 3, or 2 or 4).
131
132.. option:: --version
133
134 Print version info and exit.
135
136.. option:: --help
137
138 Print this page.
139
140.. option:: --cpuclock-test
141
142 Perform test and validation of internal CPU clock.
143
144.. option:: --crctest=test
145
146 Test the speed of the builtin checksumming functions. If no argument is
147 given, all of them are tested. Or a comma separated list can be passed, in
148 which case the given ones are tested.
149
150.. option:: --cmdhelp=command
151
152 Print help information for `command`. May be ``all`` for all commands.
153
154.. option:: --enghelp=[ioengine[,command]]
155
156 List all commands defined by :option:`ioengine`, or print help for `command`
157 defined by :option:`ioengine`. If no :option:`ioengine` is given, list all
158 available ioengines.
159
160.. option:: --showcmd=jobfile
161
162 Turn a job file into command line options.
163
164.. option:: --readonly
165
166 Turn on safety read-only checks, preventing writes. The ``--readonly``
167 option is an extra safety guard to prevent users from accidentally starting
168 a write workload when that is not desired. Fio will only write if
169 `rw=write/randwrite/rw/randrw` is given. This extra safety net can be used
170 as an extra precaution as ``--readonly`` will also enable a write check in
171 the I/O engine core to prevent writes due to unknown user space bug(s).
172
173.. option:: --eta=when
174
175 When real-time ETA estimate should be printed. May be `always`, `never` or
176 `auto`.
177
178.. option:: --eta-newline=time
179
180 Force a new line for every `time` period passed.
181
182.. option:: --status-interval=time
183
184 Force full status dump every `time` period passed.
185
186.. option:: --section=name
187
188 Only run specified section in job file. Multiple sections can be specified.
189 The ``--section`` option allows one to combine related jobs into one file.
190 E.g. one job file could define light, moderate, and heavy sections. Tell
191 fio to run only the "heavy" section by giving ``--section=heavy``
192 command line option. One can also specify the "write" operations in one
193 section and "verify" operation in another section. The ``--section`` option
194 only applies to job sections. The reserved *global* section is always
195 parsed and used.
196
197.. option:: --alloc-size=kb
198
199 Set the internal smalloc pool to this size in kb (def 1024). The
200 ``--alloc-size`` switch allows one to use a larger pool size for smalloc.
201 If running large jobs with randommap enabled, fio can run out of memory.
202 Smalloc is an internal allocator for shared structures from a fixed size
203 memory pool. The pool size defaults to 16M and can grow to 8 pools.
204
205 NOTE: While running :file:`.fio_smalloc.*` backing store files are visible
206 in :file:`/tmp`.
207
208.. option:: --warnings-fatal
209
210 All fio parser warnings are fatal, causing fio to exit with an
211 error.
212
213.. option:: --max-jobs=nr
214
215 Maximum number of threads/processes to support.
216
217.. option:: --server=args
218
219 Start a backend server, with `args` specifying what to listen to.
220 See `Client/Server`_ section.
221
222.. option:: --daemonize=pidfile
223
224 Background a fio server, writing the pid to the given `pidfile` file.
225
226.. option:: --client=hostname
227
228 Instead of running the jobs locally, send and run them on the given host or
229 set of hosts. See `Client/Server`_ section.
230
231.. option:: --remote-config=file
232
233 Tell fio server to load this local file.
234
235.. option:: --idle-prof=option
236
237 Report cpu idleness on a system or percpu basis
238 ``--idle-prof=system,percpu`` or
239 run unit work calibration only ``--idle-prof=calibrate``.
240
241.. option:: --inflate-log=log
242
243 Inflate and output compressed log.
244
245.. option:: --trigger-file=file
246
247 Execute trigger cmd when file exists.
248
249.. option:: --trigger-timeout=t
250
251 Execute trigger at this time.
252
253.. option:: --trigger=cmd
254
255 Set this command as local trigger.
256
257.. option:: --trigger-remote=cmd
258
259 Set this command as remote trigger.
260
261.. option:: --aux-path=path
262
263 Use this path for fio state generated files.
264
265Any parameters following the options will be assumed to be job files, unless
266they match a job file parameter. Multiple job files can be listed and each job
267file will be regarded as a separate group. Fio will :option:`stonewall`
268execution between each group.
269
270
271Job file format
272---------------
273
274As previously described, fio accepts one or more job files describing what it is
275supposed to do. The job file format is the classic ini file, where the names
c60ebc45 276enclosed in [] brackets define the job name. You are free to use any ASCII name
f80dba8d
MT
277you want, except *global* which has special meaning. Following the job name is
278a sequence of zero or more parameters, one per line, that define the behavior of
279the job. If the first character in a line is a ';' or a '#', the entire line is
280discarded as a comment.
281
282A *global* section sets defaults for the jobs described in that file. A job may
283override a *global* section parameter, and a job file may even have several
284*global* sections if so desired. A job is only affected by a *global* section
285residing above it.
286
287The :option:`--cmdhelp` option also lists all options. If used with an `option`
288argument, :option:`--cmdhelp` will detail the given `option`.
289
290See the `examples/` directory for inspiration on how to write job files. Note
291the copyright and license requirements currently apply to `examples/` files.
292
293So let's look at a really simple job file that defines two processes, each
294randomly reading from a 128MiB file:
295
296.. code-block:: ini
297
298 ; -- start job file --
299 [global]
300 rw=randread
301 size=128m
302
303 [job1]
304
305 [job2]
306
307 ; -- end job file --
308
309As you can see, the job file sections themselves are empty as all the described
310parameters are shared. As no :option:`filename` option is given, fio makes up a
311`filename` for each of the jobs as it sees fit. On the command line, this job
312would look as follows::
313
314$ fio --name=global --rw=randread --size=128m --name=job1 --name=job2
315
316
317Let's look at an example that has a number of processes writing randomly to
318files:
319
320.. code-block:: ini
321
322 ; -- start job file --
323 [random-writers]
324 ioengine=libaio
325 iodepth=4
326 rw=randwrite
327 bs=32k
328 direct=0
329 size=64m
330 numjobs=4
331 ; -- end job file --
332
333Here we have no *global* section, as we only have one job defined anyway. We
334want to use async I/O here, with a depth of 4 for each file. We also increased
335the buffer size used to 32KiB and define numjobs to 4 to fork 4 identical
336jobs. The result is 4 processes each randomly writing to their own 64MiB
337file. Instead of using the above job file, you could have given the parameters
338on the command line. For this case, you would specify::
339
340$ fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4
341
342When fio is utilized as a basis of any reasonably large test suite, it might be
343desirable to share a set of standardized settings across multiple job files.
344Instead of copy/pasting such settings, any section may pull in an external
345:file:`filename.fio` file with *include filename* directive, as in the following
346example::
347
348 ; -- start job file including.fio --
349 [global]
350 filename=/tmp/test
351 filesize=1m
352 include glob-include.fio
353
354 [test]
355 rw=randread
356 bs=4k
357 time_based=1
358 runtime=10
359 include test-include.fio
360 ; -- end job file including.fio --
361
362.. code-block:: ini
363
364 ; -- start job file glob-include.fio --
365 thread=1
366 group_reporting=1
367 ; -- end job file glob-include.fio --
368
369.. code-block:: ini
370
371 ; -- start job file test-include.fio --
372 ioengine=libaio
373 iodepth=4
374 ; -- end job file test-include.fio --
375
376Settings pulled into a section apply to that section only (except *global*
377section). Include directives may be nested in that any included file may contain
378further include directive(s). Include files may not contain [] sections.
379
380
381Environment variables
382~~~~~~~~~~~~~~~~~~~~~
383
384Fio also supports environment variable expansion in job files. Any sub-string of
385the form ``${VARNAME}`` as part of an option value (in other words, on the right
386of the '='), will be expanded to the value of the environment variable called
387`VARNAME`. If no such environment variable is defined, or `VARNAME` is the
388empty string, the empty string will be substituted.
389
390As an example, let's look at a sample fio invocation and job file::
391
392$ SIZE=64m NUMJOBS=4 fio jobfile.fio
393
394.. code-block:: ini
395
396 ; -- start job file --
397 [random-writers]
398 rw=randwrite
399 size=${SIZE}
400 numjobs=${NUMJOBS}
401 ; -- end job file --
402
403This will expand to the following equivalent job file at runtime:
404
405.. code-block:: ini
406
407 ; -- start job file --
408 [random-writers]
409 rw=randwrite
410 size=64m
411 numjobs=4
412 ; -- end job file --
413
414Fio ships with a few example job files, you can also look there for inspiration.
415
416Reserved keywords
417~~~~~~~~~~~~~~~~~
418
419Additionally, fio has a set of reserved keywords that will be replaced
420internally with the appropriate value. Those keywords are:
421
422**$pagesize**
423
424 The architecture page size of the running system.
425
426**$mb_memory**
427
428 Megabytes of total memory in the system.
429
430**$ncpus**
431
432 Number of online available CPUs.
433
434These can be used on the command line or in the job file, and will be
435automatically substituted with the current system values when the job is
436run. Simple math is also supported on these keywords, so you can perform actions
437like::
438
439 size=8*$mb_memory
440
441and get that properly expanded to 8 times the size of memory in the machine.
442
443
444Job file parameters
445-------------------
446
447This section describes in details each parameter associated with a job. Some
448parameters take an option of a given type, such as an integer or a
449string. Anywhere a numeric value is required, an arithmetic expression may be
450used, provided it is surrounded by parentheses. Supported operators are:
451
452 - addition (+)
453 - subtraction (-)
454 - multiplication (*)
455 - division (/)
456 - modulus (%)
457 - exponentiation (^)
458
459For time values in expressions, units are microseconds by default. This is
460different than for time values not in expressions (not enclosed in
461parentheses). The following types are used:
462
463
464Parameter types
465~~~~~~~~~~~~~~~
466
467**str**
468 String. This is a sequence of alpha characters.
469
470**time**
471 Integer with possible time suffix. In seconds unless otherwise
c60ebc45 472 specified, use e.g. 10m for 10 minutes. Accepts s/m/h for seconds, minutes,
f80dba8d
MT
473 and hours, and accepts 'ms' (or 'msec') for milliseconds, and 'us' (or
474 'usec') for microseconds.
475
476.. _int:
477
478**int**
479 Integer. A whole number value, which may contain an integer prefix
480 and an integer suffix:
481
482 [*integer prefix*] **number** [*integer suffix*]
483
484 The optional *integer prefix* specifies the number's base. The default
485 is decimal. *0x* specifies hexadecimal.
486
487 The optional *integer suffix* specifies the number's units, and includes an
488 optional unit prefix and an optional unit. For quantities of data, the
489 default unit is bytes. For quantities of time, the default unit is seconds.
490
491 With :option:`kb_base` =1000, fio follows international standards for unit
492 prefixes. To specify power-of-10 decimal values defined in the
493 International System of Units (SI):
494
495 * *Ki* -- means kilo (K) or 1000
496 * *Mi* -- means mega (M) or 1000**2
497 * *Gi* -- means giga (G) or 1000**3
498 * *Ti* -- means tera (T) or 1000**4
499 * *Pi* -- means peta (P) or 1000**5
500
501 To specify power-of-2 binary values defined in IEC 80000-13:
502
503 * *k* -- means kibi (Ki) or 1024
504 * *M* -- means mebi (Mi) or 1024**2
505 * *G* -- means gibi (Gi) or 1024**3
506 * *T* -- means tebi (Ti) or 1024**4
507 * *P* -- means pebi (Pi) or 1024**5
508
509 With :option:`kb_base` =1024 (the default), the unit prefixes are opposite
510 from those specified in the SI and IEC 80000-13 standards to provide
511 compatibility with old scripts. For example, 4k means 4096.
512
513 For quantities of data, an optional unit of 'B' may be included
514 (e.g., 'kB' is the same as 'k').
515
516 The *integer suffix* is not case sensitive (e.g., m/mi mean mebi/mega,
517 not milli). 'b' and 'B' both mean byte, not bit.
518
519 Examples with :option:`kb_base` =1000:
520
521 * *4 KiB*: 4096, 4096b, 4096B, 4ki, 4kib, 4kiB, 4Ki, 4KiB
522 * *1 MiB*: 1048576, 1mi, 1024ki
523 * *1 MB*: 1000000, 1m, 1000k
524 * *1 TiB*: 1099511627776, 1ti, 1024gi, 1048576mi
525 * *1 TB*: 1000000000, 1t, 1000m, 1000000k
526
527 Examples with :option:`kb_base` =1024 (default):
528
529 * *4 KiB*: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
530 * *1 MiB*: 1048576, 1m, 1024k
531 * *1 MB*: 1000000, 1mi, 1000ki
532 * *1 TiB*: 1099511627776, 1t, 1024g, 1048576m
533 * *1 TB*: 1000000000, 1ti, 1000mi, 1000000ki
534
535 To specify times (units are not case sensitive):
536
537 * *D* -- means days
538 * *H* -- means hours
539 * *M* -- mean minutes
540 * *s* -- or sec means seconds (default)
541 * *ms* -- or *msec* means milliseconds
542 * *us* -- or *usec* means microseconds
543
544 If the option accepts an upper and lower range, use a colon ':' or
545 minus '-' to separate such values. See :ref:`irange <irange>`.
546
547.. _bool:
548
549**bool**
550 Boolean. Usually parsed as an integer, however only defined for
551 true and false (1 and 0).
552
553.. _irange:
554
555**irange**
556 Integer range with suffix. Allows value range to be given, such as
c60ebc45 557 1024-4096. A colon may also be used as the separator, e.g. 1k:4k. If the
f80dba8d
MT
558 option allows two sets of ranges, they can be specified with a ',' or '/'
559 delimiter: 1k-4k/8k-32k. Also see :ref:`int <int>`.
560
561**float_list**
562 A list of floating point numbers, separated by a ':' character.
563
564
565Units
566~~~~~
567
568.. option:: kb_base=int
569
570 Select the interpretation of unit prefixes in input parameters.
571
572 **1000**
573 Inputs comply with IEC 80000-13 and the International
574 System of Units (SI). Use:
575
576 - power-of-2 values with IEC prefixes (e.g., KiB)
577 - power-of-10 values with SI prefixes (e.g., kB)
578
579 **1024**
580 Compatibility mode (default). To avoid breaking old scripts:
581
582 - power-of-2 values with SI prefixes
583 - power-of-10 values with IEC prefixes
584
585 See :option:`bs` for more details on input parameters.
586
587 Outputs always use correct prefixes. Most outputs include both
588 side-by-side, like::
589
590 bw=2383.3kB/s (2327.4KiB/s)
591
592 If only one value is reported, then kb_base selects the one to use:
593
594 **1000** -- SI prefixes
595
596 **1024** -- IEC prefixes
597
598.. option:: unit_base=int
599
600 Base unit for reporting. Allowed values are:
601
602 **0**
603 Use auto-detection (default).
604 **8**
605 Byte based.
606 **1**
607 Bit based.
608
609
610With the above in mind, here follows the complete list of fio job parameters.
611
612
613Job description
614~~~~~~~~~~~~~~~
615
616.. option:: name=str
617
618 ASCII name of the job. This may be used to override the name printed by fio
619 for this job. Otherwise the job name is used. On the command line this
620 parameter has the special purpose of also signaling the start of a new job.
621
622.. option:: description=str
623
624 Text description of the job. Doesn't do anything except dump this text
625 description when this job is run. It's not parsed.
626
627.. option:: loops=int
628
629 Run the specified number of iterations of this job. Used to repeat the same
630 workload a given number of times. Defaults to 1.
631
632.. option:: numjobs=int
633
634 Create the specified number of clones of this job. May be used to setup a
635 larger number of threads/processes doing the same thing. Each thread is
636 reported separately; to see statistics for all clones as a whole, use
637 :option:`group_reporting` in conjunction with :option:`new_group`.
638 See :option:`--max-jobs`.
639
640
641Time related parameters
642~~~~~~~~~~~~~~~~~~~~~~~
643
644.. option:: runtime=time
645
646 Tell fio to terminate processing after the specified number of seconds. It
647 can be quite hard to determine for how long a specified job will run, so
648 this parameter is handy to cap the total runtime to a given time.
649
650.. option:: time_based
651
652 If set, fio will run for the duration of the :option:`runtime` specified
653 even if the file(s) are completely read or written. It will simply loop over
654 the same workload as many times as the :option:`runtime` allows.
655
a881438b 656.. option:: startdelay=irange(time)
f80dba8d
MT
657
658 Delay start of job for the specified number of seconds. Supports all time
659 suffixes to allow specification of hours, minutes, seconds and milliseconds
660 -- seconds are the default if a unit is omitted. Can be given as a range
661 which causes each thread to choose randomly out of the range.
662
663.. option:: ramp_time=time
664
665 If set, fio will run the specified workload for this amount of time before
666 logging any performance numbers. Useful for letting performance settle
667 before logging results, thus minimizing the runtime required for stable
668 results. Note that the ``ramp_time`` is considered lead in time for a job,
669 thus it will increase the total runtime if a special timeout or
670 :option:`runtime` is specified.
671
672.. option:: clocksource=str
673
674 Use the given clocksource as the base of timing. The supported options are:
675
676 **gettimeofday**
677 :manpage:`gettimeofday(2)`
678
679 **clock_gettime**
680 :manpage:`clock_gettime(2)`
681
682 **cpu**
683 Internal CPU clock source
684
685 cpu is the preferred clocksource if it is reliable, as it is very fast (and
686 fio is heavy on time calls). Fio will automatically use this clocksource if
687 it's supported and considered reliable on the system it is running on,
688 unless another clocksource is specifically set. For x86/x86-64 CPUs, this
689 means supporting TSC Invariant.
690
691.. option:: gtod_reduce=bool
692
693 Enable all of the :manpage:`gettimeofday(2)` reducing options
694 (:option:`disable_clat`, :option:`disable_slat`, :option:`disable_bw`) plus
695 reduce precision of the timeout somewhat to really shrink the
696 :manpage:`gettimeofday(2)` call count. With this option enabled, we only do
697 about 0.4% of the :manpage:`gettimeofday(2)` calls we would have done if all
698 time keeping was enabled.
699
700.. option:: gtod_cpu=int
701
702 Sometimes it's cheaper to dedicate a single thread of execution to just
703 getting the current time. Fio (and databases, for instance) are very
704 intensive on :manpage:`gettimeofday(2)` calls. With this option, you can set
705 one CPU aside for doing nothing but logging current time to a shared memory
706 location. Then the other threads/processes that run I/O workloads need only
707 copy that segment, instead of entering the kernel with a
708 :manpage:`gettimeofday(2)` call. The CPU set aside for doing these time
709 calls will be excluded from other uses. Fio will manually clear it from the
710 CPU mask of other jobs.
711
712
713Target file/device
714~~~~~~~~~~~~~~~~~~
715
716.. option:: directory=str
717
718 Prefix filenames with this directory. Used to place files in a different
719 location than :file:`./`. You can specify a number of directories by
720 separating the names with a ':' character. These directories will be
721 assigned equally distributed to job clones creates with :option:`numjobs` as
722 long as they are using generated filenames. If specific `filename(s)` are
723 set fio will use the first listed directory, and thereby matching the
724 `filename` semantic which generates a file each clone if not specified, but
725 let all clones use the same if set.
726
727 See the :option:`filename` option for escaping certain characters.
728
729.. option:: filename=str
730
731 Fio normally makes up a `filename` based on the job name, thread number, and
732 file number. If you want to share files between threads in a job or several
733 jobs, specify a `filename` for each of them to override the default. If the
734 ioengine is file based, you can specify a number of files by separating the
735 names with a ':' colon. So if you wanted a job to open :file:`/dev/sda` and
736 :file:`/dev/sdb` as the two working files, you would use
737 ``filename=/dev/sda:/dev/sdb``.
738 On Windows, disk devices are accessed as :file:`\\\\.\\PhysicalDrive0` for
739 the first device, :file:`\\\\.\\PhysicalDrive1` for the second etc.
740 Note: Windows and FreeBSD prevent write access to areas
741 of the disk containing in-use data (e.g. filesystems). If the wanted
742 `filename` does need to include a colon, then escape that with a ``\``
743 character. For instance, if the `filename` is :file:`/dev/dsk/foo@3,0:c`,
744 then you would use ``filename="/dev/dsk/foo@3,0\:c"``. The
745 :file:`-` is a reserved name, meaning stdin or stdout. Which of the two
746 depends on the read/write direction set.
747
748.. option:: filename_format=str
749
750 If sharing multiple files between jobs, it is usually necessary to have fio
751 generate the exact names that you want. By default, fio will name a file
752 based on the default file format specification of
753 :file:`jobname.jobnumber.filenumber`. With this option, that can be
754 customized. Fio will recognize and replace the following keywords in this
755 string:
756
757 **$jobname**
758 The name of the worker thread or process.
759 **$jobnum**
760 The incremental number of the worker thread or process.
761 **$filenum**
762 The incremental number of the file for that worker thread or
763 process.
764
765 To have dependent jobs share a set of files, this option can be set to have
766 fio generate filenames that are shared between the two. For instance, if
767 :file:`testfiles.$filenum` is specified, file number 4 for any job will be
768 named :file:`testfiles.4`. The default of :file:`$jobname.$jobnum.$filenum`
769 will be used if no other format specifier is given.
770
771.. option:: unique_filename=bool
772
773 To avoid collisions between networked clients, fio defaults to prefixing any
774 generated filenames (with a directory specified) with the source of the
775 client connecting. To disable this behavior, set this option to 0.
776
777.. option:: opendir=str
778
779 Recursively open any files below directory `str`.
780
781.. option:: lockfile=str
782
783 Fio defaults to not locking any files before it does I/O to them. If a file
784 or file descriptor is shared, fio can serialize I/O to that file to make the
785 end result consistent. This is usual for emulating real workloads that share
786 files. The lock modes are:
787
788 **none**
789 No locking. The default.
790 **exclusive**
791 Only one thread or process may do I/O at a time, excluding all
792 others.
793 **readwrite**
794 Read-write locking on the file. Many readers may
795 access the file at the same time, but writes get exclusive access.
796
797.. option:: nrfiles=int
798
799 Number of files to use for this job. Defaults to 1.
800
801.. option:: openfiles=int
802
803 Number of files to keep open at the same time. Defaults to the same as
804 :option:`nrfiles`, can be set smaller to limit the number simultaneous
805 opens.
806
807.. option:: file_service_type=str
808
809 Defines how fio decides which file from a job to service next. The following
810 types are defined:
811
812 **random**
813 Choose a file at random.
814
815 **roundrobin**
816 Round robin over opened files. This is the default.
817
818 **sequential**
819 Finish one file before moving on to the next. Multiple files can
820 still be open depending on 'openfiles'.
821
822 **zipf**
c60ebc45 823 Use a *Zipf* distribution to decide what file to access.
f80dba8d
MT
824
825 **pareto**
c60ebc45 826 Use a *Pareto* distribution to decide what file to access.
f80dba8d
MT
827
828 **gauss**
c60ebc45 829 Use a *Gaussian* (normal) distribution to decide what file to
f80dba8d
MT
830 access.
831
832 For *random*, *roundrobin*, and *sequential*, a postfix can be appended to
833 tell fio how many I/Os to issue before switching to a new file. For example,
834 specifying ``file_service_type=random:8`` would cause fio to issue
835 8 I/Os before selecting a new file at random. For the non-uniform
836 distributions, a floating point postfix can be given to influence how the
837 distribution is skewed. See :option:`random_distribution` for a description
838 of how that would work.
839
840.. option:: ioscheduler=str
841
842 Attempt to switch the device hosting the file to the specified I/O scheduler
843 before running.
844
845.. option:: create_serialize=bool
846
847 If true, serialize the file creation for the jobs. This may be handy to
848 avoid interleaving of data files, which may greatly depend on the filesystem
849 used and even the number of processors in the system.
850
851.. option:: create_fsync=bool
852
853 fsync the data file after creation. This is the default.
854
855.. option:: create_on_open=bool
856
857 Don't pre-setup the files for I/O, just create open() when it's time to do
858 I/O to that file.
859
860.. option:: create_only=bool
861
862 If true, fio will only run the setup phase of the job. If files need to be
863 laid out or updated on disk, only that will be done. The actual job contents
864 are not executed.
865
866.. option:: allow_file_create=bool
867
868 If true, fio is permitted to create files as part of its workload. This is
869 the default behavior. If this option is false, then fio will error out if
870 the files it needs to use don't already exist. Default: true.
871
872.. option:: allow_mounted_write=bool
873
c60ebc45 874 If this isn't set, fio will abort jobs that are destructive (e.g. that write)
f80dba8d
MT
875 to what appears to be a mounted device or partition. This should help catch
876 creating inadvertently destructive tests, not realizing that the test will
877 destroy data on the mounted file system. Default: false.
878
879.. option:: pre_read=bool
880
881 If this is given, files will be pre-read into memory before starting the
882 given I/O operation. This will also clear the :option:`invalidate` flag,
883 since it is pointless to pre-read and then drop the cache. This will only
884 work for I/O engines that are seek-able, since they allow you to read the
c60ebc45 885 same data multiple times. Thus it will not work on e.g. network or splice I/O.
f80dba8d
MT
886
887.. option:: unlink=bool
888
889 Unlink the job files when done. Not the default, as repeated runs of that
890 job would then waste time recreating the file set again and again.
891
892.. option:: unlink_each_loop=bool
893
894 Unlink job files after each iteration or loop.
895
896.. option:: zonesize=int
897
898 Divide a file into zones of the specified size. See :option:`zoneskip`.
899
900.. option:: zonerange=int
901
902 Give size of an I/O zone. See :option:`zoneskip`.
903
904.. option:: zoneskip=int
905
906 Skip the specified number of bytes when :option:`zonesize` data has been
907 read. The two zone options can be used to only do I/O on zones of a file.
908
909
910I/O type
911~~~~~~~~
912
913.. option:: direct=bool
914
915 If value is true, use non-buffered I/O. This is usually O_DIRECT. Note that
916 ZFS on Solaris doesn't support direct I/O. On Windows the synchronous
917 ioengines don't support direct I/O. Default: false.
918
919.. option:: atomic=bool
920
921 If value is true, attempt to use atomic direct I/O. Atomic writes are
922 guaranteed to be stable once acknowledged by the operating system. Only
923 Linux supports O_ATOMIC right now.
924
925.. option:: buffered=bool
926
927 If value is true, use buffered I/O. This is the opposite of the
928 :option:`direct` option. Defaults to true.
929
930.. option:: readwrite=str, rw=str
931
932 Type of I/O pattern. Accepted values are:
933
934 **read**
935 Sequential reads.
936 **write**
937 Sequential writes.
938 **trim**
939 Sequential trims (Linux block devices only).
940 **randwrite**
941 Random writes.
942 **randread**
943 Random reads.
944 **randtrim**
945 Random trims (Linux block devices only).
946 **rw,readwrite**
947 Sequential mixed reads and writes.
948 **randrw**
949 Random mixed reads and writes.
950 **trimwrite**
951 Sequential trim+write sequences. Blocks will be trimmed first,
952 then the same blocks will be written to.
953
954 Fio defaults to read if the option is not specified. For the mixed I/O
955 types, the default is to split them 50/50. For certain types of I/O the
956 result may still be skewed a bit, since the speed may be different. It is
957 possible to specify a number of I/O's to do before getting a new offset,
958 this is done by appending a ``:<nr>`` to the end of the string given. For a
959 random read, it would look like ``rw=randread:8`` for passing in an offset
960 modifier with a value of 8. If the suffix is used with a sequential I/O
961 pattern, then the value specified will be added to the generated offset for
962 each I/O. For instance, using ``rw=write:4k`` will skip 4k for every
963 write. It turns sequential I/O into sequential I/O with holes. See the
964 :option:`rw_sequencer` option.
965
966.. option:: rw_sequencer=str
967
968 If an offset modifier is given by appending a number to the ``rw=<str>``
969 line, then this option controls how that number modifies the I/O offset
970 being generated. Accepted values are:
971
972 **sequential**
973 Generate sequential offset.
974 **identical**
975 Generate the same offset.
976
977 ``sequential`` is only useful for random I/O, where fio would normally
c60ebc45 978 generate a new random offset for every I/O. If you append e.g. 8 to randread,
f80dba8d
MT
979 you would get a new random offset for every 8 I/O's. The result would be a
980 seek for only every 8 I/O's, instead of for every I/O. Use ``rw=randread:8``
981 to specify that. As sequential I/O is already sequential, setting
982 ``sequential`` for that would not result in any differences. ``identical``
983 behaves in a similar fashion, except it sends the same offset 8 number of
984 times before generating a new offset.
985
986.. option:: unified_rw_reporting=bool
987
988 Fio normally reports statistics on a per data direction basis, meaning that
989 reads, writes, and trims are accounted and reported separately. If this
990 option is set fio sums the results and report them as "mixed" instead.
991
992.. option:: randrepeat=bool
993
994 Seed the random number generator used for random I/O patterns in a
995 predictable way so the pattern is repeatable across runs. Default: true.
996
997.. option:: allrandrepeat=bool
998
999 Seed all random number generators in a predictable way so results are
1000 repeatable across runs. Default: false.
1001
1002.. option:: randseed=int
1003
1004 Seed the random number generators based on this seed value, to be able to
1005 control what sequence of output is being generated. If not set, the random
1006 sequence depends on the :option:`randrepeat` setting.
1007
1008.. option:: fallocate=str
1009
1010 Whether pre-allocation is performed when laying down files.
1011 Accepted values are:
1012
1013 **none**
1014 Do not pre-allocate space.
1015
1016 **posix**
1017 Pre-allocate via :manpage:`posix_fallocate(3)`.
1018
1019 **keep**
1020 Pre-allocate via :manpage:`fallocate(2)` with
1021 FALLOC_FL_KEEP_SIZE set.
1022
1023 **0**
1024 Backward-compatible alias for **none**.
1025
1026 **1**
1027 Backward-compatible alias for **posix**.
1028
1029 May not be available on all supported platforms. **keep** is only available
1030 on Linux. If using ZFS on Solaris this must be set to **none** because ZFS
1031 doesn't support it. Default: **posix**.
1032
1033.. option:: fadvise_hint=str
1034
1035 Use :manpage:`posix_fadvise(2)` to advise the kernel on what I/O patterns
1036 are likely to be issued. Accepted values are:
1037
1038 **0**
1039 Backwards-compatible hint for "no hint".
1040
1041 **1**
1042 Backwards compatible hint for "advise with fio workload type". This
1043 uses **FADV_RANDOM** for a random workload, and **FADV_SEQUENTIAL**
1044 for a sequential workload.
1045
1046 **sequential**
1047 Advise using **FADV_SEQUENTIAL**.
1048
1049 **random**
1050 Advise using **FADV_RANDOM**.
1051
1052.. option:: fadvise_stream=int
1053
1054 Use :manpage:`posix_fadvise(2)` to advise the kernel what stream ID the
1055 writes issued belong to. Only supported on Linux. Note, this option may
1056 change going forward.
1057
1058.. option:: offset=int
1059
1060 Start I/O at the given offset in the file. The data before the given offset
1061 will not be touched. This effectively caps the file size at `real_size -
1062 offset`.
1063
1064.. option:: offset_increment=int
1065
1066 If this is provided, then the real offset becomes `offset + offset_increment
1067 * thread_number`, where the thread number is a counter that starts at 0 and
1068 is incremented for each sub-job (i.e. when :option:`numjobs` option is
1069 specified). This option is useful if there are several jobs which are
1070 intended to operate on a file in parallel disjoint segments, with even
1071 spacing between the starting points.
1072
1073.. option:: number_ios=int
1074
c60ebc45 1075 Fio will normally perform I/Os until it has exhausted the size of the region
f80dba8d
MT
1076 set by :option:`size`, or if it exhaust the allocated time (or hits an error
1077 condition). With this setting, the range/size can be set independently of
c60ebc45 1078 the number of I/Os to perform. When fio reaches this number, it will exit
f80dba8d
MT
1079 normally and report status. Note that this does not extend the amount of I/O
1080 that will be done, it will only stop fio if this condition is met before
1081 other end-of-job criteria.
1082
1083.. option:: fsync=int
1084
1085 If writing to a file, issue a sync of the dirty data for every number of
1086 blocks given. For example, if you give 32 as a parameter, fio will sync the
1087 file for every 32 writes issued. If fio is using non-buffered I/O, we may
1088 not sync the file. The exception is the sg I/O engine, which synchronizes
1089 the disk cache anyway.
1090
1091.. option:: fdatasync=int
1092
1093 Like :option:`fsync` but uses :manpage:`fdatasync(2)` to only sync data and
1094 not metadata blocks. In FreeBSD and Windows there is no
1095 :manpage:`fdatasync(2)`, this falls back to using :manpage:`fsync(2)`.
1096
1097.. option:: write_barrier=int
1098
1099 Make every `N-th` write a barrier write.
1100
1101.. option:: sync_file_range=str:val
1102
1103 Use :manpage:`sync_file_range(2)` for every `val` number of write
1104 operations. Fio will track range of writes that have happened since the last
1105 :manpage:`sync_file_range(2)` call. `str` can currently be one or more of:
1106
1107 **wait_before**
1108 SYNC_FILE_RANGE_WAIT_BEFORE
1109 **write**
1110 SYNC_FILE_RANGE_WRITE
1111 **wait_after**
1112 SYNC_FILE_RANGE_WAIT_AFTER
1113
1114 So if you do ``sync_file_range=wait_before,write:8``, fio would use
1115 ``SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE`` for every 8
1116 writes. Also see the :manpage:`sync_file_range(2)` man page. This option is
1117 Linux specific.
1118
1119.. option:: overwrite=bool
1120
1121 If true, writes to a file will always overwrite existing data. If the file
1122 doesn't already exist, it will be created before the write phase begins. If
1123 the file exists and is large enough for the specified write phase, nothing
1124 will be done.
1125
1126.. option:: end_fsync=bool
1127
1128 If true, fsync file contents when a write stage has completed.
1129
1130.. option:: fsync_on_close=bool
1131
1132 If true, fio will :manpage:`fsync(2)` a dirty file on close. This differs
1133 from end_fsync in that it will happen on every file close, not just at the
1134 end of the job.
1135
1136.. option:: rwmixread=int
1137
1138 Percentage of a mixed workload that should be reads. Default: 50.
1139
1140.. option:: rwmixwrite=int
1141
1142 Percentage of a mixed workload that should be writes. If both
1143 :option:`rwmixread` and :option:`rwmixwrite` is given and the values do not
1144 add up to 100%, the latter of the two will be used to override the
1145 first. This may interfere with a given rate setting, if fio is asked to
1146 limit reads or writes to a certain rate. If that is the case, then the
1147 distribution may be skewed. Default: 50.
1148
1149.. option:: random_distribution=str:float[,str:float][,str:float]
1150
1151 By default, fio will use a completely uniform random distribution when asked
1152 to perform random I/O. Sometimes it is useful to skew the distribution in
1153 specific ways, ensuring that some parts of the data is more hot than others.
1154 fio includes the following distribution models:
1155
1156 **random**
1157 Uniform random distribution
1158
1159 **zipf**
1160 Zipf distribution
1161
1162 **pareto**
1163 Pareto distribution
1164
1165 **gauss**
c60ebc45 1166 Normal (Gaussian) distribution
f80dba8d
MT
1167
1168 **zoned**
1169 Zoned random distribution
1170
1171 When using a **zipf** or **pareto** distribution, an input value is also
1172 needed to define the access pattern. For **zipf**, this is the `zipf
c60ebc45 1173 theta`. For **pareto**, it's the `Pareto power`. Fio includes a test
f80dba8d
MT
1174 program, :command:`genzipf`, that can be used visualize what the given input
1175 values will yield in terms of hit rates. If you wanted to use **zipf** with
1176 a `theta` of 1.2, you would use ``random_distribution=zipf:1.2`` as the
1177 option. If a non-uniform model is used, fio will disable use of the random
1178 map. For the **gauss** distribution, a normal deviation is supplied as a
1179 value between 0 and 100.
1180
1181 For a **zoned** distribution, fio supports specifying percentages of I/O
1182 access that should fall within what range of the file or device. For
1183 example, given a criteria of:
1184
1185 * 60% of accesses should be to the first 10%
1186 * 30% of accesses should be to the next 20%
1187 * 8% of accesses should be to to the next 30%
1188 * 2% of accesses should be to the next 40%
1189
1190 we can define that through zoning of the random accesses. For the above
1191 example, the user would do::
1192
1193 random_distribution=zoned:60/10:30/20:8/30:2/40
1194
1195 similarly to how :option:`bssplit` works for setting ranges and percentages
1196 of block sizes. Like :option:`bssplit`, it's possible to specify separate
1197 zones for reads, writes, and trims. If just one set is given, it'll apply to
1198 all of them.
1199
1200.. option:: percentage_random=int[,int][,int]
1201
1202 For a random workload, set how big a percentage should be random. This
1203 defaults to 100%, in which case the workload is fully random. It can be set
1204 from anywhere from 0 to 100. Setting it to 0 would make the workload fully
1205 sequential. Any setting in between will result in a random mix of sequential
1206 and random I/O, at the given percentages. Comma-separated values may be
1207 specified for reads, writes, and trims as described in :option:`blocksize`.
1208
1209.. option:: norandommap
1210
1211 Normally fio will cover every block of the file when doing random I/O. If
1212 this option is given, fio will just get a new random offset without looking
1213 at past I/O history. This means that some blocks may not be read or written,
1214 and that some blocks may be read/written more than once. If this option is
1215 used with :option:`verify` and multiple blocksizes (via :option:`bsrange`),
1216 only intact blocks are verified, i.e., partially-overwritten blocks are
1217 ignored.
1218
1219.. option:: softrandommap=bool
1220
1221 See :option:`norandommap`. If fio runs with the random block map enabled and
1222 it fails to allocate the map, if this option is set it will continue without
1223 a random block map. As coverage will not be as complete as with random maps,
1224 this option is disabled by default.
1225
1226.. option:: random_generator=str
1227
1228 Fio supports the following engines for generating
1229 I/O offsets for random I/O:
1230
1231 **tausworthe**
1232 Strong 2^88 cycle random number generator
1233 **lfsr**
1234 Linear feedback shift register generator
1235 **tausworthe64**
1236 Strong 64-bit 2^258 cycle random number generator
1237
1238 **tausworthe** is a strong random number generator, but it requires tracking
1239 on the side if we want to ensure that blocks are only read or written
1240 once. **LFSR** guarantees that we never generate the same offset twice, and
1241 it's also less computationally expensive. It's not a true random generator,
1242 however, though for I/O purposes it's typically good enough. **LFSR** only
1243 works with single block sizes, not with workloads that use multiple block
1244 sizes. If used with such a workload, fio may read or write some blocks
1245 multiple times. The default value is **tausworthe**, unless the required
1246 space exceeds 2^32 blocks. If it does, then **tausworthe64** is
1247 selected automatically.
1248
1249
1250Block size
1251~~~~~~~~~~
1252
1253.. option:: blocksize=int[,int][,int], bs=int[,int][,int]
1254
1255 The block size in bytes used for I/O units. Default: 4096. A single value
1256 applies to reads, writes, and trims. Comma-separated values may be
1257 specified for reads, writes, and trims. A value not terminated in a comma
1258 applies to subsequent types.
1259
1260 Examples:
1261
1262 **bs=256k**
1263 means 256k for reads, writes and trims.
1264
1265 **bs=8k,32k**
1266 means 8k for reads, 32k for writes and trims.
1267
1268 **bs=8k,32k,**
1269 means 8k for reads, 32k for writes, and default for trims.
1270
1271 **bs=,8k**
1272 means default for reads, 8k for writes and trims.
1273
1274 **bs=,8k,**
1275 means default for reads, 8k for writes, and default for writes.
1276
1277.. option:: blocksize_range=irange[,irange][,irange], bsrange=irange[,irange][,irange]
1278
1279 A range of block sizes in bytes for I/O units. The issued I/O unit will
1280 always be a multiple of the minimum size, unless
1281 :option:`blocksize_unaligned` is set.
1282
1283 Comma-separated ranges may be specified for reads, writes, and trims as
1284 described in :option:`blocksize`.
1285
1286 Example: ``bsrange=1k-4k,2k-8k``.
1287
1288.. option:: bssplit=str[,str][,str]
1289
1290 Sometimes you want even finer grained control of the block sizes issued, not
1291 just an even split between them. This option allows you to weight various
1292 block sizes, so that you are able to define a specific amount of block sizes
1293 issued. The format for this option is::
1294
1295 bssplit=blocksize/percentage:blocksize/percentage
1296
1297 for as many block sizes as needed. So if you want to define a workload that
1298 has 50% 64k blocks, 10% 4k blocks, and 40% 32k blocks, you would write::
1299
1300 bssplit=4k/10:64k/50:32k/40
1301
1302 Ordering does not matter. If the percentage is left blank, fio will fill in
1303 the remaining values evenly. So a bssplit option like this one::
1304
1305 bssplit=4k/50:1k/:32k/
1306
1307 would have 50% 4k ios, and 25% 1k and 32k ios. The percentages always add up
1308 to 100, if bssplit is given a range that adds up to more, it will error out.
1309
1310 Comma-separated values may be specified for reads, writes, and trims as
1311 described in :option:`blocksize`.
1312
1313 If you want a workload that has 50% 2k reads and 50% 4k reads, while having
1314 90% 4k writes and 10% 8k writes, you would specify::
1315
1316 bssplit=2k/50:4k/50,4k/90,8k/10
1317
1318.. option:: blocksize_unaligned, bs_unaligned
1319
1320 If set, fio will issue I/O units with any size within
1321 :option:`blocksize_range`, not just multiples of the minimum size. This
1322 typically won't work with direct I/O, as that normally requires sector
1323 alignment.
1324
1325.. option:: bs_is_seq_rand
1326
1327 If this option is set, fio will use the normal read,write blocksize settings
1328 as sequential,random blocksize settings instead. Any random read or write
1329 will use the WRITE blocksize settings, and any sequential read or write will
1330 use the READ blocksize settings.
1331
1332.. option:: blockalign=int[,int][,int], ba=int[,int][,int]
1333
1334 Boundary to which fio will align random I/O units. Default:
1335 :option:`blocksize`. Minimum alignment is typically 512b for using direct
1336 I/O, though it usually depends on the hardware block size. This option is
1337 mutually exclusive with using a random map for files, so it will turn off
1338 that option. Comma-separated values may be specified for reads, writes, and
1339 trims as described in :option:`blocksize`.
1340
1341
1342Buffers and memory
1343~~~~~~~~~~~~~~~~~~
1344
1345.. option:: zero_buffers
1346
1347 Initialize buffers with all zeros. Default: fill buffers with random data.
1348
1349.. option:: refill_buffers
1350
1351 If this option is given, fio will refill the I/O buffers on every
1352 submit. The default is to only fill it at init time and reuse that
1353 data. Only makes sense if zero_buffers isn't specified, naturally. If data
1354 verification is enabled, `refill_buffers` is also automatically enabled.
1355
1356.. option:: scramble_buffers=bool
1357
1358 If :option:`refill_buffers` is too costly and the target is using data
1359 deduplication, then setting this option will slightly modify the I/O buffer
1360 contents to defeat normal de-dupe attempts. This is not enough to defeat
1361 more clever block compression attempts, but it will stop naive dedupe of
1362 blocks. Default: true.
1363
1364.. option:: buffer_compress_percentage=int
1365
1366 If this is set, then fio will attempt to provide I/O buffer content (on
1367 WRITEs) that compress to the specified level. Fio does this by providing a
1368 mix of random data and a fixed pattern. The fixed pattern is either zeroes,
1369 or the pattern specified by :option:`buffer_pattern`. If the pattern option
1370 is used, it might skew the compression ratio slightly. Note that this is per
1371 block size unit, for file/disk wide compression level that matches this
1372 setting, you'll also want to set :option:`refill_buffers`.
1373
1374.. option:: buffer_compress_chunk=int
1375
1376 See :option:`buffer_compress_percentage`. This setting allows fio to manage
1377 how big the ranges of random data and zeroed data is. Without this set, fio
1378 will provide :option:`buffer_compress_percentage` of blocksize random data,
1379 followed by the remaining zeroed. With this set to some chunk size smaller
1380 than the block size, fio can alternate random and zeroed data throughout the
1381 I/O buffer.
1382
1383.. option:: buffer_pattern=str
1384
1385 If set, fio will fill the I/O buffers with this pattern. If not set, the
1386 contents of I/O buffers is defined by the other options related to buffer
1387 contents. The setting can be any pattern of bytes, and can be prefixed with
1388 0x for hex values. It may also be a string, where the string must then be
1389 wrapped with ``""``, e.g.::
1390
1391 buffer_pattern="abcd"
1392
1393 or::
1394
1395 buffer_pattern=-12
1396
1397 or::
1398
1399 buffer_pattern=0xdeadface
1400
1401 Also you can combine everything together in any order::
1402
1403 buffer_pattern=0xdeadface"abcd"-12
1404
1405.. option:: dedupe_percentage=int
1406
1407 If set, fio will generate this percentage of identical buffers when
1408 writing. These buffers will be naturally dedupable. The contents of the
1409 buffers depend on what other buffer compression settings have been set. It's
1410 possible to have the individual buffers either fully compressible, or not at
1411 all. This option only controls the distribution of unique buffers.
1412
1413.. option:: invalidate=bool
1414
1415 Invalidate the buffer/page cache parts for this file prior to starting
1416 I/O. Defaults to true.
1417
1418.. option:: sync=bool
1419
1420 Use synchronous I/O for buffered writes. For the majority of I/O engines,
1421 this means using O_SYNC. Default: false.
1422
1423.. option:: iomem=str, mem=str
1424
1425 Fio can use various types of memory as the I/O unit buffer. The allowed
1426 values are:
1427
1428 **malloc**
1429 Use memory from :manpage:`malloc(3)` as the buffers. Default memory
1430 type.
1431
1432 **shm**
1433 Use shared memory as the buffers. Allocated through
1434 :manpage:`shmget(2)`.
1435
1436 **shmhuge**
1437 Same as shm, but use huge pages as backing.
1438
1439 **mmap**
1440 Use mmap to allocate buffers. May either be anonymous memory, or can
1441 be file backed if a filename is given after the option. The format
1442 is `mem=mmap:/path/to/file`.
1443
1444 **mmaphuge**
1445 Use a memory mapped huge file as the buffer backing. Append filename
1446 after mmaphuge, ala `mem=mmaphuge:/hugetlbfs/file`.
1447
1448 **mmapshared**
1449 Same as mmap, but use a MMAP_SHARED mapping.
1450
1451 The area allocated is a function of the maximum allowed bs size for the job,
1452 multiplied by the I/O depth given. Note that for **shmhuge** and
1453 **mmaphuge** to work, the system must have free huge pages allocated. This
1454 can normally be checked and set by reading/writing
1455 :file:`/proc/sys/vm/nr_hugepages` on a Linux system. Fio assumes a huge page
1456 is 4MiB in size. So to calculate the number of huge pages you need for a
1457 given job file, add up the I/O depth of all jobs (normally one unless
1458 :option:`iodepth` is used) and multiply by the maximum bs set. Then divide
1459 that number by the huge page size. You can see the size of the huge pages in
1460 :file:`/proc/meminfo`. If no huge pages are allocated by having a non-zero
1461 number in `nr_hugepages`, using **mmaphuge** or **shmhuge** will fail. Also
1462 see :option:`hugepage-size`.
1463
1464 **mmaphuge** also needs to have hugetlbfs mounted and the file location
1465 should point there. So if it's mounted in :file:`/huge`, you would use
1466 `mem=mmaphuge:/huge/somefile`.
1467
1468.. option:: iomem_align=int
1469
1470 This indicates the memory alignment of the I/O memory buffers. Note that
1471 the given alignment is applied to the first I/O unit buffer, if using
1472 :option:`iodepth` the alignment of the following buffers are given by the
1473 :option:`bs` used. In other words, if using a :option:`bs` that is a
1474 multiple of the page sized in the system, all buffers will be aligned to
1475 this value. If using a :option:`bs` that is not page aligned, the alignment
1476 of subsequent I/O memory buffers is the sum of the :option:`iomem_align` and
1477 :option:`bs` used.
1478
1479.. option:: hugepage-size=int
1480
1481 Defines the size of a huge page. Must at least be equal to the system
1482 setting, see :file:`/proc/meminfo`. Defaults to 4MiB. Should probably
1483 always be a multiple of megabytes, so using ``hugepage-size=Xm`` is the
1484 preferred way to set this to avoid setting a non-pow-2 bad value.
1485
1486.. option:: lockmem=int
1487
1488 Pin the specified amount of memory with :manpage:`mlock(2)`. Can be used to
1489 simulate a smaller amount of memory. The amount specified is per worker.
1490
1491
1492I/O size
1493~~~~~~~~
1494
1495.. option:: size=int
1496
1497 The total size of file I/O for this job. Fio will run until this many bytes
1498 has been transferred, unless runtime is limited by other options (such as
1499 :option:`runtime`, for instance, or increased/decreased by
1500 :option:`io_size`). Unless specific :option:`nrfiles` and :option:`filesize`
1501 options are given, fio will divide this size between the available files
1502 specified by the job. If not set, fio will use the full size of the given
1503 files or devices. If the files do not exist, size must be given. It is also
1504 possible to give size as a percentage between 1 and 100. If ``size=20%`` is
1505 given, fio will use 20% of the full size of the given files or devices.
1506
1507.. option:: io_size=int, io_limit=int
1508
1509 Normally fio operates within the region set by :option:`size`, which means
1510 that the :option:`size` option sets both the region and size of I/O to be
1511 performed. Sometimes that is not what you want. With this option, it is
1512 possible to define just the amount of I/O that fio should do. For instance,
1513 if :option:`size` is set to 20GiB and :option:`io_size` is set to 5GiB, fio
1514 will perform I/O within the first 20GiB but exit when 5GiB have been
1515 done. The opposite is also possible -- if :option:`size` is set to 20GiB,
1516 and :option:`io_size` is set to 40GiB, then fio will do 40GiB of I/O within
1517 the 0..20GiB region.
1518
1519.. option:: filesize=int
1520
1521 Individual file sizes. May be a range, in which case fio will select sizes
1522 for files at random within the given range and limited to :option:`size` in
1523 total (if that is given). If not given, each created file is the same size.
1524
1525.. option:: file_append=bool
1526
1527 Perform I/O after the end of the file. Normally fio will operate within the
1528 size of a file. If this option is set, then fio will append to the file
1529 instead. This has identical behavior to setting :option:`offset` to the size
1530 of a file. This option is ignored on non-regular files.
1531
1532.. option:: fill_device=bool, fill_fs=bool
1533
1534 Sets size to something really large and waits for ENOSPC (no space left on
1535 device) as the terminating condition. Only makes sense with sequential
1536 write. For a read workload, the mount point will be filled first then I/O
1537 started on the result. This option doesn't make sense if operating on a raw
1538 device node, since the size of that is already known by the file system.
1539 Additionally, writing beyond end-of-device will not return ENOSPC there.
1540
1541
1542I/O engine
1543~~~~~~~~~~
1544
1545.. option:: ioengine=str
1546
1547 Defines how the job issues I/O to the file. The following types are defined:
1548
1549 **sync**
1550 Basic :manpage:`read(2)` or :manpage:`write(2)`
1551 I/O. :manpage:`lseek(2)` is used to position the I/O location.
1552
1553 **psync**
1554 Basic :manpage:`pread(2)` or :manpage:`pwrite(2)` I/O. Default on
1555 all supported operating systems except for Windows.
1556
1557 **vsync**
1558 Basic :manpage:`readv(2)` or :manpage:`writev(2)` I/O. Will emulate
c60ebc45 1559 queuing by coalescing adjacent I/Os into a single submission.
f80dba8d
MT
1560
1561 **pvsync**
1562 Basic :manpage:`preadv(2)` or :manpage:`pwritev(2)` I/O.
1563
1564 **pvsync2**
1565 Basic :manpage:`preadv2(2)` or :manpage:`pwritev2(2)` I/O.
1566
1567 **libaio**
1568 Linux native asynchronous I/O. Note that Linux may only support
1569 queued behaviour with non-buffered I/O (set ``direct=1`` or
1570 ``buffered=0``).
1571 This engine defines engine specific options.
1572
1573 **posixaio**
1574 POSIX asynchronous I/O using :manpage:`aio_read(3)` and
1575 :manpage:`aio_write(3)`.
1576
1577 **solarisaio**
1578 Solaris native asynchronous I/O.
1579
1580 **windowsaio**
1581 Windows native asynchronous I/O. Default on Windows.
1582
1583 **mmap**
1584 File is memory mapped with :manpage:`mmap(2)` and data copied
1585 to/from using :manpage:`memcpy(3)`.
1586
1587 **splice**
1588 :manpage:`splice(2)` is used to transfer the data and
1589 :manpage:`vmsplice(2)` to transfer data from user space to the
1590 kernel.
1591
1592 **sg**
1593 SCSI generic sg v3 I/O. May either be synchronous using the SG_IO
1594 ioctl, or if the target is an sg character device we use
1595 :manpage:`read(2)` and :manpage:`write(2)` for asynchronous
1596 I/O. Requires filename option to specify either block or character
1597 devices.
1598
1599 **null**
1600 Doesn't transfer any data, just pretends to. This is mainly used to
1601 exercise fio itself and for debugging/testing purposes.
1602
1603 **net**
1604 Transfer over the network to given ``host:port``. Depending on the
1605 :option:`protocol` used, the :option:`hostname`, :option:`port`,
1606 :option:`listen` and :option:`filename` options are used to specify
1607 what sort of connection to make, while the :option:`protocol` option
1608 determines which protocol will be used. This engine defines engine
1609 specific options.
1610
1611 **netsplice**
1612 Like **net**, but uses :manpage:`splice(2)` and
1613 :manpage:`vmsplice(2)` to map data and send/receive.
1614 This engine defines engine specific options.
1615
1616 **cpuio**
1617 Doesn't transfer any data, but burns CPU cycles according to the
1618 :option:`cpuload` and :option:`cpuchunks` options. Setting
1619 :option:`cpuload` =85 will cause that job to do nothing but burn 85%
1620 of the CPU. In case of SMP machines, use :option:`numjobs`
1621 =<no_of_cpu> to get desired CPU usage, as the cpuload only loads a
1622 single CPU at the desired rate. A job never finishes unless there is
1623 at least one non-cpuio job.
1624
1625 **guasi**
1626 The GUASI I/O engine is the Generic Userspace Asyncronous Syscall
1627 Interface approach to async I/O. See
1628
1629 http://www.xmailserver.org/guasi-lib.html
1630
1631 for more info on GUASI.
1632
1633 **rdma**
1634 The RDMA I/O engine supports both RDMA memory semantics
1635 (RDMA_WRITE/RDMA_READ) and channel semantics (Send/Recv) for the
1636 InfiniBand, RoCE and iWARP protocols.
1637
1638 **falloc**
1639 I/O engine that does regular fallocate to simulate data transfer as
1640 fio ioengine.
1641
1642 DDIR_READ
1643 does fallocate(,mode = FALLOC_FL_KEEP_SIZE,).
1644
1645 DDIR_WRITE
1646 does fallocate(,mode = 0).
1647
1648 DDIR_TRIM
1649 does fallocate(,mode = FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE).
1650
1651 **e4defrag**
1652 I/O engine that does regular EXT4_IOC_MOVE_EXT ioctls to simulate
1653 defragment activity in request to DDIR_WRITE event.
1654
1655 **rbd**
1656 I/O engine supporting direct access to Ceph Rados Block Devices
1657 (RBD) via librbd without the need to use the kernel rbd driver. This
1658 ioengine defines engine specific options.
1659
1660 **gfapi**
1661 Using Glusterfs libgfapi sync interface to direct access to
1662 Glusterfs volumes without having to go through FUSE. This ioengine
1663 defines engine specific options.
1664
1665 **gfapi_async**
1666 Using Glusterfs libgfapi async interface to direct access to
1667 Glusterfs volumes without having to go through FUSE. This ioengine
1668 defines engine specific options.
1669
1670 **libhdfs**
1671 Read and write through Hadoop (HDFS). The :file:`filename` option
1672 is used to specify host,port of the hdfs name-node to connect. This
1673 engine interprets offsets a little differently. In HDFS, files once
1674 created cannot be modified. So random writes are not possible. To
1675 imitate this, libhdfs engine expects bunch of small files to be
1676 created over HDFS, and engine will randomly pick a file out of those
1677 files based on the offset generated by fio backend. (see the example
1678 job file to create such files, use ``rw=write`` option). Please
1679 note, you might want to set necessary environment variables to work
1680 with hdfs/libhdfs properly. Each jobs uses it's own connection to
1681 HDFS.
1682
1683 **mtd**
1684 Read, write and erase an MTD character device (e.g.,
1685 :file:`/dev/mtd0`). Discards are treated as erases. Depending on the
1686 underlying device type, the I/O may have to go in a certain pattern,
1687 e.g., on NAND, writing sequentially to erase blocks and discarding
1688 before overwriting. The writetrim mode works well for this
1689 constraint.
1690
1691 **pmemblk**
1692 Read and write using filesystem DAX to a file on a filesystem
1693 mounted with DAX on a persistent memory device through the NVML
1694 libpmemblk library.
1695
1696 **dev-dax**
1697 Read and write using device DAX to a persistent memory device (e.g.,
1698 /dev/dax0.0) through the NVML libpmem library.
1699
1700 **external**
1701 Prefix to specify loading an external I/O engine object file. Append
c60ebc45 1702 the engine filename, e.g. ``ioengine=external:/tmp/foo.o`` to load
f80dba8d
MT
1703 ioengine :file:`foo.o` in :file:`/tmp`.
1704
1705
1706I/O engine specific parameters
1707~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1708
1709In addition, there are some parameters which are only valid when a specific
1710ioengine is in use. These are used identically to normal parameters, with the
1711caveat that when used on the command line, they must come after the
1712:option:`ioengine` that defines them is selected.
1713
1714.. option:: userspace_reap : [libaio]
1715
1716 Normally, with the libaio engine in use, fio will use the
1717 :manpage:`io_getevents(2)` system call to reap newly returned events. With
1718 this flag turned on, the AIO ring will be read directly from user-space to
1719 reap events. The reaping mode is only enabled when polling for a minimum of
c60ebc45 1720 0 events (e.g. when :option:`iodepth_batch_complete` `=0`).
f80dba8d
MT
1721
1722.. option:: hipri : [psyncv2]
1723
1724 Set RWF_HIPRI on I/O, indicating to the kernel that it's of higher priority
1725 than normal.
1726
1727.. option:: cpuload=int : [cpuio]
1728
1729 Attempt to use the specified percentage of CPU cycles.
1730
1731.. option:: cpuchunks=int : [cpuio]
1732
1733 Split the load into cycles of the given time. In microseconds.
1734
1735.. option:: exit_on_io_done=bool : [cpuio]
1736
1737 Detect when I/O threads are done, then exit.
1738
1739.. option:: hostname=str : [netsplice] [net]
1740
1741 The host name or IP address to use for TCP or UDP based I/O. If the job is
1742 a TCP listener or UDP reader, the host name is not used and must be omitted
1743 unless it is a valid UDP multicast address.
1744
1745.. option:: namenode=str : [libhdfs]
1746
1747 The host name or IP address of a HDFS cluster namenode to contact.
1748
1749.. option:: port=int
1750
1751 [netsplice], [net]
1752
1753 The TCP or UDP port to bind to or connect to. If this is used with
1754 :option:`numjobs` to spawn multiple instances of the same job type, then
1755 this will be the starting port number since fio will use a range of
1756 ports.
1757
1758 [libhdfs]
1759
1760 the listening port of the HFDS cluster namenode.
1761
1762.. option:: interface=str : [netsplice] [net]
1763
1764 The IP address of the network interface used to send or receive UDP
1765 multicast.
1766
1767.. option:: ttl=int : [netsplice] [net]
1768
1769 Time-to-live value for outgoing UDP multicast packets. Default: 1.
1770
1771.. option:: nodelay=bool : [netsplice] [net]
1772
1773 Set TCP_NODELAY on TCP connections.
1774
1775.. option:: protocol=str : [netsplice] [net]
1776
1777.. option:: proto=str : [netsplice] [net]
1778
1779 The network protocol to use. Accepted values are:
1780
1781 **tcp**
1782 Transmission control protocol.
1783 **tcpv6**
1784 Transmission control protocol V6.
1785 **udp**
1786 User datagram protocol.
1787 **udpv6**
1788 User datagram protocol V6.
1789 **unix**
1790 UNIX domain socket.
1791
1792 When the protocol is TCP or UDP, the port must also be given, as well as the
1793 hostname if the job is a TCP listener or UDP reader. For unix sockets, the
1794 normal filename option should be used and the port is invalid.
1795
1796.. option:: listen : [net]
1797
1798 For TCP network connections, tell fio to listen for incoming connections
1799 rather than initiating an outgoing connection. The :option:`hostname` must
1800 be omitted if this option is used.
1801
1802.. option:: pingpong : [net]
1803
1804 Normally a network writer will just continue writing data, and a network
1805 reader will just consume packages. If ``pingpong=1`` is set, a writer will
1806 send its normal payload to the reader, then wait for the reader to send the
1807 same payload back. This allows fio to measure network latencies. The
1808 submission and completion latencies then measure local time spent sending or
1809 receiving, and the completion latency measures how long it took for the
1810 other end to receive and send back. For UDP multicast traffic
1811 ``pingpong=1`` should only be set for a single reader when multiple readers
1812 are listening to the same address.
1813
1814.. option:: window_size : [net]
1815
1816 Set the desired socket buffer size for the connection.
1817
1818.. option:: mss : [net]
1819
1820 Set the TCP maximum segment size (TCP_MAXSEG).
1821
1822.. option:: donorname=str : [e4defrag]
1823
1824 File will be used as a block donor(swap extents between files).
1825
1826.. option:: inplace=int : [e4defrag]
1827
1828 Configure donor file blocks allocation strategy:
1829
1830 **0**
1831 Default. Preallocate donor's file on init.
1832 **1**
1833 Allocate space immediately inside defragment event, and free right
1834 after event.
1835
1836.. option:: clustername=str : [rbd]
1837
1838 Specifies the name of the Ceph cluster.
1839
1840.. option:: rbdname=str : [rbd]
1841
1842 Specifies the name of the RBD.
1843
1844.. option:: pool=str : [rbd]
1845
1846 Specifies the name of the Ceph pool containing RBD.
1847
1848.. option:: clientname=str : [rbd]
1849
1850 Specifies the username (without the 'client.' prefix) used to access the
1851 Ceph cluster. If the *clustername* is specified, the *clientname* shall be
1852 the full *type.id* string. If no type. prefix is given, fio will add
1853 'client.' by default.
1854
1855.. option:: skip_bad=bool : [mtd]
1856
1857 Skip operations against known bad blocks.
1858
1859.. option:: hdfsdirectory : [libhdfs]
1860
1861 libhdfs will create chunk in this HDFS directory.
1862
1863.. option:: chunk_size : [libhdfs]
1864
1865 the size of the chunk to use for each file.
1866
1867
1868I/O depth
1869~~~~~~~~~
1870
1871.. option:: iodepth=int
1872
1873 Number of I/O units to keep in flight against the file. Note that
1874 increasing *iodepth* beyond 1 will not affect synchronous ioengines (except
c60ebc45 1875 for small degrees when :option:`verify_async` is in use). Even async
f80dba8d
MT
1876 engines may impose OS restrictions causing the desired depth not to be
1877 achieved. This may happen on Linux when using libaio and not setting
1878 :option:`direct` =1, since buffered I/O is not async on that OS. Keep an
1879 eye on the I/O depth distribution in the fio output to verify that the
1880 achieved depth is as expected. Default: 1.
1881
1882.. option:: iodepth_batch_submit=int, iodepth_batch=int
1883
1884 This defines how many pieces of I/O to submit at once. It defaults to 1
1885 which means that we submit each I/O as soon as it is available, but can be
1886 raised to submit bigger batches of I/O at the time. If it is set to 0 the
1887 :option:`iodepth` value will be used.
1888
1889.. option:: iodepth_batch_complete_min=int, iodepth_batch_complete=int
1890
1891 This defines how many pieces of I/O to retrieve at once. It defaults to 1
1892 which means that we'll ask for a minimum of 1 I/O in the retrieval process
1893 from the kernel. The I/O retrieval will go on until we hit the limit set by
1894 :option:`iodepth_low`. If this variable is set to 0, then fio will always
1895 check for completed events before queuing more I/O. This helps reduce I/O
1896 latency, at the cost of more retrieval system calls.
1897
1898.. option:: iodepth_batch_complete_max=int
1899
1900 This defines maximum pieces of I/O to retrieve at once. This variable should
1901 be used along with :option:`iodepth_batch_complete_min` =int variable,
1902 specifying the range of min and max amount of I/O which should be
1903 retrieved. By default it is equal to :option:`iodepth_batch_complete_min`
1904 value.
1905
1906 Example #1::
1907
1908 iodepth_batch_complete_min=1
1909 iodepth_batch_complete_max=<iodepth>
1910
1911 which means that we will retrieve at least 1 I/O and up to the whole
1912 submitted queue depth. If none of I/O has been completed yet, we will wait.
1913
1914 Example #2::
1915
1916 iodepth_batch_complete_min=0
1917 iodepth_batch_complete_max=<iodepth>
1918
1919 which means that we can retrieve up to the whole submitted queue depth, but
1920 if none of I/O has been completed yet, we will NOT wait and immediately exit
1921 the system call. In this example we simply do polling.
1922
1923.. option:: iodepth_low=int
1924
1925 The low water mark indicating when to start filling the queue
1926 again. Defaults to the same as :option:`iodepth`, meaning that fio will
1927 attempt to keep the queue full at all times. If :option:`iodepth` is set to
c60ebc45 1928 e.g. 16 and *iodepth_low* is set to 4, then after fio has filled the queue of
f80dba8d
MT
1929 16 requests, it will let the depth drain down to 4 before starting to fill
1930 it again.
1931
1932.. option:: io_submit_mode=str
1933
1934 This option controls how fio submits the I/O to the I/O engine. The default
1935 is `inline`, which means that the fio job threads submit and reap I/O
1936 directly. If set to `offload`, the job threads will offload I/O submission
1937 to a dedicated pool of I/O threads. This requires some coordination and thus
1938 has a bit of extra overhead, especially for lower queue depth I/O where it
1939 can increase latencies. The benefit is that fio can manage submission rates
1940 independently of the device completion rates. This avoids skewed latency
1941 reporting if I/O gets back up on the device side (the coordinated omission
1942 problem).
1943
1944
1945I/O rate
1946~~~~~~~~
1947
a881438b 1948.. option:: thinktime=time
f80dba8d
MT
1949
1950 Stall the job x microseconds after an I/O has completed before issuing the
1951 next. May be used to simulate processing being done by an application. See
1952 :option:`thinktime_blocks` and :option:`thinktime_spin`.
1953
a881438b 1954.. option:: thinktime_spin=time
f80dba8d
MT
1955
1956 Only valid if :option:`thinktime` is set - pretend to spend CPU time doing
1957 something with the data received, before falling back to sleeping for the
1958 rest of the period specified by :option:`thinktime`.
1959
1960.. option:: thinktime_blocks=int
1961
1962 Only valid if :option:`thinktime` is set - control how many blocks to issue,
1963 before waiting `thinktime` usecs. If not set, defaults to 1 which will make
1964 fio wait `thinktime` usecs after every block. This effectively makes any
1965 queue depth setting redundant, since no more than 1 I/O will be queued
1966 before we have to complete it and do our thinktime. In other words, this
1967 setting effectively caps the queue depth if the latter is larger.
71bfa161 1968
f80dba8d 1969.. option:: rate=int[,int][,int]
71bfa161 1970
f80dba8d
MT
1971 Cap the bandwidth used by this job. The number is in bytes/sec, the normal
1972 suffix rules apply. Comma-separated values may be specified for reads,
1973 writes, and trims as described in :option:`blocksize`.
71bfa161 1974
f80dba8d 1975.. option:: rate_min=int[,int][,int]
71bfa161 1976
f80dba8d
MT
1977 Tell fio to do whatever it can to maintain at least this bandwidth. Failing
1978 to meet this requirement will cause the job to exit. Comma-separated values
1979 may be specified for reads, writes, and trims as described in
1980 :option:`blocksize`.
71bfa161 1981
f80dba8d 1982.. option:: rate_iops=int[,int][,int]
71bfa161 1983
f80dba8d
MT
1984 Cap the bandwidth to this number of IOPS. Basically the same as
1985 :option:`rate`, just specified independently of bandwidth. If the job is
1986 given a block size range instead of a fixed value, the smallest block size
1987 is used as the metric. Comma-separated values may be specified for reads,
1988 writes, and trims as described in :option:`blocksize`.
71bfa161 1989
f80dba8d 1990.. option:: rate_iops_min=int[,int][,int]
71bfa161 1991
f80dba8d
MT
1992 If fio doesn't meet this rate of I/O, it will cause the job to exit.
1993 Comma-separated values may be specified for reads, writes, and trims as
1994 described in :option:`blocksize`.
71bfa161 1995
f80dba8d 1996.. option:: rate_process=str
66c098b8 1997
f80dba8d
MT
1998 This option controls how fio manages rated I/O submissions. The default is
1999 `linear`, which submits I/O in a linear fashion with fixed delays between
c60ebc45 2000 I/Os that gets adjusted based on I/O completion rates. If this is set to
f80dba8d
MT
2001 `poisson`, fio will submit I/O based on a more real world random request
2002 flow, known as the Poisson process
2003 (https://en.wikipedia.org/wiki/Poisson_point_process). The lambda will be
2004 10^6 / IOPS for the given workload.
71bfa161
JA
2005
2006
f80dba8d
MT
2007I/O latency
2008~~~~~~~~~~~
71bfa161 2009
a881438b 2010.. option:: latency_target=time
71bfa161 2011
f80dba8d
MT
2012 If set, fio will attempt to find the max performance point that the given
2013 workload will run at while maintaining a latency below this target. The
2014 values is given in microseconds. See :option:`latency_window` and
2015 :option:`latency_percentile`.
71bfa161 2016
a881438b 2017.. option:: latency_window=time
71bfa161 2018
f80dba8d
MT
2019 Used with :option:`latency_target` to specify the sample window that the job
2020 is run at varying queue depths to test the performance. The value is given
2021 in microseconds.
b4692828 2022
f80dba8d 2023.. option:: latency_percentile=float
71bfa161 2024
c60ebc45 2025 The percentage of I/Os that must fall within the criteria specified by
f80dba8d 2026 :option:`latency_target` and :option:`latency_window`. If not set, this
c60ebc45 2027 defaults to 100.0, meaning that all I/Os must be equal or below to the value
f80dba8d 2028 set by :option:`latency_target`.
71bfa161 2029
a881438b 2030.. option:: max_latency=time
71bfa161 2031
f80dba8d
MT
2032 If set, fio will exit the job if it exceeds this maximum latency. It will
2033 exit with an ETIME error.
71bfa161 2034
f80dba8d 2035.. option:: rate_cycle=int
71bfa161 2036
f80dba8d
MT
2037 Average bandwidth for :option:`rate` and :option:`rate_min` over this number
2038 of milliseconds.
71bfa161 2039
71bfa161 2040
f80dba8d
MT
2041I/O replay
2042~~~~~~~~~~
71bfa161 2043
f80dba8d 2044.. option:: write_iolog=str
c2b1e753 2045
f80dba8d
MT
2046 Write the issued I/O patterns to the specified file. See
2047 :option:`read_iolog`. Specify a separate file for each job, otherwise the
2048 iologs will be interspersed and the file may be corrupt.
c2b1e753 2049
f80dba8d 2050.. option:: read_iolog=str
71bfa161 2051
f80dba8d
MT
2052 Open an iolog with the specified file name and replay the I/O patterns it
2053 contains. This can be used to store a workload and replay it sometime
2054 later. The iolog given may also be a blktrace binary file, which allows fio
2055 to replay a workload captured by :command:`blktrace`. See
2056 :manpage:`blktrace(8)` for how to capture such logging data. For blktrace
2057 replay, the file needs to be turned into a blkparse binary data file first
2058 (``blkparse <device> -o /dev/null -d file_for_fio.bin``).
71bfa161 2059
f80dba8d 2060.. option:: replay_no_stall=int
71bfa161 2061
f80dba8d
MT
2062 When replaying I/O with :option:`read_iolog` the default behavior is to
2063 attempt to respect the time stamps within the log and replay them with the
2064 appropriate delay between IOPS. By setting this variable fio will not
2065 respect the timestamps and attempt to replay them as fast as possible while
2066 still respecting ordering. The result is the same I/O pattern to a given
2067 device, but different timings.
71bfa161 2068
f80dba8d 2069.. option:: replay_redirect=str
b4692828 2070
f80dba8d
MT
2071 While replaying I/O patterns using :option:`read_iolog` the default behavior
2072 is to replay the IOPS onto the major/minor device that each IOP was recorded
2073 from. This is sometimes undesirable because on a different machine those
2074 major/minor numbers can map to a different device. Changing hardware on the
2075 same system can also result in a different major/minor mapping.
2076 ``replay_redirect`` causes all IOPS to be replayed onto the single specified
2077 device regardless of the device it was recorded
2078 from. i.e. :option:`replay_redirect` = :file:`/dev/sdc` would cause all I/O
2079 in the blktrace or iolog to be replayed onto :file:`/dev/sdc`. This means
2080 multiple devices will be replayed onto a single device, if the trace
2081 contains multiple devices. If you want multiple devices to be replayed
2082 concurrently to multiple redirected devices you must blkparse your trace
2083 into separate traces and replay them with independent fio invocations.
2084 Unfortunately this also breaks the strict time ordering between multiple
2085 device accesses.
71bfa161 2086
f80dba8d 2087.. option:: replay_align=int
74929ac2 2088
f80dba8d
MT
2089 Force alignment of I/O offsets and lengths in a trace to this power of 2
2090 value.
3c54bc46 2091
f80dba8d 2092.. option:: replay_scale=int
3c54bc46 2093
f80dba8d 2094 Scale sector offsets down by this factor when replaying traces.
3c54bc46 2095
3c54bc46 2096
f80dba8d
MT
2097Threads, processes and job synchronization
2098~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3c54bc46 2099
f80dba8d 2100.. option:: thread
3c54bc46 2101
f80dba8d
MT
2102 Fio defaults to forking jobs, however if this option is given, fio will use
2103 :manpage:`pthread_create(3)` to create threads instead.
71bfa161 2104
f80dba8d 2105.. option:: wait_for=str
74929ac2 2106
f80dba8d
MT
2107 Specifies the name of the already defined job to wait for. Single waitee
2108 name only may be specified. If set, the job won't be started until all
2109 workers of the waitee job are done.
74929ac2 2110
f80dba8d
MT
2111 ``wait_for`` operates on the job name basis, so there are a few
2112 limitations. First, the waitee must be defined prior to the waiter job
2113 (meaning no forward references). Second, if a job is being referenced as a
2114 waitee, it must have a unique name (no duplicate waitees).
74929ac2 2115
f80dba8d 2116.. option:: nice=int
892a6ffc 2117
f80dba8d 2118 Run the job with the given nice value. See man :manpage:`nice(2)`.
892a6ffc 2119
f80dba8d
MT
2120 On Windows, values less than -15 set the process class to "High"; -1 through
2121 -15 set "Above Normal"; 1 through 15 "Below Normal"; and above 15 "Idle"
2122 priority class.
74929ac2 2123
f80dba8d 2124.. option:: prio=int
71bfa161 2125
f80dba8d
MT
2126 Set the I/O priority value of this job. Linux limits us to a positive value
2127 between 0 and 7, with 0 being the highest. See man
2128 :manpage:`ionice(1)`. Refer to an appropriate manpage for other operating
2129 systems since meaning of priority may differ.
71bfa161 2130
f80dba8d 2131.. option:: prioclass=int
d59aa780 2132
f80dba8d 2133 Set the I/O priority class. See man :manpage:`ionice(1)`.
d59aa780 2134
f80dba8d 2135.. option:: cpumask=int
71bfa161 2136
f80dba8d
MT
2137 Set the CPU affinity of this job. The parameter given is a bitmask of
2138 allowed CPU's the job may run on. So if you want the allowed CPUs to be 1
2139 and 5, you would pass the decimal value of (1 << 1 | 1 << 5), or 34. See man
2140 :manpage:`sched_setaffinity(2)`. This may not work on all supported
2141 operating systems or kernel versions. This option doesn't work well for a
2142 higher CPU count than what you can store in an integer mask, so it can only
2143 control cpus 1-32. For boxes with larger CPU counts, use
2144 :option:`cpus_allowed`.
6d500c2e 2145
f80dba8d 2146.. option:: cpus_allowed=str
6d500c2e 2147
f80dba8d
MT
2148 Controls the same options as :option:`cpumask`, but it allows a text setting
2149 of the permitted CPUs instead. So to use CPUs 1 and 5, you would specify
2150 ``cpus_allowed=1,5``. This options also allows a range of CPUs. Say you
2151 wanted a binding to CPUs 1, 5, and 8-15, you would set
2152 ``cpus_allowed=1,5,8-15``.
6d500c2e 2153
f80dba8d 2154.. option:: cpus_allowed_policy=str
6d500c2e 2155
f80dba8d
MT
2156 Set the policy of how fio distributes the CPUs specified by
2157 :option:`cpus_allowed` or cpumask. Two policies are supported:
6d500c2e 2158
f80dba8d
MT
2159 **shared**
2160 All jobs will share the CPU set specified.
2161 **split**
2162 Each job will get a unique CPU from the CPU set.
6d500c2e 2163
f80dba8d
MT
2164 **shared** is the default behaviour, if the option isn't specified. If
2165 **split** is specified, then fio will will assign one cpu per job. If not
2166 enough CPUs are given for the jobs listed, then fio will roundrobin the CPUs
2167 in the set.
6d500c2e 2168
f80dba8d 2169.. option:: numa_cpu_nodes=str
6d500c2e 2170
f80dba8d
MT
2171 Set this job running on specified NUMA nodes' CPUs. The arguments allow
2172 comma delimited list of cpu numbers, A-B ranges, or `all`. Note, to enable
2173 numa options support, fio must be built on a system with libnuma-dev(el)
2174 installed.
61b9861d 2175
f80dba8d 2176.. option:: numa_mem_policy=str
61b9861d 2177
f80dba8d
MT
2178 Set this job's memory policy and corresponding NUMA nodes. Format of the
2179 arguments::
5c94b008 2180
f80dba8d 2181 <mode>[:<nodelist>]
ce35b1ec 2182
f80dba8d
MT
2183 ``mode`` is one of the following memory policy: ``default``, ``prefer``,
2184 ``bind``, ``interleave``, ``local`` For ``default`` and ``local`` memory
2185 policy, no node is needed to be specified. For ``prefer``, only one node is
2186 allowed. For ``bind`` and ``interleave``, it allow comma delimited list of
2187 numbers, A-B ranges, or `all`.
71bfa161 2188
f80dba8d 2189.. option:: cgroup=str
390b1537 2190
f80dba8d
MT
2191 Add job to this control group. If it doesn't exist, it will be created. The
2192 system must have a mounted cgroup blkio mount point for this to work. If
2193 your system doesn't have it mounted, you can do so with::
5af1c6f3 2194
f80dba8d 2195 # mount -t cgroup -o blkio none /cgroup
5af1c6f3 2196
f80dba8d 2197.. option:: cgroup_weight=int
5af1c6f3 2198
f80dba8d
MT
2199 Set the weight of the cgroup to this value. See the documentation that comes
2200 with the kernel, allowed values are in the range of 100..1000.
a086c257 2201
f80dba8d 2202.. option:: cgroup_nodelete=bool
8c07860d 2203
f80dba8d
MT
2204 Normally fio will delete the cgroups it has created after the job
2205 completion. To override this behavior and to leave cgroups around after the
2206 job completion, set ``cgroup_nodelete=1``. This can be useful if one wants
2207 to inspect various cgroup files after job completion. Default: false.
8c07860d 2208
f80dba8d 2209.. option:: flow_id=int
8c07860d 2210
f80dba8d
MT
2211 The ID of the flow. If not specified, it defaults to being a global
2212 flow. See :option:`flow`.
1907dbc6 2213
f80dba8d 2214.. option:: flow=int
71bfa161 2215
f80dba8d
MT
2216 Weight in token-based flow control. If this value is used, then there is a
2217 'flow counter' which is used to regulate the proportion of activity between
2218 two or more jobs. Fio attempts to keep this flow counter near zero. The
2219 ``flow`` parameter stands for how much should be added or subtracted to the
2220 flow counter on each iteration of the main I/O loop. That is, if one job has
2221 ``flow=8`` and another job has ``flow=-1``, then there will be a roughly 1:8
2222 ratio in how much one runs vs the other.
71bfa161 2223
f80dba8d 2224.. option:: flow_watermark=int
a31041ea 2225
f80dba8d
MT
2226 The maximum value that the absolute value of the flow counter is allowed to
2227 reach before the job must wait for a lower value of the counter.
82407585 2228
f80dba8d 2229.. option:: flow_sleep=int
82407585 2230
f80dba8d
MT
2231 The period of time, in microseconds, to wait after the flow watermark has
2232 been exceeded before retrying operations.
82407585 2233
f80dba8d 2234.. option:: stonewall, wait_for_previous
82407585 2235
f80dba8d
MT
2236 Wait for preceding jobs in the job file to exit, before starting this
2237 one. Can be used to insert serialization points in the job file. A stone
2238 wall also implies starting a new reporting group, see
2239 :option:`group_reporting`.
2240
2241.. option:: exitall
2242
2243 When one job finishes, terminate the rest. The default is to wait for each
2244 job to finish, sometimes that is not the desired action.
2245
2246.. option:: exec_prerun=str
2247
2248 Before running this job, issue the command specified through
2249 :manpage:`system(3)`. Output is redirected in a file called
2250 :file:`jobname.prerun.txt`.
2251
2252.. option:: exec_postrun=str
2253
2254 After the job completes, issue the command specified though
2255 :manpage:`system(3)`. Output is redirected in a file called
2256 :file:`jobname.postrun.txt`.
2257
2258.. option:: uid=int
2259
2260 Instead of running as the invoking user, set the user ID to this value
2261 before the thread/process does any work.
2262
2263.. option:: gid=int
2264
2265 Set group ID, see :option:`uid`.
2266
2267
2268Verification
2269~~~~~~~~~~~~
2270
2271.. option:: verify_only
2272
2273 Do not perform specified workload, only verify data still matches previous
2274 invocation of this workload. This option allows one to check data multiple
2275 times at a later date without overwriting it. This option makes sense only
2276 for workloads that write data, and does not support workloads with the
2277 :option:`time_based` option set.
2278
2279.. option:: do_verify=bool
2280
2281 Run the verify phase after a write phase. Only valid if :option:`verify` is
2282 set. Default: true.
2283
2284.. option:: verify=str
2285
2286 If writing to a file, fio can verify the file contents after each iteration
2287 of the job. Each verification method also implies verification of special
2288 header, which is written to the beginning of each block. This header also
2289 includes meta information, like offset of the block, block number, timestamp
2290 when block was written, etc. :option:`verify` can be combined with
2291 :option:`verify_pattern` option. The allowed values are:
2292
2293 **md5**
2294 Use an md5 sum of the data area and store it in the header of
2295 each block.
2296
2297 **crc64**
2298 Use an experimental crc64 sum of the data area and store it in the
2299 header of each block.
2300
2301 **crc32c**
2302 Use a crc32c sum of the data area and store it in the header of each
2303 block.
2304
2305 **crc32c-intel**
2306 Use hardware assisted crc32c calculation provided on SSE4.2 enabled
2307 processors. Falls back to regular software crc32c, if not supported
2308 by the system.
2309
2310 **crc32**
2311 Use a crc32 sum of the data area and store it in the header of each
2312 block.
2313
2314 **crc16**
2315 Use a crc16 sum of the data area and store it in the header of each
2316 block.
2317
2318 **crc7**
2319 Use a crc7 sum of the data area and store it in the header of each
2320 block.
2321
2322 **xxhash**
2323 Use xxhash as the checksum function. Generally the fastest software
2324 checksum that fio supports.
2325
2326 **sha512**
2327 Use sha512 as the checksum function.
2328
2329 **sha256**
2330 Use sha256 as the checksum function.
2331
2332 **sha1**
2333 Use optimized sha1 as the checksum function.
82407585 2334
f80dba8d
MT
2335 **meta**
2336 This option is deprecated, since now meta information is included in
2337 generic verification header and meta verification happens by
2338 default. For detailed information see the description of the
2339 :option:`verify` setting. This option is kept because of
2340 compatibility's sake with old configurations. Do not use it.
2341
2342 **pattern**
2343 Verify a strict pattern. Normally fio includes a header with some
2344 basic information and checksumming, but if this option is set, only
2345 the specific pattern set with :option:`verify_pattern` is verified.
2346
2347 **null**
2348 Only pretend to verify. Useful for testing internals with
2349 :option:`ioengine` `=null`, not for much else.
2350
2351 This option can be used for repeated burn-in tests of a system to make sure
2352 that the written data is also correctly read back. If the data direction
2353 given is a read or random read, fio will assume that it should verify a
2354 previously written file. If the data direction includes any form of write,
2355 the verify will be of the newly written data.
2356
2357.. option:: verifysort=bool
2358
2359 If true, fio will sort written verify blocks when it deems it faster to read
2360 them back in a sorted manner. This is often the case when overwriting an
2361 existing file, since the blocks are already laid out in the file system. You
2362 can ignore this option unless doing huge amounts of really fast I/O where
2363 the red-black tree sorting CPU time becomes significant. Default: true.
2364
2365.. option:: verifysort_nr=int
2366
2367 Pre-load and sort verify blocks for a read workload.
2368
2369.. option:: verify_offset=int
2370
2371 Swap the verification header with data somewhere else in the block before
2372 writing. It is swapped back before verifying.
2373
2374.. option:: verify_interval=int
2375
2376 Write the verification header at a finer granularity than the
2377 :option:`blocksize`. It will be written for chunks the size of
2378 ``verify_interval``. :option:`blocksize` should divide this evenly.
2379
2380.. option:: verify_pattern=str
2381
2382 If set, fio will fill the I/O buffers with this pattern. Fio defaults to
2383 filling with totally random bytes, but sometimes it's interesting to fill
2384 with a known pattern for I/O verification purposes. Depending on the width
2385 of the pattern, fio will fill 1/2/3/4 bytes of the buffer at the time(it can
2386 be either a decimal or a hex number). The ``verify_pattern`` if larger than
2387 a 32-bit quantity has to be a hex number that starts with either "0x" or
2388 "0X". Use with :option:`verify`. Also, ``verify_pattern`` supports %o
2389 format, which means that for each block offset will be written and then
2390 verified back, e.g.::
61b9861d
RP
2391
2392 verify_pattern=%o
2393
f80dba8d
MT
2394 Or use combination of everything::
2395
61b9861d 2396 verify_pattern=0xff%o"abcd"-12
e28218f3 2397
f80dba8d
MT
2398.. option:: verify_fatal=bool
2399
2400 Normally fio will keep checking the entire contents before quitting on a
2401 block verification failure. If this option is set, fio will exit the job on
2402 the first observed failure. Default: false.
2403
2404.. option:: verify_dump=bool
2405
2406 If set, dump the contents of both the original data block and the data block
2407 we read off disk to files. This allows later analysis to inspect just what
2408 kind of data corruption occurred. Off by default.
2409
2410.. option:: verify_async=int
2411
2412 Fio will normally verify I/O inline from the submitting thread. This option
2413 takes an integer describing how many async offload threads to create for I/O
2414 verification instead, causing fio to offload the duty of verifying I/O
2415 contents to one or more separate threads. If using this offload option, even
2416 sync I/O engines can benefit from using an :option:`iodepth` setting higher
2417 than 1, as it allows them to have I/O in flight while verifies are running.
2418
2419.. option:: verify_async_cpus=str
2420
2421 Tell fio to set the given CPU affinity on the async I/O verification
2422 threads. See :option:`cpus_allowed` for the format used.
2423
2424.. option:: verify_backlog=int
2425
2426 Fio will normally verify the written contents of a job that utilizes verify
2427 once that job has completed. In other words, everything is written then
2428 everything is read back and verified. You may want to verify continually
2429 instead for a variety of reasons. Fio stores the meta data associated with
2430 an I/O block in memory, so for large verify workloads, quite a bit of memory
2431 would be used up holding this meta data. If this option is enabled, fio will
2432 write only N blocks before verifying these blocks.
2433
2434.. option:: verify_backlog_batch=int
2435
2436 Control how many blocks fio will verify if :option:`verify_backlog` is
2437 set. If not set, will default to the value of :option:`verify_backlog`
2438 (meaning the entire queue is read back and verified). If
2439 ``verify_backlog_batch`` is less than :option:`verify_backlog` then not all
2440 blocks will be verified, if ``verify_backlog_batch`` is larger than
2441 :option:`verify_backlog`, some blocks will be verified more than once.
2442
2443.. option:: verify_state_save=bool
2444
2445 When a job exits during the write phase of a verify workload, save its
2446 current state. This allows fio to replay up until that point, if the verify
2447 state is loaded for the verify read phase. The format of the filename is,
2448 roughly::
2449
2450 <type>-<jobname>-<jobindex>-verify.state.
2451
2452 <type> is "local" for a local run, "sock" for a client/server socket
2453 connection, and "ip" (192.168.0.1, for instance) for a networked
2454 client/server connection.
2455
2456.. option:: verify_state_load=bool
2457
2458 If a verify termination trigger was used, fio stores the current write state
2459 of each thread. This can be used at verification time so that fio knows how
2460 far it should verify. Without this information, fio will run a full
2461 verification pass, according to the settings in the job file used.
2462
2463.. option:: trim_percentage=int
2464
2465 Number of verify blocks to discard/trim.
2466
2467.. option:: trim_verify_zero=bool
2468
2469 Verify that trim/discarded blocks are returned as zeroes.
2470
2471.. option:: trim_backlog=int
2472
2473 Verify that trim/discarded blocks are returned as zeroes.
2474
2475.. option:: trim_backlog_batch=int
2476
2477 Trim this number of I/O blocks.
2478
2479.. option:: experimental_verify=bool
2480
2481 Enable experimental verification.
2482
2483
2484Steady state
2485~~~~~~~~~~~~
2486
2487.. option:: steadystate=str:float, ss=str:float
2488
2489 Define the criterion and limit for assessing steady state performance. The
2490 first parameter designates the criterion whereas the second parameter sets
2491 the threshold. When the criterion falls below the threshold for the
2492 specified duration, the job will stop. For example, `iops_slope:0.1%` will
2493 direct fio to terminate the job when the least squares regression slope
2494 falls below 0.1% of the mean IOPS. If :option:`group_reporting` is enabled
2495 this will apply to all jobs in the group. Below is the list of available
2496 steady state assessment criteria. All assessments are carried out using only
2497 data from the rolling collection window. Threshold limits can be expressed
2498 as a fixed value or as a percentage of the mean in the collection window.
2499
2500 **iops**
2501 Collect IOPS data. Stop the job if all individual IOPS measurements
2502 are within the specified limit of the mean IOPS (e.g., ``iops:2``
2503 means that all individual IOPS values must be within 2 of the mean,
2504 whereas ``iops:0.2%`` means that all individual IOPS values must be
2505 within 0.2% of the mean IOPS to terminate the job).
2506
2507 **iops_slope**
2508 Collect IOPS data and calculate the least squares regression
2509 slope. Stop the job if the slope falls below the specified limit.
2510
2511 **bw**
2512 Collect bandwidth data. Stop the job if all individual bandwidth
2513 measurements are within the specified limit of the mean bandwidth.
2514
2515 **bw_slope**
2516 Collect bandwidth data and calculate the least squares regression
2517 slope. Stop the job if the slope falls below the specified limit.
2518
2519.. option:: steadystate_duration=time, ss_dur=time
2520
2521 A rolling window of this duration will be used to judge whether steady state
2522 has been reached. Data will be collected once per second. The default is 0
2523 which disables steady state detection.
2524
2525.. option:: steadystate_ramp_time=time, ss_ramp=time
2526
2527 Allow the job to run for the specified duration before beginning data
2528 collection for checking the steady state job termination criterion. The
2529 default is 0.
2530
2531
2532Measurements and reporting
2533~~~~~~~~~~~~~~~~~~~~~~~~~~
2534
2535.. option:: per_job_logs=bool
2536
2537 If set, this generates bw/clat/iops log with per file private filenames. If
2538 not set, jobs with identical names will share the log filename. Default:
2539 true.
2540
2541.. option:: group_reporting
2542
2543 It may sometimes be interesting to display statistics for groups of jobs as
2544 a whole instead of for each individual job. This is especially true if
2545 :option:`numjobs` is used; looking at individual thread/process output
2546 quickly becomes unwieldy. To see the final report per-group instead of
2547 per-job, use :option:`group_reporting`. Jobs in a file will be part of the
2548 same reporting group, unless if separated by a :option:`stonewall`, or by
2549 using :option:`new_group`.
2550
2551.. option:: new_group
2552
2553 Start a new reporting group. See: :option:`group_reporting`. If not given,
2554 all jobs in a file will be part of the same reporting group, unless
2555 separated by a :option:`stonewall`.
2556
2557.. option:: write_bw_log=str
2558
2559 If given, write a bandwidth log for this job. Can be used to store data of
2560 the bandwidth of the jobs in their lifetime. The included
2561 :command:`fio_generate_plots` script uses :command:`gnuplot` to turn these
2562 text files into nice graphs. See :option:`write_lat_log` for behaviour of
2563 given filename. For this option, the postfix is :file:`_bw.x.log`, where `x`
2564 is the index of the job (`1..N`, where `N` is the number of jobs). If
2565 :option:`per_job_logs` is false, then the filename will not include the job
2566 index. See `Log File Formats`_.
2567
2568.. option:: write_lat_log=str
2569
2570 Same as :option:`write_bw_log`, except that this option stores I/O
2571 submission, completion, and total latencies instead. If no filename is given
2572 with this option, the default filename of :file:`jobname_type.log` is
2573 used. Even if the filename is given, fio will still append the type of
2574 log. So if one specifies::
e3cedca7
JA
2575
2576 write_lat_log=foo
2577
f80dba8d
MT
2578 The actual log names will be :file:`foo_slat.x.log`, :file:`foo_clat.x.log`,
2579 and :file:`foo_lat.x.log`, where `x` is the index of the job (1..N, where N
2580 is the number of jobs). This helps :command:`fio_generate_plot` find the
2581 logs automatically. If :option:`per_job_logs` is false, then the filename
2582 will not include the job index. See `Log File Formats`_.
be4ecfdf 2583
f80dba8d 2584.. option:: write_hist_log=str
06842027 2585
f80dba8d
MT
2586 Same as :option:`write_lat_log`, but writes I/O completion latency
2587 histograms. If no filename is given with this option, the default filename
2588 of :file:`jobname_clat_hist.x.log` is used, where `x` is the index of the
2589 job (1..N, where `N` is the number of jobs). Even if the filename is given,
2590 fio will still append the type of log. If :option:`per_job_logs` is false,
2591 then the filename will not include the job index. See `Log File Formats`_.
06842027 2592
f80dba8d 2593.. option:: write_iops_log=str
06842027 2594
f80dba8d
MT
2595 Same as :option:`write_bw_log`, but writes IOPS. If no filename is given
2596 with this option, the default filename of :file:`jobname_type.x.log` is
2597 used,where `x` is the index of the job (1..N, where `N` is the number of
2598 jobs). Even if the filename is given, fio will still append the type of
2599 log. If :option:`per_job_logs` is false, then the filename will not include
2600 the job index. See `Log File Formats`_.
06842027 2601
f80dba8d 2602.. option:: log_avg_msec=int
06842027 2603
f80dba8d
MT
2604 By default, fio will log an entry in the iops, latency, or bw log for every
2605 I/O that completes. When writing to the disk log, that can quickly grow to a
2606 very large size. Setting this option makes fio average the each log entry
2607 over the specified period of time, reducing the resolution of the log. See
2608 :option:`log_max_value` as well. Defaults to 0, logging all entries.
06842027 2609
f80dba8d 2610.. option:: log_hist_msec=int
06842027 2611
f80dba8d
MT
2612 Same as :option:`log_avg_msec`, but logs entries for completion latency
2613 histograms. Computing latency percentiles from averages of intervals using
c60ebc45 2614 :option:`log_avg_msec` is inaccurate. Setting this option makes fio log
f80dba8d
MT
2615 histogram entries over the specified period of time, reducing log sizes for
2616 high IOPS devices while retaining percentile accuracy. See
2617 :option:`log_hist_coarseness` as well. Defaults to 0, meaning histogram
2618 logging is disabled.
06842027 2619
f80dba8d 2620.. option:: log_hist_coarseness=int
06842027 2621
f80dba8d
MT
2622 Integer ranging from 0 to 6, defining the coarseness of the resolution of
2623 the histogram logs enabled with :option:`log_hist_msec`. For each increment
2624 in coarseness, fio outputs half as many bins. Defaults to 0, for which
2625 histogram logs contain 1216 latency bins. See `Log File Formats`_.
8b28bd41 2626
f80dba8d 2627.. option:: log_max_value=bool
66c098b8 2628
f80dba8d
MT
2629 If :option:`log_avg_msec` is set, fio logs the average over that window. If
2630 you instead want to log the maximum value, set this option to 1. Defaults to
2631 0, meaning that averaged values are logged.
a696fa2a 2632
f80dba8d 2633.. option:: log_offset=int
a696fa2a 2634
f80dba8d
MT
2635 If this is set, the iolog options will include the byte offset for the I/O
2636 entry as well as the other data values.
71bfa161 2637
f80dba8d 2638.. option:: log_compression=int
7de87099 2639
f80dba8d
MT
2640 If this is set, fio will compress the I/O logs as it goes, to keep the
2641 memory footprint lower. When a log reaches the specified size, that chunk is
2642 removed and compressed in the background. Given that I/O logs are fairly
2643 highly compressible, this yields a nice memory savings for longer runs. The
2644 downside is that the compression will consume some background CPU cycles, so
2645 it may impact the run. This, however, is also true if the logging ends up
2646 consuming most of the system memory. So pick your poison. The I/O logs are
2647 saved normally at the end of a run, by decompressing the chunks and storing
2648 them in the specified log file. This feature depends on the availability of
2649 zlib.
e0b0d892 2650
f80dba8d 2651.. option:: log_compression_cpus=str
e0b0d892 2652
f80dba8d
MT
2653 Define the set of CPUs that are allowed to handle online log compression for
2654 the I/O jobs. This can provide better isolation between performance
2655 sensitive jobs, and background compression work.
9e684a49 2656
f80dba8d 2657.. option:: log_store_compressed=bool
9e684a49 2658
f80dba8d
MT
2659 If set, fio will store the log files in a compressed format. They can be
2660 decompressed with fio, using the :option:`--inflate-log` command line
2661 parameter. The files will be stored with a :file:`.fz` suffix.
9e684a49 2662
f80dba8d 2663.. option:: log_unix_epoch=bool
9e684a49 2664
f80dba8d
MT
2665 If set, fio will log Unix timestamps to the log files produced by enabling
2666 write_type_log for each log type, instead of the default zero-based
2667 timestamps.
2668
2669.. option:: block_error_percentiles=bool
2670
2671 If set, record errors in trim block-sized units from writes and trims and
2672 output a histogram of how many trims it took to get to errors, and what kind
2673 of error was encountered.
2674
2675.. option:: bwavgtime=int
2676
2677 Average the calculated bandwidth over the given time. Value is specified in
2678 milliseconds. If the job also does bandwidth logging through
2679 :option:`write_bw_log`, then the minimum of this option and
2680 :option:`log_avg_msec` will be used. Default: 500ms.
2681
2682.. option:: iopsavgtime=int
2683
2684 Average the calculated IOPS over the given time. Value is specified in
2685 milliseconds. If the job also does IOPS logging through
2686 :option:`write_iops_log`, then the minimum of this option and
2687 :option:`log_avg_msec` will be used. Default: 500ms.
2688
2689.. option:: disk_util=bool
2690
2691 Generate disk utilization statistics, if the platform supports it.
2692 Default: true.
2693
2694.. option:: disable_lat=bool
2695
2696 Disable measurements of total latency numbers. Useful only for cutting back
2697 the number of calls to :manpage:`gettimeofday(2)`, as that does impact
2698 performance at really high IOPS rates. Note that to really get rid of a
2699 large amount of these calls, this option must be used with
2700 :option:`disable_slat` and :option:`disable_bw` as well.
2701
2702.. option:: disable_clat=bool
2703
2704 Disable measurements of completion latency numbers. See
2705 :option:`disable_lat`.
2706
2707.. option:: disable_slat=bool
2708
2709 Disable measurements of submission latency numbers. See
2710 :option:`disable_slat`.
2711
2712.. option:: disable_bw=bool
2713
2714 Disable measurements of throughput/bandwidth numbers. See
2715 :option:`disable_lat`.
2716
2717.. option:: clat_percentiles=bool
2718
2719 Enable the reporting of percentiles of completion latencies.
2720
2721.. option:: percentile_list=float_list
2722
2723 Overwrite the default list of percentiles for completion latencies and the
2724 block error histogram. Each number is a floating number in the range
2725 (0,100], and the maximum length of the list is 20. Use ``:`` to separate the
2726 numbers, and list the numbers in ascending order. For example,
2727 ``--percentile_list=99.5:99.9`` will cause fio to report the values of
2728 completion latency below which 99.5% and 99.9% of the observed latencies
2729 fell, respectively.
2730
2731
2732Error handling
2733~~~~~~~~~~~~~~
2734
2735.. option:: exitall_on_error
2736
2737 When one job finishes in error, terminate the rest. The default is to wait
2738 for each job to finish.
2739
2740.. option:: continue_on_error=str
2741
2742 Normally fio will exit the job on the first observed failure. If this option
2743 is set, fio will continue the job when there is a 'non-fatal error' (EIO or
2744 EILSEQ) until the runtime is exceeded or the I/O size specified is
2745 completed. If this option is used, there are two more stats that are
2746 appended, the total error count and the first error. The error field given
2747 in the stats is the first error that was hit during the run.
2748
2749 The allowed values are:
2750
2751 **none**
2752 Exit on any I/O or verify errors.
2753
2754 **read**
2755 Continue on read errors, exit on all others.
2756
2757 **write**
2758 Continue on write errors, exit on all others.
2759
2760 **io**
2761 Continue on any I/O error, exit on all others.
2762
2763 **verify**
2764 Continue on verify errors, exit on all others.
2765
2766 **all**
2767 Continue on all errors.
2768
2769 **0**
2770 Backward-compatible alias for 'none'.
2771
2772 **1**
2773 Backward-compatible alias for 'all'.
2774
2775.. option:: ignore_error=str
2776
2777 Sometimes you want to ignore some errors during test in that case you can
2778 specify error list for each error type.
2779 ``ignore_error=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST`` errors for
2780 given error type is separated with ':'. Error may be symbol ('ENOSPC',
2781 'ENOMEM') or integer. Example::
2782
2783 ignore_error=EAGAIN,ENOSPC:122
2784
2785 This option will ignore EAGAIN from READ, and ENOSPC and 122(EDQUOT) from
2786 WRITE.
2787
2788.. option:: error_dump=bool
2789
2790 If set dump every error even if it is non fatal, true by default. If
2791 disabled only fatal error will be dumped.
2792
2793
2794Interpreting the output
2795-----------------------
2796
2797Fio spits out a lot of output. While running, fio will display the status of the
2798jobs created. An example of that would be::
2799
2800 Jobs: 1: [_r] [24.8% done] [r=20992KiB/s,w=24064KiB/s,t=0KiB/s] [r=82,w=94,t=0 iops] [eta 00h:01m:31s]
2801
2802The characters inside the square brackets denote the current status of each
2803thread. The possible values (in typical life cycle order) are:
2804
2805+------+-----+-----------------------------------------------------------+
2806| Idle | Run | |
2807+======+=====+===========================================================+
2808| P | | Thread setup, but not started. |
2809+------+-----+-----------------------------------------------------------+
2810| C | | Thread created. |
2811+------+-----+-----------------------------------------------------------+
2812| I | | Thread initialized, waiting or generating necessary data. |
2813+------+-----+-----------------------------------------------------------+
2814| | p | Thread running pre-reading file(s). |
2815+------+-----+-----------------------------------------------------------+
2816| | R | Running, doing sequential reads. |
2817+------+-----+-----------------------------------------------------------+
2818| | r | Running, doing random reads. |
2819+------+-----+-----------------------------------------------------------+
2820| | W | Running, doing sequential writes. |
2821+------+-----+-----------------------------------------------------------+
2822| | w | Running, doing random writes. |
2823+------+-----+-----------------------------------------------------------+
2824| | M | Running, doing mixed sequential reads/writes. |
2825+------+-----+-----------------------------------------------------------+
2826| | m | Running, doing mixed random reads/writes. |
2827+------+-----+-----------------------------------------------------------+
2828| | F | Running, currently waiting for :manpage:`fsync(2)` |
2829+------+-----+-----------------------------------------------------------+
2830| | V | Running, doing verification of written data. |
2831+------+-----+-----------------------------------------------------------+
2832| E | | Thread exited, not reaped by main thread yet. |
2833+------+-----+-----------------------------------------------------------+
2834| _ | | Thread reaped, or |
2835+------+-----+-----------------------------------------------------------+
2836| X | | Thread reaped, exited with an error. |
2837+------+-----+-----------------------------------------------------------+
2838| K | | Thread reaped, exited due to signal. |
2839+------+-----+-----------------------------------------------------------+
2840
2841Fio will condense the thread string as not to take up more space on the command
2842line as is needed. For instance, if you have 10 readers and 10 writers running,
2843the output would look like this::
2844
2845 Jobs: 20 (f=20): [R(10),W(10)] [4.0% done] [r=20992KiB/s,w=24064KiB/s,t=0KiB/s] [r=82,w=94,t=0 iops] [eta 57m:36s]
2846
2847Fio will still maintain the ordering, though. So the above means that jobs 1..10
2848are readers, and 11..20 are writers.
2849
2850The other values are fairly self explanatory -- number of threads currently
2851running and doing I/O, rate of I/O since last check (read speed listed first,
2852then write speed), and the estimated completion percentage and time for the
2853running group. It's impossible to estimate runtime of the following groups (if
2854any). Note that the string is displayed in order, so it's possible to tell which
2855of the jobs are currently doing what. The first character is the first job
2856defined in the job file, and so forth.
2857
2858When fio is done (or interrupted by :kbd:`ctrl-c`), it will show the data for
2859each thread, group of threads, and disks in that order. For each data direction,
2860the output looks like::
2861
2862 Client1 (g=0): err= 0:
2863 write: io= 32MiB, bw= 666KiB/s, iops=89 , runt= 50320msec
2864 slat (msec): min= 0, max= 136, avg= 0.03, stdev= 1.92
2865 clat (msec): min= 0, max= 631, avg=48.50, stdev=86.82
2866 bw (KiB/s) : min= 0, max= 1196, per=51.00%, avg=664.02, stdev=681.68
2867 cpu : usr=1.49%, sys=0.25%, ctx=7969, majf=0, minf=17
2868 IO depths : 1=0.1%, 2=0.3%, 4=0.5%, 8=99.0%, 16=0.0%, 32=0.0%, >32=0.0%
2869 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
2870 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
2871 issued r/w: total=0/32768, short=0/0
2872 lat (msec): 2=1.6%, 4=0.0%, 10=3.2%, 20=12.8%, 50=38.4%, 100=24.8%,
2873 lat (msec): 250=15.2%, 500=0.0%, 750=0.0%, 1000=0.0%, >=2048=0.0%
71bfa161
JA
2874
2875The client number is printed, along with the group id and error of that
f80dba8d
MT
2876thread. Below is the I/O statistics, here for writes. In the order listed, they
2877denote:
2878
2879**io**
2880 Number of megabytes I/O performed.
2881
2882**bw**
2883 Average bandwidth rate.
2884
2885**iops**
c60ebc45 2886 Average I/Os performed per second.
f80dba8d
MT
2887
2888**runt**
2889 The runtime of that thread.
2890
2891**slat**
2892 Submission latency (avg being the average, stdev being the standard
2893 deviation). This is the time it took to submit the I/O. For sync I/O,
2894 the slat is really the completion latency, since queue/complete is one
2895 operation there. This value can be in milliseconds or microseconds, fio
2896 will choose the most appropriate base and print that. In the example
2897 above, milliseconds is the best scale. Note: in :option:`--minimal` mode
0d237712 2898 latencies are always expressed in microseconds.
f80dba8d
MT
2899
2900**clat**
2901 Completion latency. Same names as slat, this denotes the time from
2902 submission to completion of the I/O pieces. For sync I/O, clat will
2903 usually be equal (or very close) to 0, as the time from submit to
2904 complete is basically just CPU time (I/O has already been done, see slat
2905 explanation).
2906
2907**bw**
2908 Bandwidth. Same names as the xlat stats, but also includes an
2909 approximate percentage of total aggregate bandwidth this thread received
2910 in this group. This last value is only really useful if the threads in
2911 this group are on the same disk, since they are then competing for disk
2912 access.
2913
2914**cpu**
2915 CPU usage. User and system time, along with the number of context
2916 switches this thread went through, usage of system and user time, and
2917 finally the number of major and minor page faults. The CPU utilization
2918 numbers are averages for the jobs in that reporting group, while the
23a8e176 2919 context and fault counters are summed.
f80dba8d
MT
2920
2921**IO depths**
2922 The distribution of I/O depths over the job life time. The numbers are
2923 divided into powers of 2, so for example the 16= entries includes depths
2924 up to that value but higher than the previous entry. In other words, it
2925 covers the range from 16 to 31.
2926
2927**IO submit**
2928 How many pieces of I/O were submitting in a single submit call. Each
c60ebc45
SW
2929 entry denotes that amount and below, until the previous entry -- e.g.,
2930 8=100% mean that we submitted anywhere in between 5-8 I/Os per submit
f80dba8d
MT
2931 call.
2932
2933**IO complete**
2934 Like the above submit number, but for completions instead.
2935
2936**IO issued**
2937 The number of read/write requests issued, and how many of them were
2938 short.
2939
2940**IO latencies**
2941 The distribution of I/O completion latencies. This is the time from when
2942 I/O leaves fio and when it gets completed. The numbers follow the same
2943 pattern as the I/O depths, meaning that 2=1.6% means that 1.6% of the
2944 I/O completed within 2 msecs, 20=12.8% means that 12.8% of the I/O took
2945 more than 10 msecs, but less than (or equal to) 20 msecs.
71bfa161
JA
2946
2947After each client has been listed, the group statistics are printed. They
f80dba8d 2948will look like this::
71bfa161 2949
f80dba8d
MT
2950 Run status group 0 (all jobs):
2951 READ: io=64MB, aggrb=22178, minb=11355, maxb=11814, mint=2840msec, maxt=2955msec
2952 WRITE: io=64MB, aggrb=1302, minb=666, maxb=669, mint=50093msec, maxt=50320msec
71bfa161
JA
2953
2954For each data direction, it prints:
2955
f80dba8d
MT
2956**io**
2957 Number of megabytes I/O performed.
2958**aggrb**
2959 Aggregate bandwidth of threads in this group.
2960**minb**
2961 The minimum average bandwidth a thread saw.
2962**maxb**
2963 The maximum average bandwidth a thread saw.
2964**mint**
2965 The smallest runtime of the threads in that group.
2966**maxt**
2967 The longest runtime of the threads in that group.
71bfa161 2968
f80dba8d 2969And finally, the disk statistics are printed. They will look like this::
71bfa161 2970
f80dba8d
MT
2971 Disk stats (read/write):
2972 sda: ios=16398/16511, merge=30/162, ticks=6853/819634, in_queue=826487, util=100.00%
71bfa161
JA
2973
2974Each value is printed for both reads and writes, with reads first. The
2975numbers denote:
2976
f80dba8d 2977**ios**
c60ebc45 2978 Number of I/Os performed by all groups.
f80dba8d
MT
2979**merge**
2980 Number of merges I/O the I/O scheduler.
2981**ticks**
2982 Number of ticks we kept the disk busy.
2983**io_queue**
2984 Total time spent in the disk queue.
2985**util**
2986 The disk utilization. A value of 100% means we kept the disk
71bfa161
JA
2987 busy constantly, 50% would be a disk idling half of the time.
2988
f80dba8d
MT
2989It is also possible to get fio to dump the current output while it is running,
2990without terminating the job. To do that, send fio the **USR1** signal. You can
2991also get regularly timed dumps by using the :option:`--status-interval`
2992parameter, or by creating a file in :file:`/tmp` named
2993:file:`fio-dump-status`. If fio sees this file, it will unlink it and dump the
2994current output status.
8423bd11 2995
71bfa161 2996
f80dba8d
MT
2997Terse output
2998------------
71bfa161 2999
f80dba8d
MT
3000For scripted usage where you typically want to generate tables or graphs of the
3001results, fio can output the results in a semicolon separated format. The format
3002is one long line of values, such as::
71bfa161 3003
f80dba8d
MT
3004 2;card0;0;0;7139336;121836;60004;1;10109;27.932460;116.933948;220;126861;3495.446807;1085.368601;226;126864;3523.635629;1089.012448;24063;99944;50.275485%;59818.274627;5540.657370;7155060;122104;60004;1;8338;29.086342;117.839068;388;128077;5032.488518;1234.785715;391;128085;5061.839412;1236.909129;23436;100928;50.287926%;59964.832030;5644.844189;14.595833%;19.394167%;123706;0;7313;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.02%;0.05%;0.16%;6.04%;40.40%;52.68%;0.64%;0.01%;0.00%;0.01%;0.00%;0.00%;0.00%;0.00%;0.00%
3005 A description of this job goes here.
562c2d2f
DN
3006
3007The job description (if provided) follows on a second line.
71bfa161 3008
f80dba8d
MT
3009To enable terse output, use the :option:`--minimal` command line option. The
3010first value is the version of the terse output format. If the output has to be
3011changed for some reason, this number will be incremented by 1 to signify that
3012change.
6820cb3b 3013
71bfa161
JA
3014Split up, the format is as follows:
3015
f80dba8d
MT
3016 ::
3017
3018 terse version, fio version, jobname, groupid, error
3019
3020 READ status::
3021
3022 Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
3023 Submission latency: min, max, mean, stdev (usec)
3024 Completion latency: min, max, mean, stdev (usec)
3025 Completion latency percentiles: 20 fields (see below)
3026 Total latency: min, max, mean, stdev (usec)
3027 Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev
3028
3029 WRITE status:
3030
3031 ::
3032
3033 Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
3034 Submission latency: min, max, mean, stdev (usec)
3035 Completion latency: min, max, mean, stdev(usec)
3036 Completion latency percentiles: 20 fields (see below)
3037 Total latency: min, max, mean, stdev (usec)
3038 Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev
3039
3040 CPU usage::
3041
3042 user, system, context switches, major faults, minor faults
3043
3044 I/O depths::
3045
3046 <=1, 2, 4, 8, 16, 32, >=64
3047
3048 I/O latencies microseconds::
3049
3050 <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000
3051
3052 I/O latencies milliseconds::
3053
3054 <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000
3055
3056 Disk utilization::
3057
3058 Disk name, Read ios, write ios,
3059 Read merges, write merges,
3060 Read ticks, write ticks,
3061 Time spent in queue, disk utilization percentage
3062
3063 Additional Info (dependent on continue_on_error, default off)::
3064
3065 total # errors, first error code
3066
3067 Additional Info (dependent on description being set)::
3068
3069 Text description
3070
3071Completion latency percentiles can be a grouping of up to 20 sets, so for the
3072terse output fio writes all of them. Each field will look like this::
1db92cb6
JA
3073
3074 1.00%=6112
3075
f80dba8d 3076which is the Xth percentile, and the `usec` latency associated with it.
1db92cb6 3077
f80dba8d
MT
3078For disk utilization, all disks used by fio are shown. So for each disk there
3079will be a disk utilization section.
f2f788dd 3080
25c8b9d7 3081
f80dba8d
MT
3082Trace file format
3083-----------------
3084
3085There are two trace file format that you can encounter. The older (v1) format is
3086unsupported since version 1.20-rc3 (March 2008). It will still be described
25c8b9d7
PD
3087below in case that you get an old trace and want to understand it.
3088
3089In any case the trace is a simple text file with a single action per line.
3090
3091
f80dba8d
MT
3092Trace file format v1
3093~~~~~~~~~~~~~~~~~~~~
3094
3095Each line represents a single I/O action in the following format::
3096
3097 rw, offset, length
25c8b9d7 3098
f80dba8d 3099where `rw=0/1` for read/write, and the offset and length entries being in bytes.
25c8b9d7 3100
f80dba8d 3101This format is not supported in fio versions => 1.20-rc3.
25c8b9d7 3102
25c8b9d7 3103
f80dba8d
MT
3104Trace file format v2
3105~~~~~~~~~~~~~~~~~~~~
25c8b9d7 3106
f80dba8d
MT
3107The second version of the trace file format was added in fio version 1.17. It
3108allows to access more then one file per trace and has a bigger set of possible
3109file actions.
25c8b9d7 3110
f80dba8d 3111The first line of the trace file has to be::
25c8b9d7 3112
f80dba8d 3113 fio version 2 iolog
25c8b9d7
PD
3114
3115Following this can be lines in two different formats, which are described below.
3116
f80dba8d 3117The file management format::
25c8b9d7 3118
f80dba8d 3119 filename action
25c8b9d7
PD
3120
3121The filename is given as an absolute path. The action can be one of these:
3122
f80dba8d
MT
3123**add**
3124 Add the given filename to the trace.
3125**open**
3126 Open the file with the given filename. The filename has to have
3127 been added with the **add** action before.
3128**close**
3129 Close the file with the given filename. The file has to have been
3130 opened before.
3131
3132
3133The file I/O action format::
3134
3135 filename action offset length
3136
3137The `filename` is given as an absolute path, and has to have been added and
3138opened before it can be used with this format. The `offset` and `length` are
3139given in bytes. The `action` can be one of these:
3140
3141**wait**
3142 Wait for `offset` microseconds. Everything below 100 is discarded.
3143 The time is relative to the previous `wait` statement.
3144**read**
3145 Read `length` bytes beginning from `offset`.
3146**write**
3147 Write `length` bytes beginning from `offset`.
3148**sync**
3149 :manpage:`fsync(2)` the file.
3150**datasync**
3151 :manpage:`fdatasync(2)` the file.
3152**trim**
3153 Trim the given file from the given `offset` for `length` bytes.
3154
3155CPU idleness profiling
3156----------------------
3157
3158In some cases, we want to understand CPU overhead in a test. For example, we
3159test patches for the specific goodness of whether they reduce CPU usage.
3160Fio implements a balloon approach to create a thread per CPU that runs at idle
3161priority, meaning that it only runs when nobody else needs the cpu.
3162By measuring the amount of work completed by the thread, idleness of each CPU
3163can be derived accordingly.
3164
3165An unit work is defined as touching a full page of unsigned characters. Mean and
3166standard deviation of time to complete an unit work is reported in "unit work"
3167section. Options can be chosen to report detailed percpu idleness or overall
3168system idleness by aggregating percpu stats.
3169
3170
3171Verification and triggers
3172-------------------------
3173
3174Fio is usually run in one of two ways, when data verification is done. The first
3175is a normal write job of some sort with verify enabled. When the write phase has
3176completed, fio switches to reads and verifies everything it wrote. The second
3177model is running just the write phase, and then later on running the same job
3178(but with reads instead of writes) to repeat the same I/O patterns and verify
3179the contents. Both of these methods depend on the write phase being completed,
3180as fio otherwise has no idea how much data was written.
3181
3182With verification triggers, fio supports dumping the current write state to
3183local files. Then a subsequent read verify workload can load this state and know
3184exactly where to stop. This is useful for testing cases where power is cut to a
3185server in a managed fashion, for instance.
99b9a85a
JA
3186
3187A verification trigger consists of two things:
3188
f80dba8d
MT
31891) Storing the write state of each job.
31902) Executing a trigger command.
99b9a85a 3191
f80dba8d
MT
3192The write state is relatively small, on the order of hundreds of bytes to single
3193kilobytes. It contains information on the number of completions done, the last X
3194completions, etc.
99b9a85a 3195
f80dba8d
MT
3196A trigger is invoked either through creation ('touch') of a specified file in
3197the system, or through a timeout setting. If fio is run with
3198:option:`--trigger-file` = :file:`/tmp/trigger-file`, then it will continually
3199check for the existence of :file:`/tmp/trigger-file`. When it sees this file, it
3200will fire off the trigger (thus saving state, and executing the trigger
99b9a85a
JA
3201command).
3202
f80dba8d
MT
3203For client/server runs, there's both a local and remote trigger. If fio is
3204running as a server backend, it will send the job states back to the client for
3205safe storage, then execute the remote trigger, if specified. If a local trigger
3206is specified, the server will still send back the write state, but the client
3207will then execute the trigger.
99b9a85a 3208
f80dba8d
MT
3209Verification trigger example
3210~~~~~~~~~~~~~~~~~~~~~~~~~~~~
99b9a85a 3211
f80dba8d
MT
3212Lets say we want to run a powercut test on the remote machine 'server'. Our
3213write workload is in :file:`write-test.fio`. We want to cut power to 'server' at
3214some point during the run, and we'll run this test from the safety or our local
3215machine, 'localbox'. On the server, we'll start the fio backend normally::
99b9a85a 3216
f80dba8d 3217 server# fio --server
99b9a85a 3218
f80dba8d 3219and on the client, we'll fire off the workload::
99b9a85a 3220
f80dba8d 3221 localbox$ fio --client=server --trigger-file=/tmp/my-trigger --trigger-remote="bash -c \"echo b > /proc/sysrq-triger\""
99b9a85a 3222
f80dba8d 3223We set :file:`/tmp/my-trigger` as the trigger file, and we tell fio to execute::
99b9a85a 3224
f80dba8d 3225 echo b > /proc/sysrq-trigger
99b9a85a 3226
f80dba8d
MT
3227on the server once it has received the trigger and sent us the write state. This
3228will work, but it's not **really** cutting power to the server, it's merely
3229abruptly rebooting it. If we have a remote way of cutting power to the server
3230through IPMI or similar, we could do that through a local trigger command
3231instead. Lets assume we have a script that does IPMI reboot of a given hostname,
3232ipmi-reboot. On localbox, we could then have run fio with a local trigger
3233instead::
99b9a85a 3234
f80dba8d 3235 localbox$ fio --client=server --trigger-file=/tmp/my-trigger --trigger="ipmi-reboot server"
99b9a85a 3236
f80dba8d
MT
3237For this case, fio would wait for the server to send us the write state, then
3238execute ``ipmi-reboot server`` when that happened.
3239
3240Loading verify state
3241~~~~~~~~~~~~~~~~~~~~
3242
3243To load store write state, read verification job file must contain the
3244:option:`verify_state_load` option. If that is set, fio will load the previously
99b9a85a 3245stored state. For a local fio run this is done by loading the files directly,
f80dba8d
MT
3246and on a client/server run, the server backend will ask the client to send the
3247files over and load them from there.
a3ae5b05
JA
3248
3249
f80dba8d
MT
3250Log File Formats
3251----------------
a3ae5b05
JA
3252
3253Fio supports a variety of log file formats, for logging latencies, bandwidth,
3254and IOPS. The logs share a common format, which looks like this:
3255
f80dba8d 3256 *time* (`msec`), *value*, *data direction*, *offset*
a3ae5b05 3257
f80dba8d 3258Time for the log entry is always in milliseconds. The *value* logged depends
a3ae5b05
JA
3259on the type of log, it will be one of the following:
3260
f80dba8d
MT
3261 **Latency log**
3262 Value is latency in usecs
3263 **Bandwidth log**
3264 Value is in KiB/sec
3265 **IOPS log**
3266 Value is IOPS
3267
3268*Data direction* is one of the following:
3269
3270 **0**
3271 I/O is a READ
3272 **1**
3273 I/O is a WRITE
3274 **2**
3275 I/O is a TRIM
3276
3277The *offset* is the offset, in bytes, from the start of the file, for that
3278particular I/O. The logging of the offset can be toggled with
3279:option:`log_offset`.
3280
3281If windowed logging is enabled through :option:`log_avg_msec` then fio doesn't
c60ebc45 3282log individual I/Os. Instead of logs the average values over the specified period
f80dba8d
MT
3283of time. Since 'data direction' and 'offset' are per-I/O values, they aren't
3284applicable if windowed logging is enabled. If windowed logging is enabled and
3285:option:`log_max_value` is set, then fio logs maximum values in that window
3286instead of averages.
3287
3288
3289Client/server
3290-------------
3291
3292Normally fio is invoked as a stand-alone application on the machine where the
3293I/O workload should be generated. However, the frontend and backend of fio can
3294be run separately. Ie the fio server can generate an I/O workload on the "Device
3295Under Test" while being controlled from another machine.
3296
3297Start the server on the machine which has access to the storage DUT::
3298
3299 fio --server=args
3300
3301where args defines what fio listens to. The arguments are of the form
3302``type,hostname`` or ``IP,port``. *type* is either ``ip`` (or ip4) for TCP/IP
3303v4, ``ip6`` for TCP/IP v6, or ``sock`` for a local unix domain socket.
3304*hostname* is either a hostname or IP address, and *port* is the port to listen
3305to (only valid for TCP/IP, not a local socket). Some examples:
3306
33071) ``fio --server``
3308
3309 Start a fio server, listening on all interfaces on the default port (8765).
3310
33112) ``fio --server=ip:hostname,4444``
3312
3313 Start a fio server, listening on IP belonging to hostname and on port 4444.
3314
33153) ``fio --server=ip6:::1,4444``
3316
3317 Start a fio server, listening on IPv6 localhost ::1 and on port 4444.
3318
33194) ``fio --server=,4444``
3320
3321 Start a fio server, listening on all interfaces on port 4444.
3322
33235) ``fio --server=1.2.3.4``
3324
3325 Start a fio server, listening on IP 1.2.3.4 on the default port.
3326
33276) ``fio --server=sock:/tmp/fio.sock``
3328
3329 Start a fio server, listening on the local socket /tmp/fio.sock.
3330
3331Once a server is running, a "client" can connect to the fio server with::
3332
3333 fio <local-args> --client=<server> <remote-args> <job file(s)>
3334
3335where `local-args` are arguments for the client where it is running, `server`
3336is the connect string, and `remote-args` and `job file(s)` are sent to the
3337server. The `server` string follows the same format as it does on the server
3338side, to allow IP/hostname/socket and port strings.
3339
3340Fio can connect to multiple servers this way::
3341
3342 fio --client=<server1> <job file(s)> --client=<server2> <job file(s)>
3343
3344If the job file is located on the fio server, then you can tell the server to
3345load a local file as well. This is done by using :option:`--remote-config` ::
3346
3347 fio --client=server --remote-config /path/to/file.fio
3348
3349Then fio will open this local (to the server) job file instead of being passed
3350one from the client.
3351
3352If you have many servers (example: 100 VMs/containers), you can input a pathname
3353of a file containing host IPs/names as the parameter value for the
3354:option:`--client` option. For example, here is an example :file:`host.list`
3355file containing 2 hostnames::
3356
3357 host1.your.dns.domain
3358 host2.your.dns.domain
3359
3360The fio command would then be::
a3ae5b05 3361
f80dba8d 3362 fio --client=host.list <job file(s)>
a3ae5b05 3363
f80dba8d
MT
3364In this mode, you cannot input server-specific parameters or job files -- all
3365servers receive the same job file.
a3ae5b05 3366
f80dba8d
MT
3367In order to let ``fio --client`` runs use a shared filesystem from multiple
3368hosts, ``fio --client`` now prepends the IP address of the server to the
3369filename. For example, if fio is using directory :file:`/mnt/nfs/fio` and is
3370writing filename :file:`fileio.tmp`, with a :option:`--client` `hostfile`
3371containing two hostnames ``h1`` and ``h2`` with IP addresses 192.168.10.120 and
3372192.168.10.121, then fio will create two files::
a3ae5b05 3373
f80dba8d
MT
3374 /mnt/nfs/fio/192.168.10.120.fileio.tmp
3375 /mnt/nfs/fio/192.168.10.121.fileio.tmp