7. Terse output
8. Trace file format
9. CPU idleness profiling
+10. Verification and triggers
+11. Log File Formats
+
1.0 Overview and history
------------------------
IO engine How do we issue io? We could be memory mapping the
file, we could be using regular read/write, we
- could be using splice, async io, syslet, or even
- SG (SCSI generic sg).
+ could be using splice, async io, or even SG
+ (SCSI generic sg).
IO depth If the io engine is async, how large a queuing
depth do we want to maintain?
'#', the entire line is discarded as a comment.
So let's look at a really simple job file that defines two processes, each
-randomly reading from a 128MB file.
+randomly reading from a 128MiB file.
; -- start job file --
[global]
Here we have no global section, as we only have one job defined anyway.
We want to use async io here, with a depth of 4 for each file. We also
-increased the buffer size used to 32KB and define numjobs to 4 to
+increased the buffer size used to 32KiB and define numjobs to 4 to
fork 4 identical jobs. The result is 4 processes each randomly writing
-to their own 64MB file. Instead of using the above job file, you could
+to their own 64MiB file. Instead of using the above job file, you could
have given the parameters on the command line. For this case, you would
specify:
-------------------------
fio also supports environment variable expansion in job files. Any
-substring of the form "${VARNAME}" as part of an option value (in other
+sub-string of the form "${VARNAME}" as part of an option value (in other
words, on the right of the `='), will be expanded to the value of the
environment variable called VARNAME. If no such environment variable
is defined, or VARNAME is the empty string, the empty string will be
This section describes in details each parameter associated with a job.
Some parameters take an option of a given type, such as an integer or
-a string. The following types are used:
+a string. Anywhere a numeric value is required, an arithmetic expression
+may be used, provided it is surrounded by parentheses. Supported operators
+are:
+
+ addition (+)
+ subtraction (-)
+ multiplication (*)
+ division (/)
+ modulus (%)
+ exponentiation (^)
+
+For time values in expressions, units are microseconds by default. This is
+different than for time values not in expressions (not enclosed in
+parentheses). The following types are used:
str String. This is a sequence of alpha characters.
time Integer with possible time suffix. In seconds unless otherwise
specified, use eg 10m for 10 minutes. Accepts s/m/h for seconds,
minutes, and hours, and accepts 'ms' (or 'msec') for milliseconds,
and 'us' (or 'usec') for microseconds.
-int SI integer. A whole number value, which may contain a suffix
- describing the base of the number. Accepted suffixes are k/m/g/t/p,
- meaning kilo, mega, giga, tera, and peta. The suffix is not case
- sensitive, and you may also include trailing 'b' (eg 'kb' is the same
- as 'k'). So if you want to specify 4096, you could either write
- out '4096' or just give 4k. The suffixes signify base 2 values, so
- 1024 is 1k and 1024k is 1m and so on, unless the suffix is explicitly
- set to a base 10 value using 'kib', 'mib', 'gib', etc. If that is the
- case, then 1000 is used as the multiplier. This can be handy for
- disks, since manufacturers generally use base 10 values when listing
- the capacity of a drive. If the option accepts an upper and lower
- range, use a colon ':' or minus '-' to separate such values. May also
- include a prefix to indicate numbers base. If 0x is used, the number
- is assumed to be hexadecimal. See irange.
+
+int Integer. A whole number value, which may contain an integer prefix
+ and an integer suffix.
+ [integer prefix]number[integer suffix]
+
+ The optional integer prefix specifies the number's base. The default
+ is decimal. 0x specifies hexadecimal.
+
+ The optional integer suffix specifies the number's units, and includes
+ an optional unit prefix and an optional unit. For quantities of data,
+ the default unit is bytes. For quantities of time, the default unit
+ is seconds.
+
+ With kb_base=1000, fio follows international standards for unit prefixes.
+ To specify power-of-10 decimal values defined in the International
+ System of Units (SI):
+ Ki means kilo (K) or 1000
+ Mi means mega (M) or 1000**2
+ Gi means giga (G) or 1000**3
+ Ti means tera (T) or 1000**4
+ Pi means peta (P) or 1000**5
+
+ To specify power-of-2 binary values defined in IEC 80000-13:
+ k means kibi (Ki) or 1024
+ M means mebi (Mi) or 1024**2
+ G means gibi (Gi) or 1024**3
+ T means tebi (Ti) or 1024**4
+ P means pebi (Pi) or 1024**5
+
+ With kb_base=1024 (the default), the unit prefixes are opposite from
+ those specified in the SI and IEC 80000-13 standards to provide
+ compatibility with old scripts. For example, 4k means 4096.
+
+ For quantities of data, an optional unit of 'B' may be included
+ (e.g., 'kB' is the same as 'k').
+
+ The integer suffix is not case sensitive (e.g., m/mi mean mebi/mega,
+ not milli). 'b' and 'B' both mean byte, not bit.
+
+ Examples with kb_base=1000:
+ 4 KiB: 4096, 4096b, 4096B, 4ki, 4kib, 4kiB, 4Ki, 4KiB
+ 1 MiB: 1048576, 1mi, 1024ki
+ 1 MB: 1000000, 1m, 1000k
+ 1 TiB: 1073741824, 1ti, 1024mi, 1048576ki
+ 1 TB: 1000000000, 1t, 1000m, 1000000k
+
+ Examples with kb_base=1024 (default):
+ 4 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
+ 1 MiB: 1048576, 1m, 1024k
+ 1 MB: 1000000, 1mi, 1000ki
+ 1 TiB: 1073741824, 1t, 1024m, 1048576k
+ 1 TB: 1000000000, 1ti, 1000mi, 1000000ki
+
+ To specify times (units are not case sensitive):
+ D means days
+ H means hours
+ M mean minutes
+ s or sec means seconds (default)
+ ms or msec means milliseconds
+ us or usec means microseconds
+
+ If the option accepts an upper and lower range, use a colon ':' or
+ minus '-' to separate such values. See irange.
+
bool Boolean. Usually parsed as an integer, however only defined for
true and false (1 and 0).
irange Integer range with suffix. Allows value range to be given, such
1k:4k. If the option allows two sets of ranges, they can be
specified with a ',' or '/' delimiter: 1k-4k/8k-32k. Also see
int.
-float_list A list of floating numbers, separated by a ':' character.
+float_list A list of floating point numbers, separated by a ':' character.
With the above in mind, here follows the complete list of fio job
parameters.
special purpose of also signaling the start of a new
job.
+wait_for=str Specifies the name of the already defined job to wait
+ for. Single waitee name only may be specified. If set, the job
+ won't be started until all workers of the waitee job are done.
+
+ Wait_for operates on the job name basis, so there are a few
+ limitations. First, the waitee must be defined prior to the
+ waiter job (meaning no forward references). Second, if a job
+ is being referenced as a waitee, it must have a unique name
+ (no duplicate waitees).
+
description=str Text description of the job. Doesn't do anything except
dump this text description when this job is run. It's
not parsed.
filename=str Fio normally makes up a filename based on the job name,
thread number, and file number. If you want to share
files between threads in a job or several jobs, specify
- a filename for each of them to override the default. If
- the ioengine used is 'net', the filename is the host, port,
- and protocol to use in the format of =host,port,protocol.
- See ioengine=net for more. If the ioengine is file based, you
- can specify a number of files by separating the names with a
- ':' colon. So if you wanted a job to open /dev/sda and /dev/sdb
- as the two working files, you would use
- filename=/dev/sda:/dev/sdb. On Windows, disk devices are
- accessed as \\.\PhysicalDrive0 for the first device,
- \\.\PhysicalDrive1 for the second etc. Note: Windows and
- FreeBSD prevent write access to areas of the disk containing
- in-use data (e.g. filesystems).
+ a filename for each of them to override the default.
+ If the ioengine is file based, you can specify a number of
+ files by separating the names with a ':' colon. So if you
+ wanted a job to open /dev/sda and /dev/sdb as the two working
+ files, you would use filename=/dev/sda:/dev/sdb. On Windows,
+ disk devices are accessed as \\.\PhysicalDrive0 for the first
+ device, \\.\PhysicalDrive1 for the second etc. Note: Windows
+ and FreeBSD prevent write access to areas of the disk
+ containing in-use data (e.g. filesystems).
If the wanted filename does need to include a colon, then
escape that with a '\' character. For instance, if the filename
is "/dev/dsk/foo@3,0:c", then you would use
default of $jobname.$jobnum.$filenum will be used if
no other format specifier is given.
+unique_filename=bool To avoid collisions between networked clients, fio
+ defaults to prefixing any generated filenames (with a directory
+ specified) with the source of the client connecting. To disable
+ this behavior, set this option to 0.
+
opendir=str Tell fio to recursively add any file it can find in this
directory and down the file system tree.
read Sequential reads
write Sequential writes
+ trim Sequential trims
randwrite Random writes
randread Random reads
+ randtrim Random trims
rw,readwrite Sequential mixed reads and writes
randrw Random mixed reads and writes
+ trimwrite Sequential trim+write sequences
+ Fio defaults to read if the option is not specified.
For the mixed io types, the default is to split them 50/50.
For certain types of io the result may still be skewed a bit,
since the speed may be different. It is possible to specify
the same offset 8 number of times before generating a new
offset.
-kb_base=int The base unit for a kilobyte. The defacto base is 2^10, 1024.
- Storage manufacturers like to use 10^3 or 1000 as a base
- ten unit instead, for obvious reasons. Allow values are
- 1024 or 1000, with 1024 being the default.
+kb_base=int Select the interpretation of unit prefixes in input parameters.
+ 1000 = Inputs comply with IEC 80000-13 and the International
+ System of Units (SI). Use:
+ - power-of-2 values with IEC prefixes (e.g., KiB)
+ - power-of-10 values with SI prefixes (e.g., kB)
+ 1024 = Compatibility mode (default). To avoid breaking
+ old scripts:
+ - power-of-2 values with SI prefixes
+ - power-of-10 values with IEC prefixes
+ See bs= for more details on input parameters.
+
+ Outputs always use correct prefixes. Most outputs include
+ both side-by-side, like:
+ bw=2383.3kB/s (2327.4KiB/s)
+ If only one value is reported, then kb_base selects the
+ one to use:
+ 1000 = SI prefixes
+ 1024 = IEC prefixes
unified_rw_reporting=bool Fio normally reports statistics on a per
- data direction basis, meaning that read, write, and trim are
+ data direction basis, meaning that reads, writes, and trims are
accounted and reported separately. If this option is set,
the fio will sum the results and report them as "mixed"
instead.
randrepeat=bool For random IO workloads, seed the generator in a predictable
way so that results are repeatable across repetitions.
+ Defaults to true.
randseed=int Seed the random number generators based on this seed value, to
be able to control what sequence of output is being generated.
If not set, the random sequence depends on the randrepeat
setting.
-use_os_rand=bool Fio can either use the random generator supplied by the OS
- to generator random offsets, or it can use it's own internal
- generator (based on Tausworthe). Default is to use the
- internal generator, which is often of better quality and
- faster.
-
fallocate=str Whether pre-allocation is performed when laying down files.
Accepted values are:
on what IO patterns it is likely to issue. Sometimes you
want to test specific IO patterns without telling the
kernel about it, in which case you can disable this option.
- If set, fio will use POSIX_FADV_SEQUENTIAL for sequential
- IO and POSIX_FADV_RANDOM for random IO.
+ The following options are supported:
+
+ sequential Use FADV_SEQUENTIAL
+ random Use FADV_RANDOM
+ 1 Backwards-compatible hint for basing
+ the hint on the fio workload. Will use
+ FADV_SEQUENTIAL for a sequential
+ workload, and FADV_RANDOM for a random
+ workload.
+ 0 Backwards-compatible setting for not
+ issing a fadvise hint.
+
+fadvise_stream=int Notify the kernel what write stream ID to place these
+ writes under. Only supported on Linux. Note, this option
+ may change going forward.
size=int The total size of file io for this job. Fio will run until
this many bytes has been transferred, unless runtime is
- limited by other options (such as 'runtime', for instance).
- Unless specific nrfiles and filesize options are given,
- fio will divide this size between the available files
- specified by the job. If not set, fio will use the full
- size of the given files or devices. If the files do not
- exist, size must be given. It is also possible to give
- size as a percentage between 1 and 100. If size=20% is
- given, fio will use 20% of the full size of the given
- files or devices.
-
+ limited by other options (such as 'runtime', for instance,
+ or increased/decreased by 'io_size'). Unless specific nrfiles
+ and filesize options are given, fio will divide this size
+ between the available files specified by the job. If not set,
+ fio will use the full size of the given files or devices.
+ If the files do not exist, size must be given. It is also
+ possible to give size as a percentage between 1 and 100. If
+ size=20% is given, fio will use 20% of the full size of the
+ given files or devices.
+
+io_size=int
io_limit=int Normally fio operates within the region set by 'size', which
means that the 'size' option sets both the region and size of
IO to be performed. Sometimes that is not what you want. With
this option, it is possible to define just the amount of IO
- that fio should do. For instance, if 'size' is set to 20G and
- 'io_limit' is set to 5G, fio will perform IO within the first
- 20G but exit when 5G have been done.
+ that fio should do. For instance, if 'size' is set to 20GiB and
+ 'io_size' is set to 5GiB, fio will perform IO within the first
+ 20GiB but exit when 5GiB have been done. The opposite is also
+ possible - if 'size' is set to 20GiB, and 'io_size' is set to
+ 40GiB, then fio will do 40GiB of IO within the 0..20GiB region.
filesize=int Individual file sizes. May be a range, in which case fio
will select sizes for files at random within the given range
Additionally, writing beyond end-of-device will not return
ENOSPC there.
-blocksize=int
-bs=int The block size used for the io units. Defaults to 4k. Values
- can be given for both read and writes. If a single int is
- given, it will apply to both. If a second int is specified
- after a comma, it will apply to writes only. In other words,
- the format is either bs=read_and_write or bs=read,write,trim.
- bs=4k,8k will thus use 4k blocks for reads, 8k blocks for
- writes, and 8k for trims. You can terminate the list with
- a trailing comma. bs=4k,8k, would use the default value for
- trims.. If you only wish to set the write size, you
- can do so by passing an empty read size - bs=,8k will set
- 8k for writes and leave the read default value.
-
-blockalign=int
-ba=int At what boundary to align random IO offsets. Defaults to
- the same as 'blocksize' the minimum blocksize given.
- Minimum alignment is typically 512b for using direct IO,
- though it usually depends on the hardware block size. This
- option is mutually exclusive with using a random map for
- files, so it will turn off that option.
-
-blocksize_range=irange
-bsrange=irange Instead of giving a single block size, specify a range
- and fio will mix the issued io block sizes. The issued
- io unit will always be a multiple of the minimum value
- given (also see bs_unaligned). Applies to both reads and
- writes, however a second range can be given after a comma.
- See bs=.
-
-bssplit=str Sometimes you want even finer grained control of the
+blocksize=int[,int][,int]
+bs=int[,int][,int]
+ The block size in bytes used for I/O units. Default: 4096.
+ A single value applies to reads, writes, and trims.
+ Comma-separated values may be specified for reads, writes,
+ and trims. A value not terminated in a comma applies to
+ subsequent types.
+
+ Examples:
+ bs=256k means 256k for reads, writes and trims
+ bs=8k,32k means 8k for reads, 32k for writes and trims
+ bs=8k,32k, means 8k for reads, 32k for writes, and
+ default for trims
+ bs=,8k means default for reads, 8k for writes and trims
+ bs=,8k, means default for reads, 8k for writes, and
+ default for writes
+
+blocksize_range=irange[,irange][,irange]
+bsrange=irange[,irange][,irange]
+ A range of block sizes in bytes for I/O units.
+ The issued I/O unit will always be a multiple of the minimum
+ size, unless blocksize_unaligned is set.
+
+ Comma-separated ranges may be specified for reads, writes,
+ and trims as described in 'blocksize'.
+
+ Example: bsrange=1k-4k,2k-8k
+
+bssplit=str[,str][,str]
+ Sometimes you want even finer grained control of the
block sizes issued, not just an even split between them.
This option allows you to weight various block sizes,
so that you are able to define a specific amount of
always add up to 100, if bssplit is given a range that adds
up to more, it will error out.
- bssplit also supports giving separate splits to reads and
- writes. The format is identical to what bs= accepts. You
- have to separate the read and write parts with a comma. So
- if you want a workload that has 50% 2k reads and 50% 4k reads,
+ Comma-separated values may be specified for reads, writes,
+ and trims as described in 'blocksize'.
+
+ If you want a workload that has 50% 2k reads and 50% 4k reads,
while having 90% 4k writes and 10% 8k writes, you would
specify:
bssplit=2k/50:4k/50,4k/90:8k/10
blocksize_unaligned
-bs_unaligned If this option is given, any byte size value within bsrange
- may be used as a block range. This typically wont work with
- direct IO, as that normally requires sector alignment.
+bs_unaligned If set, fio will issue I/O units with any size within
+ blocksize_range, not just multiples of the minimum size.
+ This typically won't work with direct I/O, as that normally
+ requires sector alignment.
bs_is_seq_rand If this option is set, fio will use the normal read,write
- blocksize settings as sequential,random instead. Any random
- read or write will use the WRITE blocksize settings, and any
- sequential read or write will use the READ blocksize setting.
+ blocksize settings as sequential,random blocksize settings
+ instead. Any random read or write will use the WRITE blocksize
+ settings, and any sequential read or write will use the READ
+ blocksize settings.
+
+blockalign=int[,int][,int]
+ba=int[,int][,int]
+ Boundary to which fio will align random I/O units.
+ Default: 'blocksize'.
+ Minimum alignment is typically 512b for using direct IO,
+ though it usually depends on the hardware block size. This
+ option is mutually exclusive with using a random map for
+ files, so it will turn off that option.
+ Comma-separated values may be specified for reads, writes,
+ and trims as described in 'blocksize'.
zero_buffers If this option is given, fio will init the IO buffers to
all zeroes. The default is to fill them with random data.
- The resulting IO buffers will not be completely zeroed,
- unless scramble_buffers is also turned off.
refill_buffers If this option is given, fio will refill the IO buffers
on every submit. The default is to only fill it at init
buffer_compress_percentage=int If this is set, then fio will attempt to
provide IO buffer content (on WRITEs) that compress to
the specified level. Fio does this by providing a mix of
- random data and zeroes. Note that this is per block size
- unit, for file/disk wide compression level that matches
- this setting, you'll also want to set refill_buffers.
+ random data and a fixed pattern. The fixed pattern is either
+ zeroes, or the pattern specified by buffer_pattern. If the
+ pattern option is used, it might skew the compression ratio
+ slightly. Note that this is per block size unit, for file/disk
+ wide compression level that matches this setting, you'll also
+ want to set refill_buffers.
buffer_compress_chunk=int See buffer_compress_percentage. This
setting allows fio to manage how big the ranges of random
the other options related to buffer contents. The setting can
be any pattern of bytes, and can be prefixed with 0x for hex
values. It may also be a string, where the string must then
- be wrapped with "".
+ be wrapped with "", e.g.:
+
+ buffer_pattern="abcd"
+ or
+ buffer_pattern=-12
+ or
+ buffer_pattern=0xdeadface
+
+ Also you can combine everything together in any order:
+ buffer_pattern=0xdeadface"abcd"-12
dedupe_percentage=int If set, fio will generate this percentage of
identical buffers when writing. These buffers will be
the next. Multiple files can still be
open depending on 'openfiles'.
- The string can have a number appended, indicating how
- often to switch to a new file. So if option random:4 is
- given, fio will switch to a new random file after 4 ios
- have been issued.
+ zipf Use a zipfian distribution to decide what file
+ to access.
+
+ pareto Use a pareto distribution to decide what file
+ to access.
+
+ gauss Use a gaussian (normal) distribution to decide
+ what file to access.
+
+ For random, roundrobin, and sequential, a postfix can be
+ appended to tell fio how many I/Os to issue before switching
+ to a new file. For example, specifying
+ 'file_service_type=random:8' would cause fio to issue 8 I/Os
+ before selecting a new file at random. For the non-uniform
+ distributions, a floating point postfix can be given to
+ influence how the distribution is skewed. See
+ 'random_distribution' for a description of how that would work.
ioengine=str Defines how the job issues io to the file. The following
types are defined:
sync Basic read(2) or write(2) io. lseek(2) is
used to position the io location.
- psync Basic pread(2) or pwrite(2) io.
+ psync Basic pread(2) or pwrite(2) io. Default on all
+ supported operating systems except for Windows.
vsync Basic readv(2) or writev(2) IO.
- psyncv Basic preadv(2) or pwritev(2) IO.
+ pvsync Basic preadv(2) or pwritev(2) IO.
+
+ pvsync2 Basic preadv2(2) or pwritev2(2) IO.
libaio Linux native asynchronous io. Note that Linux
may only support queued behaviour with
solarisaio Solaris native asynchronous io.
windowsaio Windows native asynchronous io.
+ Default on Windows.
mmap File is memory mapped and data copied
to/from using memcpy(3).
vmsplice(2) to transfer data from user
space to the kernel.
- syslet-rw Use the syslet system calls to make
- regular read/write async.
-
sg SCSI generic sg v3 io. May either be
synchronous using the SG_IO ioctl, or if
the target is an sg character device
cpuio Doesn't transfer any data, but burns CPU
cycles according to the cpuload= and
- cpucycle= options. Setting cpuload=85
+ cpuchunks= options. Setting cpuload=85
will cause that job to do nothing but burn
85% of the CPU. In case of SMP machines,
use numjobs=<no_of_cpu> to get desired CPU
usage, as the cpuload only loads a single
- CPU at the desired rate.
+ CPU at the desired rate. A job never finishes
+ unless there is at least one non-cpuio job.
guasi The GUASI IO engine is the Generic Userspace
Asyncronous Syscall Interface approach
defines engine specific options.
libhdfs Read and write through Hadoop (HDFS).
- The 'filename' option is used to specify host,
- port of the hdfs name-node to connect. This
- engine interprets offsets a little
+ This engine interprets offsets a little
differently. In HDFS, files once created
cannot be modified. So random writes are not
possible. To imitate this, libhdfs engine
- expects bunch of small files to be created
- over HDFS, and engine will randomly pick a
- file out of those files based on the offset
- generated by fio backend. (see the example
- job file to create such files, use rw=write
- option). Please note, you might want to set
- necessary environment variables to work with
- hdfs/libhdfs properly.
+ creates bunch of small files, and engine will
+ pick a file out of those files based on the
+ offset generated by fio backend. Each jobs uses
+ it's own connection to HDFS.
+
+ mtd Read, write and erase an MTD character device
+ (e.g., /dev/mtd0). Discards are treated as
+ erases. Depending on the underlying device
+ type, the I/O may have to go in a certain
+ pattern, e.g., on NAND, writing sequentially
+ to erase blocks and discarding before
+ overwriting. The writetrim mode works well
+ for this constraint.
+
+ pmemblk Read and write through the NVML libpmemblk
+ interface.
+
+ dev-dax Read and write through a DAX device exposed
+ from persistent memory.
external Prefix to specify loading an external
IO engine object file. Append the engine
filename, eg ioengine=external:/tmp/foo.o
to load ioengine foo.o in /tmp.
-iodepth=int This defines how many io units to keep in flight against
+iodepth=int This defines how many I/O units to keep in flight against
the file. The default is 1 for each file defined in this
job, can be overridden with a larger value for higher
concurrency. Note that increasing iodepth beyond 1 will not
iodepth_batch=int This defines how many pieces of IO to submit at once.
It defaults to 1 which means that we submit each IO
as soon as it is available, but can be raised to submit
- bigger batches of IO at the time.
+ bigger batches of IO at the time. If it is set to 0 the iodepth
+ value will be used.
+iodepth_batch_complete_min=int
iodepth_batch_complete=int This defines how many pieces of IO to retrieve
at once. It defaults to 1 which means that we'll ask
for a minimum of 1 IO in the retrieval process from
events before queuing more IO. This helps reduce
IO latency, at the cost of more retrieval system calls.
+iodepth_batch_complete_max=int This defines maximum pieces of IO to
+ retrieve at once. This variable should be used along with
+ iodepth_batch_complete_min=int variable, specifying the range
+ of min and max amount of IO which should be retrieved. By default
+ it is equal to iodepth_batch_complete_min value.
+
+ Example #1:
+
+ iodepth_batch_complete_min=1
+ iodepth_batch_complete_max=<iodepth>
+
+ which means that we will retrieve at least 1 IO and up to the
+ whole submitted queue depth. If none of IO has been completed
+ yet, we will wait.
+
+ Example #2:
+
+ iodepth_batch_complete_min=0
+ iodepth_batch_complete_max=<iodepth>
+
+ which means that we can retrieve up to the whole submitted
+ queue depth, but if none of IO has been completed yet, we will
+ NOT wait and immediately exit the system call. In this example
+ we simply do polling.
+
iodepth_low=int The low water mark indicating when to start filling
the queue again. Defaults to the same as iodepth, meaning
that fio will attempt to keep the queue full at all times.
after fio has filled the queue of 16 requests, it will let
the depth drain down to 4 before starting to fill it again.
+io_submit_mode=str This option controls how fio submits the IO to
+ the IO engine. The default is 'inline', which means that the
+ fio job threads submit and reap IO directly. If set to
+ 'offload', the job threads will offload IO submission to a
+ dedicated pool of IO threads. This requires some coordination
+ and thus has a bit of extra overhead, especially for lower
+ queue depth IO where it can increase latencies. The benefit
+ is that fio can manage submission rates independently of
+ the device completion rates. This avoids skewed latency
+ reporting if IO gets back up on the device side (the
+ coordinated omission problem).
+
direct=bool If value is true, use non-buffered io. This is usually
O_DIRECT. Note that ZFS on Solaris doesn't support direct io.
On Windows the synchronous ioengines don't support direct io.
fdatasync=int Like fsync= but uses fdatasync() to only sync data and not
metadata blocks.
- In FreeBSD and Windows there is no fdatasync(), this falls back to
- using fsync()
+ In FreeBSD and Windows there is no fdatasync(), this falls back
+ to using fsync()
sync_file_range=str:val Use sync_file_range() for every 'val' number of
write operations. Fio will track range of writes that
if fio is asked to limit reads or writes to a certain rate.
If that is the case, then the distribution may be skewed.
-random_distribution=str:float By default, fio will use a completely uniform
+random_distribution=str:float[,str:float][,str:float]
+ By default, fio will use a completely uniform
random distribution when asked to perform random IO. Sometimes
it is useful to skew the distribution in specific ways,
ensuring that some parts of the data is more hot than others.
random Uniform random distribution
zipf Zipf distribution
pareto Pareto distribution
+ gauss Normal (gaussian) distribution
+ zoned Zoned random distribution
When using a zipf or pareto distribution, an input value
is also needed to define the access pattern. For zipf, this
what the given input values will yield in terms of hit rates.
If you wanted to use zipf with a theta of 1.2, you would use
random_distribution=zipf:1.2 as the option. If a non-uniform
- model is used, fio will disable use of the random map.
+ model is used, fio will disable use of the random map. For
+ the gauss distribution, a normal deviation is supplied as
+ a value between 0 and 100.
+
+ For a zoned distribution, fio supports specifying percentages
+ of IO access that should fall within what range of the file or
+ device. For example, given a criteria of:
-percentage_random=int For a random workload, set how big a percentage should
+ 60% of accesses should be to the first 10%
+ 30% of accesses should be to the next 20%
+ 8% of accesses should be to to the next 30%
+ 2% of accesses should be to the next 40%
+
+ we can define that through zoning of the random accesses. For
+ the above example, the user would do:
+
+ random_distribution=zoned:60/10:30/20:8/30:2/40
+
+ similarly to how bssplit works for setting ranges and
+ percentages of block sizes. Like bssplit, it's possible to
+ specify separate zones for reads, writes, and trims. If just
+ one set is given, it'll apply to all of them.
+
+percentage_random=int[,int][,int]
+ For a random workload, set how big a percentage should
be random. This defaults to 100%, in which case the workload
is fully random. It can be set from anywhere from 0 to 100.
Setting it to 0 would make the workload fully sequential. Any
setting in between will result in a random mix of sequential
- and random IO, at the given percentages. It is possible to
- set different values for reads, writes, and trim. To do so,
- simply use a comma separated list. See blocksize.
-
+ and random IO, at the given percentages.
+ Comma-separated values may be specified for reads, writes,
+ and trims as described in 'blocksize'.
+
norandommap Normally fio will cover every block of the file when doing
random IO. If this option is given, fio will just get a
new random offset without looking at past io history. This
means that some blocks may not be read or written, and that
- some blocks may be read/written more than once. This option
- is mutually exclusive with verify= if and only if multiple
- blocksizes (via bsrange=) are used, since fio only tracks
- complete rewrites of blocks.
+ some blocks may be read/written more than once. If this option
+ is used with verify= and multiple blocksizes (via bsrange=),
+ only intact blocks are verified, i.e., partially-overwritten
+ blocks are ignored.
softrandommap=bool See norandommap. If fio runs with the random block map
enabled and it fails to allocate the map, if this option is
tausworthe Strong 2^88 cycle random number generator
lfsr Linear feedback shift register generator
+ tausworthe64 Strong 64-bit 2^258 cycle random number
+ generator
Tausworthe is a strong random number generator, but it
requires tracking on the side if we want to ensure that
typically good enough. LFSR only works with single
block sizes, not with workloads that use multiple block
sizes. If used with such a workload, fio may read or write
- some blocks multiple times.
+ some blocks multiple times. The default value is tausworthe,
+ unless the required space exceeds 2^32 blocks. If it does,
+ then tausworthe64 is selected automatically.
nice=int Run the job with the given nice value. See man nice(2).
+ On Windows, values less than -15 set the process class to "High";
+ -1 through -15 set "Above Normal"; 1 through 15 "Below Normal";
+ and above 15 "Idle" priority class.
+
prio=int Set the io priority value of this job. Linux limits us to
a positive value between 0 and 7, with 0 being the highest.
- See man ionice(1).
+ See man ionice(1). Refer to an appropriate manpage for
+ other operating systems since meaning of priority may differ.
prioclass=int Set the io priority class. See man ionice(1).
other words, this setting effectively caps the queue depth
if the latter is larger.
-rate=int Cap the bandwidth used by this job. The number is in bytes/sec,
- the normal suffix rules apply. You can use rate=500k to limit
- reads and writes to 500k each, or you can specify read and
- writes separately. Using rate=1m,500k would limit reads to
- 1MB/sec and writes to 500KB/sec. Capping only reads or
- writes can be done with rate=,500k or rate=500k,. The former
- will only limit writes (to 500KB/sec), the latter will only
- limit reads.
-
-ratemin=int Tell fio to do whatever it can to maintain at least this
- bandwidth. Failing to meet this requirement, will cause
- the job to exit. The same format as rate is used for
- read vs write separation.
-
-rate_iops=int Cap the bandwidth to this number of IOPS. Basically the same
+rate=int[,int][,int]
+ Cap the bandwidth used by this job. The number is in bytes/sec,
+ the normal suffix rules apply.
+ Comma-separated values may be specified for reads, writes,
+ and trims as described in 'blocksize'.
+
+rate_min=int[,int][,int]
+ Tell fio to do whatever it can to maintain at least this
+ bandwidth. Failing to meet this requirement will cause
+ the job to exit.
+ Comma-separated values may be specified for reads, writes,
+ and trims as described in 'blocksize'.
+
+rate_iops=int[,int][,int]
+ Cap the bandwidth to this number of IOPS. Basically the same
as rate, just specified independently of bandwidth. If the
job is given a block size range instead of a fixed value,
- the smallest block size is used as the metric. The same format
- as rate is used for read vs write separation.
-
-rate_iops_min=int If fio doesn't meet this rate of IO, it will cause
- the job to exit. The same format as rate is used for read vs
- write separation.
+ the smallest block size is used as the metric.
+ Comma-separated values may be specified for reads, writes,
+ and trims as described in 'blocksize'.
+
+rate_iops_min=int[,int][,int]
+ If fio doesn't meet this rate of IO, it will cause
+ the job to exit.
+ Comma-separated values may be specified for reads, writes,
+ and trims as described in 'blocksize'.
+
+rate_process=str This option controls how fio manages rated IO
+ submissions. The default is 'linear', which submits IO in a
+ linear fashion with fixed delays between IOs that gets
+ adjusted based on IO completion rates. If this is set to
+ 'poisson', fio will submit IO based on a more real world
+ random request flow, known as the Poisson process
+ (https://en.wikipedia.org/wiki/Poisson_process). The lambda
+ will be 10^6 / IOPS for the given workload.
latency_target=int If set, fio will attempt to find the max performance
point that the given workload will run at while maintaining a
max_latency=int If set, fio will exit the job if it exceeds this maximum
latency. It will exit with an ETIME error.
-ratecycle=int Average bandwidth for 'rate' and 'ratemin' over this number
+rate_cycle=int Average bandwidth for 'rate' and 'rate_min' over this number
of milliseconds.
cpumask=int Set the CPU affinity of this job. The parameter given is a
one cpu per job. If not enough CPUs are given for the jobs
listed, then fio will roundrobin the CPUs in the set.
-numa_cpu_nodes=str Set this job running on spcified NUMA nodes' CPUs. The
+numa_cpu_nodes=str Set this job running on specified NUMA nodes' CPUs. The
arguments allow comma delimited list of cpu numbers,
A-B ranges, or 'all'. Note, to enable numa options support,
fio must be built on a system with libnuma-dev(el) installed.
numa_mem_policy=str Set this job's memory policy and corresponding NUMA
- nodes. Format of the argements:
+ nodes. Format of the arguments:
<mode>[:<nodelist>]
`mode' is one of the following memory policy:
default, prefer, bind, interleave, local
thus it will increase the total runtime if a special timeout
or runtime is specified.
+steadystate=str:float
+ss=str:float Define the criterion and limit for assessing steady state
+ performance. The first parameter designates the criterion
+ whereas the second parameter sets the threshold. When the
+ criterion falls below the threshold for the specified duration,
+ the job will stop. For example, iops_slope:0.1% will direct fio
+ to terminate the job when the least squares regression slope
+ falls below 0.1% of the mean IOPS. If group_reporting is
+ enabled this will apply to all jobs in the group. Below is the
+ list of available steady state assessment criteria. All
+ assessments are carried out using only data from the rolling
+ collection window. Threshold limits can be expressed as a fixed
+ value or as a percentage of the mean in the collection window.
+ iops Collect IOPS data. Stop the job if all
+ individual IOPS measurements are within the
+ specified limit of the mean IOPS (e.g., iops:2
+ means that all individual IOPS values must be
+ within 2 of the mean, whereas iops:0.2% means
+ that all individual IOPS values must be within
+ 0.2% of the mean IOPS to terminate the job).
+ iops_slope
+ Collect IOPS data and calculate the least
+ squares regression slope. Stop the job if the
+ slope falls below the specified limit.
+ bw Collect bandwidth data. Stop the job if all
+ individual bandwidth measurements are within
+ the specified limit of the mean bandwidth.
+ bw_slope
+ Collect bandwidth data and calculate the least
+ squares regression slope. Stop the job if the
+ slope falls below the specified limit.
+
+steadystate_duration=time
+ss_dur=time A rolling window of this duration will be used to judge whether
+ steady state has been reached. Data will be collected once per
+ second. The default is 0 which disables steady state detection.
+
+steadystate_ramp_time=time
+ss_ramp=time Allow the job to run for the specified duration before
+ beginning data collection for checking the steady state job
+ termination criterion. The default is 0.
+
invalidate=bool Invalidate the buffer/page cache parts for this file prior
to starting io. Defaults to true.
io engines, this means using O_SYNC.
iomem=str
-mem=str Fio can use various types of memory as the io unit buffer.
+mem=str Fio can use various types of memory as the I/O unit buffer.
The allowed values are:
malloc Use memory from malloc(3) as the buffers.
+ Default memory type.
shm Use shared memory as the buffers. Allocated
through shmget(2).
backing. Append filename after mmaphuge, ala
mem=mmaphuge:/hugetlbfs/file
+ mmapshared Same as mmap, but use a MMAP_SHARED
+ mapping.
+
The area allocated is a function of the maximum allowed
bs size for the job, multiplied by the io depth given. Note
that for shmhuge and mmaphuge to work, the system must have
free huge pages allocated. This can normally be checked
and set by reading/writing /proc/sys/vm/nr_hugepages on a
- Linux system. Fio assumes a huge page is 4MB in size. So
+ Linux system. Fio assumes a huge page is 4MiB in size. So
to calculate the number of huge pages you need for a given
job file, add up the io depth of all jobs (normally one unless
iodepth= is used) and multiply by the maximum bs set. Then
location should point there. So if it's mounted in /huge,
you would use mem=mmaphuge:/huge/somefile.
-iomem_align=int This indiciates the memory alignment of the IO memory buffers.
- Note that the given alignment is applied to the first IO unit
+iomem_align=int This indicates the memory alignment of the IO memory buffers.
+ Note that the given alignment is applied to the first I/O unit
buffer, if using iodepth the alignment of the following buffers
are given by the bs used. In other words, if using a bs that is
a multiple of the page sized in the system, all buffers will
hugepage-size=int
Defines the size of a huge page. Must at least be equal
- to the system setting, see /proc/meminfo. Defaults to 4MB.
+ to the system setting, see /proc/meminfo. Defaults to 4MiB.
Should probably always be a multiple of megabytes, so using
hugepage-size=Xm is the preferred way to set this to avoid
setting a non-pow-2 bad value.
to wait for each job to finish, sometimes that is not the
desired action.
+exitall_on_error When one job finishes in error, terminate the rest. The
+ default is to wait for each job to finish.
+
bwavgtime=int Average the calculated bandwidth over the given time. Value
- is specified in milliseconds.
+ is specified in milliseconds. If the job also does bandwidth
+ logging through 'write_bw_log', then the minimum of this option
+ and 'log_avg_msec' will be used. Default: 500ms.
iopsavgtime=int Average the calculated IOPS over the given time. Value
- is specified in milliseconds.
+ is specified in milliseconds. If the job also does IOPS logging
+ through 'write_iops_log', then the minimum of this option and
+ 'log_avg_msec' will be used. Default: 500ms.
-create_serialize=bool If true, serialize the file creating for the jobs.
+create_serialize=bool If true, serialize the file creation for the jobs.
This may be handy to avoid interleaving of data
files, which may greatly depend on the filesystem
used and even the number of processors in the system.
that will be done. The actual job contents are not
executed.
+allow_file_create=bool If true, fio is permitted to create files as part
+ of its workload. This is the default behavior. If this
+ option is false, then fio will error out if the files it
+ needs to use don't already exist. Default: true.
+
+allow_mounted_write=bool If this isn't set, fio will abort jobs that
+ are destructive (eg that write) to what appears to be a
+ mounted device or partition. This should help catch creating
+ inadvertently destructive tests, not realizing that the test
+ will destroy data on the mounted file system. Default: false.
+
pre_read=bool If this is given, files will be pre-read into memory before
starting the given IO operation. This will also clear
the 'invalidate' flag, since it is pointless to pre-read
and then drop the cache. This will only work for IO engines
- that are seekable, since they allow you to read the same data
+ that are seek-able, since they allow you to read the same data
multiple times. Thus it will not work on eg network or splice
IO.
runs of that job would then waste time recreating the file
set again and again.
+unlink_each_loop=bool Unlink job files after each iteration or loop.
+
loops=int Run the specified number of iterations of this job. Used
to repeat the same workload a given number of times. Defaults
to 1.
verify is set. Defaults to 1.
verify=str If writing to a file, fio can verify the file contents
- after each iteration of the job. The allowed values are:
+ after each iteration of the job. Each verification method also implies
+ verification of special header, which is written to the beginning of
+ each block. This header also includes meta information, like offset
+ of the block, block number, timestamp when block was written, etc.
+ verify=str can be combined with verify_pattern=str option.
+ The allowed values are:
md5 Use an md5 sum of the data area and store
it in the header of each block.
crc32c Use a crc32c sum of the data area and store
it in the header of each block.
- crc32c-intel Use hardware assisted crc32c calcuation
+ crc32c-intel Use hardware assisted crc32c calculation
provided on SSE4.2 enabled processors. Falls
back to regular software crc32c, if not
supported by the system.
sha1 Use optimized sha1 as the checksum function.
- meta Write extra information about each io
- (timestamp, block number etc.). The block
- number is verified. The io sequence number is
- verified for workloads that write data.
- See also verify_pattern.
+ meta This option is deprecated, since now meta information is
+ included in generic verification header and meta verification
+ happens by default. For detailed information see the description
+ of the verify=str setting. This option is kept because of
+ compatibility's sake with old configurations. Do not use it.
+
+ pattern Verify a strict pattern. Normally fio includes
+ a header with some basic information and
+ checksumming, but if this option is set, only
+ the specific pattern set with 'verify_pattern'
+ is verified.
null Only pretend to verify. Useful for testing
internals with ioengine=null, not for much
buffer at the time(it can be either a decimal or a hex number).
The verify_pattern if larger than a 32-bit quantity has to
be a hex number that starts with either "0x" or "0X". Use
- with verify=meta.
+ with verify=str. Also, verify_pattern supports %o format,
+ which means that for each block offset will be written and
+ then verified back, e.g.:
+
+ verify_pattern=%o
+
+ Or use combination of everything:
+ verify_pattern=0xff%o"abcd"-12
verify_fatal=bool Normally fio will keep checking the entire contents
before quitting on a block verification failure. If this
if verify_backlog_batch is larger than verify_backlog, some
blocks will be verified more than once.
+verify_state_save=bool When a job exits during the write phase of a verify
+ workload, save its current state. This allows fio to replay
+ up until that point, if the verify state is loaded for the
+ verify read phase. The format of the filename is, roughly,
+ <type>-<jobname>-<jobindex>-verify.state. <type> is "local"
+ for a local run, "sock" for a client/server socket connection,
+ and "ip" (192.168.0.1, for instance) for a networked
+ client/server connection.
+
+verify_state_load=bool If a verify termination trigger was used, fio stores
+ the current write state of each thread. This can be used at
+ verification time so that fio knows how far it should verify.
+ Without this information, fio will run a full verification
+ pass, according to the settings in the job file used.
+
stonewall
wait_for_previous Wait for preceding jobs in the job file to exit, before
starting this one. Can be used to insert serialization
replay_no_stall=int When replaying I/O with read_iolog the default behavior
is to attempt to respect the time stamps within the log and
- replay them with the appropriate delay between IOPS. By
+ replay them with the appropriate delay between IOPS. By
setting this variable fio will not respect the timestamps and
attempt to replay them as fast as possible while still
- respecting ordering. The result is the same I/O pattern to a
+ respecting ordering. The result is the same I/O pattern to a
given device, but different timings.
replay_redirect=str While replaying I/O patterns using read_iolog the
mapping. Replay_redirect causes all IOPS to be replayed onto
the single specified device regardless of the device it was
recorded from. i.e. replay_redirect=/dev/sdc would cause all
- IO in the blktrace to be replayed onto /dev/sdc. This means
- multiple devices will be replayed onto a single, if the trace
- contains multiple devices. If you want multiple devices to be
- replayed concurrently to multiple redirected devices you must
- blkparse your trace into separate traces and replay them with
- independent fio invocations. Unfortuantely this also breaks
- the strict time ordering between multiple device accesses.
+ IO in the blktrace or iolog to be replayed onto /dev/sdc.
+ This means multiple devices will be replayed onto a single
+ device, if the trace contains multiple devices. If you want
+ multiple devices to be replayed concurrently to multiple
+ redirected devices you must blkparse your trace into separate
+ traces and replay them with independent fio invocations.
+ Unfortunately this also breaks the strict time ordering
+ between multiple device accesses.
+
+replay_align=int Force alignment of IO offsets and lengths in a trace
+ to this power of 2 value.
+
+replay_scale=int Scale sector offsets down by this factor when
+ replaying traces.
+
+per_job_logs=bool If set, this generates bw/clat/iops log with per
+ file private filenames. If not set, jobs with identical names
+ will share the log filename. Default: true.
write_bw_log=str If given, write a bandwidth log of the jobs in this job
file. Can be used to store data of the bandwidth of the
graphs. See write_lat_log for behaviour of given
filename. For this option, the suffix is _bw.x.log, where
x is the index of the job (1..N, where N is the number of
- jobs).
+ jobs). If 'per_job_logs' is false, then the filename will not
+ include the job index. See 'Log File Formats'.
write_lat_log=str Same as write_bw_log, except that this option stores io
submission, completion, and total latencies instead. If no
The actual log names will be foo_slat.x.log, foo_clat.x.log,
and foo_lat.x.log, where x is the index of the job (1..N,
where N is the number of jobs). This helps fio_generate_plot
- fine the logs automatically.
+ find the logs automatically. If 'per_job_logs' is false, then
+ the filename will not include the job index. See 'Log File
+ Formats'.
+
+write_hist_log=str Same as write_lat_log, but writes I/O completion
+ latency histograms. If no filename is given with this option, the
+ default filename of "jobname_clat_hist.x.log" is used, where x is
+ the index of the job (1..N, where N is the number of jobs). Even
+ if the filename is given, fio will still append the type of log.
+ If per_job_logs is false, then the filename will not include the
+ job index. See 'Log File Formats'.
write_iops_log=str Same as write_bw_log, but writes IOPS. If no filename is
given with this option, the default filename of
"jobname_type.x.log" is used,where x is the index of the job
(1..N, where N is the number of jobs). Even if the filename
- is given, fio will still append the type of log.
+ is given, fio will still append the type of log. If
+ 'per_job_logs' is false, then the filename will not include
+ the job index. See 'Log File Formats'.
log_avg_msec=int By default, fio will log an entry in the iops, latency,
or bw log for every IO that completes. When writing to the
disk log, that can quickly grow to a very large size. Setting
this option makes fio average the each log entry over the
specified period of time, reducing the resolution of the log.
- Defaults to 0.
+ See log_max_value as well. Defaults to 0, logging all entries.
+
+log_hist_msec=int Same as log_avg_msec, but logs entries for completion
+ latency histograms. Computing latency percentiles from averages of
+ intervals using log_avg_msec is innacurate. Setting this option makes
+ fio log histogram entries over the specified period of time, reducing
+ log sizes for high IOPS devices while retaining percentile accuracy.
+ See log_hist_coarseness as well. Defaults to 0, meaning histogram
+ logging is disabled.
+
+log_hist_coarseness=int Integer ranging from 0 to 6, defining the coarseness
+ of the resolution of the histogram logs enabled with log_hist_msec. For
+ each increment in coarseness, fio outputs half as many bins. Defaults to
+ 0, for which histogram logs contain 1216 latency bins. See
+ 'Log File Formats'.
+
+log_max_value=bool If log_avg_msec is set, fio logs the average over that
+ window. If you instead want to log the maximum value, set this
+ option to 1. Defaults to 0, meaning that averaged values are
+ logged.
log_offset=int If this is set, the iolog options will include the byte
offset for the IO entry as well as the other data values.
in the specified log file. This feature depends on the
availability of zlib.
-log_store_compressed=bool If set, and log_compression is also set,
- fio will store the log files in a compressed format. They
- can be decompressed with fio, using the --inflate-log
- command line parameter. The files will be stored with a
- .fz suffix.
+log_compression_cpus=str Define the set of CPUs that are allowed to
+ handle online log compression for the IO jobs. This can
+ provide better isolation between performance sensitive jobs,
+ and background compression work.
+
+log_store_compressed=bool If set, fio will store the log files in a
+ compressed format. They can be decompressed with fio, using
+ the --inflate-log command line parameter. The files will be
+ stored with a .fz suffix.
+
+log_unix_epoch=bool If set, fio will log Unix timestamps to the log
+ files produced by enabling write_type_log for each log type, instead
+ of the default zero-based timestamps.
+
+block_error_percentiles=bool If set, record errors in trim block-sized
+ units from writes and trims and output a histogram of
+ how many trims it took to get to errors, and what kind
+ of error was encountered.
lockmem=int Pin down the specified amount of memory with mlock(2). Can
potentially be used instead of removing memory or booting
completion latencies.
percentile_list=float_list Overwrite the default list of percentiles
- for completion latencies. Each number is a floating
- number in the range (0,100], and the maximum length of
- the list is 20. Use ':' to separate the numbers, and
- list the numbers in ascending order. For example,
- --percentile_list=99.5:99.9 will cause fio to report
- the values of completion latency below which 99.5% and
- 99.9% of the observed latencies fell, respectively.
+ for completion latencies and the block error histogram.
+ Each number is a floating number in the range (0,100],
+ and the maximum length of the list is 20. Use ':'
+ to separate the numbers, and list the numbers in ascending
+ order. For example, --percentile_list=99.5:99.9 will cause
+ fio to report the values of completion latency below which
+ 99.5% and 99.9% of the observed latencies fell, respectively.
clocksource=str Use the given clocksource as the base of timing. The
supported options are:
enabled when polling for a minimum of 0 events (eg when
iodepth_batch_complete=0).
-[cpu] cpuload=int Attempt to use the specified percentage of CPU cycles.
+[psyncv2] hipri Set RWF_HIPRI on IO, indicating to the kernel that
+ it's of higher priority than normal.
-[cpu] cpuchunks=int Split the load into cycles of the given time. In
+[cpuio] cpuload=int Attempt to use the specified percentage of CPU cycles.
+
+[cpuio] cpuchunks=int Split the load into cycles of the given time. In
microseconds.
-[cpu] exit_on_io_done=bool Detect when IO threads are done, then exit.
+[cpuio] exit_on_io_done=bool Detect when IO threads are done, then exit.
[netsplice] hostname=str
[net] hostname=str The host name or IP address to use for TCP or UDP based IO.
If the job is a TCP listener or UDP reader, the hostname is not
used and must be omitted unless it is a valid UDP multicast
address.
+[libhdfs] namenode=str The host name or IP address of a HDFS cluster namenode to contact.
[netsplice] port=int
-[net] port=int The TCP or UDP port to bind to or connect to.
+[net] port=int The TCP or UDP port to bind to or connect to. If this is used
+with numjobs to spawn multiple instances of the same job type, then this will
+be the starting port number since fio will use a range of ports.
+[libhdfs] port=int the listening port of the HFDS cluster namenode.
[netsplice] interface=str
[net] interface=str The IP address of the network interface used to send or
[net] listen For TCP network connections, tell fio to listen for incoming
connections rather than initiating an outgoing connection. The
hostname must be omitted if this option is used.
-[net] pingpong Normaly a network writer will just continue writing data, and
+
+[net] pingpong Normally a network writer will just continue writing data, and
a network reader will just consume packages. If pingpong=1
is set, a writer will send its normal payload to the reader,
then wait for the reader to send the same payload back. This
single reader when multiple readers are listening to the same
address.
+[net] window_size Set the desired socket buffer size for the connection.
+
+[net] mss Set the TCP maximum segment size (TCP_MAXSEG).
+
[e4defrag] donorname=str
File will be used as a block donor(swap extents between files)
[e4defrag] inplace=int
Configure donor file blocks allocation strategy
0(default): Preallocate donor's file on init
- 1 : allocate space immidietly inside defragment event,
+ 1 : allocate space immediately inside defragment event,
and free right after event
+[rbd] clustername=str Specifies the name of the Ceph cluster.
+[rbd] rbdname=str Specifies the name of the RBD.
+[rbd] pool=str Specifies the name of the Ceph pool containing RBD.
+[rbd] clientname=str Specifies the username (without the 'client.' prefix)
+ used to access the Ceph cluster. If the clustername is
+ specified, the clientname shall be the full type.id
+ string. If no type. prefix is given, fio will add
+ 'client.' by default.
+
+[mtd] skip_bad=bool Skip operations against known bad blocks.
+
+[libhdfs] hdfsdirectory libhdfs will create chunk in this HDFS directory
+[libhdfs] chunk_size the size of the chunk to use for each file.
6.0 Interpreting the output
fio spits out a lot of output. While running, fio will display the
status of the jobs created. An example of that would be:
-Threads: 1: [_r] [24.8% done] [ 13509/ 8334 kb/s] [eta 00h:01m:31s]
+Jobs: 1: [_r] [24.8% done] [r=20992KiB/s,w=24064KiB/s,t=0KiB/s] [r=82,w=94,t=0 iops] [eta 00h:01m:31s]
The characters inside the square brackets denote the current status of
each thread. The possible values (in typical life cycle order) are:
command line as is needed. For instance, if you have 10 readers and 10
writers running, the output would look like this:
-Jobs: 20 (f=20): [R(10),W(10)] [4.0% done] [2103MB/0KB/0KB /s] [538K/0/0 iops] [eta 57m:36s]
+Jobs: 20 (f=20): [R(10),W(10)] [4.0% done] [r=20992KiB/s,w=24064KiB/s,t=0KiB/s] [r=82,w=94,t=0 iops] [eta 57m:36s]
Fio will still maintain the ordering, though. So the above means that jobs
1..10 are readers, and 11..20 are writers.
direction, the output looks like:
Client1 (g=0): err= 0:
- write: io= 32MB, bw= 666KB/s, iops=89 , runt= 50320msec
+ write: io= 32MiB, bw= 666KiB/s, iops=89 , runt= 50320msec
slat (msec): min= 0, max= 136, avg= 0.03, stdev= 1.92
clat (msec): min= 0, max= 631, avg=48.50, stdev=86.82
- bw (KB/s) : min= 0, max= 1196, per=51.00%, avg=664.02, stdev=681.68
+ bw (KiB/s) : min= 0, max= 1196, per=51.00%, avg=664.02, stdev=681.68
cpu : usr=1.49%, sys=0.25%, ctx=7969, majf=0, minf=17
IO depths : 1=0.1%, 2=0.3%, 4=0.5%, 8=99.0%, 16=0.0%, 32=0.0%, >32=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
cpu= CPU usage. User and system time, along with the number
of context switches this thread went through, usage of
system and user time, and finally the number of major
- and minor page faults.
+ and minor page faults. The CPU utilization numbers are
+ averages for the jobs in that reporting group, while the
+ context and fault counters are summed.
IO depths= The distribution of io depths over the job life time. The
numbers are divided into powers of 2, so for example the
16= entries includes depths up to that value but higher
terse version, fio version, jobname, groupid, error
READ status:
- Total IO (KB), bandwidth (KB/sec), IOPS, runtime (msec)
- Submission latency: min, max, mean, deviation (usec)
- Completion latency: min, max, mean, deviation (usec)
+ Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
+ Submission latency: min, max, mean, stdev (usec)
+ Completion latency: min, max, mean, stdev (usec)
Completion latency percentiles: 20 fields (see below)
- Total latency: min, max, mean, deviation (usec)
- Bw (KB/s): min, max, aggregate percentage of total, mean, deviation
+ Total latency: min, max, mean, stdev (usec)
+ Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev
WRITE status:
- Total IO (KB), bandwidth (KB/sec), IOPS, runtime (msec)
- Submission latency: min, max, mean, deviation (usec)
- Completion latency: min, max, mean, deviation (usec)
+ Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
+ Submission latency: min, max, mean, stdev (usec)
+ Completion latency: min, max, mean, stdev(usec)
Completion latency percentiles: 20 fields (see below)
- Total latency: min, max, mean, deviation (usec)
- Bw (KB/s): min, max, aggregate percentage of total, mean, deviation
+ Total latency: min, max, mean, stdev (usec)
+ Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev
CPU usage: user, system, context switches, major faults, minor faults
IO depths: <=1, 2, 4, 8, 16, 32, >=64
IO latencies microseconds: <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000
bytes. The action can be one of these:
wait Wait for 'offset' microseconds. Everything below 100 is discarded.
+ The time is relative to the previous wait statement.
read Read 'length' bytes beginning from 'offset'
write Write 'length' bytes beginning from 'offset'
sync fsync() the file
and standard deviation of time to complete an unit work is reported in "unit
work" section. Options can be chosen to report detailed percpu idleness or
overall system idleness by aggregating percpu stats.
+
+
+10.0 Verification and triggers
+------------------------------
+Fio is usually run in one of two ways, when data verification is done. The
+first is a normal write job of some sort with verify enabled. When the
+write phase has completed, fio switches to reads and verifies everything
+it wrote. The second model is running just the write phase, and then later
+on running the same job (but with reads instead of writes) to repeat the
+same IO patterns and verify the contents. Both of these methods depend
+on the write phase being completed, as fio otherwise has no idea how much
+data was written.
+
+With verification triggers, fio supports dumping the current write state
+to local files. Then a subsequent read verify workload can load this state
+and know exactly where to stop. This is useful for testing cases where
+power is cut to a server in a managed fashion, for instance.
+
+A verification trigger consists of two things:
+
+1) Storing the write state of each job
+2) Executing a trigger command
+
+The write state is relatively small, on the order of hundreds of bytes
+to single kilobytes. It contains information on the number of completions
+done, the last X completions, etc.
+
+A trigger is invoked either through creation ('touch') of a specified
+file in the system, or through a timeout setting. If fio is run with
+--trigger-file=/tmp/trigger-file, then it will continually check for
+the existence of /tmp/trigger-file. When it sees this file, it will
+fire off the trigger (thus saving state, and executing the trigger
+command).
+
+For client/server runs, there's both a local and remote trigger. If
+fio is running as a server backend, it will send the job states back
+to the client for safe storage, then execute the remote trigger, if
+specified. If a local trigger is specified, the server will still send
+back the write state, but the client will then execute the trigger.
+
+10.1 Verification trigger example
+---------------------------------
+Lets say we want to run a powercut test on the remote machine 'server'.
+Our write workload is in write-test.fio. We want to cut power to 'server'
+at some point during the run, and we'll run this test from the safety
+or our local machine, 'localbox'. On the server, we'll start the fio
+backend normally:
+
+server# fio --server
+
+and on the client, we'll fire off the workload:
+
+localbox$ fio --client=server --trigger-file=/tmp/my-trigger --trigger-remote="bash -c \"echo b > /proc/sysrq-triger\""
+
+We set /tmp/my-trigger as the trigger file, and we tell fio to execute
+
+echo b > /proc/sysrq-trigger
+
+on the server once it has received the trigger and sent us the write
+state. This will work, but it's not _really_ cutting power to the server,
+it's merely abruptly rebooting it. If we have a remote way of cutting
+power to the server through IPMI or similar, we could do that through
+a local trigger command instead. Lets assume we have a script that does
+IPMI reboot of a given hostname, ipmi-reboot. On localbox, we could
+then have run fio with a local trigger instead:
+
+localbox$ fio --client=server --trigger-file=/tmp/my-trigger --trigger="ipmi-reboot server"
+
+For this case, fio would wait for the server to send us the write state,
+then execute 'ipmi-reboot server' when that happened.
+
+10.2 Loading verify state
+-------------------------
+To load store write state, read verification job file must contain
+the verify_state_load option. If that is set, fio will load the previously
+stored state. For a local fio run this is done by loading the files directly,
+and on a client/server run, the server backend will ask the client to send
+the files over and load them from there.
+
+
+11.0 Log File Formats
+---------------------
+
+Fio supports a variety of log file formats, for logging latencies, bandwidth,
+and IOPS. The logs share a common format, which looks like this:
+
+time (msec), value, data direction, offset
+
+Time for the log entry is always in milliseconds. The value logged depends
+on the type of log, it will be one of the following:
+
+ Latency log Value is latency in usecs
+ Bandwidth log Value is in KiB/sec
+ IOPS log Value is IOPS
+
+Data direction is one of the following:
+
+ 0 IO is a READ
+ 1 IO is a WRITE
+ 2 IO is a TRIM
+
+The offset is the offset, in bytes, from the start of the file, for that
+particular IO. The logging of the offset can be toggled with 'log_offset'.
+
+If windowed logging is enabled through 'log_avg_msec', then fio doesn't log
+individual IOs. Instead of logs the average values over the specified
+period of time. Since 'data direction' and 'offset' are per-IO values,
+they aren't applicable if windowed logging is enabled. If windowed logging
+is enabled and 'log_max_value' is set, then fio logs maximum values in
+that window instead of averages.
projects / fio.git / blobdiff