.TP
.BI \-\-eta \fR=\fPwhen
Specifies when real\-time ETA estimate should be printed. \fIwhen\fR may
-be `always', `never' or `auto'.
+be `always', `never' or `auto'. `auto' is the default, it prints ETA when
+requested if the output is a TTY. `always' disregards the output type, and
+prints ETA when requested. `never' never prints ETA.
+.TP
+.BI \-\-eta\-interval \fR=\fPtime
+By default, fio requests client ETA status roughly every second. With this
+option, the interval is configurable. Fio imposes a minimum allowed time to
+avoid flooding the console, less than 250 msec is not supported.
.TP
.BI \-\-eta\-newline \fR=\fPtime
Force a new line for every \fItime\fR period passed. When the unit is omitted,
Tell fio to terminate processing after the specified period of time. It
can be quite hard to determine for how long a specified job will run, so
this parameter is handy to cap the total runtime to a given time. When
-the unit is omitted, the value is intepreted in seconds.
+the unit is omitted, the value is interpreted in seconds.
.TP
.BI time_based
If set, fio will run for the duration of the \fBruntime\fR specified
.RE
.TP
.BI fadvise_hint \fR=\fPstr
-Use \fBposix_fadvise\fR\|(2) to advise the kernel what I/O patterns
-are likely to be issued. Accepted values are:
+Use \fBposix_fadvise\fR\|(2) or \fBposix_madvise\fR\|(2) to advise the kernel
+what I/O patterns are likely to be issued. Accepted values are:
.RS
.RS
.TP
.TP
.B zoned
Zoned random distribution
+.B zoned_abs
+Zoned absolute random distribution
.RE
.P
When using a \fBzipf\fR or \fBpareto\fR distribution, an input value is also
random_distribution=zoned:60/10:30/20:8/30:2/40
.RE
.P
-similarly to how \fBbssplit\fR works for setting ranges and percentages
+A \fBzoned_abs\fR distribution works exactly like the\fBzoned\fR, except that
+it takes absolute sizes. For example, let's say you wanted to define access
+according to the following criteria:
+.RS
+.P
+.PD 0
+60% of accesses should be to the first 20G
+.P
+30% of accesses should be to the next 100G
+.P
+10% of accesses should be to the next 500G
+.PD
+.RE
+.P
+we can define an absolute zoning distribution with:
+.RS
+.P
+random_distribution=zoned:60/10:30/20:8/30:2/40
+.RE
+.P
+For both \fBzoned\fR and \fBzoned_abs\fR, fio supports defining up to 256
+separate zones.
+.P
+Similarly to how \fBbssplit\fR works for setting ranges and percentages
of block sizes. Like \fBbssplit\fR, it's possible to specify separate
zones for reads, writes, and trims. If just one set is given, it'll apply to
all of them.
90% 4k writes and 10% 8k writes, you would specify:
.RS
.P
-bssplit=2k/50:4k/50,4k/90,8k/10
+bssplit=2k/50:4k/50,4k/90:8k/10
.RE
+.P
+Fio supports defining up to 64 different weights for each data direction.
.RE
.TP
.BI blocksize_unaligned "\fR,\fB bs_unaligned"
blocks. Default: true.
.TP
.BI buffer_compress_percentage \fR=\fPint
-If this is set, then fio will attempt to provide I/O buffer content (on
-WRITEs) that compresses to the specified level. Fio does this by providing a
-mix of random data and a fixed pattern. The fixed pattern is either zeros,
-or the pattern specified by \fBbuffer_pattern\fR. If the pattern option
-is used, it might skew the compression ratio slightly. Note that this is per
-block size unit, for file/disk wide compression level that matches this
-setting, you'll also want to set \fBrefill_buffers\fR.
+If this is set, then fio will attempt to provide I/O buffer content
+(on WRITEs) that compresses to the specified level. Fio does this by
+providing a mix of random data followed by fixed pattern data. The
+fixed pattern is either zeros, or the pattern specified by
+\fBbuffer_pattern\fR. If the \fBbuffer_pattern\fR option is used, it
+might skew the compression ratio slightly. Setting
+\fBbuffer_compress_percentage\fR to a value other than 100 will also
+enable \fBrefill_buffers\fR in order to reduce the likelihood that
+adjacent blocks are so similar that they over compress when seen
+together. See \fBbuffer_compress_chunk\fR for how to set a finer or
+coarser granularity of the random/fixed data regions. Defaults to unset
+i.e., buffer data will not adhere to any compression level.
.TP
.BI buffer_compress_chunk \fR=\fPint
-See \fBbuffer_compress_percentage\fR. This setting allows fio to manage
-how big the ranges of random data and zeroed data is. Without this set, fio
-will provide \fBbuffer_compress_percentage\fR of blocksize random data,
-followed by the remaining zeroed. With this set to some chunk size smaller
-than the block size, fio can alternate random and zeroed data throughout the
-I/O buffer.
+This setting allows fio to manage how big the random/fixed data region
+is when using \fBbuffer_compress_percentage\fR. When
+\fBbuffer_compress_chunk\fR is set to some non-zero value smaller than the
+block size, fio can repeat the random/fixed region throughout the I/O
+buffer at the specified interval (which particularly useful when
+bigger block sizes are used for a job). When set to 0, fio will use a
+chunk size that matches the block size resulting in a single
+random/fixed region within the I/O buffer. Defaults to 512. When the
+unit is omitted, the value is interpreted in bytes.
.TP
.BI buffer_pattern \fR=\fPstr
If set, fio will fill the I/O buffers with this pattern or with the contents
writing. These buffers will be naturally dedupable. The contents of the
buffers depend on what other buffer compression settings have been set. It's
possible to have the individual buffers either fully compressible, or not at
-all. This option only controls the distribution of unique buffers.
+all \-\- this option only controls the distribution of unique buffers. Setting
+this option will also enable \fBrefill_buffers\fR to prevent every buffer
+being identical.
.TP
.BI invalidate \fR=\fPbool
Invalidate the buffer/page cache parts of the files to be used prior to
ioctl, or if the target is an sg character device we use
\fBread\fR\|(2) and \fBwrite\fR\|(2) for asynchronous
I/O. Requires \fBfilename\fR option to specify either block or
-character devices.
+character devices. The sg engine includes engine specific options.
.TP
.B null
Doesn't transfer any data, just pretends to. This is mainly used to
at least one non\-cpuio job.
.TP
.B guasi
-The GUASI I/O engine is the Generic Userspace Asyncronous Syscall
+The GUASI I/O engine is the Generic Userspace Asynchronous Syscall
Interface approach to async I/O. See \fIhttp://www.xmailserver.org/guasi\-lib.html\fR
for more info on GUASI.
.TP
.B rdma
The RDMA I/O engine supports both RDMA memory semantics
(RDMA_WRITE/RDMA_READ) and channel semantics (Send/Recv) for the
-InfiniBand, RoCE and iWARP protocols.
+InfiniBand, RoCE and iWARP protocols. This engine defines engine
+specific options.
.TP
.B falloc
I/O engine that does regular fallocate to simulate data transfer as
I/O engine that does regular EXT4_IOC_MOVE_EXT ioctls to simulate
defragment activity in request to DDIR_WRITE event.
.TP
+.B rados
+I/O engine supporting direct access to Ceph Reliable Autonomic Distributed
+Object Store (RADOS) via librados. This ioengine defines engine specific
+options.
+.TP
.B rbd
I/O engine supporting direct access to Ceph Rados Block Devices
(RBD) via librbd without the need to use the kernel rbd driver. This
.TP
.B pmemblk
Read and write using filesystem DAX to a file on a filesystem
-mounted with DAX on a persistent memory device through the NVML
+mounted with DAX on a persistent memory device through the PMDK
libpmemblk library.
.TP
.B dev\-dax
Read and write using device DAX to a persistent memory device (e.g.,
-/dev/dax0.0) through the NVML libpmem library.
+/dev/dax0.0) through the PMDK libpmem library.
.TP
.B external
Prefix to specify loading an external I/O engine object file. Append
Simply create the files and do no I/O to them. You still need to set
\fBfilesize\fR so that all the accounting still occurs, but no actual I/O will be
done other than creating the file.
+.TP
+.B libpmem
+Read and write using mmap I/O to a file on a filesystem
+mounted with DAX on a persistent memory device through the PMDK
+libpmem library.
.SS "I/O engine specific parameters"
In addition, there are some parameters which are only valid when a specific
\fBioengine\fR is in use. These are used identically to normal parameters,
this will be the starting port number since fio will use a range of
ports.
.TP
-.BI (netsplice,net)hostname \fR=\fPstr
-The hostname or IP address to use for TCP or UDP based I/O. If the job is
-a TCP listener or UDP reader, the hostname is not used and must be omitted
-unless it is a valid UDP multicast address.
+.BI (rdma)port
+The port to use for RDMA-CM communication. This should be the same
+value on the client and the server side.
+.TP
+.BI (netsplice,net, rdma)hostname \fR=\fPstr
+The hostname or IP address to use for TCP, UDP or RDMA-CM based I/O.
+If the job is a TCP listener or UDP reader, the hostname is not used
+and must be omitted unless it is a valid UDP multicast address.
.TP
.BI (netsplice,net)interface \fR=\fPstr
The IP address of the network interface used to send or receive UDP
.RE
.RE
.TP
-.BI (rbd)clustername \fR=\fPstr
+.BI (rbd,rados)clustername \fR=\fPstr
Specifies the name of the Ceph cluster.
.TP
.BI (rbd)rbdname \fR=\fPstr
Specifies the name of the RBD.
.TP
-.BI (rbd)pool \fR=\fPstr
-Specifies the name of the Ceph pool containing RBD.
+.BI (rbd,rados)pool \fR=\fPstr
+Specifies the name of the Ceph pool containing RBD or RADOS data.
.TP
-.BI (rbd)clientname \fR=\fPstr
+.BI (rbd,rados)clientname \fR=\fPstr
Specifies the username (without the 'client.' prefix) used to access the
Ceph cluster. If the \fBclustername\fR is specified, the \fBclientname\fR shall be
the full *type.id* string. If no type. prefix is given, fio will add 'client.'
by default.
.TP
+.BI (rbd,rados)busy_poll \fR=\fPbool
+Poll store instead of waiting for completion. Usually this provides better
+throughput at cost of higher(up to 100%) CPU utilization.
+.TP
.BI (mtd)skip_bad \fR=\fPbool
Skip operations against known bad blocks.
.TP
.TP
.BI (libhdfs)chunk_size
The size of the chunk to use for each file.
+.TP
+.BI (rdma)verb \fR=\fPstr
+The RDMA verb to use on this side of the RDMA ioengine
+connection. Valid values are write, read, send and recv. These
+correspond to the equivalent RDMA verbs (e.g. write = rdma_write
+etc.). Note that this only needs to be specified on the client side of
+the connection. See the examples folder.
+.TP
+.BI (rdma)bindname \fR=\fPstr
+The name to use to bind the local RDMA-CM connection to a local RDMA
+device. This could be a hostname or an IPv4 or IPv6 address. On the
+server side this will be passed into the rdma_bind_addr() function and
+on the client site it will be used in the rdma_resolve_add()
+function. This can be useful when multiple paths exist between the
+client and the server or in certain loopback configurations.
+.TP
+.BI (sg)readfua \fR=\fPbool
+With readfua option set to 1, read operations include the force
+unit access (fua) flag. Default: 0.
+.TP
+.BI (sg)writefua \fR=\fPbool
+With writefua option set to 1, write operations include the force
+unit access (fua) flag. Default: 0.
+.TP
+.BI (sg)sg_write_mode \fR=\fPstr
+Specify the type of write commands to issue. This option can take three
+values:
+.RS
+.RS
+.TP
+.B write (default)
+Write opcodes are issued as usual
+.TP
+.B verify
+Issue WRITE AND VERIFY commands. The BYTCHK bit is set to 0. This
+directs the device to carry out a medium verification with no data
+comparison. The writefua option is ignored with this selection.
+.TP
+.B same
+Issue WRITE SAME commands. This transfers a single block to the device
+and writes this same block of data to a contiguous sequence of LBAs
+beginning at the specified offset. fio's block size parameter
+specifies the amount of data written with each command. However, the
+amount of data actually transferred to the device is equal to the
+device's block (sector) size. For a device with 512 byte sectors,
+blocksize=8k will write 16 sectors with each command. fio will still
+generate 8k of data for each command butonly the first 512 bytes will
+be used and transferred to the device. The writefua option is ignored
+with this selection.
+
.SS "I/O depth"
.TP
.BI iodepth \fR=\fPint
flow, known as the Poisson process
(\fIhttps://en.wikipedia.org/wiki/Poisson_point_process\fR). The lambda will be
10^6 / IOPS for the given workload.
+.TP
+.BI rate_ignore_thinktime \fR=\fPbool
+By default, fio will attempt to catch up to the specified rate setting, if any
+kind of thinktime setting was used. If this option is set, then fio will
+ignore the thinktime and continue doing IO at the specified rate, instead of
+entering a catch-up mode after thinktime is done.
.SS "I/O latency"
.TP
.BI latency_target \fR=\fPtime
still respecting ordering. The result is the same I/O pattern to a given
device, but different timings.
.TP
+.BI replay_time_scale \fR=\fPint
+When replaying I/O with \fBread_iolog\fR, fio will honor the original timing
+in the trace. With this option, it's possible to scale the time. It's a
+percentage option, if set to 50 it means run at 50% the original IO rate in
+the trace. If set to 200, run at twice the original IO rate. Defaults to 100.
+.TP
.BI replay_redirect \fR=\fPstr
While replaying I/O patterns using \fBread_iolog\fR the default behavior
is to replay the IOPS onto the major/minor device that each IOP was recorded
Scale sector offsets down by this factor when replaying traces.
.SS "Threads, processes and job synchronization"
.TP
+.BI replay_skip \fR=\fPstr
+Sometimes it's useful to skip certain IO types in a replay trace. This could
+be, for instance, eliminating the writes in the trace. Or not replaying the
+trims/discards, if you are redirecting to a device that doesn't support them.
+This option takes a comma separated list of read, write, trim, sync.
+.TP
.BI thread
Fio defaults to creating jobs by using fork, however if this option is
given, fio will create jobs by using POSIX Threads' function
.BI prioclass \fR=\fPint
Set the I/O priority class. See man \fBionice\fR\|(1).
.TP
-.BI cpumask \fR=\fPint
-Set the CPU affinity of this job. The parameter given is a bit mask of
-allowed CPUs the job may run on. So if you want the allowed CPUs to be 1
-and 5, you would pass the decimal value of (1 << 1 | 1 << 5), or 34. See man
-\fBsched_setaffinity\fR\|(2). This may not work on all supported
-operating systems or kernel versions. This option doesn't work well for a
-higher CPU count than what you can store in an integer mask, so it can only
-control cpus 1\-32. For boxes with larger CPU counts, use
-\fBcpus_allowed\fR.
-.TP
.BI cpus_allowed \fR=\fPstr
Controls the same options as \fBcpumask\fR, but accepts a textual
-specification of the permitted CPUs instead. So to use CPUs 1 and 5 you
-would specify `cpus_allowed=1,5'. This option also allows a range of CPUs
-to be specified \-\- say you wanted a binding to CPUs 1, 5, and 8 to 15, you
-would set `cpus_allowed=1,5,8\-15'.
+specification of the permitted CPUs instead and CPUs are indexed from 0. So
+to use CPUs 0 and 5 you would specify `cpus_allowed=0,5'. This option also
+allows a range of CPUs to be specified \-\- say you wanted a binding to CPUs
+0, 5, and 8 to 15, you would set `cpus_allowed=0,5,8\-15'.
+.RS
+.P
+On Windows, when `cpus_allowed' is unset only CPUs from fio's current
+processor group will be used and affinity settings are inherited from the
+system. An fio build configured to target Windows 7 makes options that set
+CPUs processor group aware and values will set both the processor group
+and a CPU from within that group. For example, on a system where processor
+group 0 has 40 CPUs and processor group 1 has 32 CPUs, `cpus_allowed'
+values between 0 and 39 will bind CPUs from processor group 0 and
+`cpus_allowed' values between 40 and 71 will bind CPUs from processor
+group 1. When using `cpus_allowed_policy=shared' all CPUs specified by a
+single `cpus_allowed' option must be from the same processor group. For
+Windows fio builds not built for Windows 7, CPUs will only be selected from
+(and be relative to) whatever processor group fio happens to be running in
+and CPUs from other processor groups cannot be used.
+.RE
.TP
.BI cpus_allowed_policy \fR=\fPstr
Set the policy of how fio distributes the CPUs specified by
in the set.
.RE
.TP
+.BI cpumask \fR=\fPint
+Set the CPU affinity of this job. The parameter given is a bit mask of
+allowed CPUs the job may run on. So if you want the allowed CPUs to be 1
+and 5, you would pass the decimal value of (1 << 1 | 1 << 5), or 34. See man
+\fBsched_setaffinity\fR\|(2). This may not work on all supported
+operating systems or kernel versions. This option doesn't work well for a
+higher CPU count than what you can store in an integer mask, so it can only
+control cpus 1\-32. For boxes with larger CPU counts, use
+\fBcpus_allowed\fR.
+.TP
.BI numa_cpu_nodes \fR=\fPstr
Set this job running on specified NUMA nodes' CPUs. The arguments allow
comma delimited list of cpu numbers, A\-B ranges, or `all'. Note, to enable
<mode>[:<nodelist>]
.RE
.P
-`mode' is one of the following memory poicies: `default', `prefer',
+`mode' is one of the following memory policies: `default', `prefer',
`bind', `interleave' or `local'. For `default' and `local' memory
policies, no node needs to be specified. For `prefer', only one node is
allowed. For `bind' and `interleave' the `nodelist' may be as
each block. This will automatically use hardware acceleration
(e.g. SSE4.2 on an x86 or CRC crypto extensions on ARM64) but will
fall back to software crc32c if none is found. Generally the
-fatest checksum fio supports when hardware accelerated.
+fastest checksum fio supports when hardware accelerated.
.TP
.B crc32c\-intel
Synonym for crc32c.
the verify will be of the newly written data.
.RE
.TP
-.BI verifysort \fR=\fPbool
-If true, fio will sort written verify blocks when it deems it faster to read
-them back in a sorted manner. This is often the case when overwriting an
-existing file, since the blocks are already laid out in the file system. You
-can ignore this option unless doing huge amounts of really fast I/O where
-the red\-black tree sorting CPU time becomes significant. Default: true.
-.TP
-.BI verifysort_nr \fR=\fPint
-Pre\-load and sort verify blocks for a read workload.
-.TP
.BI verify_offset \fR=\fPint
Swap the verification header with data somewhere else in the block before
writing. It is swapped back before verifying.
.TP
.BI write_bw_log \fR=\fPstr
If given, write a bandwidth log for this job. Can be used to store data of
-the bandwidth of the jobs in their lifetime. The included
-\fBfio_generate_plots\fR script uses gnuplot to turn these
-text files into nice graphs. See \fBwrite_lat_log\fR for behavior of
-given filename. For this option, the postfix is `_bw.x.log', where `x'
-is the index of the job (1..N, where N is the number of jobs). If
-\fBper_job_logs\fR is false, then the filename will not include the job
-index. See \fBLOG FILE FORMATS\fR section.
-.TP
-.BI write_lat_log \fR=\fPstr
-Same as \fBwrite_bw_log\fR, except that this option stores I/O
-submission, completion, and total latencies instead. If no filename is given
-with this option, the default filename of `jobname_type.log' is
-used. Even if the filename is given, fio will still append the type of
-log. So if one specifies:
+the bandwidth of the jobs in their lifetime.
.RS
+.P
+If no str argument is given, the default filename of
+`jobname_type.x.log' is used. Even when the argument is given, fio
+will still append the type of log. So if one specifies:
.RS
.P
-write_lat_log=foo
+write_bw_log=foo
.RE
.P
-The actual log names will be `foo_slat.x.log', `foo_clat.x.log',
-and `foo_lat.x.log', where `x' is the index of the job (1..N, where N
-is the number of jobs). This helps \fBfio_generate_plots\fR find the
-logs automatically. If \fBper_job_logs\fR is false, then the filename
-will not include the job index. See \fBLOG FILE FORMATS\fR section.
+The actual log name will be `foo_bw.x.log' where `x' is the index
+of the job (1..N, where N is the number of jobs). If
+\fBper_job_logs\fR is false, then the filename will not include the
+`.x` job index.
+.P
+The included \fBfio_generate_plots\fR script uses gnuplot to turn these
+text files into nice graphs. See the \fBLOG FILE FORMATS\fR section for how data is
+structured within the file.
.RE
.TP
+.BI write_lat_log \fR=\fPstr
+Same as \fBwrite_bw_log\fR, except this option creates I/O
+submission (e.g., `name_slat.x.log'), completion (e.g.,
+`name_clat.x.log'), and total (e.g., `name_lat.x.log') latency
+files instead. See \fBwrite_bw_log\fR for details about the
+filename format and the \fBLOG FILE FORMATS\fR section for how data is structured
+within the files.
+.TP
.BI write_hist_log \fR=\fPstr
-Same as \fBwrite_lat_log\fR, but writes I/O completion latency
-histograms. If no filename is given with this option, the default filename
-of `jobname_clat_hist.x.log' is used, where `x' is the index of the
-job (1..N, where N is the number of jobs). Even if the filename is given,
-fio will still append the type of log. If \fBper_job_logs\fR is false,
-then the filename will not include the job index. See \fBLOG FILE FORMATS\fR section.
+Same as \fBwrite_bw_log\fR but writes an I/O completion latency
+histogram file (e.g., `name_hist.x.log') instead. Note that this
+file will be empty unless \fBlog_hist_msec\fR has also been set.
+See \fBwrite_bw_log\fR for details about the filename format and
+the \fBLOG FILE FORMATS\fR section for how data is structured
+within the file.
.TP
.BI write_iops_log \fR=\fPstr
-Same as \fBwrite_bw_log\fR, but writes IOPS. If no filename is given
-with this option, the default filename of `jobname_type.x.log' is
-used, where `x' is the index of the job (1..N, where N is the number of
-jobs). Even if the filename is given, fio will still append the type of
-log. If \fBper_job_logs\fR is false, then the filename will not include
-the job index. See \fBLOG FILE FORMATS\fR section.
+Same as \fBwrite_bw_log\fR, but writes an IOPS file (e.g.
+`name_iops.x.log') instead. See \fBwrite_bw_log\fR for
+details about the filename format and the \fBLOG FILE FORMATS\fR section for how data
+is structured within the file.
.TP
.BI log_avg_msec \fR=\fPint
By default, fio will log an entry in the iops, latency, or bw log for every
\fBlog_avg_msec\fR is inaccurate. Setting this option makes fio log
histogram entries over the specified period of time, reducing log sizes for
high IOPS devices while retaining percentile accuracy. See
-\fBlog_hist_coarseness\fR as well. Defaults to 0, meaning histogram
-logging is disabled.
+\fBlog_hist_coarseness\fR and \fBwrite_hist_log\fR as well.
+Defaults to 0, meaning histogram logging is disabled.
.TP
.BI log_hist_coarseness \fR=\fPint
Integer ranging from 0 to 6, defining the coarseness of the resolution of
.BI log_compression_cpus \fR=\fPstr
Define the set of CPUs that are allowed to handle online log compression for
the I/O jobs. This can provide better isolation between performance
-sensitive jobs, and background compression work.
+sensitive jobs, and background compression work. See \fBcpus_allowed\fR for
+the format used.
.TP
.BI log_store_compressed \fR=\fPbool
If set, fio will store the log files in a compressed format. They can be
`\-\-percentile_list=99.5:99.9' will cause fio to report the values of
completion latency below which 99.5% and 99.9% of the observed latencies
fell, respectively.
+.TP
+.BI significant_figures \fR=\fPint
+If using \fB\-\-output\-format\fR of `normal', set the significant figures
+to this value. Higher values will yield more precise IOPS and throughput
+units, while lower values will round. Requires a minimum value of 1 and a
+maximum value of 10. Defaults to 4.
.SS "Error handling"
.TP
.BI exitall_on_error
.B I
Thread initialized, waiting or generating necessary data.
.TP
-.B P
+.B p
Thread running pre\-reading file(s).
.TP
.B /
short or dropped.
.TP
.B IO latency
-These values are for \fBlatency-target\fR and related options. When
+These values are for \fBlatency_target\fR and related options. When
these options are engaged, this section describes the I/O depth required
to meet the specified latency target.
.RE
.RE
.SH AUTHORS
.B fio
-was written by Jens Axboe <jens.axboe@oracle.com>,
-now Jens Axboe <axboe@fb.com>.
+was written by Jens Axboe <axboe@kernel.dk>.
.br
This man page was written by Aaron Carroll <aaronc@cse.unsw.edu.au> based
on documentation by Jens Axboe.
projects / fio.git / blobdiff