defined by \fIioengine\fR. If no \fIioengine\fR is given, list all
available ioengines.
.TP
-.BI \-\-showcmd \fR=\fPjobfile
-Convert \fIjobfile\fR to a set of command\-line options.
+.BI \-\-showcmd
+Convert given \fIjobfile\fRs to a set of command\-line options.
.TP
.BI \-\-readonly
Turn on safety read\-only checks, preventing writes and trims. The \fB\-\-readonly\fR
.RS
.P
.PD 0
-z means Zone
+z means Zone
.P
.PD
.RE
explicit size is specified by \fBfilesize\fR.
.RS
.P
-Each colon in the wanted path must be escaped with a '\\'
+Each colon in the wanted path must be escaped with a '\e'
character. For instance, if the path is `/dev/dsk/foo@3,0:c' then you
would use `filename=/dev/dsk/foo@3,0\\:c' and if the path is
`F:\\filename' then you would use `filename=F\\:\\filename'.
.BI max_open_zones \fR=\fPint
When running a random write test across an entire drive many more zones will be
open than in a typical application workload. Hence this command line option
-that allows to limit the number of open zones. The number of open zones is
+that allows one to limit the number of open zones. The number of open zones is
defined as the number of zones to which write commands are issued by all
threads/processes.
.TP
.TP
.B trimwrite
Sequential trim+write sequences. Blocks will be trimmed first,
-then the same blocks will be written to.
+then the same blocks will be written to. So if `io_size=64K' is specified,
+Fio will trim a total of 64K bytes and also write 64K bytes on the same
+trimmed blocks. This behaviour will be consistent with `number_ios' or
+other Fio options limiting the total bytes or number of I/O's.
+.TP
+.B randtrimwrite
+Like
+.B trimwrite ,
+but uses random offsets rather than sequential writes.
.RE
.P
Fio defaults to read if the option is not specified. For the mixed I/O
effectively caps the file size at `real_size \- offset'. Can be combined with
\fBsize\fR to constrain the start and end range of the I/O workload.
A percentage can be specified by a number between 1 and 100 followed by '%',
-for example, `offset=20%' to specify 20%. In ZBD mode, value can be set as
+for example, `offset=20%' to specify 20%. In ZBD mode, value can be set as
number of zones using 'z'.
.TP
.BI offset_align \fR=\fPint
intended to operate on a file in parallel disjoint segments, with even
spacing between the starting points. Percentages can be used for this option.
If a percentage is given, the generated offset will be aligned to the minimum
-\fBblocksize\fR or to the value of \fBoffset_align\fR if provided.In ZBD mode, value
+\fBblocksize\fR or to the value of \fBoffset_align\fR if provided.In ZBD mode, value
can be set as number of zones using 'z'.
.TP
.BI number_ios \fR=\fPint
supplied as a value between 0 and 100.
.P
The second, optional float is allowed for \fBpareto\fR, \fBzipf\fR and \fBnormal\fR
-distributions. It allows to set base of distribution in non-default place, giving
+distributions. It allows one to set base of distribution in non-default place, giving
more control over most probable outcome. This value is in range [0-1] which maps linearly to
range of possible random values.
Defaults are: random for \fBpareto\fR and \fBzipf\fR, and 0.5 for \fBnormal\fR.
per job is supported
.RE
.TP
+.BI dedupe_global \fR=\fPbool
+This controls whether the deduplication buffers will be shared amongst
+all jobs that have this option set. The buffers are spread evenly between
+participating jobs.
+.P
+.RS
+Note that \fBdedupe_mode\fR must be set to \fBworking_set\fR for this to work.
+Can be used in combination with compression
+.TP
.BI invalidate \fR=\fPbool
Invalidate the buffer/page cache parts of the files to be used prior to
starting I/O if the platform and file type support it. Defaults to true.
\fBmmaphuge\fR to work, the system must have free huge pages allocated. This
can normally be checked and set by reading/writing
`/proc/sys/vm/nr_hugepages' on a Linux system. Fio assumes a huge page
-is 4MiB in size. So to calculate the number of huge pages you need for a
-given job file, add up the I/O depth of all jobs (normally one unless
-\fBiodepth\fR is used) and multiply by the maximum bs set. Then divide
-that number by the huge page size. You can see the size of the huge pages in
-`/proc/meminfo'. If no huge pages are allocated by having a non-zero
+is 2 or 4MiB in size depending on the platform. So to calculate the number of
+huge pages you need for a given job file, add up the I/O depth of all jobs
+(normally one unless \fBiodepth\fR is used) and multiply by the maximum bs set.
+Then divide that number by the huge page size. You can see the size of the huge
+pages in `/proc/meminfo'. If no huge pages are allocated by having a non-zero
number in `nr_hugepages', using \fBmmaphuge\fR or \fBshmhuge\fR will fail. Also
see \fBhugepage\-size\fR.
.P
\fBbs\fR used.
.TP
.BI hugepage\-size \fR=\fPint
-Defines the size of a huge page. Must at least be equal to the system
-setting, see `/proc/meminfo'. Defaults to 4MiB. Should probably
-always be a multiple of megabytes, so using `hugepage\-size=Xm' is the
-preferred way to set this to avoid setting a non-pow-2 bad value.
+Defines the size of a huge page. Must at least be equal to the system setting,
+see `/proc/meminfo' and `/sys/kernel/mm/hugepages/'. Defaults to 2 or 4MiB
+depending on the platform. Should probably always be a multiple of megabytes,
+so using `hugepage\-size=Xm' is the preferred way to set this to avoid setting
+a non-pow-2 bad value.
.TP
.BI lockmem \fR=\fPint
Pin the specified amount of memory with \fBmlock\fR\|(2). Can be used to
.TP
.BI size \fR=\fPint[%|z]
The total size of file I/O for each thread of this job. Fio will run until
-this many bytes has been transferred, unless runtime is limited by other options
-(such as \fBruntime\fR, for instance, or increased/decreased by \fBio_size\fR).
+this many bytes has been transferred, unless runtime is altered by other means
+such as (1) \fBruntime\fR, (2) \fBio_size\fR, (3) \fBnumber_ios\fR, (4)
+gaps/holes while doing I/O's such as `rw=read:16K', or (5) sequential I/O
+reaching end of the file which is possible when \fBpercentage_random\fR is
+less than 100.
Fio will divide this size between the available files determined by options
such as \fBnrfiles\fR, \fBfilename\fR, unless \fBfilesize\fR is
specified by the job. If the result of division happens to be 0, the size is
files or devices. If the files do not exist, size must be given. It is also
possible to give size as a percentage between 1 and 100. If `size=20%' is
given, fio will use 20% of the full size of the given files or devices. In ZBD mode,
-size can be given in units of number of zones using 'z'. Can be combined with \fBoffset\fR to
+size can be given in units of number of zones using 'z'. Can be combined with \fBoffset\fR to
constrain the start and end range that I/O will be done within.
.TP
.BI io_size \fR=\fPint[%|z] "\fR,\fB io_limit" \fR=\fPint[%|z]
.BI filesize \fR=\fPirange(int)
Individual file sizes. May be a range, in which case fio will select sizes
for files at random within the given range. If not given, each created file
-is the same size. This option overrides \fBsize\fR in terms of file size,
+is the same size. This option overrides \fBsize\fR in terms of file size,
i.e. \fBsize\fR becomes merely the default for \fBio_size\fR (and
has no effect it all if \fBio_size\fR is set explicitly).
.TP
.B pvsync2
Basic \fBpreadv2\fR\|(2) or \fBpwritev2\fR\|(2) I/O.
.TP
+.B io_uring
+Fast Linux native asynchronous I/O. Supports async IO
+for both direct and buffered IO.
+This engine defines engine specific options.
+.TP
+.B io_uring_cmd
+Fast Linux native asynchronous I/O for passthrough commands.
+This engine defines engine specific options.
+.TP
.B libaio
Linux native asynchronous I/O. Note that Linux may only support
queued behavior with non-buffered I/O (set `direct=1' or
.TP
.B exec
Execute 3rd party tools. Could be used to perform monitoring during jobs runtime.
+.TP
+.B xnvme
+I/O engine using the xNVMe C API, for NVMe devices. The xnvme engine provides
+flexibility to access GNU/Linux Kernel NVMe driver via libaio, IOCTLs, io_uring,
+the SPDK NVMe driver, or your own custom NVMe driver. The xnvme engine includes
+engine specific options. (See \fIhttps://xnvme.io/\fR).
.SS "I/O engine specific parameters"
In addition, there are some parameters which are only valid when a specific
\fBioengine\fR is in use. These are used identically to normal parameters,
\fBcmdprio_percentage\fR option.
.RE
.TP
-.BI (io_uring)fixedbufs
+.BI (io_uring,io_uring_cmd)fixedbufs
If fio is asked to do direct IO, then Linux will map pages for each IO call, and
release them when IO is done. If this option is set, the pages are pre-mapped
before IO is started. This eliminates the need to map and release for each IO.
This is more efficient, and reduces the IO latency as well.
.TP
-.BI (io_uring)hipri
+.BI (io_uring,io_uring_cmd)nonvectored \fR=\fPint
+With this option, fio will use non-vectored read/write commands, where address
+must contain the address directly. Default is -1.
+.TP
+.BI (io_uring,io_uring_cmd)force_async
+Normal operation for io_uring is to try and issue an sqe as non-blocking first,
+and if that fails, execute it in an async manner. With this option set to N,
+then every N request fio will ask sqe to be issued in an async manner. Default
+is 0.
+.TP
+.BI (io_uring,io_uring_cmd,xnvme)hipri
If this option is set, fio will attempt to use polled IO completions. Normal IO
completions generate interrupts to signal the completion of IO, polled
completions do not. Hence they are require active reaping by the application.
The benefits are more efficient IO for high IOPS scenarios, and lower latencies
for low queue depth IO.
.TP
-.BI (io_uring)registerfiles
+.BI (io_uring,io_uring_cmd)registerfiles
With this option, fio registers the set of files being used with the kernel.
This avoids the overhead of managing file counts in the kernel, making the
submission and completion part more lightweight. Required for the below
sqthread_poll option.
.TP
-.BI (io_uring)sqthread_poll
+.BI (io_uring,io_uring_cmd,xnvme)sqthread_poll
Normally fio will submit IO by issuing a system call to notify the kernel of
available items in the SQ ring. If this option is set, the act of submitting IO
will be done by a polling thread in the kernel. This frees up cycles for fio, at
-the cost of using more CPU in the system.
+the cost of using more CPU in the system. As submission is just the time it
+takes to fill in the sqe entries and any syscall required to wake up the idle
+kernel thread, fio will not report submission latencies.
.TP
-.BI (io_uring)sqthread_poll_cpu
+.BI (io_uring,io_uring_cmd)sqthread_poll_cpu \fR=\fPint
When `sqthread_poll` is set, this option provides a way to define which CPU
should be used for the polling thread.
.TP
+.BI (io_uring_cmd)cmd_type \fR=\fPstr
+Specifies the type of uring passthrough command to be used. Supported
+value is nvme. Default is nvme.
+.TP
.BI (libaio)userspace_reap
Normally, with the libaio engine in use, fio will use the
\fBio_getevents\fR\|(3) system call to reap newly returned events. With
When hipri is set this determines the probability of a pvsync2 I/O being high
priority. The default is 100%.
.TP
-.BI (pvsync2,libaio,io_uring)nowait
+.BI (pvsync2,libaio,io_uring,io_uring_cmd)nowait \fR=\fPbool
By default if a request cannot be executed immediately (e.g. resource starvation,
waiting on locks) it is queued and the initiating process will be blocked until
the required resource becomes free.
the full *type.id* string. If no type. prefix is given, fio will add 'client.'
by default.
.TP
+.BI (rados)conf \fR=\fPstr
+Specifies the configuration path of ceph cluster, so conf file does not
+have to be /etc/ceph/ceph.conf.
+.TP
.BI (rbd,rados)busy_poll \fR=\fPbool
Poll store instead of waiting for completion. Usually this provides better
throughput at cost of higher(up to 100%) CPU utilization.
.BI (http)http_s3_keyid \fR=\fPstr
The S3 key/access id.
.TP
+.BI (http)http_s3_sse_customer_key \fR=\fPstr
+The encryption customer key in SSE server side.
+.TP
+.BI (http)http_s3_sse_customer_algorithm \fR=\fPstr
+The encryption customer algorithm in SSE server side. Default is \fBAES256\fR
+.TP
+.BI (http)http_s3_storage_class \fR=\fPstr
+Which storage class to access. User-customizable settings. Default is \fBSTANDARD\fR
+.TP
.BI (http)http_swift_auth_token \fR=\fPstr
The Swift auth token. See the example configuration file on how to
retrieve this.
Specify the label or UUID of the DAOS container to open.
.TP
.BI (dfs)chunk_size
-Specificy a different chunk size (in bytes) for the dfs file.
+Specify a different chunk size (in bytes) for the dfs file.
Use DAOS container's chunk size by default.
.TP
.BI (dfs)object_class
-Specificy a different object class for the dfs file.
+Specify a different object class for the dfs file.
Use DAOS container's object class by default.
.TP
.BI (nfs)nfs_url
.TP
.BI (exec)std_redirect\fR=\fbool
If set, stdout and stderr streams are redirected to files named from the job name. Default is true.
+.TP
+.BI (xnvme)xnvme_async\fR=\fPstr
+Select the xnvme async command interface. This can take these values.
+.RS
+.RS
+.TP
+.B emu
+This is default and use to emulate asynchronous I/O by using a single thread to
+create a queue pair on top of a synchronous I/O interface using the NVMe driver
+IOCTL.
+.TP
+.BI thrpool
+Emulate an asynchronous I/O interface with a pool of userspace threads on top
+of a synchronous I/O interface using the NVMe driver IOCTL. By default four
+threads are used.
+.TP
+.BI io_uring
+Linux native asynchronous I/O interface which supports both direct and buffered
+I/O.
+.TP
+.BI libaio
+Use Linux aio for Asynchronous I/O
+.TP
+.BI posix
+Use the posix asynchronous I/O interface to perform one or more I/O operations
+asynchronously.
+.TP
+.BI nil
+Do not transfer any data; just pretend to. This is mainly used for
+introspective performance evaluation.
+.RE
+.RE
+.TP
+.BI (xnvme)xnvme_sync\fR=\fPstr
+Select the xnvme synchronous command interface. This can take these values.
+.RS
+.RS
+.TP
+.B nvme
+This is default and uses Linux NVMe Driver ioctl() for synchronous I/O.
+.TP
+.BI psync
+This supports regular as well as vectored pread() and pwrite() commands.
+.TP
+.BI block
+This is the same as psync except that it also supports zone management
+commands using Linux block layer IOCTLs.
+.RE
+.RE
+.TP
+.BI (xnvme)xnvme_admin\fR=\fPstr
+Select the xnvme admin command interface. This can take these values.
+.RS
+.RS
+.TP
+.B nvme
+This is default and uses Linux NVMe Driver ioctl() for admin commands.
+.TP
+.BI block
+Use Linux Block Layer ioctl() and sysfs for admin commands.
+.RE
+.RE
+.TP
+.BI (xnvme)xnvme_dev_nsid\fR=\fPint
+xnvme namespace identifier for userspace NVMe driver such as SPDK.
+.TP
+.BI (xnvme)xnvme_iovec
+If this option is set, xnvme will use vectored read/write commands.
.SS "I/O depth"
.TP
.BI iodepth \fR=\fPint
.BI write_iolog \fR=\fPstr
Write the issued I/O patterns to the specified file. See
\fBread_iolog\fR. Specify a separate file for each job, otherwise the
-iologs will be interspersed and the file may be corrupt.
+iologs will be interspersed and the file may be corrupt. This file will be
+opened in append mode.
.TP
.BI read_iolog \fR=\fPstr
Open an iolog with the specified filename and replay the I/O patterns it
Trim this number of I/O blocks.
.TP
.BI experimental_verify \fR=\fPbool
-Enable experimental verification.
+Enable experimental verification. Standard verify records I/O metadata for
+later use during the verification phase. Experimental verify instead resets the
+file after the write phase and then replays I/Os for the verification phase.
.SS "Steady state"
.TP
.BI steadystate \fR=\fPstr:float "\fR,\fP ss" \fR=\fPstr:float
completed. If this option is used, there are two more stats that are
appended, the total error count and the first error. The error field given
in the stats is the first error that was hit during the run.
+.RS
+.P
+Note: a write error from the device may go unnoticed by fio when using buffered
+IO, as the write() (or similar) system call merely dirties the kernel pages,
+unless `sync' or `direct' is used. Device IO errors occur when the dirty data is
+actually written out to disk. If fully sync writes aren't desirable, `fsync' or
+`fdatasync' can be used as well. This is specific to writes, as reads are always
+synchronous.
+.RS
+.P
The allowed values are:
.RS
.RS
.TP
.B Trace file format v2
The second version of the trace file format was added in fio version 1.17. It
-allows to access more then one file per trace and has a bigger set of possible
+allows one to access more than one file per trace and has a bigger set of possible
file actions.
.RS
.P
.TP
.B wait
Wait for `offset' microseconds. Everything below 100 is discarded.
-The time is relative to the previous `wait' statement.
+The time is relative to the previous `wait' statement. Note that action `wait`
+is not allowed as of version 3, as the same behavior can be achieved using
+timestamps.
.TP
.B read
Read `length' bytes beginning from `offset'.
Trim the given file from the given `offset' for `length' bytes.
.RE
.RE
+.RE
+.TP
+.B Trace file format v3
+The third version of the trace file format was added in fio version 3.31. It
+forces each action to have a timestamp associated with it.
+.RS
+.P
+The first line of the trace file has to be:
+.RS
+.P
+"fio version 3 iolog"
+.RE
+.P
+Following this can be lines in two different formats, which are described below.
+.P
+.B
+The file management format:
+.RS
+timestamp filename action
+.P
+.RE
+.B
+The file I/O action format:
+.RS
+timestamp filename action offset length
+.P
+The `timestamp` is relative to the beginning of the run (ie starts at 0). The
+`filename`, `action`, `offset` and `length` are identical to version 2, except
+that version 3 does not allow the `wait` action.
+.RE
+.RE
.SH I/O REPLAY \- MERGING TRACES
Colocation is a common practice used to get the most out of a machine.
Knowing which workloads play nicely with each other and which ones don't is
projects / fio.git / blobdiff