Merge branch 'minor_fixes' of https://github.com/sitsofe/fio

[fio.git] / HOWTO

diff --git a/HOWTO b/HOWTO
index 92c3b7371219bfc2bccbf46836d5c9d41232eae1..fe5c3cb193e9172275e0df47730158309fd33e5b 100644 (file)
--- a/HOWTO
+++ b/HOWTO
@@ -54,56 +54,66 @@ Command line options
 
 .. option:: --debug=type
 
 
 .. option:: --debug=type
 

-    Enable verbose tracing of various fio actions.  May be ``all`` for all types
-    or individual types separated by a comma (e.g. ``--debug=file,mem`` will
-    enable file and memory debugging).  Currently, additional logging is
-    available for:
+       Enable verbose tracing of various fio actions.  May be ``all`` for all types
+       or individual types separated by a comma (e.g. ``--debug=file,mem`` will
+       enable file and memory debugging).  Currently, additional logging is
+       available for:

 
 

-    *process*
+       *process*

                        Dump info related to processes.
                        Dump info related to processes.

-    *file*
+       *file*

                        Dump info related to file actions.
                        Dump info related to file actions.

-    *io*
+       *io*

                        Dump info related to I/O queuing.
                        Dump info related to I/O queuing.

-    *mem*
+       *mem*

                        Dump info related to memory allocations.
                        Dump info related to memory allocations.

-    *blktrace*
+       *blktrace*

                        Dump info related to blktrace setup.
                        Dump info related to blktrace setup.

-    *verify*
+       *verify*

                        Dump info related to I/O verification.
                        Dump info related to I/O verification.

-    *all*
+       *all*

                        Enable all debug options.
                        Enable all debug options.

-    *random*
+       *random*

                        Dump info related to random offset generation.
                        Dump info related to random offset generation.

-    *parse*
+       *parse*

                        Dump info related to option matching and parsing.
                        Dump info related to option matching and parsing.

-    *diskutil*
+       *diskutil*

                        Dump info related to disk utilization updates.
                        Dump info related to disk utilization updates.

-    *job:x*
+       *job:x*

                        Dump info only related to job number x.
                        Dump info only related to job number x.

-    *mutex*
+       *mutex*

                        Dump info only related to mutex up/down ops.
                        Dump info only related to mutex up/down ops.

-    *profile*
+       *profile*

                        Dump info related to profile extensions.
                        Dump info related to profile extensions.

-    *time*
+       *time*

                        Dump info related to internal time keeping.
                        Dump info related to internal time keeping.

-    *net*
+       *net*

                        Dump info related to networking connections.
                        Dump info related to networking connections.

-    *rate*
+       *rate*

                        Dump info related to I/O rate switching.
                        Dump info related to I/O rate switching.

-    *compress*
+       *compress*

                        Dump info related to log compress/decompress.
                        Dump info related to log compress/decompress.

-    *?* or *help*
+       *?* or *help*

                        Show available debug options.
 
 .. option:: --parse-only
 
                        Show available debug options.
 
 .. option:: --parse-only
 

-    Parse options only, don\'t start any I/O.
+       Parse options only, don't start any I/O.

 
 .. option:: --output=filename
 
        Write output to file `filename`.
 
 
 .. option:: --output=filename
 
        Write output to file `filename`.
 

+.. option:: --output-format=type
+
+       Set the reporting format to `normal`, `terse`, `json`, or `json+`.  Multiple
+       formats can be selected, separated by a comma.  `terse` is a CSV based
+       format.  `json+` is like `json`, except it adds a full dump of the latency
+       buckets.
+
+.. option:: --runtime
+       Limit run time to runtime seconds.
+

 .. option:: --bandwidth-log
 
        Generate aggregate bandwidth logs.
 .. option:: --bandwidth-log
 
        Generate aggregate bandwidth logs.


@@ -114,16 +124,9 @@ Command line options
 
 .. option:: --append-terse
 
 
 .. option:: --append-terse
 

-    Print statistics in selected mode AND terse, semicolon-delimited format.
-    **deprecated**, use :option:`--output-format` instead to select multiple
-    formats.
-
-.. option:: --output-format=type
-
-       Set the reporting format to `normal`, `terse`, `json`, or `json+`.  Multiple
-       formats can be selected, separated by a comma.  `terse` is a CSV based
-       format.  `json+` is like `json`, except it adds a full dump of the latency
-       buckets.
+       Print statistics in selected mode AND terse, semicolon-delimited format.
+       **Deprecated**, use :option:`--output-format` instead to select multiple
+       formats.

 
 .. option:: --terse-version=type
 
 
 .. option:: --terse-version=type
 


@@ -131,7 +134,7 @@ Command line options
 
 .. option:: --version
 
 
 .. option:: --version
 

-       Print version info and exit.
+       Print version information and exit.

 
 .. option:: --help
 
 
 .. option:: --help
 


@@ -143,9 +146,9 @@ Command line options
 
 .. option:: --crctest=[test]
 
 
 .. option:: --crctest=[test]
 

-    Test the speed of the built-in checksumming functions. If no argument is
-    given all of them are tested. Alternatively, a comma separated list can be passed, in
-    which case the given ones are tested.
+       Test the speed of the built-in checksumming functions. If no argument is
+       given, all of them are tested. Alternatively, a comma separated list can
+       be passed, in which case the given ones are tested.

 
 .. option:: --cmdhelp=command
 
 
 .. option:: --cmdhelp=command
 


@@ -153,27 +156,27 @@ Command line options
 
 .. option:: --enghelp=[ioengine[,command]]
 
 
 .. option:: --enghelp=[ioengine[,command]]
 

-    List all commands defined by :option:`ioengine`, or print help for `command`
-    defined by :option:`ioengine`.  If no :option:`ioengine` is given, list all
-    available ioengines.
+       List all commands defined by :option:`ioengine`, or print help for `command`
+       defined by :option:`ioengine`.  If no :option:`ioengine` is given, list all
+       available ioengines.

 
 .. option:: --showcmd=jobfile
 
 
 .. option:: --showcmd=jobfile
 

-       Turn a job file into command line options.
+       Convert `jobfile` to a set of command-line options.

 
 .. option:: --readonly
 
 
 .. option:: --readonly
 

-    Turn on safety read-only checks, preventing writes.  The ``--readonly``
-    option is an extra safety guard to prevent users from accidentally starting
-    a write workload when that is not desired.  Fio will only write if
-    `rw=write/randwrite/rw/randrw` is given.  This extra safety net can be used
-    as an extra precaution as ``--readonly`` will also enable a write check in
-    the I/O engine core to prevent writes due to unknown user space bug(s).
+       Turn on safety read-only checks, preventing writes.  The ``--readonly``
+       option is an extra safety guard to prevent users from accidentally starting
+       a write workload when that is not desired.  Fio will only write if
+       `rw=write/randwrite/rw/randrw` is given.  This extra safety net can be used
+       as an extra precaution as ``--readonly`` will also enable a write check in
+       the I/O engine core to prevent writes due to unknown user space bug(s).

 
 .. option:: --eta=when
 
 
 .. option:: --eta=when
 

-       When real-time ETA estimate should be printed.  May be `always`, `never` or
-       `auto`.
+       Specifies when real-time ETA estimate should be printed.  `when` may be
+       `always`, `never` or `auto`.

 
 .. option:: --eta-newline=time
 
 
 .. option:: --eta-newline=time
 


@@ -187,48 +190,48 @@ Command line options
 
 .. option:: --section=name
 
 
 .. option:: --section=name
 

-    Only run specified section in job file.  Multiple sections can be specified.
-    The ``--section`` option allows one to combine related jobs into one file.
-    E.g. one job file could define light, moderate, and heavy sections. Tell
-    fio to run only the "heavy" section by giving ``--section=heavy``
-    command line option.  One can also specify the "write" operations in one
-    section and "verify" operation in another section.  The ``--section`` option
-    only applies to job sections.  The reserved *global* section is always
-    parsed and used.
+       Only run specified section `name` in job file.  Multiple sections can be specified.
+       The ``--section`` option allows one to combine related jobs into one file.
+       E.g. one job file could define light, moderate, and heavy sections. Tell
+       fio to run only the "heavy" section by giving ``--section=heavy``
+       command line option.  One can also specify the "write" operations in one
+       section and "verify" operation in another section.  The ``--section`` option
+       only applies to job sections.  The reserved *global* section is always
+       parsed and used.

 
 .. option:: --alloc-size=kb
 
 
 .. option:: --alloc-size=kb
 

-    Set the internal smalloc pool to this size in KiB.  The
-    ``--alloc-size`` switch allows one to use a larger pool size for smalloc.
-    If running large jobs with randommap enabled, fio can run out of memory.
-    Smalloc is an internal allocator for shared structures from a fixed size
-    memory pool and can grow to 16 pools. The pool size defaults to 16MiB.
+       Set the internal smalloc pool size to `kb` in KiB.  The
+       ``--alloc-size`` switch allows one to use a larger pool size for smalloc.
+       If running large jobs with randommap enabled, fio can run out of memory.
+       Smalloc is an internal allocator for shared structures from a fixed size
+       memory pool and can grow to 16 pools. The pool size defaults to 16MiB.

 
 

-    NOTE: While running :file:`.fio_smalloc.*` backing store files are visible
-    in :file:`/tmp`.
+       NOTE: While running :file:`.fio_smalloc.*` backing store files are visible
+       in :file:`/tmp`.

 
 .. option:: --warnings-fatal
 
 
 .. option:: --warnings-fatal
 

-    All fio parser warnings are fatal, causing fio to exit with an
-    error.
+       All fio parser warnings are fatal, causing fio to exit with an
+       error.

 
 .. option:: --max-jobs=nr
 
 
 .. option:: --max-jobs=nr
 

-       Maximum number of threads/processes to support.
+       Set the maximum number of threads/processes to support.

 
 .. option:: --server=args
 
 
 .. option:: --server=args
 

-    Start a backend server, with `args` specifying what to listen to.
-    See `Client/Server`_ section.
+       Start a backend server, with `args` specifying what to listen to.
+       See `Client/Server`_ section.

 
 .. option:: --daemonize=pidfile
 
 
 .. option:: --daemonize=pidfile
 

-    Background a fio server, writing the pid to the given `pidfile` file.
+       Background a fio server, writing the pid to the given `pidfile` file.

 
 .. option:: --client=hostname
 
 
 .. option:: --client=hostname
 

-    Instead of running the jobs locally, send and run them on the given host or
-    set of hosts.  See `Client/Server`_ section.
+       Instead of running the jobs locally, send and run them on the given host or
+       set of hosts.  See `Client/Server`_ section.

 
 .. option:: --remote-config=file
 
 
 .. option:: --remote-config=file
 


@@ -236,7 +239,7 @@ Command line options
 
 .. option:: --idle-prof=option
 
 
 .. option:: --idle-prof=option
 

-       Report CPU idleness. *option* is one of the following:
+       Report CPU idleness. `option` is one of the following:

 
                **calibrate**
                        Run unit work calibration only and exit.
 
                **calibrate**
                        Run unit work calibration only and exit.


@@ -445,7 +448,7 @@ automatically substituted with the current system values when the job is
 run. Simple math is also supported on these keywords, so you can perform actions
 like::
 
 run. Simple math is also supported on these keywords, so you can perform actions
 like::
 

-        size=8*$mb_memory
+       size=8*$mb_memory

 
 and get that properly expanded to 8 times the size of memory in the machine.
 
 
 and get that properly expanded to 8 times the size of memory in the machine.
 


@@ -474,7 +477,7 @@ Parameter types
 ~~~~~~~~~~~~~~~
 
 **str**
 ~~~~~~~~~~~~~~~
 
 **str**

-    String. This is a sequence of alpha characters.
+       String: A sequence of alphanumeric characters.

 
 **time**
        Integer with possible time suffix.  Without a unit value is interpreted as
 
 **time**
        Integer with possible time suffix.  Without a unit value is interpreted as


@@ -488,7 +491,7 @@ Parameter types
        Integer. A whole number value, which may contain an integer prefix
        and an integer suffix:
 
        Integer. A whole number value, which may contain an integer prefix
        and an integer suffix:
 

-        [*integer prefix*] **number** [*integer suffix*]
+       [*integer prefix*] **number** [*integer suffix*]

 
        The optional *integer prefix* specifies the number's base. The default
        is decimal. *0x* specifies hexadecimal.
 
        The optional *integer prefix* specifies the number's base. The default
        is decimal. *0x* specifies hexadecimal.


@@ -521,7 +524,7 @@ Parameter types
        compatibility with old scripts.  For example, 4k means 4096.
 
        For quantities of data, an optional unit of 'B' may be included
        compatibility with old scripts.  For example, 4k means 4096.
 
        For quantities of data, an optional unit of 'B' may be included

-       (e.g.,  'kB' is the same as 'k').
+       (e.g., 'kB' is the same as 'k').

 
        The *integer suffix* is not case sensitive (e.g., m/mi mean mebi/mega,
        not milli). 'b' and 'B' both mean byte, not bit.
 
        The *integer suffix* is not case sensitive (e.g., m/mi mean mebi/mega,
        not milli). 'b' and 'B' both mean byte, not bit.


@@ -1012,8 +1015,8 @@ I/O type
 
        ``sequential`` is only useful for random I/O, where fio would normally
        generate a new random offset for every I/O. If you append e.g. 8 to randread,
 
        ``sequential`` is only useful for random I/O, where fio would normally
        generate a new random offset for every I/O. If you append e.g. 8 to randread,

-       you would get a new random offset for every 8 I/O's. The result would be a
-       seek for only every 8 I/O's, instead of for every I/O. Use ``rw=randread:8``
+       you would get a new random offset for every 8 I/Os. The result would be a
+       seek for only every 8 I/Os, instead of for every I/O. Use ``rw=randread:8``

        to specify that. As sequential I/O is already sequential, setting
        ``sequential`` for that would not result in any differences.  ``identical``
        behaves in a similar fashion, except it sends the same offset 8 number of
        to specify that. As sequential I/O is already sequential, setting
        ``sequential`` for that would not result in any differences.  ``identical``
        behaves in a similar fashion, except it sends the same offset 8 number of


@@ -1090,11 +1093,29 @@ I/O type
                **random**
                        Advise using **FADV_RANDOM**.
 
                **random**
                        Advise using **FADV_RANDOM**.
 

-.. option:: fadvise_stream=int
+.. option:: write_hint=str
+
+       Use :manpage:`fcntl(2)` to advise the kernel what life time to expect
+       from a write. Only supported on Linux, as of version 4.13. Accepted
+       values are:
+
+               **none**
+                       No particular life time associated with this file.
+
+               **short**
+                       Data written to this file has a short life time.
+
+               **medium**
+                       Data written to this file has a medium life time.
+
+               **long**
+                       Data written to this file has a long life time.
+
+               **extreme**
+                       Data written to this file has a very long life time.

 
 

-       Use :manpage:`posix_fadvise(2)` to advise the kernel what stream ID the
-       writes issued belong to. Only supported on Linux. Note, this option may
-       change going forward.
+       The values are all relative to each other, and no absolute meaning
+       should be associated with them.

 
 .. option:: offset=int
 
 
 .. option:: offset=int
 


@@ -1234,7 +1255,7 @@ I/O type
 
        * 60% of accesses should be to the first 10%
        * 30% of accesses should be to the next 20%
 
        * 60% of accesses should be to the first 10%
        * 30% of accesses should be to the next 20%

-       * 8% of accesses should be to to the next 30%
+       * 8% of accesses should be to the next 30%

        * 2% of accesses should be to the next 40%
 
        we can define that through zoning of the random accesses. For the above
        * 2% of accesses should be to the next 40%
 
        we can define that through zoning of the random accesses. For the above


@@ -1868,13 +1889,13 @@ caveat that when used on the command line, they must come after the
        hostname if the job is a TCP listener or UDP reader. For unix sockets, the
        normal filename option should be used and the port is invalid.
 
        hostname if the job is a TCP listener or UDP reader. For unix sockets, the
        normal filename option should be used and the port is invalid.
 

-.. option:: listen : [net]
+.. option:: listen : [netsplice] [net]

 
        For TCP network connections, tell fio to listen for incoming connections
        rather than initiating an outgoing connection. The :option:`hostname` must
        be omitted if this option is used.
 
 
        For TCP network connections, tell fio to listen for incoming connections
        rather than initiating an outgoing connection. The :option:`hostname` must
        be omitted if this option is used.
 

-.. option:: pingpong : [net]
+.. option:: pingpong : [netsplice] [net]

 
        Normally a network writer will just continue writing data, and a network
        reader will just consume packages. If ``pingpong=1`` is set, a writer will
 
        Normally a network writer will just continue writing data, and a network
        reader will just consume packages. If ``pingpong=1`` is set, a writer will


@@ -1886,11 +1907,11 @@ caveat that when used on the command line, they must come after the
        ``pingpong=1`` should only be set for a single reader when multiple readers
        are listening to the same address.
 
        ``pingpong=1`` should only be set for a single reader when multiple readers
        are listening to the same address.
 

-.. option:: window_size : [net]
+.. option:: window_size : [netsplice] [net]

 
        Set the desired socket buffer size for the connection.
 
 
        Set the desired socket buffer size for the connection.
 

-.. option:: mss : [net]
+.. option:: mss : [netsplice] [net]

 
        Set the TCP maximum segment size (TCP_MAXSEG).
 
 
        Set the TCP maximum segment size (TCP_MAXSEG).
 


@@ -2740,7 +2761,8 @@ Measurements and reporting
 .. option:: log_offset=int
 
        If this is set, the iolog options will include the byte offset for the I/O
 .. option:: log_offset=int
 
        If this is set, the iolog options will include the byte offset for the I/O

-       entry as well as the other data values.
+       entry as well as the other data values. Defaults to 0 meaning that
+       offsets are not present in logs. Also see `Log File Formats`_.

 
 .. option:: log_compression=int
 
 
 .. option:: log_compression=int
 


@@ -3221,7 +3243,7 @@ numbers denote:
 **ios**
                Number of I/Os performed by all groups.
 **merge**
 **ios**
                Number of I/Os performed by all groups.
 **merge**

-               Number of merges I/O the I/O scheduler.
+               Number of merges performed by the I/O scheduler.

 **ticks**
                Number of ticks we kept the disk busy.
 **in_queue**
 **ticks**
                Number of ticks we kept the disk busy.
 **in_queue**


@@ -3257,7 +3279,7 @@ changed for some reason, this number will be incremented by 1 to signify that
 change.
 
 Split up, the format is as follows (comments in brackets denote when a
 change.
 
 Split up, the format is as follows (comments in brackets denote when a

-field was introduced or whether its specific to some terse version):
+field was introduced or whether it's specific to some terse version):

 
     ::
 
 
     ::
 


@@ -3510,9 +3532,10 @@ Log File Formats
 Fio supports a variety of log file formats, for logging latencies, bandwidth,
 and IOPS. The logs share a common format, which looks like this:
 
 Fio supports a variety of log file formats, for logging latencies, bandwidth,
 and IOPS. The logs share a common format, which looks like this:
 

-    *time* (`msec`), *value*, *data direction*, *offset*
+    *time* (`msec`), *value*, *data direction*, *block size* (`bytes`),
+    *offset* (`bytes`)

 
 

-Time for the log entry is always in milliseconds. The *value* logged depends
+*Time* for the log entry is always in milliseconds. The *value* logged depends

 on the type of log, it will be one of the following:
 
     **Latency log**
 on the type of log, it will be one of the following:
 
     **Latency log**


@@ -3531,18 +3554,19 @@ on the type of log, it will be one of the following:
        **2**
                I/O is a TRIM
 
        **2**
                I/O is a TRIM
 

-The *offset* is the offset, in bytes, from the start of the file, for that
-particular I/O. The logging of the offset can be toggled with
-:option:`log_offset`.
+The entry's *block size* is always in bytes. The *offset* is the offset, in bytes,
+from the start of the file, for that particular I/O. The logging of the offset can be
+toggled with :option:`log_offset`.

 
 Fio defaults to logging every individual I/O.  When IOPS are logged for individual
 
 Fio defaults to logging every individual I/O.  When IOPS are logged for individual

-I/Os the value entry will always be 1.  If windowed logging is enabled through
+I/Os the *value* entry will always be 1. If windowed logging is enabled through

 :option:`log_avg_msec`, fio logs the average values over the specified period of time.
 If windowed logging is enabled and :option:`log_max_value` is set, then fio logs
 :option:`log_avg_msec`, fio logs the average values over the specified period of time.
 If windowed logging is enabled and :option:`log_max_value` is set, then fio logs

-maximum values in that window instead of averages.  Since 'data direction' and
-'offset' are per-I/O values, they aren't applicable if windowed logging is enabled.
+maximum values in that window instead of averages. Since *data direction*, *block
+size* and *offset* are per-I/O values, if windowed logging is enabled they
+aren't applicable and will be 0.

 
 

-Client/server
+Client/Server

 -------------
 
 Normally fio is invoked as a stand-alone application on the machine where the
 -------------
 
 Normally fio is invoked as a stand-alone application on the machine where the