t/zbd/test-zbd-support: Ensure that an assertion failure causes this test to fail

[fio.git] / HOWTO

diff --git a/HOWTO b/HOWTO
index 743144f03d14f76d46cf43d5191cf7217c4cd849..95289040dd3fd58a0bb70dfb0a94fadf1475e175 100644 (file)
--- a/HOWTO
+++ b/HOWTO
@@ -100,6 +100,10 @@ Command line options
 
        Parse options only, don't start any I/O.
 
 
        Parse options only, don't start any I/O.
 

+.. option:: --merge-blktrace-only
+
+       Merge blktraces only, don't start any I/O.
+

 .. option:: --output=filename
 
        Write output to file `filename`.
 .. option:: --output=filename
 
        Write output to file `filename`.


@@ -194,7 +198,10 @@ Command line options
        Force a full status dump of cumulative (from job start) values at `time`
        intervals. This option does *not* provide per-period measurements. So
        values such as bandwidth are running averages. When the time unit is omitted,
        Force a full status dump of cumulative (from job start) values at `time`
        intervals. This option does *not* provide per-period measurements. So
        values such as bandwidth are running averages. When the time unit is omitted,

-       `time` is interpreted in seconds.
+       `time` is interpreted in seconds. Note that using this option with
+       ``--output-format=json`` will yield output that technically isn't valid
+       json, since the output will be collated sets of valid json. It will need
+       to be split into valid sets of json after the run.

 
 .. option:: --section=name
 
 
 .. option:: --section=name
 


@@ -952,24 +959,92 @@ Target file/device
 
        Unlink job files after each iteration or loop.  Default: false.
 
 
        Unlink job files after each iteration or loop.  Default: false.
 

+.. option:: zonemode=str
+
+       Accepted values are:
+
+               **none**
+                               The :option:`zonerange`, :option:`zonesize` and
+                               :option:`zoneskip` parameters are ignored.
+               **strided**
+                               I/O happens in a single zone until
+                               :option:`zonesize` bytes have been transferred.
+                               After that number of bytes has been
+                               transferred processing of the next zone
+                               starts.
+               **zbd**
+                               Zoned block device mode. I/O happens
+                               sequentially in each zone, even if random I/O
+                               has been selected. Random I/O happens across
+                               all zones instead of being restricted to a
+                               single zone. The :option:`zoneskip` parameter
+                               is ignored. :option:`zonerange` and
+                               :option:`zonesize` must be identical.
+

 .. option:: zonerange=int
 
 .. option:: zonerange=int
 

-       Size of a single zone in which I/O occurs. See also :option:`zonesize`
-       and :option:`zoneskip`.
+       Size of a single zone. See also :option:`zonesize` and
+       :option:`zoneskip`.

 
 .. option:: zonesize=int
 
 
 .. option:: zonesize=int
 

-       Number of bytes to transfer before skipping :option:`zoneskip`
-       bytes. If this parameter is smaller than :option:`zonerange` then only
-       a fraction of each zone with :option:`zonerange` bytes will be
-       accessed.  If this parameter is larger than :option:`zonerange` then
-       each zone will be accessed multiple times before skipping
+       For :option:`zonemode` =strided, this is the number of bytes to
+       transfer before skipping :option:`zoneskip` bytes. If this parameter
+       is smaller than :option:`zonerange` then only a fraction of each zone
+       with :option:`zonerange` bytes will be accessed.  If this parameter is
+       larger than :option:`zonerange` then each zone will be accessed
+       multiple times before skipping to the next zone.
+
+       For :option:`zonemode` =zbd, this is the size of a single zone. The
+       :option:`zonerange` parameter is ignored in this mode.

 
 .. option:: zoneskip=int
 
 
 .. option:: zoneskip=int
 

-       Skip the specified number of bytes when :option:`zonesize` data have
-       been transferred. The three zone options can be used to do strided I/O
-       on a file.
+       For :option:`zonemode` =strided, the number of bytes to skip after
+       :option:`zonesize` bytes of data have been transferred. This parameter
+       must be zero for :option:`zonemode` =zbd.
+
+.. option:: read_beyond_wp=bool
+
+       This parameter applies to :option:`zonemode` =zbd only.
+
+       Zoned block devices are block devices that consist of multiple zones.
+       Each zone has a type, e.g. conventional or sequential. A conventional
+       zone can be written at any offset that is a multiple of the block
+       size. Sequential zones must be written sequentially. The position at
+       which a write must occur is called the write pointer. A zoned block
+       device can be either drive managed, host managed or host aware. For
+       host managed devices the host must ensure that writes happen
+       sequentially. Fio recognizes host managed devices and serializes
+       writes to sequential zones for these devices.
+
+       If a read occurs in a sequential zone beyond the write pointer then
+       the zoned block device will complete the read without reading any data
+       from the storage medium. Since such reads lead to unrealistically high
+       bandwidth and IOPS numbers fio only reads beyond the write pointer if
+       explicitly told to do so. Default: false.
+
+.. option:: max_open_zones=int
+
+       When running a random write test across an entire drive many more
+       zones will be open than in a typical application workload. Hence this
+       command line option that allows to limit the number of open zones. The
+       number of open zones is defined as the number of zones to which write
+       commands are issued.
+
+.. option:: zone_reset_threshold=float
+
+       A number between zero and one that indicates the ratio of logical
+       blocks with data to the total number of logical blocks in the test
+       above which zones should be reset periodically.
+
+.. option:: zone_reset_frequency=float
+
+       A number between zero and one that indicates how often a zone reset
+       should be issued if the zone reset threshold has been exceeded. A zone
+       reset is submitted after each (1 / zone_reset_frequency) write
+       requests. This and the previous parameter can be used to simulate
+       garbage collection activity.

 
 
 I/O type
 
 
 I/O type


@@ -1901,6 +1976,22 @@ I/O engine
                        mounted with DAX on a persistent memory device through the PMDK
                        libpmem library.
 
                        mounted with DAX on a persistent memory device through the PMDK
                        libpmem library.
 

+               **ime_psync**
+                       Synchronous read and write using DDN's Infinite Memory Engine (IME).
+                       This engine is very basic and issues calls to IME whenever an IO is
+                       queued.
+
+               **ime_psyncv**
+                       Synchronous read and write using DDN's Infinite Memory Engine (IME).
+                       This engine uses iovecs and will try to stack as much IOs as possible
+                       (if the IOs are "contiguous" and the IO depth is not exceeded)
+                       before issuing a call to IME.
+
+               **ime_aio**
+                       Asynchronous read and write using DDN's Infinite Memory Engine (IME).
+                       This engine will try to stack as much IOs as possible by creating
+                       requests for IME. FIO will then decide when to commit these requests.
+

 I/O engine specific parameters
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 I/O engine specific parameters
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 


@@ -2393,6 +2484,10 @@ I/O replay
        :manpage:`blktrace(8)` for how to capture such logging data. For blktrace
        replay, the file needs to be turned into a blkparse binary data file first
        (``blkparse <device> -o /dev/null -d file_for_fio.bin``).
        :manpage:`blktrace(8)` for how to capture such logging data. For blktrace
        replay, the file needs to be turned into a blkparse binary data file first
        (``blkparse <device> -o /dev/null -d file_for_fio.bin``).

+       You can specify a number of files by separating the names with a ':'
+       character. See the :option:`filename` option for information on how to
+       escape ':' and '\' characters within the file names. These files will
+       be sequentially assigned to job clones created by :option:`numjobs`.

 
 .. option:: read_iolog_chunked=bool
 
 
 .. option:: read_iolog_chunked=bool
 


@@ -2400,6 +2495,33 @@ I/O replay
        will be read at once. If selected true, input from iolog will be read
        gradually. Useful when iolog is very large, or it is generated.
 
        will be read at once. If selected true, input from iolog will be read
        gradually. Useful when iolog is very large, or it is generated.
 

+.. option:: merge_blktrace_file=str
+
+       When specified, rather than replaying the logs passed to :option:`read_iolog`,
+       the logs go through a merge phase which aggregates them into a single
+       blktrace. The resulting file is then passed on as the :option:`read_iolog`
+       parameter. The intention here is to make the order of events consistent.
+       This limits the influence of the scheduler compared to replaying multiple
+       blktraces via concurrent jobs.
+
+.. option:: merge_blktrace_scalars=float_list
+
+       This is a percentage based option that is index paired with the list of
+       files passed to :option:`read_iolog`. When merging is performed, scale
+       the time of each event by the corresponding amount. For example,
+       ``--merge_blktrace_scalars="50:100"`` runs the first trace in halftime
+       and the second trace in realtime. This knob is separately tunable from
+       :option:`replay_time_scale` which scales the trace during runtime and
+       does not change the output of the merge unlike this option.
+
+.. option:: merge_blktrace_iters=float_list
+
+       This is a whole number option that is index paired with the list of files
+       passed to :option:`read_iolog`. When merging is performed, run each trace
+       for the specified number of iterations. For example,
+       ``--merge_blktrace_iters="2:1"`` runs the first trace for two iterations
+       and the second trace for one iteration.
+

 .. option:: replay_no_stall=bool
 
        When replaying I/O with :option:`read_iolog` the default behavior is to
 .. option:: replay_no_stall=bool
 
        When replaying I/O with :option:`read_iolog` the default behavior is to


@@ -2878,6 +3000,10 @@ Steady state
        data from the rolling collection window. Threshold limits can be expressed
        as a fixed value or as a percentage of the mean in the collection window.
 
        data from the rolling collection window. Threshold limits can be expressed
        as a fixed value or as a percentage of the mean in the collection window.
 

+       When using this feature, most jobs should include the :option:`time_based`
+       and :option:`runtime` options or the :option:`loops` option so that fio does not
+       stop running after it has covered the full size of the specified file(s) or device(s).
+

                **iops**
                        Collect IOPS data. Stop the job if all individual IOPS measurements
                        are within the specified limit of the mean IOPS (e.g., ``iops:2``
                **iops**
                        Collect IOPS data. Stop the job if all individual IOPS measurements
                        are within the specified limit of the mean IOPS (e.g., ``iops:2``


@@ -3748,6 +3874,46 @@ given in bytes. The `action` can be one of these:
 **trim**
           Trim the given file from the given `offset` for `length` bytes.
 
 **trim**
           Trim the given file from the given `offset` for `length` bytes.
 

+
+I/O Replay - Merging Traces
+---------------------------
+
+Colocation is a common practice used to get the most out of a machine.
+Knowing which workloads play nicely with each other and which ones don't is
+a much harder task. While fio can replay workloads concurrently via multiple
+jobs, it leaves some variability up to the scheduler making results harder to
+reproduce. Merging is a way to make the order of events consistent.
+
+Merging is integrated into I/O replay and done when a
+:option:`merge_blktrace_file` is specified. The list of files passed to
+:option:`read_iolog` go through the merge process and output a single file
+stored to the specified file. The output file is passed on as if it were the
+only file passed to :option:`read_iolog`. An example would look like::
+
+       $ fio --read_iolog="<file1>:<file2>" --merge_blktrace_file="<output_file>"
+
+Creating only the merged file can be done by passing the command line argument
+:option:`merge-blktrace-only`.
+
+Scaling traces can be done to see the relative impact of any particular trace
+being slowed down or sped up. :option:`merge_blktrace_scalars` takes in a colon
+separated list of percentage scalars. It is index paired with the files passed
+to :option:`read_iolog`.
+
+With scaling, it may be desirable to match the running time of all traces.
+This can be done with :option:`merge_blktrace_iters`. It is index paired with
+:option:`read_iolog` just like :option:`merge_blktrace_scalars`.
+
+In an example, given two traces, A and B, each 60s long. If we want to see
+the impact of trace A issuing IOs twice as fast and repeat trace A over the
+runtime of trace B, the following can be done::
+
+       $ fio --read_iolog="<trace_a>:"<trace_b>" --merge_blktrace_file"<output_file>" --merge_blktrace_scalars="50:100" --merge_blktrace_iters="2:1"
+
+This runs trace A at 2x the speed twice for approximately the same runtime as
+a single run of trace B.
+
+

 CPU idleness profiling
 ----------------------
 
 CPU idleness profiling
 ----------------------