X-Git-Url: https://git.kernel.dk/?p=fio.git;a=blobdiff_plain;f=HOWTO;h=4fd4da1471758c14808f1d545d33b2a267557b42;hp=743144f03d14f76d46cf43d5191cf7217c4cd849;hb=59097e775dce8ccf3127963972ca61c6ce2a3141;hpb=09fd2966b1047a0ba0a66adde68a35d86c9c520a diff --git a/HOWTO b/HOWTO index 743144f0..4fd4da14 100644 --- a/HOWTO +++ b/HOWTO @@ -100,6 +100,10 @@ Command line options Parse options only, don't start any I/O. +.. option:: --merge-blktrace-only + + Merge blktraces only, don't start any I/O. + .. option:: --output=filename Write output to file `filename`. @@ -194,7 +198,10 @@ Command line options Force a full status dump of cumulative (from job start) values at `time` intervals. This option does *not* provide per-period measurements. So values such as bandwidth are running averages. When the time unit is omitted, - `time` is interpreted in seconds. + `time` is interpreted in seconds. Note that using this option with + ``--output-format=json`` will yield output that technically isn't valid + json, since the output will be collated sets of valid json. It will need + to be split into valid sets of json after the run. .. option:: --section=name @@ -952,24 +959,92 @@ Target file/device Unlink job files after each iteration or loop. Default: false. +.. option:: zonemode=str + + Accepted values are: + + **none** + The :option:`zonerange`, :option:`zonesize` and + :option:`zoneskip` parameters are ignored. + **strided** + I/O happens in a single zone until + :option:`zonesize` bytes have been transferred. + After that number of bytes has been + transferred processing of the next zone + starts. + **zbd** + Zoned block device mode. I/O happens + sequentially in each zone, even if random I/O + has been selected. Random I/O happens across + all zones instead of being restricted to a + single zone. The :option:`zoneskip` parameter + is ignored. :option:`zonerange` and + :option:`zonesize` must be identical. + .. option:: zonerange=int - Size of a single zone in which I/O occurs. See also :option:`zonesize` - and :option:`zoneskip`. + Size of a single zone. See also :option:`zonesize` and + :option:`zoneskip`. .. option:: zonesize=int - Number of bytes to transfer before skipping :option:`zoneskip` - bytes. If this parameter is smaller than :option:`zonerange` then only - a fraction of each zone with :option:`zonerange` bytes will be - accessed. If this parameter is larger than :option:`zonerange` then - each zone will be accessed multiple times before skipping + For :option:`zonemode` =strided, this is the number of bytes to + transfer before skipping :option:`zoneskip` bytes. If this parameter + is smaller than :option:`zonerange` then only a fraction of each zone + with :option:`zonerange` bytes will be accessed. If this parameter is + larger than :option:`zonerange` then each zone will be accessed + multiple times before skipping to the next zone. + + For :option:`zonemode` =zbd, this is the size of a single zone. The + :option:`zonerange` parameter is ignored in this mode. .. option:: zoneskip=int - Skip the specified number of bytes when :option:`zonesize` data have - been transferred. The three zone options can be used to do strided I/O - on a file. + For :option:`zonemode` =strided, the number of bytes to skip after + :option:`zonesize` bytes of data have been transferred. This parameter + must be zero for :option:`zonemode` =zbd. + +.. option:: read_beyond_wp=bool + + This parameter applies to :option:`zonemode` =zbd only. + + Zoned block devices are block devices that consist of multiple zones. + Each zone has a type, e.g. conventional or sequential. A conventional + zone can be written at any offset that is a multiple of the block + size. Sequential zones must be written sequentially. The position at + which a write must occur is called the write pointer. A zoned block + device can be either drive managed, host managed or host aware. For + host managed devices the host must ensure that writes happen + sequentially. Fio recognizes host managed devices and serializes + writes to sequential zones for these devices. + + If a read occurs in a sequential zone beyond the write pointer then + the zoned block device will complete the read without reading any data + from the storage medium. Since such reads lead to unrealistically high + bandwidth and IOPS numbers fio only reads beyond the write pointer if + explicitly told to do so. Default: false. + +.. option:: max_open_zones=int + + When running a random write test across an entire drive many more + zones will be open than in a typical application workload. Hence this + command line option that allows to limit the number of open zones. The + number of open zones is defined as the number of zones to which write + commands are issued. + +.. option:: zone_reset_threshold=float + + A number between zero and one that indicates the ratio of logical + blocks with data to the total number of logical blocks in the test + above which zones should be reset periodically. + +.. option:: zone_reset_frequency=float + + A number between zero and one that indicates how often a zone reset + should be issued if the zone reset threshold has been exceeded. A zone + reset is submitted after each (1 / zone_reset_frequency) write + requests. This and the previous parameter can be used to simulate + garbage collection activity. I/O type @@ -1901,6 +1976,24 @@ I/O engine mounted with DAX on a persistent memory device through the PMDK libpmem library. + **ime_psync** + Synchronous read and write using DDN's Infinite Memory Engine (IME). + This engine is very basic and issues calls to IME whenever an IO is + queued. + + **ime_psyncv** + Synchronous read and write using DDN's Infinite Memory Engine (IME). + This engine uses iovecs and will try to stack as much IOs as possible + (if the IOs are "contiguous" and the IO depth is not exceeded) + before issuing a call to IME. + + **ime_aio** + Asynchronous read and write using DDN's Infinite Memory Engine (IME). + This engine will try to stack as much IOs as possible by creating + requests for IME. FIO will then decide when to commit these requests. + **libiscsi** + Read and write iscsi lun with libiscsi. + I/O engine specific parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2248,8 +2341,13 @@ I/O depth ``serialize_overlap`` tells fio to avoid provoking this behavior by explicitly serializing in-flight I/Os that have a non-zero overlap. Note that setting this option can reduce both performance and the :option:`iodepth` achieved. - Additionally this option does not work when :option:`io_submit_mode` is set to - offload. Default: false. + + This option only applies to I/Os issued for a single job except when it is + enabled along with :option:`io_submit_mode`=offload. In offload mode, fio + will check for overlap among all I/Os submitted by offload jobs with :option:`serialize_overlap` + enabled. + + Default: false. .. option:: io_submit_mode=str @@ -2393,6 +2491,10 @@ I/O replay :manpage:`blktrace(8)` for how to capture such logging data. For blktrace replay, the file needs to be turned into a blkparse binary data file first (``blkparse -o /dev/null -d file_for_fio.bin``). + You can specify a number of files by separating the names with a ':' + character. See the :option:`filename` option for information on how to + escape ':' and '\' characters within the file names. These files will + be sequentially assigned to job clones created by :option:`numjobs`. .. option:: read_iolog_chunked=bool @@ -2400,6 +2502,33 @@ I/O replay will be read at once. If selected true, input from iolog will be read gradually. Useful when iolog is very large, or it is generated. +.. option:: merge_blktrace_file=str + + When specified, rather than replaying the logs passed to :option:`read_iolog`, + the logs go through a merge phase which aggregates them into a single + blktrace. The resulting file is then passed on as the :option:`read_iolog` + parameter. The intention here is to make the order of events consistent. + This limits the influence of the scheduler compared to replaying multiple + blktraces via concurrent jobs. + +.. option:: merge_blktrace_scalars=float_list + + This is a percentage based option that is index paired with the list of + files passed to :option:`read_iolog`. When merging is performed, scale + the time of each event by the corresponding amount. For example, + ``--merge_blktrace_scalars="50:100"`` runs the first trace in halftime + and the second trace in realtime. This knob is separately tunable from + :option:`replay_time_scale` which scales the trace during runtime and + does not change the output of the merge unlike this option. + +.. option:: merge_blktrace_iters=float_list + + This is a whole number option that is index paired with the list of files + passed to :option:`read_iolog`. When merging is performed, run each trace + for the specified number of iterations. For example, + ``--merge_blktrace_iters="2:1"`` runs the first trace for two iterations + and the second trace for one iteration. + .. option:: replay_no_stall=bool When replaying I/O with :option:`read_iolog` the default behavior is to @@ -2437,12 +2566,13 @@ I/O replay .. option:: replay_align=int - Force alignment of I/O offsets and lengths in a trace to this power of 2 - value. + Force alignment of the byte offsets in a trace to this value. The value + must be a power of 2. .. option:: replay_scale=int - Scale sector offsets down by this factor when replaying traces. + Scale byte offsets down by this factor when replaying traces. Should most + likely use :option:`replay_align` as well. .. option:: replay_skip=str @@ -2878,6 +3008,10 @@ Steady state data from the rolling collection window. Threshold limits can be expressed as a fixed value or as a percentage of the mean in the collection window. + When using this feature, most jobs should include the :option:`time_based` + and :option:`runtime` options or the :option:`loops` option so that fio does not + stop running after it has covered the full size of the specified file(s) or device(s). + **iops** Collect IOPS data. Stop the job if all individual IOPS measurements are within the specified limit of the mean IOPS (e.g., ``iops:2`` @@ -3748,6 +3882,46 @@ given in bytes. The `action` can be one of these: **trim** Trim the given file from the given `offset` for `length` bytes. + +I/O Replay - Merging Traces +--------------------------- + +Colocation is a common practice used to get the most out of a machine. +Knowing which workloads play nicely with each other and which ones don't is +a much harder task. While fio can replay workloads concurrently via multiple +jobs, it leaves some variability up to the scheduler making results harder to +reproduce. Merging is a way to make the order of events consistent. + +Merging is integrated into I/O replay and done when a +:option:`merge_blktrace_file` is specified. The list of files passed to +:option:`read_iolog` go through the merge process and output a single file +stored to the specified file. The output file is passed on as if it were the +only file passed to :option:`read_iolog`. An example would look like:: + + $ fio --read_iolog=":" --merge_blktrace_file="" + +Creating only the merged file can be done by passing the command line argument +:option:`merge-blktrace-only`. + +Scaling traces can be done to see the relative impact of any particular trace +being slowed down or sped up. :option:`merge_blktrace_scalars` takes in a colon +separated list of percentage scalars. It is index paired with the files passed +to :option:`read_iolog`. + +With scaling, it may be desirable to match the running time of all traces. +This can be done with :option:`merge_blktrace_iters`. It is index paired with +:option:`read_iolog` just like :option:`merge_blktrace_scalars`. + +In an example, given two traces, A and B, each 60s long. If we want to see +the impact of trace A issuing IOs twice as fast and repeat trace A over the +runtime of trace B, the following can be done:: + + $ fio --read_iolog=":"" --merge_blktrace_file"" --merge_blktrace_scalars="50:100" --merge_blktrace_iters="2:1" + +This runs trace A at 2x the speed twice for approximately the same runtime as +a single run of trace B. + + CPU idleness profiling ----------------------