Update compression documentation

[fio.git] / HOWTO

diff --git a/HOWTO b/HOWTO
index df79e2dffb3043e771fa437f6f1845b423a41160..e510e18c4c782a3335f6edaf29fed89799b9445c 100644 (file)
--- a/HOWTO
+++ b/HOWTO
@@ -217,6 +217,9 @@ Command line options
 .. option:: --max-jobs=nr
 
        Set the maximum number of threads/processes to support to `nr`.
 .. option:: --max-jobs=nr
 
        Set the maximum number of threads/processes to support to `nr`.

+       NOTE: On Linux, it may be necessary to increase the shared-memory
+       limit (:file:`/proc/sys/kernel/shmmax`) if fio runs into errors while
+       creating jobs.

 
 .. option:: --server=args
 
 
 .. option:: --server=args
 


@@ -230,7 +233,7 @@ Command line options
 .. option:: --client=hostname
 
        Instead of running the jobs locally, send and run them on the given `hostname`
 .. option:: --client=hostname
 
        Instead of running the jobs locally, send and run them on the given `hostname`

-       or set of `hostname`s.  See `Client/Server`_ section.
+       or set of `hostname`\s.  See `Client/Server`_ section.

 
 .. option:: --remote-config=file
 
 
 .. option:: --remote-config=file
 


@@ -792,6 +795,13 @@ Target file/device
        named :file:`testfiles.4`. The default of :file:`$jobname.$jobnum.$filenum`
        will be used if no other format specifier is given.
 
        named :file:`testfiles.4`. The default of :file:`$jobname.$jobnum.$filenum`
        will be used if no other format specifier is given.
 

+       If you specify a path then the directories will be created up to the
+       main directory for the file.  So for example if you specify
+       ``filename_format=a/b/c/$jobnum`` then the directories a/b/c will be
+       created before the file setup part of the job.  If you specify
+       :option:`directory` then the path will be relative that directory,
+       otherwise it is treated as the absolute path.
+

 .. option:: unique_filename=bool
 
        To avoid collisions between networked clients, fio defaults to prefixing any
 .. option:: unique_filename=bool
 
        To avoid collisions between networked clients, fio defaults to prefixing any


@@ -1118,13 +1128,20 @@ I/O type
 .. option:: offset=int
 
        Start I/O at the provided offset in the file, given as either a fixed size in
 .. option:: offset=int
 
        Start I/O at the provided offset in the file, given as either a fixed size in

-       bytes or a percentage. If a percentage is given, the next ``blockalign``-ed
-       offset will be used. Data before the given offset will not be touched. This
+       bytes or a percentage. If a percentage is given, the generated offset will be
+       aligned to the minimum ``blocksize`` or to the value of ``offset_align`` if
+       provided. Data before the given offset will not be touched. This

        effectively caps the file size at `real_size - offset`. Can be combined with
        :option:`size` to constrain the start and end range of the I/O workload.
        A percentage can be specified by a number between 1 and 100 followed by '%',
        for example, ``offset=20%`` to specify 20%.
 
        effectively caps the file size at `real_size - offset`. Can be combined with
        :option:`size` to constrain the start and end range of the I/O workload.
        A percentage can be specified by a number between 1 and 100 followed by '%',
        for example, ``offset=20%`` to specify 20%.
 

+.. option:: offset_align=int
+
+       If set to non-zero value, the byte offset generated by a percentage ``offset``
+       is aligned upwards to this value. Defaults to 0 meaning that a percentage
+       offset is aligned to the minimum block size.
+

 .. option:: offset_increment=int
 
        If this is provided, then the real offset becomes `offset + offset_increment
 .. option:: offset_increment=int
 
        If this is provided, then the real offset becomes `offset + offset_increment


@@ -1436,8 +1453,8 @@ Buffers and memory
        mix of random data and a fixed pattern. The fixed pattern is either zeros,
        or the pattern specified by :option:`buffer_pattern`. If the pattern option
        is used, it might skew the compression ratio slightly. Note that this is per
        mix of random data and a fixed pattern. The fixed pattern is either zeros,
        or the pattern specified by :option:`buffer_pattern`. If the pattern option
        is used, it might skew the compression ratio slightly. Note that this is per

-       block size unit, for file/disk wide compression level that matches this
-       setting, you'll also want to set :option:`refill_buffers`.
+       block size unit, see :option:`buffer_compress_chunk` for setting a finer
+       granularity of compression regions.

 
 .. option:: buffer_compress_chunk=int
 
 
 .. option:: buffer_compress_chunk=int
 


@@ -1446,7 +1463,8 @@ Buffers and memory
        will provide :option:`buffer_compress_percentage` of blocksize random data,
        followed by the remaining zeroed. With this set to some chunk size smaller
        than the block size, fio can alternate random and zeroed data throughout the
        will provide :option:`buffer_compress_percentage` of blocksize random data,
        followed by the remaining zeroed. With this set to some chunk size smaller
        than the block size, fio can alternate random and zeroed data throughout the

-       I/O buffer.
+       I/O buffer. This is particularly useful when bigger block sizes are used
+       for a job.

 
 .. option:: buffer_pattern=str
 
 
 .. option:: buffer_pattern=str
 


@@ -1705,7 +1723,7 @@ I/O engine
                        Doesn't transfer any data, but burns CPU cycles according to the
                        :option:`cpuload` and :option:`cpuchunks` options. Setting
                        :option:`cpuload`\=85 will cause that job to do nothing but burn 85%
                        Doesn't transfer any data, but burns CPU cycles according to the
                        :option:`cpuload` and :option:`cpuchunks` options. Setting
                        :option:`cpuload`\=85 will cause that job to do nothing but burn 85%

-                       of the CPU. In case of SMP machines, use :option:`numjobs`=<nr_of_cpu>
+                       of the CPU. In case of SMP machines, use :option:`numjobs`\=<nr_of_cpu>

                        to get desired CPU usage, as the cpuload only loads a
                        single CPU at the desired rate. A job never finishes unless there is
                        at least one non-cpuio job.
                        to get desired CPU usage, as the cpuload only loads a
                        single CPU at the desired rate. A job never finishes unless there is
                        at least one non-cpuio job.


@@ -1798,9 +1816,9 @@ I/O engine
                        details of writing an external I/O engine.
 
                **filecreate**
                        details of writing an external I/O engine.
 
                **filecreate**

-                       Simply create the files and do no IO to them.  You still need to
+                       Simply create the files and do no I/O to them.  You still need to

                        set  `filesize` so that all the accounting still occurs, but no
                        set  `filesize` so that all the accounting still occurs, but no

-                       actual IO will be done other than creating the file.
+                       actual I/O will be done other than creating the file.

 
 I/O engine specific parameters
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 I/O engine specific parameters
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


@@ -2043,7 +2061,7 @@ I/O depth
        changing data and the overlapping region has a non-zero size. Setting
        ``serialize_overlap`` tells fio to avoid provoking this behavior by explicitly
        serializing in-flight I/Os that have a non-zero overlap. Note that setting
        changing data and the overlapping region has a non-zero size. Setting
        ``serialize_overlap`` tells fio to avoid provoking this behavior by explicitly
        serializing in-flight I/Os that have a non-zero overlap. Note that setting

-       this option can reduce both performance and the `:option:iodepth` achieved.
+       this option can reduce both performance and the :option:`iodepth` achieved.

        Additionally this option does not work when :option:`io_submit_mode` is set to
        offload. Default: false.
 
        Additionally this option does not work when :option:`io_submit_mode` is set to
        offload. Default: false.
 


@@ -2705,47 +2723,46 @@ Measurements and reporting
 .. option:: write_bw_log=str
 
        If given, write a bandwidth log for this job. Can be used to store data of
 .. option:: write_bw_log=str
 
        If given, write a bandwidth log for this job. Can be used to store data of

-       the bandwidth of the jobs in their lifetime. The included
-       :command:`fio_generate_plots` script uses :command:`gnuplot` to turn these
-       text files into nice graphs. See :option:`write_lat_log` for behavior of
-       given filename. For this option, the postfix is :file:`_bw.x.log`, where `x`
-       is the index of the job (`1..N`, where `N` is the number of jobs). If
-       :option:`per_job_logs` is false, then the filename will not include the job
-       index.  See `Log File Formats`_.
+       the bandwidth of the jobs in their lifetime.

 
 

-.. option:: write_lat_log=str
+       If no str argument is given, the default filename of
+       :file:`jobname_type.x.log` is used. Even when the argument is given, fio
+       will still append the type of log. So if one specifies::

 
 

-       Same as :option:`write_bw_log`, except that this option stores I/O
-       submission, completion, and total latencies instead. If no filename is given
-       with this option, the default filename of :file:`jobname_type.log` is
-       used. Even if the filename is given, fio will still append the type of
-       log. So if one specifies::
+               write_bw_log=foo

 
 

-               write_lat_log=foo
+       The actual log name will be :file:`foo_bw.x.log` where `x` is the index
+       of the job (`1..N`, where `N` is the number of jobs). If
+       :option:`per_job_logs` is false, then the filename will not include the
+       `.x` job index.
+
+       The included :command:`fio_generate_plots` script uses :command:`gnuplot` to turn these
+       text files into nice graphs. See `Log File Formats`_ for how data is
+       structured within the file.
+
+.. option:: write_lat_log=str

 
 

-       The actual log names will be :file:`foo_slat.x.log`, :file:`foo_clat.x.log`,
-       and :file:`foo_lat.x.log`, where `x` is the index of the job (`1..N`, where `N`
-       is the number of jobs). This helps :command:`fio_generate_plots` find the
-       logs automatically. If :option:`per_job_logs` is false, then the filename
-       will not include the job index.  See `Log File Formats`_.
+       Same as :option:`write_bw_log`, except this option creates I/O
+       submission (e.g., `file:`name_slat.x.log`), completion (e.g.,
+       `file:`name_clat.x.log`), and total (e.g., `file:`name_lat.x.log`)
+       latency files instead. See :option:`write_bw_log` for details about
+       the filename format and `Log File Formats`_ for how data is structured
+       within the files.

 
 .. option:: write_hist_log=str
 
 
 .. option:: write_hist_log=str
 

-       Same as :option:`write_lat_log`, but writes I/O completion latency
-       histograms. If no filename is given with this option, the default filename
-       of :file:`jobname_clat_hist.x.log` is used, where `x` is the index of the
-       job (`1..N`, where `N` is the number of jobs). Even if the filename is given,
-       fio will still append the type of log.  If :option:`per_job_logs` is false,
-       then the filename will not include the job index. See `Log File Formats`_.
+       Same as :option:`write_bw_log` but writes an I/O completion latency
+       histogram file (e.g., `file:`name_hist.x.log`) instead. Note that this
+       file will be empty unless :option:`log_hist_msec` has also been set.
+       See :option:`write_bw_log` for details about the filename format and
+       `Log File Formats`_ for how data is structured within the file.

 
 .. option:: write_iops_log=str
 
 
 .. option:: write_iops_log=str
 

-       Same as :option:`write_bw_log`, but writes IOPS. If no filename is given
-       with this option, the default filename of :file:`jobname_type.x.log` is
-       used, where `x` is the index of the job (`1..N`, where `N` is the number of
-       jobs). Even if the filename is given, fio will still append the type of
-       log. If :option:`per_job_logs` is false, then the filename will not include
-       the job index. See `Log File Formats`_.
+       Same as :option:`write_bw_log`, but writes an IOPS file (e.g.
+       `file:`name_iops.x.log`) instead. See :option:`write_bw_log` for
+       details about the filename format and `Log File Formats`_ for how data
+       is structured within the file.

 
 .. option:: log_avg_msec=int
 
 
 .. option:: log_avg_msec=int
 


@@ -2763,15 +2780,16 @@ Measurements and reporting
        :option:`log_avg_msec` is inaccurate. Setting this option makes fio log
        histogram entries over the specified period of time, reducing log sizes for
        high IOPS devices while retaining percentile accuracy.  See
        :option:`log_avg_msec` is inaccurate. Setting this option makes fio log
        histogram entries over the specified period of time, reducing log sizes for
        high IOPS devices while retaining percentile accuracy.  See

-       :option:`log_hist_coarseness` as well. Defaults to 0, meaning histogram
-       logging is disabled.
+       :option:`log_hist_coarseness` and :option:`write_hist_log` as well.
+       Defaults to 0, meaning histogram logging is disabled.

 
 .. option:: log_hist_coarseness=int
 
        Integer ranging from 0 to 6, defining the coarseness of the resolution of
        the histogram logs enabled with :option:`log_hist_msec`. For each increment
        in coarseness, fio outputs half as many bins. Defaults to 0, for which
 
 .. option:: log_hist_coarseness=int
 
        Integer ranging from 0 to 6, defining the coarseness of the resolution of
        the histogram logs enabled with :option:`log_hist_msec`. For each increment
        in coarseness, fio outputs half as many bins. Defaults to 0, for which

-       histogram logs contain 1216 latency bins. See `Log File Formats`_.
+       histogram logs contain 1216 latency bins. See :option:`write_hist_log`
+       and `Log File Formats`_.

 
 .. option:: log_max_value=bool
 
 
 .. option:: log_max_value=bool
 


@@ -2871,7 +2889,7 @@ Measurements and reporting
 
 .. option:: lat_percentiles=bool
 
 
 .. option:: lat_percentiles=bool
 

-       Enable the reporting of percentiles of IO latencies. This is similar
+       Enable the reporting of percentiles of I/O latencies. This is similar

        to :option:`clat_percentiles`, except that this includes the
        submission latency. This option is mutually exclusive with
        :option:`clat_percentiles`.
        to :option:`clat_percentiles`, except that this includes the
        submission latency. This option is mutually exclusive with
        :option:`clat_percentiles`.


@@ -3241,7 +3259,7 @@ writes in the example above).  In the order listed, they denote:
                short or dropped.
 
 **IO latency**
                short or dropped.
 
 **IO latency**

-               These values are for `--latency-target` and related options. When
+               These values are for :option:`latency_target` and related options. When

                these options are engaged, this section describes the I/O depth required
                to meet the specified latency target.
 
                these options are engaged, this section describes the I/O depth required
                to meet the specified latency target.