Fix crash on threads being reaped before they are created

[fio.git] / HOWTO

diff --git a/HOWTO b/HOWTO
index d7283535db0d8ddff7908422185fe5cdb9480b76..73be9d7b7aa8bfc555941899b88f81834d2d78ce 100644 (file)
--- a/HOWTO
+++ b/HOWTO
@@ -159,6 +159,41 @@ specify:
 
 $ fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4
 
 
 $ fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4
 

+When fio is utilized as a basis of any reasonably large test suite, it might be
+desirable to share a set of standardized settings across multiple job files.
+Instead of copy/pasting such settings, any section may pull in an external
+.fio file with 'include filename' directive, as in the following example:
+
+; -- start job file including.fio --
+[global]
+filename=/tmp/test
+filesize=1m
+include glob-include.fio
+
+[test]
+rw=randread
+bs=4k
+time_based=1
+runtime=10
+include test-include.fio
+; -- end job file including.fio --
+
+; -- start job file glob-include.fio --
+thread=1
+group_reporting=1
+; -- end job file glob-include.fio --
+
+; -- start job file test-include.fio --
+ioengine=libaio
+iodepth=4
+; -- end job file test-include.fio --
+
+Settings pulled into a section apply to that section only (except global
+section). Include directives may be nested in that any included file may
+contain further include directive(s). Include files may not contain []
+sections.
+
+

 4.1 Environment variables
 -------------------------
 
 4.1 Environment variables
 -------------------------
 


@@ -218,7 +253,20 @@ machine.
 
 This section describes in details each parameter associated with a job.
 Some parameters take an option of a given type, such as an integer or
 
 This section describes in details each parameter associated with a job.
 Some parameters take an option of a given type, such as an integer or

-a string. The following types are used:
+a string. Anywhere a numeric value is required, an arithmetic expression
+may be used, provided it is surrounded by parentheses. Supported operators
+are:
+
+       addition (+)
+       subtraction (-)
+       multiplication (*)
+       division (/)
+       modulus (%)
+       exponentiation (^)
+
+For time values in expressions, units are microseconds by default. This is
+different than for time values not in expressions (not enclosed in
+parentheses). The following types are used:

 
 str    String. This is a sequence of alpha characters.
 time   Integer with possible time suffix. In seconds unless otherwise
 
 str    String. This is a sequence of alpha characters.
 time   Integer with possible time suffix. In seconds unless otherwise


@@ -344,7 +392,7 @@ rw=str              Type of io pattern. Accepted values are:
                For certain types of io the result may still be skewed a bit,
                since the speed may be different. It is possible to specify
                a number of IO's to do before getting a new offset, this is
                For certain types of io the result may still be skewed a bit,
                since the speed may be different. It is possible to specify
                a number of IO's to do before getting a new offset, this is

-               one by appending a ':<nr>' to the end of the string given.
+               done by appending a ':<nr>' to the end of the string given.

                For a random read, it would look like 'rw=randread:8' for
                passing in an offset modifier with a value of 8. If the
                suffix is used with a sequential IO pattern, then the value
                For a random read, it would look like 'rw=randread:8' for
                passing in an offset modifier with a value of 8. If the
                suffix is used with a sequential IO pattern, then the value


@@ -391,12 +439,6 @@ randseed=int       Seed the random number generators based on this seed value, to
                If not set, the random sequence depends on the randrepeat
                setting.
 
                If not set, the random sequence depends on the randrepeat
                setting.
 

-use_os_rand=bool Fio can either use the random generator supplied by the OS
-               to generator random offsets, or it can use it's own internal
-               generator (based on Tausworthe). Default is to use the
-               internal generator, which is often of better quality and
-               faster.
-

 fallocate=str  Whether pre-allocation is performed when laying down files.
                Accepted values are:
 
 fallocate=str  Whether pre-allocation is performed when laying down files.
                Accepted values are:
 


@@ -519,7 +561,7 @@ bssplit=str Sometimes you want even finer grained control of the
                while having 90% 4k writes and 10% 8k writes, you would
                specify:
 
                while having 90% 4k writes and 10% 8k writes, you would
                specify:
 

-               bssplit=2k/50:4k/50,4k/90,8k/10
+               bssplit=2k/50:4k/50,4k/90:8k/10

 
 blocksize_unaligned
 bs_unaligned   If this option is given, any byte size value within bsrange
 
 blocksize_unaligned
 bs_unaligned   If this option is given, any byte size value within bsrange


@@ -565,10 +607,20 @@ buffer_compress_chunk=int See buffer_compress_percentage. This
                alternate random and zeroed data throughout the IO
                buffer.
 
                alternate random and zeroed data throughout the IO
                buffer.
 

-buffer_pattern=str     If set, fio will fill the io buffers with this pattern.
-               If not set, the contents of io buffers is defined by the other
-               options related to buffer contents. The setting can be any
-               pattern of bytes, and can be prefixed with 0x for hex values.
+buffer_pattern=str     If set, fio will fill the io buffers with this
+               pattern. If not set, the contents of io buffers is defined by
+               the other options related to buffer contents. The setting can
+               be any pattern of bytes, and can be prefixed with 0x for hex
+               values. It may also be a string, where the string must then
+               be wrapped with "".
+
+dedupe_percentage=int  If set, fio will generate this percentage of
+               identical buffers when writing. These buffers will be
+               naturally dedupable. The contents of the buffers depend on
+               what other buffer compression settings have been set. It's
+               possible to have the individual buffers either fully
+               compressible, or not at all. This option only controls the
+               distribution of unique buffers.

 
 nrfiles=int    Number of files to use for this job. Defaults to 1.
 
 
 nrfiles=int    Number of files to use for this job. Defaults to 1.
 


@@ -694,7 +746,21 @@ ioengine=str       Defines how the job issues io to the file. The following
                                having to go through FUSE. This ioengine
                                defines engine specific options.
 
                                having to go through FUSE. This ioengine
                                defines engine specific options.
 

-                       hdfs    Read and write through Hadoop (HDFS).
+                       libhdfs Read and write through Hadoop (HDFS).
+                               The 'filename' option is used to specify host,
+                               port of the hdfs name-node to connect. This
+                               engine interprets offsets a little
+                               differently. In HDFS, files once created
+                               cannot be modified. So random writes are not
+                               possible. To imitate this, libhdfs engine
+                               expects bunch of small files to be created
+                               over HDFS, and engine will randomly pick a
+                               file out of those files based on the offset
+                               generated by fio backend. (see the example
+                               job file to create such files, use rw=write
+                               option). Please note, you might want to set
+                               necessary environment variables to work with
+                               hdfs/libhdfs properly.

 
                        external Prefix to specify loading an external
                                IO engine object file. Append the engine
 
                        external Prefix to specify loading an external
                                IO engine object file. Append the engine


@@ -764,7 +830,9 @@ number_ios=int      Fio will normally perform IOs until it has exhausted the size
                time (or hits an error condition). With this setting, the
                range/size can be set independently of the number of IOs to
                perform. When fio reaches this number, it will exit normally
                time (or hits an error condition). With this setting, the
                range/size can be set independently of the number of IOs to
                perform. When fio reaches this number, it will exit normally

-               and report status.
+               and report status. Note that this does not extend the amount
+               of IO that will be done, it will only stop fio if this
+               condition is met before other end-of-job criteria.

 
 fsync=int      If writing to a file, issue a sync of the dirty data
                for every number of blocks given. For example, if you give
 
 fsync=int      If writing to a file, issue a sync of the dirty data
                for every number of blocks given. For example, if you give


@@ -1566,7 +1634,9 @@ that defines them is selected.
                address.
 
 [netsplice] port=int
                address.
 
 [netsplice] port=int

-[net] port=int The TCP or UDP port to bind to or connect to.
+[net] port=int The TCP or UDP port to bind to or connect to. If this is used
+with numjobs to spawn multiple instances of the same job type, then this will
+be the starting port number since fio will use a range of ports.

 
 [netsplice] interface=str
 [net] interface=str  The IP address of the network interface used to send or
 
 [netsplice] interface=str
 [net] interface=str  The IP address of the network interface used to send or


@@ -1598,6 +1668,7 @@ that defines them is selected.
 [net] listen   For TCP network connections, tell fio to listen for incoming
                connections rather than initiating an outgoing connection. The
                hostname must be omitted if this option is used.
 [net] listen   For TCP network connections, tell fio to listen for incoming
                connections rather than initiating an outgoing connection. The
                hostname must be omitted if this option is used.

+

 [net] pingpong Normaly a network writer will just continue writing data, and
                a network reader will just consume packages. If pingpong=1
                is set, a writer will send its normal payload to the reader,
 [net] pingpong Normaly a network writer will just continue writing data, and
                a network reader will just consume packages. If pingpong=1
                is set, a writer will send its normal payload to the reader,


@@ -1610,6 +1681,10 @@ that defines them is selected.
                single reader when multiple readers are listening to the same
                address.
 
                single reader when multiple readers are listening to the same
                address.
 

+[net] window_size      Set the desired socket buffer size for the connection.
+
+[net] mss      Set the TCP maximum segment size (TCP_MAXSEG).
+

 [e4defrag] donorname=str
                File will be used as a block donor(swap extents between files)
 [e4defrag] inplace=int
 [e4defrag] donorname=str
                File will be used as a block donor(swap extents between files)
 [e4defrag] inplace=int