fio.git
3 years agodoc: minor consistency and spelling changes
Sitsofe Wheeler [Fri, 27 Jan 2017 09:43:12 +0000 (09:43 +0000)]
doc: minor consistency and spelling changes

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodoc: minor documentation changes
Sitsofe Wheeler [Thu, 26 Jan 2017 17:02:47 +0000 (17:02 +0000)]
doc: minor documentation changes

- Change some parameter types to be time in the HOWTO.
- Put a space before the end of sentence but after links in the README.
- Update the Red Hat fio package information.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoMerge branch 'shm_rm' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 26 Jan 2017 17:07:48 +0000 (10:07 -0700)]
Merge branch 'shm_rm' of https://github.com/sitsofe/fio

3 years agoMerge branch 'sphinx-doc' of https://github.com/termim/fio
Jens Axboe [Thu, 26 Jan 2017 17:04:40 +0000 (10:04 -0700)]
Merge branch 'sphinx-doc' of https://github.com/termim/fio

3 years agoshm: have os remove shared memory if fio dies unexpectedly
Sitsofe Wheeler [Tue, 24 Jan 2017 06:31:58 +0000 (06:31 +0000)]
shm: have os remove shared memory if fio dies unexpectedly

When fio doesn't exit cleanly shared memory segments are left behind on
those platforms where shared memory is supported.

Due to how fio manages shared memory lifetime we can combat the above by
marking the shared memory segment as removed shortly after we attach to
it for the first time if the OS offers both the following (non-standard)
behaviours:

1. A shared memory segment marked for removal with IPC_RMID isn't
   destroyed until all processes have detached from it.
2. After 1. has occurred it is still possible to (re)attach to the
   marked-for-removal shared memory segment.

Linux has always done the above so enable the workaround there. OpenBSD
switched to having the above behaviours in 5.1 (see shmctl in
https://www.openbsd.org/plus51.html ) so add a comment about it in
os/os-openbsd.h . FreeBSD only has the above behaviours if
kern.ipc.shm_allow_removed (introduced in 5.2) is 1 and this only became
the default in FreeBSD 11
(https://svnweb.freebsd.org/base?view=revision&revision=289112 ) so add
a comment in os/os-freebsd.h . DragonFly also has the FreeBSD flag but
at the time of writing it defaults to 0 so add a comment to
os/os-dragonfly.h .

There's a nice (if dated) post discussing how IPC_RMID behaves on
various OSes in https://www.cygwin.com/ml/cygwin/2007-11/msg00078.html
("Re: cygwin stable and cvs snapshot - fork() bug").  While fio doesn't
support the Cygwin "OS", Cygwin itself has the
kern.ipc.shm_allow_removed option in a config file and defaults it to
no.

3 years agoRemove/Move Linux specific sysfs_root field from thread_data
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:11 +0000 (00:13 +0900)]
Remove/Move Linux specific sysfs_root field from thread_data

struct thread_data is better without having a Linux specific field
(even if sysfs_root is simply unused on non Linux environment) given
that other Linux specific code such as diskutil/cgroup/blktrace/etc
are compiled only on Linux.

This commit uses td->files[0]->du->sysfs_root instead of td->sysfs_root,
but results the same from the way diskutil is currently implemented.
Besides being platform independent, it's also better in terms of data
structure since sysfs_root isn't always per-td attribute as mentioned
below.

For example, when N(>1) files are specified via filename= option
(e.g. filename=/dev/sdb:/dev/sdc for N=2), td:files is 1:N, therefore
td:du and td:du->sysfs_root are also 1:N in case of /dev/sdb:/dev/sdc,
however td:td->sysfs_root is obviously 1:1 which in this case means
td->sysfs_root is sysfs root for /dev/sdb (regardless of other files)
from the way td->sysfs_root is set only if it hasn't yet been set.

In other words, if filename=/dev/sdb:/dev/sdc is given, ioscheduler=
option only works against /dev/sdb, which could be different from what
users would expect, but this commit just removes sysfs_root from
struct thread_data without any functional change.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix bad pointer du->sysfs_root
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:10 +0000 (00:13 +0900)]
Fix bad pointer du->sysfs_root

There are two call paths for disk_util_add() (usually the second one)
which assigns ->sysfs_root for a newly allocated disk_util*, but both
temppath/foo are local char[] within their stack, so strdup/free it.

-> find_add_disk_slaves(...)
  -> __init_per_file_disk_util(..., temppath)
    -> disk_util_add(..., path)
      -> du->sysfs_root = path; /* == &temppath[0] */

-> init_per_file_disk_util(...)
  -> __init_per_file_disk_util(..., foo)
    -> disk_util_add(..., path)
      -> du->sysfs_root = path; /* == &foo[0] */

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing free(td->sysfs_root);
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:09 +0000 (00:13 +0900)]
Add missing free(td->sysfs_root);

->sysfs_root for td (not du) is always strdup'd.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove unused disk_util's name field
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:08 +0000 (00:13 +0900)]
Remove unused disk_util's name field

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove unused io_u's priv union field
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:07 +0000 (00:13 +0900)]
Remove unused io_u's priv union field

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix wrong comment on exit condition of threads/processes
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:06 +0000 (00:13 +0900)]
Fix wrong comment on exit condition of threads/processes

74f4b020 in 2013 has changed the exit condition from min to max,
so sync the comment with 74f4b020.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse ARRAY_SIZE()
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:05 +0000 (00:13 +0900)]
Use ARRAY_SIZE()

They can assume ARRAY_SIZE() is defined and usable as far as fio's
internal code is concerned (i.e. FIO_INTERNAL defined).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDefine pointer alignment macro in fio.h
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:04 +0000 (00:13 +0900)]
Define pointer alignment macro in fio.h

Define PTR_ALIGN() as a common utility within the entire fio,
but inside FIO_INTERNAL guard since it could be a common name.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofix manpage heading issue
Mikhail Terekhov [Sat, 21 Jan 2017 02:44:56 +0000 (21:44 -0500)]
fix manpage heading issue

Sphinx skips zero level headins when generating manpages.
Increasing sections depth solves missing headers problem.

3 years agoAdd missing opt/cat group entries
Tomohiro Kusumi [Fri, 20 Jan 2017 17:46:43 +0000 (02:46 +0900)]
Add missing opt/cat group entries

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoChange .category of cpuio to FIO_OPT_C_ENGINE
Tomohiro Kusumi [Fri, 20 Jan 2017 17:46:42 +0000 (02:46 +0900)]
Change .category of cpuio to FIO_OPT_C_ENGINE

This is supposed to be ENGINE instead of GENERAL.

 # grep "\.category" engines -rI | grep -v ENGINE
 engines/cpu.c:          .category = FIO_OPT_C_GENERAL,
 engines/cpu.c:          .category = FIO_OPT_C_GENERAL,
 engines/cpu.c:          .category = FIO_OPT_C_GENERAL,

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'sphinx-doc' of https://github.com/termim/fio
Jens Axboe [Fri, 20 Jan 2017 05:01:48 +0000 (22:01 -0700)]
Merge branch 'sphinx-doc' of https://github.com/termim/fio

3 years agoremove documentation output in clean
Mikhail Terekhov [Fri, 20 Jan 2017 04:50:11 +0000 (23:50 -0500)]
remove documentation output in clean

3 years agoMerge branch 'sphinx-doc' of https://github.com/termim/fio
Jens Axboe [Fri, 20 Jan 2017 02:50:09 +0000 (19:50 -0700)]
Merge branch 'sphinx-doc' of https://github.com/termim/fio

3 years agoignore documentation output directory
Mikhail Terekhov [Fri, 20 Jan 2017 02:08:39 +0000 (21:08 -0500)]
ignore documentation output directory

3 years agoChange td_var() to take void* for the first arg
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:27 +0000 (02:38 +0900)]
Change td_var() to take void* for the first arg

Callers don't always pass thread_options* to td_var(),
nor is it dependent on a certain structure type,
so just take void* without pointless cast.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMove options_mem_dupe() to parse.c
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:26 +0000 (02:38 +0900)]
Move options_mem_dupe() to parse.c

This function should be in parse.c with fio_option* argument first.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRefactor fio_show_ioengine_help()
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:25 +0000 (02:38 +0900)]
Refactor fio_show_ioengine_help()

Since td is there only to call load_ioengine(),
ioengine ops doesn't need to be refered to as td.io_ops.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix typo for "job" in plural
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:24 +0000 (02:38 +0900)]
Fix typo for "job" in plural

It seems this message was meant to be 1 or more "job".

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRename FIO_TYPE_BD to FIO_TYPE_BLOCK
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:23 +0000 (02:38 +0900)]
Rename FIO_TYPE_BD to FIO_TYPE_BLOCK

Since character device is FIO_TYPE_CHAR, BLOCK (or BLK) makes
better sense for block device.

This may break external ioengines on compile-time, but ioengines
usually don't refer to file type which are regfile/blk/chr/pipe
(which I believe makes sense to commit this change as there's
been changes that break not only api but also abi).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'sphinx-doc' of https://github.com/termim/fio into sphinx
Jens Axboe [Wed, 18 Jan 2017 16:51:33 +0000 (09:51 -0700)]
Merge branch 'sphinx-doc' of https://github.com/termim/fio into sphinx

3 years agoinit: fix double free of pid_file
Jens Axboe [Wed, 18 Jan 2017 15:18:28 +0000 (08:18 -0700)]
init: fix double free of pid_file

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRevert README and HOWTO files renaming.
Mikhail Terekhov [Wed, 18 Jan 2017 13:56:35 +0000 (08:56 -0500)]
Revert README and HOWTO files renaming.

github doesn't support Sphinx extencions to ReStructured text format.

3 years agoadd .rst suffix to README and HOWTO
Mikhail Terekhov [Wed, 18 Jan 2017 02:01:50 +0000 (21:01 -0500)]
add .rst suffix to README and HOWTO

3 years agoConvert documentation to reStructured text.
Mikhail Terekhov [Mon, 26 Dec 2016 03:17:57 +0000 (22:17 -0500)]
Convert documentation to reStructured text.

Reshuffle contents of README and HOWTO
Add some job file examples to HTML docs
Automatically generate fio.1 manpage

3 years agorun sphinx-quickstart
Mikhail Terekhov [Mon, 26 Dec 2016 03:13:57 +0000 (22:13 -0500)]
run sphinx-quickstart

3 years agoFio 2.17 fio-2.17
Jens Axboe [Tue, 17 Jan 2017 15:51:31 +0000 (08:51 -0700)]
Fio 2.17

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDrop crc32c-arm64 option
wei xiao [Fri, 6 Jan 2017 02:37:03 +0000 (02:37 +0000)]
Drop crc32c-arm64 option

Crc32c will use ARM64 hardware assist, if available.

Jira: ENTLLT-359
Change-Id: Ibdf88f14bd4669ed68a93078c43eb5c8412da543

3 years agoSupport zlib in the Windows build (enabled latency histogram logging)
Rebecca Cran [Mon, 9 Jan 2017 21:45:38 +0000 (21:45 +0000)]
Support zlib in the Windows build (enabled latency histogram logging)

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agopmemblk: Clarify fsize is in MiB not MB
Robert Elliott [Tue, 10 Jan 2017 21:21:27 +0000 (15:21 -0600)]
pmemblk: Clarify fsize is in MiB not MB

Clarify that the FSIZE argument is in MiB (power of 2) rather
than MB (power of 10) in comments and variable names.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agopmemblk, dev-dax: clean up error logs
Robert Elliott [Tue, 10 Jan 2017 21:21:26 +0000 (15:21 -0600)]
pmemblk, dev-dax: clean up error logs

Print the errno string rather than the number, and prefix
all log_err prints with pmemblk or dev-dax rather than fio.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agopmemblk, dev-dax: Update descriptions
Robert Elliott [Tue, 10 Jan 2017 21:21:25 +0000 (15:21 -0600)]
pmemblk, dev-dax: Update descriptions

Make the descriptions of the pmemblk and dev-dax ioengines
more symmetric.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agopmemblk, dev-dax: load libpmem and libpmemblk at startup
Robert Elliott [Tue, 10 Jan 2017 21:21:24 +0000 (15:21 -0600)]
pmemblk, dev-dax: load libpmem and libpmemblk at startup

The pmemblk and dev-dax ioengines were loading libpmem.so and
libpmemblk.so using dlopen() at runtime.

Although the upstream nvml Makefile installs a libpmem.so
symbolic link, some of the distros (e.g. SUSE Linux Enterprise
Server 12 SP2) are just installing so.1 symbolic links.  So,
fio fails to find the libpmem.so and libpmemblk.so library files.

http://www.ibm.com/developerworks/linux/library/l-shlibs/index.html
says applications should always load the versioned filenames to
avoid compatibility issues; the non-versioned filenames are just
for the compiler and linker.

Change ./configure to pass -lpmem and -lpmemblk to the compiler
so the fio binary loads the versioned filename at startup like
the other libraries, rather than open the non-versioned filename
with dlopen() during runtime.

This way they show up in ldd:
$ ldd fio
        linux-vdso.so.1 (0x00007ffc290aa000)
        libpmemblk.so.1 => /lib64/libpmemblk.so.1 (0x00007f65b5bc4000)
        libpmem.so.1 => /lib64/libpmem.so.1 (0x00007f65b59be000)
        libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f65b57b3000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f65b55ab000)
        libaio.so.1 => /lib64/libaio.so.1 (0x00007f65b53a9000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f65b5191000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f65b4e88000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f65b4c6a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f65b4a66000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f65b46a0000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f65b449b000)
        /lib64/ld-linux-x86-64.so.2 (0x0000559f4d872000)

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoPython style/portability fix
Tomohiro Kusumi [Tue, 10 Jan 2017 17:39:47 +0000 (02:39 +0900)]
Python style/portability fix

In practice, one would normally explicitly derive a class from
object class (was called new-style class back then).
https://docs.python.org/release/2.5.2/ref/node33.html
https://wiki.python.org/moin/NewClassVsClassicClass

print needs parentheses for portability with Python3.x.

xrange() only exists in Python2.x (i.e. breaks on Python3.x).
Using range() (which pre-allocates a whole list in Python2.x)
won't be a problem unless len(averages) is huge enough to give
any pressure to vm subsystem.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix README regarding fio snapshots
Tomohiro Kusumi [Tue, 10 Jan 2017 17:39:46 +0000 (02:39 +0900)]
Fix README regarding fio snapshots

As of now, fio-git-*.tar.{gz,bz2} are git repositories,
but fio-`version`.tar.gz are archives without .git/ meta data.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing trailing \n in dprint()
Tomohiro Kusumi [Tue, 10 Jan 2017 17:39:45 +0000 (02:39 +0900)]
Add missing trailing \n in dprint()

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoPartly revert 8172fe97 in 2008 (A few debug debug log fixes)
Tomohiro Kusumi [Tue, 10 Jan 2017 17:39:44 +0000 (02:39 +0900)]
Partly revert 8172fe97 in 2008 (A few debug debug log fixes)

Not sure what the point of 8172fe97 was, but shouldn't all the
prints within a single line be dprint() (or all be log_info()) ?

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix compile time errors for skeleton_external ioengine
Tomohiro Kusumi [Mon, 9 Jan 2017 14:42:03 +0000 (23:42 +0900)]
Fix compile time errors for skeleton_external ioengine

though this file is not compiled by default.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMention sg ioengine requires filename option
Tomohiro Kusumi [Mon, 9 Jan 2017 14:42:02 +0000 (23:42 +0900)]
Mention sg ioengine requires filename option

similar to the way net ioengine documents filename option.

Also bring in a file type check from fio_sgio_type_check()
to avoid pointless read capacity error against regfiles.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoNon functional fixup for 16 bytes read capacity for sg ioengine
Tomohiro Kusumi [Mon, 9 Jan 2017 14:42:01 +0000 (23:42 +0900)]
Non functional fixup for 16 bytes read capacity for sg ioengine

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix conditional/message for max lba for sg ioengine
Tomohiro Kusumi [Mon, 9 Jan 2017 14:42:00 +0000 (23:42 +0900)]
Fix conditional/message for max lba for sg ioengine

Whether sg ioengine uses 10 or 16 bytes read/write command (by 5ad7be56)
is determined by ->prep() depending on lba to read/write vs 0xFFFFFFFFULL,
so let the message show it depends.

Also remove unused sgio_data::max_lba which is essentially the same
(divide by bs minus 1) as fio_file::real_file_size.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoStaticize pmemblk ioengine_ops
Tomohiro Kusumi [Mon, 9 Jan 2017 14:41:59 +0000 (23:41 +0900)]
Staticize pmemblk ioengine_ops

Inbox engines can have static ops.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoBring in additional sg error cases from Linux kernel (and sg3) header
Tomohiro Kusumi [Mon, 9 Jan 2017 14:41:58 +0000 (23:41 +0900)]
Bring in additional sg error cases from Linux kernel (and sg3) header

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix comment on SCSI commands
Tomohiro Kusumi [Fri, 6 Jan 2017 18:15:18 +0000 (03:15 +0900)]
Fix comment on SCSI commands

[0] is service action (unless this is intentional).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove doubled ; in .c
Tomohiro Kusumi [Fri, 6 Jan 2017 18:15:17 +0000 (03:15 +0900)]
Remove doubled ; in .c

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: cleaner gcd()
Jens Axboe [Thu, 5 Jan 2017 17:38:41 +0000 (10:38 -0700)]
init: cleaner gcd()

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDon't malloc ioengine_ops for cpp_null
Tomohiro Kusumi [Thu, 5 Jan 2017 17:32:17 +0000 (02:32 +0900)]
Don't malloc ioengine_ops for cpp_null

fio assumes opses aren't dynamically allocated which means malloc'd
engine here never gets freed.

According to 46a67478 and a8075704, the point of get_ioengine() is
how non-static ops gets initialized by g++, but not how space for ops
gets allocated, thus it should just define a static global variable
within "C" scope and pass the address of ops to a caller via **.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix invalid ioengine initialization for cpp_null
Tomohiro Kusumi [Thu, 5 Jan 2017 17:32:16 +0000 (02:32 +0900)]
Fix invalid ioengine initialization for cpp_null

->name is a pointer now, so it can't be strcpy'd without a valid
address set to ->name.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: fill in vc->name for pattern verify
Jens Axboe [Thu, 5 Jan 2017 17:33:23 +0000 (10:33 -0700)]
verify: fill in vc->name for pattern verify

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: ensure that verify_interval is always a factor of min/max bs
Jens Axboe [Thu, 5 Jan 2017 17:26:08 +0000 (10:26 -0700)]
verify: ensure that verify_interval is always a factor of min/max bs

A previous commit force aligned the buffer length to the
verify_interval, but that breaks if max_bs isn't a multiple
of min_bs. If that isn't the case, we can align the block size
bigger than the IO unit really is. This causes corruption and
failures.

Fixes: a9f70b1f5087 ("Make sure io_u->buflen is aligned to the verify_interval")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove '--runtime' command line option
Jens Axboe [Thu, 5 Jan 2017 16:49:06 +0000 (09:49 -0700)]
Remove '--runtime' command line option

It conflicts with the job file option, and we (in most cases) set
the default before we have parsed it, depending on where in the
command line the --runtime option is added.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: use log_verify_failure() for pattern verifies
Jens Axboe [Thu, 5 Jan 2017 16:38:50 +0000 (09:38 -0700)]
verify: use log_verify_failure() for pattern verifies

Similar to the other verify types, otherwise we don't get
the absolute offset logged.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoStyle cleanups for arm crc32c hw support
Jens Axboe [Thu, 5 Jan 2017 02:44:35 +0000 (19:44 -0700)]
Style cleanups for arm crc32c hw support

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd arm64 hardware assisted crc32c support
wei xiao [Thu, 5 Jan 2017 02:32:40 +0000 (10:32 +0800)]
Add arm64 hardware assisted crc32c support

HW assisted crc32c get ~ x39.6 speedups on ARM Cortex-A57@2GHz

Auto-detect whether to use hw assisted crc32c.
If the hardware assisted crypto is available, always use it.
Otherwise, fallback to software.

3 years agoAdd missing .help string for io_size option
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:11 +0000 (23:40 +0900)]
Add missing .help string for io_size option

fio --cmdhelp currently prints "(null)" for io_size.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix README - change just type "configure" to "./configure"
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:10 +0000 (23:40 +0900)]
Fix README - change just type "configure" to "./configure"

Literally typing "configure" may or may not work depending on visible paths.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd BSD package/building info to README
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:09 +0000 (23:40 +0900)]
Add BSD package/building info to README

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix canonical name for runtime/timeout option
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:08 +0000 (23:40 +0900)]
Fix canonical name for runtime/timeout option

.name should be set to "runtime" than "timeout".
fio_options[] as well as the existing documentation consider
"runtime" as an option name, and "timeout" as an alias.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoSync README with fio usage output
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:07 +0000 (23:40 +0900)]
Sync README with fio usage output

Some of the commits only update one of these.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing trailing \n in log_err/info()
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:06 +0000 (23:40 +0900)]
Add missing trailing \n in log_err/info()

Callers need trailing \n in case of fwrite(3) (if not syslog(3)).

These are the ones that seem to need trailing \n not followed
by another log function call as consequence.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix trivial calloc argument order
Tomohiro Kusumi [Tue, 3 Jan 2017 14:40:05 +0000 (23:40 +0900)]
Fix trivial calloc argument order

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agot/stest: remove old test
Jens Axboe [Tue, 3 Jan 2017 01:21:14 +0000 (18:21 -0700)]
t/stest: remove old test

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDocumentation for IEC binary and SI decimal prefix handling
Robert Elliott [Thu, 22 Dec 2016 20:50:51 +0000 (14:50 -0600)]
Documentation for IEC binary and SI decimal prefix handling

Documentation for the previous patch improving IEC and SI
decimal prefix handling (e.g., KiB, MiB, GiB vs. KB, MB, GB).

Includes updates to int, kb_base, and the parameters using
int to express quantities of data (e.g., bs, ba, bsrange, bssplit).

Includes updates to parameters that accept comma-separated
values (usually ints) for reads, writes, and trims; trims were
often not mentioned.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoImprove IEC binary and SI decimal prefix handling
Robert Elliott [Thu, 22 Dec 2016 20:50:50 +0000 (14:50 -0600)]
Improve IEC binary and SI decimal prefix handling

Use kb_base=1000 to follow international standards for unit prefixes.

To specify power-of-10 decimal values defined in the International
System of Units (SI):
Ki means kilo (K) or 1000
Mi means mega (M) or 1000**2
Gi means giga (G) or 1000**3
Ti means tera (T) or 1000**4
Pi means peta (P) or 1000**5

To specify power-of-2 binary values defined in IEC 80000-13:
k means kibi (Ki) or 1024
M means mebi (Mi) or 1024**2
G means gibi (Gi) or 1024**3
T means tebi (Ti) or 1024**4
P means pebi (Pi) or 1024**5

For example, this specifies a blocksize of 4096 bytes:
kb_base=1000
bs=4KiB

With kb_base=1024 (the default), the unit prefixes are opposite from
those specified in the SI and IEC 80000-13 standards to provide
compatibility with old scripts.  For example, this specifies a
blocksize of 4096 bytes:
kb_base=1024
bs=4K

For outputs printing quantities and bandwidths:
* eta stats only use the preferred prefix
* final stats include both (non-preferred prefix in parenthesis)
* in gfio, all windows include both

Text outputs are rearranged to try to obviously break any scripts
parsing the output rather than silently confuse them.  The terse
and json outputs, which are intended for parsing, are unchanged.

Old:
Jobs: 576 (f=576), CR=86.4GB/576MB KB/s: [w(288),r(288)] [0.0% done] [33884MB/27114MB/0KB /s] [8471K/6778K/0 iops] [eta 06h:59m:57s]
   read: io=363338MB, bw=34014MB/s, iops=8304.3K, runt= 10682msec
  write: io=282447MB, bw=26225MB/s, iops=6402.7K, runt= 10770msec
   READ: io=363338MB, aggrb=34014MB/s, minb=34014MB/s, maxb=34014MB/s, mint=10682msec, maxt=10682msec
  WRITE: io=282447MB, aggrb=26225MB/s, minb=26225MB/s, maxb=26225MB/s, mint=10770msec, maxt=10770msec

New:
Jobs: 576 (f=576), 590MB/s-89.2GB/s: [w(288),r(288)][0.0%][r=34.2GB/s,w=26.3GB/s][r=8542k,w=6572k IOPS][eta 06h:59m:55s]
   read: IOPS=8362k, BW=34.3GB/s (31.1GiB/s)(665GB/19417msec)
  write: IOPS=6423k, BW=26.4GB/s (24.6GiB/s)(511GB/19431msec)
   READ: bw=34.3GB/s (31.1GiB/s), 34.3GB/s-34.3GB/s (31.1GiB/s-31.1GiB/s), io=665GB (619GiB), run=19417-19417msec
  WRITE: bw=26.4GB/s (24.6GiB/s), 26.4GB/s-26.4GB/s (24.6GiB/s-24.6GiB/s), io=511GB (476GiB), run=19431-19431msec

Documentation changes are in a subsequent patch.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoClean up unit prefixes for binary multiples in comments and prints
Robert Elliott [Thu, 22 Dec 2016 20:50:49 +0000 (14:50 -0600)]
Clean up unit prefixes for binary multiples in comments and prints

Change comments and prints that are referring to values based on
binary multiples to use IEC 80000-13 prefixes for binary
multiples rather than SI prefixes for decimal multiples (e.g.,
KiB, MiB, and GiB rather than KB, MB, and GB).

Includes a fix for the rate_min print that was printing "KB/s"
although it was really printing bytes per second.

Includes a fix for goptions confusing the PB and TB units (while
changing them to PiB and TiB).

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoConvert group_run_stats to use bytes instead of KiB/KB
Robert Elliott [Thu, 22 Dec 2016 20:50:48 +0000 (14:50 -0600)]
Convert group_run_stats to use bytes instead of KiB/KB

In group_run_stats, convert these fields:
* io_kb (bytes transferred)
* agg (aggregate average bandwidth)
* min_bw
* max_bw

from tracking either KiB or KB depending on kb_base to simply tracking bytes.
Rename io_kb to iobytes to match this new meaning.

This will simplify computing quantities and bandwidth rates with both IEC
binary and SI decimal unit prefixes (e.g., GiB/s and GB/s) in future patches.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogclient: Use proper time units in latency buckets chart
Robert Elliott [Thu, 22 Dec 2016 20:50:47 +0000 (14:50 -0600)]
gclient: Use proper time units in latency buckets chart

In the Results page Latency buckets chart, use us and ms rather
than u and s for the time units.  This matches the use of s for
seconds that is already being done.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogclient: Delete unused code
Robert Elliott [Thu, 22 Dec 2016 20:50:46 +0000 (14:50 -0600)]
gclient: Delete unused code

Delete two #if 0 blocks from pre-2012 that have old calls to
num2str without the current number of arguments and hardcode
"KB/s" units.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoLine up colons across read, write, and trim thread stats
Robert Elliott [Thu, 22 Dec 2016 20:50:45 +0000 (14:50 -0600)]
Line up colons across read, write, and trim thread stats

Line up the colons across the read, write, and trim
thread stats rows.

Example:
drive_0: (groupid=0, jobs=576): err= 0: pid=42198: Wed Dec 21 19:51:35 2016
  read : io=420642MB, bw=33953MB/s, iops=8289.3K, runt= 12389msec
  write: io=325276MB, bw=26177MB/s, iops=6390.9K, runt= 12426msec
  trim: io=0MB, bw=0MB/s, iops=0, runt= 12426msec

becomes:
drive_0: (groupid=0, jobs=576): err= 0: pid=42198: Wed Dec 21 19:51:35 2016
   read: io=420642MB, bw=33953MB/s, iops=8289.3K, runt= 12389msec
  write: io=325276MB, bw=26177MB/s, iops=6390.9K, runt= 12426msec
   trim: io=0MB, bw=0MB/s, iops=0, runt= 12426msec

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix unit_base kb_base mixup in thread option conversion functions
Robert Elliott [Thu, 22 Dec 2016 20:50:44 +0000 (14:50 -0600)]
Fix unit_base kb_base mixup in thread option conversion functions

Don't set unit_base, which is 1 or 8, to kb_base, which is 1000
or 1024, during the convert functions used by the initial cconv test.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agotests, profiles: Use IEC prefixes for binary multiples
Robert Elliott [Thu, 22 Dec 2016 20:50:43 +0000 (14:50 -0600)]
tests, profiles: Use IEC prefixes for binary multiples

Use IEC 80000-13 prefixes for binary multiples of bytes
(KiB, MiB, etc.) rather than SI prefixes for decimal multiples
of bytes (KB, MB, etc.) in places in which the code has fixed
divide by 1024, 1024*1024, >>20, etc.  and is clearly always
using binary multiples.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDocument trim workload choices and other nits
Robert Elliott [Thu, 22 Dec 2016 20:50:42 +0000 (14:50 -0600)]
Document trim workload choices and other nits

Add some missing trim workloads (rw=trim, randtrim) to HOWTO,
improve some wording about trims in HOWTO, fio.1 (the manpage),
and the help text in options.c.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogfio: Improve IOPS textbox labels
Robert Elliott [Thu, 22 Dec 2016 20:50:41 +0000 (14:50 -0600)]
gfio: Improve IOPS textbox labels

Change the three textboxes labeled IOPS to Read IOPS, Write IOPS,
and Trim IOPS, add Trim to the Blocksize label, and fix Steve's
last name in the Credits.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAvoid using units in option defaults
Robert Elliott [Thu, 22 Dec 2016 20:50:40 +0000 (14:50 -0600)]
Avoid using units in option defaults

Change option defaults and some generated option strings to specify
the number of bytes without units, so they are immune to the
interpretation of the units (e.g. is 1M interpreted as
1048576 or 1000000).

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUpdate Windows maintainer contact details
Rebecca Cran [Tue, 3 Jan 2017 00:20:06 +0000 (17:20 -0700)]
Update Windows maintainer contact details

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: mark steadystate option parents
Jens Axboe [Thu, 29 Dec 2016 16:07:29 +0000 (09:07 -0700)]
options: mark steadystate option parents

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agorbd: style fixups
Jens Axboe [Sat, 24 Dec 2016 02:54:47 +0000 (19:54 -0700)]
rbd: style fixups

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agorbd: remove duplicate _fio_rbd_connect
Pan Liu [Fri, 23 Dec 2016 16:35:55 +0000 (00:35 +0800)]
rbd: remove duplicate _fio_rbd_connect

Signed-off-by: Pan Liu <pan.liu@istuary.com>
3 years agofio: Add support for auto detect dev-dax and libpmemblk engines
Dave Jiang [Tue, 20 Dec 2016 23:31:45 +0000 (16:31 -0700)]
fio: Add support for auto detect dev-dax and libpmemblk engines

Added support code in configure to detect whether libpmem and libpmemblk
are installed. dev-dax engine is enabled if libpmem is installed.
pmemblk engine is installed if libpmem and libpmemblk are installed.
A --disable-pmem flag is added to explicitly disable the libpmem based
engines even if the libs are detected.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix compile on OSX
Jens Axboe [Tue, 20 Dec 2016 21:58:06 +0000 (14:58 -0700)]
Fix compile on OSX

And probably other platforms, that don't have the posix fadvise
hints.

Fixes: ecb2083d3955 ("options: full control of fadvise hinting")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFio 2.16 fio-2.16
Jens Axboe [Tue, 20 Dec 2016 06:12:56 +0000 (23:12 -0700)]
Fio 2.16

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: full control of fadvise hinting
Jens Axboe [Mon, 19 Dec 2016 20:58:42 +0000 (13:58 -0700)]
options: full control of fadvise hinting

Previously fio supported only 0/1 for this hint. 0 would not do any
hints, but 1 (the default) would hint based on what the fio workload
was - fio would tell the kernel it was a random workload, if it was
random, or a sequential for a sequential workload.

Now fio also supports setting what the hint type should be explicitly,
by allowing a setting of 'random' or 'sequential' for full control
of the hint.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: add additional support for dev-dax ioengine
Dave Jiang [Fri, 16 Dec 2016 22:59:06 +0000 (15:59 -0700)]
fio: add additional support for dev-dax ioengine

Fixup options.c to add the dev-dax to ioengine list. Update fio.1 and
HOWTO as well. Also adding example fio job file.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge https://bitbucket.org/vincentfu/fio-steadystate into steady-state-2
Jens Axboe [Fri, 16 Dec 2016 17:58:55 +0000 (10:58 -0700)]
Merge https://bitbucket.org/vincentfu/fio-steadystate into steady-state-2

3 years agouse type double in creating floating point JSON values
Vincent Fu [Tue, 13 Dec 2016 20:40:46 +0000 (15:40 -0500)]
use type double in creating floating point JSON values

struct json_value uses a double to store floating point values. Have json_create_value_float also take a double (instead of a float) as an argument.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: move _FORTIFY_SOURCE to only when optimization is turned on
Dave Jiang [Thu, 15 Dec 2016 21:51:28 +0000 (14:51 -0700)]
fio: move _FORTIFY_SOURCE to only when optimization is turned on

When compile with debug, without the proper -O flag the compiler complains:
/usr/include/features.h:331:4: warning: #warning _FORTIFY_SOURCE requires
compiling with optimization (-O) [-Wcpp]
#  warning _FORTIFY_SOURCE requires compiling with optimization (-O)
Moving _FORTIFY_SOURCE to optimized compile option.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: add device dax engine
Dave Jiang [Wed, 7 Dec 2016 22:54:30 +0000 (15:54 -0700)]
fio: add device dax engine

Adding an ioengine that access DAX device directly. Very similar to the
mmap engine. Uses the libpmem's pmem_memcpy_persist() call for writes.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoChange misleading error message for invalid size= value
Jakub Sitnicki [Tue, 13 Dec 2016 16:27:24 +0000 (17:27 +0100)]
Change misleading error message for invalid size= value

The error message refers to the IO size, yet another parameter, but we
actually compare the size parameter value against the block size.

Change the message to reflect the code, and make it easier for the user
to see what values are we comparing.

Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge git://git.kernel.dk/fio into steady-state
Vincent Fu [Tue, 13 Dec 2016 14:20:52 +0000 (09:20 -0500)]
Merge git://git.kernel.dk/fio into steady-state

3 years agoiolog: add support for replay_scale option
Sitsofe Wheeler [Sat, 3 Dec 2016 09:24:30 +0000 (09:24 +0000)]
iolog: add support for replay_scale option

Enable replay_scale (and replay_align) when playing back fio's log
format by making ipo_bytes_align() available from a header.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoblktrace: Fix replay_align 32 bit truncation
Sitsofe Wheeler [Sat, 3 Dec 2016 09:31:34 +0000 (09:31 +0000)]
blktrace: Fix replay_align 32 bit truncation

When using replay_align the aligned offset was truncated when the
original offset was greater than 2^32. Fix this by forcing the
calculation to happen with 64 bits.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoiolog: Ignore re-add/re-open with replay_redirect
Sitsofe Wheeler [Mon, 12 Dec 2016 12:11:39 +0000 (12:11 +0000)]
iolog: Ignore re-add/re-open with replay_redirect

The HOWTO entry for replay_redirect says:
"Replay_redirect causes all IOPS to be replayed onto
the single specified device regardless of the device it was
recorded from. i.e. replay_redirect=/dev/sdc would cause all
IO in the blktrace or iolog to be replayed onto /dev/sdc.
This means multiple devices will be replayed onto a single
device, if the trace contains multiple devices."

Unfortunately trying to replay the following iolog (saved as
redirtest.fiolog):
fio version 2 iolog
/dev/sdb add
/dev/sdc add
/dev/sdb open
/dev/sdc open
/dev/sdb read 0 512
/dev/sdc read 512 512
/dev/sdb close
/dev/sdc close

with an fio line like
fio --name=redirtest --replay_redirect=/dev/nullb0 --read_iolog=redirtest.fiolog
winds up hitting this assertion:
fio: ioengines.c:413: td_io_open_file: Assertion `!fio_file_open(f)' failed.
which occurs because after redirection is applied /dev/nullb0 is added and
opened twice.

Fix this by ignoring subsequent adds or opens for the same file during
iolog replay setup when replay_redirect is in effect.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agosteadystate: make test script work better under Windows
Vincent Fu [Fri, 9 Dec 2016 20:54:32 +0000 (15:54 -0500)]
steadystate: make test script work better under Windows

3 years agosteadystate: make file for read testing optional on posix systems
Vincent Fu [Fri, 9 Dec 2016 18:36:25 +0000 (13:36 -0500)]
steadystate: make file for read testing optional on posix systems

3 years agosteadystate: improve output of test script
Vincent Fu [Thu, 8 Dec 2016 20:21:51 +0000 (15:21 -0500)]
steadystate: improve output of test script