fio.git
3 years agoio_u: don't add slat samples if we are in ramp time
Jens Axboe [Tue, 7 Mar 2017 17:18:53 +0000 (10:18 -0700)]
io_u: don't add slat samples if we are in ramp time

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: fix overflow trying to use 'd' suffix
Sitsofe Wheeler [Fri, 24 Feb 2017 01:38:18 +0000 (01:38 +0000)]
fio: fix overflow trying to use 'd' suffix

Fix overflow that happened when using the 'd' time suffix on platforms
with 32 bit longs which led to a time of less than 10 minutes being
used.

3 years agoFio 2.18 fio-2.18
Jens Axboe [Thu, 23 Feb 2017 15:44:32 +0000 (08:44 -0700)]
Fio 2.18

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: disable compile time asserts for !opt and !static_assert
Jens Axboe [Wed, 22 Feb 2017 16:00:06 +0000 (09:00 -0700)]
configure: disable compile time asserts for !opt and !static_assert

If we don't have _Static_assert(), disable compile time
assertion checks if optimizations are disabled.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: use 'bool' for get_new_job()
Jens Axboe [Wed, 22 Feb 2017 03:53:48 +0000 (20:53 -0700)]
init: use 'bool' for get_new_job()

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: exit on failure to add all jobs
Jens Axboe [Tue, 21 Feb 2017 15:16:21 +0000 (08:16 -0700)]
init: exit on failure to add all jobs

It's too late to set do_exit here, so just exit manually if we
fail adding jobs.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoappveyor: add CI building of the Windows version of fio
Sitsofe Wheeler [Sun, 19 Feb 2017 08:52:28 +0000 (08:52 +0000)]
appveyor: add CI building of the Windows version of fio

Add AppVeyor continuous integration build of the Windows port. This also
builds the Windows installer and the resulting MSI can be downloaded.

Note that AppVeyor limits you to one build at a time.

3 years agoRevert "configure: Drop default CONFIG_LITTLE_ENDIAN for Cygwin"
Tomohiro Kusumi [Sun, 19 Feb 2017 18:24:44 +0000 (20:24 +0200)]
Revert "configure: Drop default CONFIG_LITTLE_ENDIAN for Cygwin"

This reverts commit d0f2828f6e49613d944a980e018cbfc02901264b
to unbreak on Cygwin.

As mentioned at below, it also needed to remove "exit 0",
but I'd rather like to revert this commit for now since there are
some other small niche to be fixed if we remove "exit 0".

https://github.com/axboe/fio/issues/314

The orginal commit was to get rid of redundant config setup on
endianness on Cygwin, but it turned out many of the unconditional
output_sym() calls in this CYGWIN*) case were done under Cygwin
specific way without going through regular configure procedure
(see "Forcing known good options on Windows" part for details),
although there are some unconditional output_sym() calls that need
to be there (e.g. CONFIG_WINDOWSAIO).

Since fixing endianness alone (by removing "exit 0") doesn't
have much meaning as a whole, it makes more sense to just revert
this for the time being.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoSilence Cygwin warning "'vsprintf_s' redeclared without dllimport..."
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:22 +0000 (20:22 +0200)]
Silence Cygwin warning "'vsprintf_s' redeclared without dllimport..."

Silence a compiler warning which used to didn't show up with some
old version of Cygwin. ad9c0fbc in 2012 added this, but it seems
this prototype should have been declared by an appropriate header
that's already been included here, so drop this prototype.
It also sounds what this message means according to Google.

Feel free to drop this if this is really needed for some reason.

  os/windows/posix.c:43:5: warning: 'vsprintf_s' redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
   int vsprintf_s(
       ^

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove irrelevant Cygwin config flag CONFIG_FADVISE
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:21 +0000 (20:22 +0200)]
Remove irrelevant Cygwin config flag CONFIG_FADVISE

This is unused and nothing happens regardless of this flag being
set on Cygwin. The one for POSIX fadvise is CONFIG_POSIX_FADVISE.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd details of file number/size related options to HOWTO
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:20 +0000 (20:22 +0200)]
Add details of file number/size related options to HOWTO

File number/size related options for each job/thread are difficult
to understand without looking at source.

fio(1) hasn't been updated (yet). There's already quite significant
amount of diff between these two, and people are basically only
updating HOWTO.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAvoid irrelevant "offset extend ends" error message
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:19 +0000 (20:22 +0200)]
Avoid irrelevant "offset extend ends" error message

When ->real_file_size is -1 due to unsupported file type/etc,
"if (f->file_offset > f->real_file_size)" conditional is always true
(unless ->file_offset is also negative which is an another problem),
and results in irrelevant error message. To avoid this, just return
1 without showing the irrelevant error message and then return 1.

It's still possible that char_size() returns with ->real_file_size
set to -1 (though platform dependent), and the irrelevant error
message be shown.

Also this message itself isn't likely to show any relevant info
due to the timing it's been called as mentioned in a comment.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRename thread_options' ->io_limit to io_size
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:18 +0000 (20:22 +0200)]
Rename thread_options' ->io_limit to io_size

This is less confusing since io_size is the option name itself,
and not much reason to use io_limit which is an alias of io_size.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRevert "Always set ->real_file_size to -1 when failed to get file size"
Jens Axboe [Fri, 17 Feb 2017 20:30:43 +0000 (13:30 -0700)]
Revert "Always set ->real_file_size to -1 when failed to get file size"

This reverts commit 34c9f82301a584899a9e65e4b9f56d98c38a52e7.

It breaks a very basic use case, ala:

./fio --ioengine=libaio --rw=randread --numjobs=1 --iodepth=256 --bs=1M --size=10G --direct=1 --filename=/tmp/test100 --name=test

if /tmp/test100 doesn't already exist.

3 years agoMerge branch 'travis_osx' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 16 Feb 2017 16:00:08 +0000 (09:00 -0700)]
Merge branch 'travis_osx' of https://github.com/sitsofe/fio

3 years agotravis: add OS X builds
Sitsofe Wheeler [Wed, 15 Feb 2017 18:48:10 +0000 (18:48 +0000)]
travis: add OS X builds

Add two enabled OS X builds (Travis' default OS X and macOS 10.12 with
Xcode 8.2) and one disabled OS X build (OS X 10.11 with the 10.12 SDK
but targeting 10.11).

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agotravis: prepare for additional builds
Sitsofe Wheeler [Wed, 15 Feb 2017 18:44:48 +0000 (18:44 +0000)]
travis: prepare for additional builds

Prepare the .travis.yml file to accept new builds by making the Linux
OS choice more explicit and protecting some Linux specific steps.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agotravis: run parallel make jobs
Sitsofe Wheeler [Mon, 6 Feb 2017 07:00:41 +0000 (07:00 +0000)]
travis: run parallel make jobs

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoMakefile: add test using the null ioengine
Sitsofe Wheeler [Sat, 11 Feb 2017 10:05:15 +0000 (10:05 +0000)]
Makefile: add test using the null ioengine

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoExplicitly check td_trim(td) for the direction of next io_u
Tomohiro Kusumi [Wed, 15 Feb 2017 20:40:55 +0000 (22:40 +0200)]
Explicitly check td_trim(td) for the direction of next io_u

This else case should explicitly check td_trim(td) since there is
a potential unused enum (e.g. DDIR_WAIT only used by iolog for now).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDrop obsolete comment on a race condition
Tomohiro Kusumi [Wed, 15 Feb 2017 20:40:54 +0000 (22:40 +0200)]
Drop obsolete comment on a race condition

which is also mentioned in below posts.

http://www.spinics.net/lists/fio/msg05548.html
http://www.spinics.net/lists/fio/msg05552.html
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse 0 instead of DDIR_READ to iterate from 0 to DDIR_RWDIR_CNT
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:48 +0000 (17:19 +0200)]
Use 0 instead of DDIR_READ to iterate from 0 to DDIR_RWDIR_CNT

I think this is better since we want to count up DDIR_RWDIR_CNT,
but not specifically (count - something) even if something is 0,
and that's basically what DDIR_RWDIR_CNT is there for to equal
SYNC=3 after READ/WRITE/TRIM in io_ddir.h.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing "rand"/"trimwrite" strings to corresponding ddir slots
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:46 +0000 (17:19 +0200)]
Add missing "rand"/"trimwrite" strings to corresponding ddir slots

though "rand" alone is unused.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAlways set ->real_file_size to -1 when failed to get file size
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:45 +0000 (17:19 +0200)]
Always set ->real_file_size to -1 when failed to get file size

In some cases it's set to -1, but on others it's not,
while the existing code (e.g. get_fs_free_counts(), setup_files(), etc)
seem to expect ->real_file_size to be -1 in case of any error.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDrop conditional declaration of disk_list
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:44 +0000 (17:19 +0200)]
Drop conditional declaration of disk_list

disk_list is used unconditionally whether it's Linux or not,
so leave the one in libfio.c, and remove the one in diskutil.c.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: Use x86 instead of i386 for $cpu for IA32
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:43 +0000 (17:19 +0200)]
configure: Use x86 instead of i386 for $cpu for IA32

fio internally uses "x86" for any sort of IA32, so use x86 for $cpu
as well which gets printed on ./configure.

This change should have been a part of e12f4ede in 2016.

 # grep i386 . -rI
 ./arch/arch.h:#if defined(__i386__)
 ./configure:elif check_define __i386__ ; then
 ./configure:  cpu="i386"
 ./configure:  i386|i486|i586|i686|i86pc|BePC)
 ./configure:    cpu="i386"
 ./crc/sha1.c:#if defined(__i386__) || defined(__x86_64__)
 ./crc/xxhash.c:#if defined(__ARM_FEATURE_UNALIGNED) || defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: Drop default CONFIG_LITTLE_ENDIAN for Cygwin
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:42 +0000 (17:19 +0200)]
configure: Drop default CONFIG_LITTLE_ENDIAN for Cygwin

It's a bit strange that Cygwin is assumed to be le by default when
no other platforms have such assumption even if they only target
a single or certain arch (e.g. non-Android Linux often runs on x86,
DragonFlyBSD only runs on x86, and so on..).

Endianness will be detected by pointer arithmetic anyway, and it's
better to rely on it to handle any sort of newly supported archs
in the future.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoWindows: re-enable the mmap ioengine and fix static asserts
Rebecca Cran [Thu, 9 Feb 2017 22:17:27 +0000 (22:17 +0000)]
Windows: re-enable the mmap ioengine and fix static asserts

Windows supports mmap functionality via the CreateFileMapping and
MapViewOfFile functions. Update posix.c and re-enable the mmap
ioengine in the Makefile.

Also, enable the built-in static asserts, without which the build
breaks when configured with --disable-optimizations.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix the return type of log_err and log_info functions
Rebecca Cran [Tue, 7 Feb 2017 16:13:19 +0000 (09:13 -0700)]
Fix the return type of log_err and log_info functions

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoWindows: Update the EULA year and add more examples to the installer
Rebecca Cran [Mon, 6 Feb 2017 16:17:24 +0000 (16:17 +0000)]
Windows: Update the EULA year and add more examples to the installer

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogfio: fix git location for fio
Jens Axboe [Mon, 13 Feb 2017 19:43:17 +0000 (12:43 -0700)]
gfio: fix git location for fio

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoBe more verbose on endianness detection failure
Tomohiro Kusumi [Mon, 13 Feb 2017 16:13:43 +0000 (18:13 +0200)]
Be more verbose on endianness detection failure

> I think this deserves and enum. That also enables the compiler to check
> if the below switch covers all the potential values.

Updated with enum ENDIAN_XXX, which you can change the naming if it
doesn't sound good.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosteadystate: Use calloc(3)
Tomohiro Kusumi [Sat, 11 Feb 2017 14:28:54 +0000 (16:28 +0200)]
steadystate: Use calloc(3)

which guarantees zero clear.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd a comment to clarify 941bda94
Tomohiro Kusumi [Sat, 11 Feb 2017 14:28:53 +0000 (16:28 +0200)]
Add a comment to clarify 941bda94

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: try to disable weak linking on OSX
Sitsofe Wheeler [Sat, 11 Feb 2017 09:19:46 +0000 (09:19 +0000)]
configure: try to disable weak linking on OSX

macOS 10.12 introduced support for clock_gettime() but when compiling
with Xcode 8 (or later) and targeting 10.11 (or older) the clock_gettime
symbol is now found at configure/compile time but referenced weakly.
Running the created binary on 10.11 results in a "dyld: lazy symbol
binding failed: Symbol not found: _clock_gettime" error.

This drama has played out across other projects:
- https://github.com/Homebrew/homebrew-core/issues/2674
- https://github.com/curl/curl/issues/1069
- https://svn.filezilla-project.org/filezilla?view=revision&revision=7824OO
- https://github.com/mesonbuild/meson/pull/949
- https://github.com/Homebrew/homebrew-core/issues?utf8=%E2%9C%93&q=label%3Aclock_gettime%20

Options for working around this issue include:
1. Turn references to functions that might be weak into compilation
   errors (-Werror=partial-availability).
2. Disable visibility of functions that might be weak at compile link
   time (-Wl,-no_weak_imports).
3. Change configure tests to directly check the targeted OSX version and
   fail the test if it is too old to have the required function.
4. Make an empty function declaration marked with
   __attribute__((weak_import)) that is used if the symbol would
   otherwise be unavailable. It is then possible to check if the symbol
   is NULL at runtime to determine availability.

This commit does 2. when targeting the Darwin platform after checking if
the compiler/linker work with that option. It also changes the clockid_t
configure test to no longer use clock_gettime() because targeting OSX
10.11 when using the 10.12 SDK results in having the clockid_t typedef
without having the clock_gettime function. The combination of these
fixes should solve https://github.com/axboe/fio/issues/305 .

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agofnv: work with non-64-bit aligned chunks of data
Jens Axboe [Tue, 7 Feb 2017 22:11:37 +0000 (15:11 -0700)]
fnv: work with non-64-bit aligned chunks of data

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobloom: if we're not setting bits, break after first failed mask check
Jens Axboe [Tue, 7 Feb 2017 22:10:26 +0000 (15:10 -0700)]
bloom: if we're not setting bits, break after first failed mask check

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: don't crash server on failure to open output log
Jens Axboe [Tue, 7 Feb 2017 16:06:16 +0000 (09:06 -0700)]
init: don't crash server on failure to open output log

We don't use it on the server side anyway.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agodoc: minor HOWTO fixes
Sitsofe Wheeler [Sat, 4 Feb 2017 09:10:13 +0000 (09:10 +0000)]
doc: minor HOWTO fixes

- Add a note to to size/offset to see the other for constraining I/O to
  a range (requested in https://github.com/axboe/fio/issues/298 ).
- Minor grammar and typo fixes.
- Update example fio ETA status to match what modern fio would likely
  display.
- Explain f= is open files and note that trim may be displayed.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agofix to replay_align on iolog
Dylan Fairchild [Thu, 2 Feb 2017 18:37:00 +0000 (10:37 -0800)]
fix to replay_align on iolog
previous code was rejecting all positive alignment values and
only accepting align to 0 value, opposite of what it should be doing

3 years agodoc: document profiles, minor fixes
Sitsofe Wheeler [Sun, 29 Jan 2017 21:20:08 +0000 (21:20 +0000)]
doc: document profiles, minor fixes

- Add documentation for profile to the HOWTO.
- Explictly document the how parameters that take time are interpreted
  when the unit is omitted.
- Add disable_bw_measurement default alias.
- Minor sentence rewording.
- Change max_latency to say it returns ETIMEDOUT rather than ETIME
  to match the commit c5b3adc41e0a91be5c6accdbe6506522ebddfbdd .

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodoc: minor consistency and spelling changes
Sitsofe Wheeler [Fri, 27 Jan 2017 09:43:12 +0000 (09:43 +0000)]
doc: minor consistency and spelling changes

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodoc: minor documentation changes
Sitsofe Wheeler [Thu, 26 Jan 2017 17:02:47 +0000 (17:02 +0000)]
doc: minor documentation changes

- Change some parameter types to be time in the HOWTO.
- Put a space before the end of sentence but after links in the README.
- Update the Red Hat fio package information.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoMerge branch 'shm_rm' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 26 Jan 2017 17:07:48 +0000 (10:07 -0700)]
Merge branch 'shm_rm' of https://github.com/sitsofe/fio

3 years agoMerge branch 'sphinx-doc' of https://github.com/termim/fio
Jens Axboe [Thu, 26 Jan 2017 17:04:40 +0000 (10:04 -0700)]
Merge branch 'sphinx-doc' of https://github.com/termim/fio

3 years agoshm: have os remove shared memory if fio dies unexpectedly
Sitsofe Wheeler [Tue, 24 Jan 2017 06:31:58 +0000 (06:31 +0000)]
shm: have os remove shared memory if fio dies unexpectedly

When fio doesn't exit cleanly shared memory segments are left behind on
those platforms where shared memory is supported.

Due to how fio manages shared memory lifetime we can combat the above by
marking the shared memory segment as removed shortly after we attach to
it for the first time if the OS offers both the following (non-standard)
behaviours:

1. A shared memory segment marked for removal with IPC_RMID isn't
   destroyed until all processes have detached from it.
2. After 1. has occurred it is still possible to (re)attach to the
   marked-for-removal shared memory segment.

Linux has always done the above so enable the workaround there. OpenBSD
switched to having the above behaviours in 5.1 (see shmctl in
https://www.openbsd.org/plus51.html ) so add a comment about it in
os/os-openbsd.h . FreeBSD only has the above behaviours if
kern.ipc.shm_allow_removed (introduced in 5.2) is 1 and this only became
the default in FreeBSD 11
(https://svnweb.freebsd.org/base?view=revision&revision=289112 ) so add
a comment in os/os-freebsd.h . DragonFly also has the FreeBSD flag but
at the time of writing it defaults to 0 so add a comment to
os/os-dragonfly.h .

There's a nice (if dated) post discussing how IPC_RMID behaves on
various OSes in https://www.cygwin.com/ml/cygwin/2007-11/msg00078.html
("Re: cygwin stable and cvs snapshot - fork() bug").  While fio doesn't
support the Cygwin "OS", Cygwin itself has the
kern.ipc.shm_allow_removed option in a config file and defaults it to
no.

3 years agoRemove/Move Linux specific sysfs_root field from thread_data
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:11 +0000 (00:13 +0900)]
Remove/Move Linux specific sysfs_root field from thread_data

struct thread_data is better without having a Linux specific field
(even if sysfs_root is simply unused on non Linux environment) given
that other Linux specific code such as diskutil/cgroup/blktrace/etc
are compiled only on Linux.

This commit uses td->files[0]->du->sysfs_root instead of td->sysfs_root,
but results the same from the way diskutil is currently implemented.
Besides being platform independent, it's also better in terms of data
structure since sysfs_root isn't always per-td attribute as mentioned
below.

For example, when N(>1) files are specified via filename= option
(e.g. filename=/dev/sdb:/dev/sdc for N=2), td:files is 1:N, therefore
td:du and td:du->sysfs_root are also 1:N in case of /dev/sdb:/dev/sdc,
however td:td->sysfs_root is obviously 1:1 which in this case means
td->sysfs_root is sysfs root for /dev/sdb (regardless of other files)
from the way td->sysfs_root is set only if it hasn't yet been set.

In other words, if filename=/dev/sdb:/dev/sdc is given, ioscheduler=
option only works against /dev/sdb, which could be different from what
users would expect, but this commit just removes sysfs_root from
struct thread_data without any functional change.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix bad pointer du->sysfs_root
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:10 +0000 (00:13 +0900)]
Fix bad pointer du->sysfs_root

There are two call paths for disk_util_add() (usually the second one)
which assigns ->sysfs_root for a newly allocated disk_util*, but both
temppath/foo are local char[] within their stack, so strdup/free it.

-> find_add_disk_slaves(...)
  -> __init_per_file_disk_util(..., temppath)
    -> disk_util_add(..., path)
      -> du->sysfs_root = path; /* == &temppath[0] */

-> init_per_file_disk_util(...)
  -> __init_per_file_disk_util(..., foo)
    -> disk_util_add(..., path)
      -> du->sysfs_root = path; /* == &foo[0] */

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing free(td->sysfs_root);
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:09 +0000 (00:13 +0900)]
Add missing free(td->sysfs_root);

->sysfs_root for td (not du) is always strdup'd.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove unused disk_util's name field
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:08 +0000 (00:13 +0900)]
Remove unused disk_util's name field

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove unused io_u's priv union field
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:07 +0000 (00:13 +0900)]
Remove unused io_u's priv union field

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix wrong comment on exit condition of threads/processes
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:06 +0000 (00:13 +0900)]
Fix wrong comment on exit condition of threads/processes

74f4b020 in 2013 has changed the exit condition from min to max,
so sync the comment with 74f4b020.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse ARRAY_SIZE()
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:05 +0000 (00:13 +0900)]
Use ARRAY_SIZE()

They can assume ARRAY_SIZE() is defined and usable as far as fio's
internal code is concerned (i.e. FIO_INTERNAL defined).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDefine pointer alignment macro in fio.h
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:04 +0000 (00:13 +0900)]
Define pointer alignment macro in fio.h

Define PTR_ALIGN() as a common utility within the entire fio,
but inside FIO_INTERNAL guard since it could be a common name.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofix manpage heading issue
Mikhail Terekhov [Sat, 21 Jan 2017 02:44:56 +0000 (21:44 -0500)]
fix manpage heading issue

Sphinx skips zero level headins when generating manpages.
Increasing sections depth solves missing headers problem.

3 years agoAdd missing opt/cat group entries
Tomohiro Kusumi [Fri, 20 Jan 2017 17:46:43 +0000 (02:46 +0900)]
Add missing opt/cat group entries

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoChange .category of cpuio to FIO_OPT_C_ENGINE
Tomohiro Kusumi [Fri, 20 Jan 2017 17:46:42 +0000 (02:46 +0900)]
Change .category of cpuio to FIO_OPT_C_ENGINE

This is supposed to be ENGINE instead of GENERAL.

 # grep "\.category" engines -rI | grep -v ENGINE
 engines/cpu.c:          .category = FIO_OPT_C_GENERAL,
 engines/cpu.c:          .category = FIO_OPT_C_GENERAL,
 engines/cpu.c:          .category = FIO_OPT_C_GENERAL,

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'sphinx-doc' of https://github.com/termim/fio
Jens Axboe [Fri, 20 Jan 2017 05:01:48 +0000 (22:01 -0700)]
Merge branch 'sphinx-doc' of https://github.com/termim/fio

3 years agoremove documentation output in clean
Mikhail Terekhov [Fri, 20 Jan 2017 04:50:11 +0000 (23:50 -0500)]
remove documentation output in clean

3 years agoMerge branch 'sphinx-doc' of https://github.com/termim/fio
Jens Axboe [Fri, 20 Jan 2017 02:50:09 +0000 (19:50 -0700)]
Merge branch 'sphinx-doc' of https://github.com/termim/fio

3 years agoignore documentation output directory
Mikhail Terekhov [Fri, 20 Jan 2017 02:08:39 +0000 (21:08 -0500)]
ignore documentation output directory

3 years agoChange td_var() to take void* for the first arg
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:27 +0000 (02:38 +0900)]
Change td_var() to take void* for the first arg

Callers don't always pass thread_options* to td_var(),
nor is it dependent on a certain structure type,
so just take void* without pointless cast.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMove options_mem_dupe() to parse.c
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:26 +0000 (02:38 +0900)]
Move options_mem_dupe() to parse.c

This function should be in parse.c with fio_option* argument first.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRefactor fio_show_ioengine_help()
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:25 +0000 (02:38 +0900)]
Refactor fio_show_ioengine_help()

Since td is there only to call load_ioengine(),
ioengine ops doesn't need to be refered to as td.io_ops.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix typo for "job" in plural
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:24 +0000 (02:38 +0900)]
Fix typo for "job" in plural

It seems this message was meant to be 1 or more "job".

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRename FIO_TYPE_BD to FIO_TYPE_BLOCK
Tomohiro Kusumi [Wed, 18 Jan 2017 17:38:23 +0000 (02:38 +0900)]
Rename FIO_TYPE_BD to FIO_TYPE_BLOCK

Since character device is FIO_TYPE_CHAR, BLOCK (or BLK) makes
better sense for block device.

This may break external ioengines on compile-time, but ioengines
usually don't refer to file type which are regfile/blk/chr/pipe
(which I believe makes sense to commit this change as there's
been changes that break not only api but also abi).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'sphinx-doc' of https://github.com/termim/fio into sphinx
Jens Axboe [Wed, 18 Jan 2017 16:51:33 +0000 (09:51 -0700)]
Merge branch 'sphinx-doc' of https://github.com/termim/fio into sphinx

3 years agoinit: fix double free of pid_file
Jens Axboe [Wed, 18 Jan 2017 15:18:28 +0000 (08:18 -0700)]
init: fix double free of pid_file

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRevert README and HOWTO files renaming.
Mikhail Terekhov [Wed, 18 Jan 2017 13:56:35 +0000 (08:56 -0500)]
Revert README and HOWTO files renaming.

github doesn't support Sphinx extencions to ReStructured text format.

3 years agoadd .rst suffix to README and HOWTO
Mikhail Terekhov [Wed, 18 Jan 2017 02:01:50 +0000 (21:01 -0500)]
add .rst suffix to README and HOWTO

3 years agoConvert documentation to reStructured text.
Mikhail Terekhov [Mon, 26 Dec 2016 03:17:57 +0000 (22:17 -0500)]
Convert documentation to reStructured text.

Reshuffle contents of README and HOWTO
Add some job file examples to HTML docs
Automatically generate fio.1 manpage

3 years agorun sphinx-quickstart
Mikhail Terekhov [Mon, 26 Dec 2016 03:13:57 +0000 (22:13 -0500)]
run sphinx-quickstart

3 years agoFio 2.17 fio-2.17
Jens Axboe [Tue, 17 Jan 2017 15:51:31 +0000 (08:51 -0700)]
Fio 2.17

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDrop crc32c-arm64 option
wei xiao [Fri, 6 Jan 2017 02:37:03 +0000 (02:37 +0000)]
Drop crc32c-arm64 option

Crc32c will use ARM64 hardware assist, if available.

Jira: ENTLLT-359
Change-Id: Ibdf88f14bd4669ed68a93078c43eb5c8412da543

3 years agoSupport zlib in the Windows build (enabled latency histogram logging)
Rebecca Cran [Mon, 9 Jan 2017 21:45:38 +0000 (21:45 +0000)]
Support zlib in the Windows build (enabled latency histogram logging)

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agopmemblk: Clarify fsize is in MiB not MB
Robert Elliott [Tue, 10 Jan 2017 21:21:27 +0000 (15:21 -0600)]
pmemblk: Clarify fsize is in MiB not MB

Clarify that the FSIZE argument is in MiB (power of 2) rather
than MB (power of 10) in comments and variable names.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agopmemblk, dev-dax: clean up error logs
Robert Elliott [Tue, 10 Jan 2017 21:21:26 +0000 (15:21 -0600)]
pmemblk, dev-dax: clean up error logs

Print the errno string rather than the number, and prefix
all log_err prints with pmemblk or dev-dax rather than fio.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agopmemblk, dev-dax: Update descriptions
Robert Elliott [Tue, 10 Jan 2017 21:21:25 +0000 (15:21 -0600)]
pmemblk, dev-dax: Update descriptions

Make the descriptions of the pmemblk and dev-dax ioengines
more symmetric.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agopmemblk, dev-dax: load libpmem and libpmemblk at startup
Robert Elliott [Tue, 10 Jan 2017 21:21:24 +0000 (15:21 -0600)]
pmemblk, dev-dax: load libpmem and libpmemblk at startup

The pmemblk and dev-dax ioengines were loading libpmem.so and
libpmemblk.so using dlopen() at runtime.

Although the upstream nvml Makefile installs a libpmem.so
symbolic link, some of the distros (e.g. SUSE Linux Enterprise
Server 12 SP2) are just installing so.1 symbolic links.  So,
fio fails to find the libpmem.so and libpmemblk.so library files.

http://www.ibm.com/developerworks/linux/library/l-shlibs/index.html
says applications should always load the versioned filenames to
avoid compatibility issues; the non-versioned filenames are just
for the compiler and linker.

Change ./configure to pass -lpmem and -lpmemblk to the compiler
so the fio binary loads the versioned filename at startup like
the other libraries, rather than open the non-versioned filename
with dlopen() during runtime.

This way they show up in ldd:
$ ldd fio
        linux-vdso.so.1 (0x00007ffc290aa000)
        libpmemblk.so.1 => /lib64/libpmemblk.so.1 (0x00007f65b5bc4000)
        libpmem.so.1 => /lib64/libpmem.so.1 (0x00007f65b59be000)
        libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f65b57b3000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f65b55ab000)
        libaio.so.1 => /lib64/libaio.so.1 (0x00007f65b53a9000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f65b5191000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f65b4e88000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f65b4c6a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f65b4a66000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f65b46a0000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f65b449b000)
        /lib64/ld-linux-x86-64.so.2 (0x0000559f4d872000)

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoPython style/portability fix
Tomohiro Kusumi [Tue, 10 Jan 2017 17:39:47 +0000 (02:39 +0900)]
Python style/portability fix

In practice, one would normally explicitly derive a class from
object class (was called new-style class back then).
https://docs.python.org/release/2.5.2/ref/node33.html
https://wiki.python.org/moin/NewClassVsClassicClass

print needs parentheses for portability with Python3.x.

xrange() only exists in Python2.x (i.e. breaks on Python3.x).
Using range() (which pre-allocates a whole list in Python2.x)
won't be a problem unless len(averages) is huge enough to give
any pressure to vm subsystem.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix README regarding fio snapshots
Tomohiro Kusumi [Tue, 10 Jan 2017 17:39:46 +0000 (02:39 +0900)]
Fix README regarding fio snapshots

As of now, fio-git-*.tar.{gz,bz2} are git repositories,
but fio-`version`.tar.gz are archives without .git/ meta data.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing trailing \n in dprint()
Tomohiro Kusumi [Tue, 10 Jan 2017 17:39:45 +0000 (02:39 +0900)]
Add missing trailing \n in dprint()

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoPartly revert 8172fe97 in 2008 (A few debug debug log fixes)
Tomohiro Kusumi [Tue, 10 Jan 2017 17:39:44 +0000 (02:39 +0900)]
Partly revert 8172fe97 in 2008 (A few debug debug log fixes)

Not sure what the point of 8172fe97 was, but shouldn't all the
prints within a single line be dprint() (or all be log_info()) ?

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix compile time errors for skeleton_external ioengine
Tomohiro Kusumi [Mon, 9 Jan 2017 14:42:03 +0000 (23:42 +0900)]
Fix compile time errors for skeleton_external ioengine

though this file is not compiled by default.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMention sg ioengine requires filename option
Tomohiro Kusumi [Mon, 9 Jan 2017 14:42:02 +0000 (23:42 +0900)]
Mention sg ioengine requires filename option

similar to the way net ioengine documents filename option.

Also bring in a file type check from fio_sgio_type_check()
to avoid pointless read capacity error against regfiles.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoNon functional fixup for 16 bytes read capacity for sg ioengine
Tomohiro Kusumi [Mon, 9 Jan 2017 14:42:01 +0000 (23:42 +0900)]
Non functional fixup for 16 bytes read capacity for sg ioengine

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix conditional/message for max lba for sg ioengine
Tomohiro Kusumi [Mon, 9 Jan 2017 14:42:00 +0000 (23:42 +0900)]
Fix conditional/message for max lba for sg ioengine

Whether sg ioengine uses 10 or 16 bytes read/write command (by 5ad7be56)
is determined by ->prep() depending on lba to read/write vs 0xFFFFFFFFULL,
so let the message show it depends.

Also remove unused sgio_data::max_lba which is essentially the same
(divide by bs minus 1) as fio_file::real_file_size.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoStaticize pmemblk ioengine_ops
Tomohiro Kusumi [Mon, 9 Jan 2017 14:41:59 +0000 (23:41 +0900)]
Staticize pmemblk ioengine_ops

Inbox engines can have static ops.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoBring in additional sg error cases from Linux kernel (and sg3) header
Tomohiro Kusumi [Mon, 9 Jan 2017 14:41:58 +0000 (23:41 +0900)]
Bring in additional sg error cases from Linux kernel (and sg3) header

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix comment on SCSI commands
Tomohiro Kusumi [Fri, 6 Jan 2017 18:15:18 +0000 (03:15 +0900)]
Fix comment on SCSI commands

[0] is service action (unless this is intentional).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove doubled ; in .c
Tomohiro Kusumi [Fri, 6 Jan 2017 18:15:17 +0000 (03:15 +0900)]
Remove doubled ; in .c

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: cleaner gcd()
Jens Axboe [Thu, 5 Jan 2017 17:38:41 +0000 (10:38 -0700)]
init: cleaner gcd()

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDon't malloc ioengine_ops for cpp_null
Tomohiro Kusumi [Thu, 5 Jan 2017 17:32:17 +0000 (02:32 +0900)]
Don't malloc ioengine_ops for cpp_null

fio assumes opses aren't dynamically allocated which means malloc'd
engine here never gets freed.

According to 46a67478 and a8075704, the point of get_ioengine() is
how non-static ops gets initialized by g++, but not how space for ops
gets allocated, thus it should just define a static global variable
within "C" scope and pass the address of ops to a caller via **.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix invalid ioengine initialization for cpp_null
Tomohiro Kusumi [Thu, 5 Jan 2017 17:32:16 +0000 (02:32 +0900)]
Fix invalid ioengine initialization for cpp_null

->name is a pointer now, so it can't be strcpy'd without a valid
address set to ->name.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: fill in vc->name for pattern verify
Jens Axboe [Thu, 5 Jan 2017 17:33:23 +0000 (10:33 -0700)]
verify: fill in vc->name for pattern verify

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: ensure that verify_interval is always a factor of min/max bs
Jens Axboe [Thu, 5 Jan 2017 17:26:08 +0000 (10:26 -0700)]
verify: ensure that verify_interval is always a factor of min/max bs

A previous commit force aligned the buffer length to the
verify_interval, but that breaks if max_bs isn't a multiple
of min_bs. If that isn't the case, we can align the block size
bigger than the IO unit really is. This causes corruption and
failures.

Fixes: a9f70b1f5087 ("Make sure io_u->buflen is aligned to the verify_interval")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove '--runtime' command line option
Jens Axboe [Thu, 5 Jan 2017 16:49:06 +0000 (09:49 -0700)]
Remove '--runtime' command line option

It conflicts with the job file option, and we (in most cases) set
the default before we have parsed it, depending on where in the
command line the --runtime option is added.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: use log_verify_failure() for pattern verifies
Jens Axboe [Thu, 5 Jan 2017 16:38:50 +0000 (09:38 -0700)]
verify: use log_verify_failure() for pattern verifies

Similar to the other verify types, otherwise we don't get
the absolute offset logged.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoStyle cleanups for arm crc32c hw support
Jens Axboe [Thu, 5 Jan 2017 02:44:35 +0000 (19:44 -0700)]
Style cleanups for arm crc32c hw support

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd arm64 hardware assisted crc32c support
wei xiao [Thu, 5 Jan 2017 02:32:40 +0000 (10:32 +0800)]
Add arm64 hardware assisted crc32c support

HW assisted crc32c get ~ x39.6 speedups on ARM Cortex-A57@2GHz

Auto-detect whether to use hw assisted crc32c.
If the hardware assisted crypto is available, always use it.
Otherwise, fallback to software.