fio.git
3 years agofilesetup: keep OS_O_DIRECT flag when pre-allocating file
weiping zhang [Sat, 29 Jul 2017 08:47:59 +0000 (16:47 +0800)]
filesetup: keep OS_O_DIRECT flag when pre-allocating file

When direct=1 the file open for pre-allocation should also set OS_O_DIRECT so
we fail earlier if the filesystem/file doesn't support direct operations.

Signed-off-by: weiping zhang <zhangweiping@didichuxing.com>
3 years agodon't print native_fallocate() error if ENOSYS
Tomohiro Kusumi [Thu, 27 Jul 2017 20:38:37 +0000 (23:38 +0300)]
don't print native_fallocate() error if ENOSYS

log_err(ENOSYS) on native_fallocate() failure which was newly added
by 2c3e17be('filesetup: add native fallocate') should be silenced
or somehow be printed only once instead of being a per file message.

This happens on a platform like FreeBSD where posix_fallocate(3)
exists, but native_fallocate() is ENOSYS.

This commit just silences it if errno is set to ENOSYS on return.
"native" is the default mode, thus not printing ENOSYS won't be any
confusing unless fallocate=native is explicitly specified, and
native_fallocate() has dprint() for ENOSYS case anyway.

--
 # uname
 FreeBSD
 # ./fio --name=xxx --ioengine=sync --rw=read --bs=4k --size=10m --nrfiles=20
 xxx: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
 fio-2.99-23-gdad0
 Starting 1 process
 xxx: Laying out IO files (20 files / total 10MiB)
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 fio: native_fallocate call failed: Function not implemented
 ...

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agopvsync2: Add hipri_percentage option
Stephen Bates [Thu, 27 Jul 2017 20:02:15 +0000 (14:02 -0600)]
pvsync2: Add hipri_percentage option

Allow the user to set the probability of a pvsync2 IO being high
priority. Add a new option for the pvsync2 engine called
--hipri_percentage will takes values in the range (0, 100) and use
that as a probability for setting RWF_HIPRI. The default for this
option is 100 so exisiting users of --hirpi are unaffected.

Changes since v3:
  Remove unwanted #ifdefs

Changes since v2:
  s/_probability/_percentage/

Changes since v1:
  Use --hipri_probability rather than impacting --hipri
  Correct the random number generator call

Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'minor_fixes' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 27 Jul 2017 14:11:40 +0000 (08:11 -0600)]
Merge branch 'minor_fixes' of https://github.com/sitsofe/fio

3 years agofio: refactor fallocate defines
Sitsofe Wheeler [Thu, 27 Jul 2017 05:06:10 +0000 (06:06 +0100)]
fio: refactor fallocate defines

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoinit: force fallocate_mode to none when fallocate is unsupported
Sitsofe Wheeler [Thu, 27 Jul 2017 04:50:41 +0000 (05:50 +0100)]
init: force fallocate_mode to none when fallocate is unsupported

Commit 2c3e17be ("filesetup: add native fallocate") broke fio on Windows
because fallocate_mode wound up being set to 0 on platforms without any
type of fallocate. Work around this by forcing fallocate_mode to
FIO_FALLOCATE_NONE on such platforms.

This should address the issue mention in the mail
http://www.spinics.net/lists/fio/msg06085.html ("The commit 'filesetup:
add native fallocate' breaks on Cygwin").

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodoc: minor grammar fixes
Sitsofe Wheeler [Thu, 20 Jul 2017 06:10:52 +0000 (07:10 +0100)]
doc: minor grammar fixes

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodoc: add block size to log file format
Sitsofe Wheeler [Tue, 18 Jul 2017 06:46:52 +0000 (07:46 +0100)]
doc: add block size to log file format

Commit a3ae5b057588f555a1544ec749167a44c5a071aa ("Update documentation
on log file formats") added documentation for the log format but left
out the block size column so let's add it here.

While we're here clean up some formatting in log_offset, add a reference
to Log File Formats and mention that it defaults to off.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agotime: Add chosen clocksource debug
Sitsofe Wheeler [Sun, 9 Jul 2017 19:54:38 +0000 (20:54 +0100)]
time: Add chosen clocksource debug

Make it a bit easier to see which clocksource was used by printing its
enumeration value when --debug=time is used.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoexamples: add a butterfly seek job file
Sitsofe Wheeler [Sun, 9 Jul 2017 16:10:42 +0000 (17:10 +0100)]
examples: add a butterfly seek job file

Add a job file that approximates doing
read at start
read at end
read at start+1
read at end-1
[...]
read at end-1
read at start+1
read at end
read at start

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoarch: raise an error when compiling for an unknown ARM platform
Sitsofe Wheeler [Sun, 9 Jul 2017 07:32:47 +0000 (08:32 +0100)]
arch: raise an error when compiling for an unknown ARM platform

The nop, read_barrier() and write_barrier() macros have to always be
defined but due to the way os/arch-arm.h is written it is possible for
them to be undefined when the ARM platform is not recognized.

Change things so compiling for an unknown ARM platform raises an
explicit compilation error at the point of the problem.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoos: add missing include for bswap_* on BSDs
Sitsofe Wheeler [Sun, 9 Jul 2017 07:09:35 +0000 (08:09 +0100)]
os: add missing include for bswap_* on BSDs

Commit 0dcebdf4a70ef0d8144b8fcba763ae87e7fc74b5 ("configure: add endian
check") removed the sys/endian.h include from the os/*.h BSD headers but
this include is needed to access the bswap_* macros. The lack of this
include wouldn't show up unless you were compiling for a big-endian BSD
platform.

This commit adds the include back for {Free,Net,Open}BSD and adds it for
the first time to the DragonFlyBSD header.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agotravis: get rid of non-breaking space characters
Sitsofe Wheeler [Sun, 9 Jul 2017 06:48:16 +0000 (07:48 +0100)]
travis: get rid of non-breaking space characters

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: remove unnecessary escaping
Sitsofe Wheeler [Sun, 9 Jul 2017 06:28:56 +0000 (07:28 +0100)]
HOWTO: remove unnecessary escaping

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodrop logging when blkdev invalidation failed on unsupported platforms
Tomohiro Kusumi [Tue, 25 Jul 2017 19:05:44 +0000 (22:05 +0300)]
drop logging when blkdev invalidation failed on unsupported platforms

22de5d77('Use ENOTSUP if OS doesn't support blkdev page cache invalidation')
changed errno values being used on supported platforms to ENOTSUP,
and also log_info'd it via strerror(), but strerror() isn't working
on some platform and instead prints irrelevant string.

Re: fio Windows Invalidate
http://www.spinics.net/lists/fio/msg06084.html

The logging of ENOTSUP could just be dropped since this isn't a real
failure, as many platforms either don't have the idea of invalidating
blkdev's page cache mapping, or use chrdev for block devices.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO: add missing [netsplice] for ioengine specific options
Tomohiro Kusumi [Tue, 25 Jul 2017 19:05:43 +0000 (22:05 +0300)]
HOWTO: add missing [netsplice] for ioengine specific options

These seem to be supported whether a socket is read/written by
send/recv or splice.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman: fix broken ioengine specific option format
Tomohiro Kusumi [Tue, 25 Jul 2017 19:05:42 +0000 (22:05 +0300)]
man: fix broken ioengine specific option format

The parentheses here are supposed to be for the name of ioengines,
but not for the name of options, according to other correct ones.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO: fix "should be to to"
Tomohiro Kusumi [Tue, 25 Jul 2017 19:05:41 +0000 (22:05 +0300)]
HOWTO: fix "should be to to"

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoCorrectly detect whether ioengine_load can exit early
Ben Walker [Wed, 19 Jul 2017 19:27:57 +0000 (12:27 -0700)]
Correctly detect whether ioengine_load can exit early

Just the presence of io_ops isn't enough - the requested
ioengine may have changed.

Signed-off-by: Ben Walker <benjamin.walker@intel.com>
3 years agobuild: Sort file list
Bernhard M. Wiedemann [Sun, 16 Jul 2017 19:46:31 +0000 (21:46 +0200)]
build: Sort file list

to make build reproducible.
Because filesystems/readdir returns entries in underministic order
and the linker orders functions depending on input order
the resulting fio binary differed across builds
(which we do in disposable VMs for openSUSE)

See https://reproducible-builds.org/ for why this matters.

Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman: add missing \ for fcntl(2) in write_hint option
Tomohiro Kusumi [Sun, 16 Jul 2017 12:40:20 +0000 (15:40 +0300)]
man: add missing \ for fcntl(2) in write_hint option

8f4b9f24 ('Update documentation for write_hint') was missing \.
Without this '|' becomes visible.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman: fix wrong info on sync_file_range=
Tomohiro Kusumi [Fri, 14 Jul 2017 07:24:03 +0000 (10:24 +0300)]
man: fix wrong info on sync_file_range=

This part isn't in sync with HOWTO, but also simply wrong.
HOWTO as well as the actual option have SYNC_FILE_RANGE_WAIT_AFTER for this.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoUpdate documentation for write_hint
Jens Axboe [Thu, 13 Jul 2017 15:39:01 +0000 (09:39 -0600)]
Update documentation for write_hint

We no longer have fadvise_stream, adapt the documentation to
describe the stable 'write_hint' option.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoparse: enable options to be marked dont-free
Jens Axboe [Wed, 12 Jul 2017 22:44:07 +0000 (16:44 -0600)]
parse: enable options to be marked dont-free

For profiles, we can't free the options prematurely.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoprofiles/act: use the right options
Jens Axboe [Wed, 12 Jul 2017 22:32:24 +0000 (16:32 -0600)]
profiles/act: use the right options

We have a double set - one we actually set, and then one we
check. Oops, fix that up.

Fixes: 7b504eddc4b0 ("parse: get rid of raw option offsets")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoreturn correct error code for unhandled addr.
Pan Liu [Mon, 10 Jul 2017 16:28:45 +0000 (00:28 +0800)]
return correct error code for unhandled addr.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
3 years agoFio 2.99 fio-2.99
Jens Axboe [Fri, 7 Jul 2017 18:45:04 +0000 (12:45 -0600)]
Fio 2.99

Test release before we wrap it up and make a 3.0.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoinit: add comment as to how we seed read/write/trim generators
Jens Axboe [Fri, 7 Jul 2017 14:37:49 +0000 (08:37 -0600)]
init: add comment as to how we seed read/write/trim generators

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoio_u: sequence random buflen generation individually
Jens Axboe [Fri, 7 Jul 2017 14:33:57 +0000 (08:33 -0600)]
io_u: sequence random buflen generation individually

For trimwrite, we rely on write+trim generating identical
buffer lengths, if they have identical settings. If we use
bssplit, then we generate them using a random generator
according to the percentages set. This means we may not get
the same offsets in sequence.

Split the seeding into separate random generators, and ensure
that we seed the write and trim ones identically, if we are
doing trim writes.

Fixes: https://github.com/axboe/fio/issues/399
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agooslib/libmtd: kill dead code
Jens Axboe [Thu, 6 Jul 2017 14:09:22 +0000 (08:09 -0600)]
oslib/libmtd: kill dead code

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO: fix indentation for options and job parameters
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:16 +0000 (21:08 +0300)]
HOWTO: fix indentation for options and job parameters

HOWTO mostly uses tab indentations for command line options and job
file parameters, so replace white spaces with tabs.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO: minor backports from the man page
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:15 +0000 (21:08 +0300)]
HOWTO: minor backports from the man page

This commit mainly backports some minor fixups from the man page
sections covered in the several previous commits.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman: refer to REPORTING-BUGS for bug reporting
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:14 +0000 (21:08 +0300)]
man: refer to REPORTING-BUGS for bug reporting

REPORTING-BUGS should be mentioned here rather than README, provided
REPORTING-BUGS also mentions to see README for the mailing list
in addition to details of bug reporting.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman: sync "PARAMETER TYPES" section with HOWTO
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:13 +0000 (21:08 +0300)]
man: sync "PARAMETER TYPES" section with HOWTO

This commit brings in updates from HOWTO to man page. As a result it
replaces some entire paragraphs with the corresponding ones in HOWTO,
though not much difference for the replaced part as it only had small
modification basically saying the same thing.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman: sync "JOB FILE PARAMETERS" section with HOWTO
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:12 +0000 (21:08 +0300)]
man: sync "JOB FILE PARAMETERS" section with HOWTO

This commit replaces the entire section with the corresponding section
in HOWTO, since the man page is only slightly different from HOWTO
regarding this section.

It also renames the following section names to sync with HOWTO.
  "JOB PARAMETERS" to "JOB FILE PARAMETERS"
  "JOB PARAMETERS"'s sub section "Types" to non-sub section "PARAMETER TYPES"
  "JOB PARAMETERS"'s sub section "Parameter List" to non-sub section "JOB DESCRIPTION"

Since section names aren't following the typical man page convention
(man-pages(7) in Linux) to begin with, changing the section names
to conform to HOWTO does no harm unless there's a reason to keep names
that were once made.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman: sync "JOB FILE FORMAT" section with HOWTO
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:11 +0000 (21:08 +0300)]
man: sync "JOB FILE FORMAT" section with HOWTO

This commit brings in updates from HOWTO to man page. As a result it
replaces some entire paragraphs with the corresponding ones in HOWTO,
though not much difference for the replaced part as it only had small
modification basically saying the same thing.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman: sync OPTIONS section with HOWTO
Tomohiro Kusumi [Tue, 4 Jul 2017 18:08:10 +0000 (21:08 +0300)]
man: sync OPTIONS section with HOWTO

This commit brings in updates from HOWTO. The exception is --debug
option, which is too verbose yet not that important for all debug types
to be listed in details.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'fallocate_native' of https://github.com/sitsofe/fio
Jens Axboe [Mon, 3 Jul 2017 22:51:31 +0000 (16:51 -0600)]
Merge branch 'fallocate_native' of https://github.com/sitsofe/fio

3 years agoHOWTO: indent v3 terse output definition
Sitsofe Wheeler [Mon, 3 Jul 2017 21:16:13 +0000 (22:16 +0100)]
HOWTO: indent v3 terse output definition

Without indentation Sphinx complains:
../HOWTO:3336: WARNING: Literal block expected; none found.
../HOWTO:3336: WARNING: Literal block expected; none found.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agofilesetup: add native fallocate
Sitsofe Wheeler [Fri, 30 Jun 2017 06:32:19 +0000 (07:32 +0100)]
filesetup: add native fallocate

- Implement a native fallocate mode that only logs an error (if it's
  implemented) and falls back to the same behaviour as fallocate=none if
  it fails
- Add a native OSX and Linux fallocate
- Update man page and HOWTO with new native and defaults

v4:
- Update commit message
- Update man page and HOWTO

Fixes https://github.com/axboe/fio/issues/22 - we now print a message if
fallocate fails which gives the user a hint that behaviour might differ
from other case along with a workaround suggestion (fallocate=none).

Fixes https://github.com/axboe/fio/issues/376 - we now have
fallocate=none fallback behaviour if native fallocate fails. On at least
glibc platforms, posix_fallocate emulates fallocate if the
platform/filesystem doesn't support it (see
https://www.gnu.org/software/libc/manual/html_node/Storage-Allocation.html#index-posix_005ffallocate
for some of the caveats) resulting in strange layout performance thus
making it less attractive than fallocate=none behaviour for fio.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agogettime: reduce test CPU clock entries to 1000
Jens Axboe [Sun, 2 Jul 2017 22:21:47 +0000 (16:21 -0600)]
gettime: reduce test CPU clock entries to 1000

We don't need 10,000 and it's just wasting time.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agofilesetup: abstract out fallocate helper
Jens Axboe [Sun, 2 Jul 2017 22:15:32 +0000 (16:15 -0600)]
filesetup: abstract out fallocate helper

This is in preparation for improving how we handle fallocate.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agofio: make gauss a duplicate of normal for file_service_type
Sitsofe Wheeler [Wed, 28 Jun 2017 21:53:06 +0000 (22:53 +0100)]
fio: make gauss a duplicate of normal for file_service_type

To be consistent with random_distribution add a normal parameter to
file_service_type and state that gauss is an alias of it.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodoc: fix random_distribution Gaussian parameter name
Sitsofe Wheeler [Wed, 28 Jun 2017 21:30:24 +0000 (22:30 +0100)]
doc: fix random_distribution Gaussian parameter name

random_distribution's parameter for a Gaussian distribution is called
"normal". Note this differs to file_service_type where the Gaussian
distribution really is selected with the "gauss" parameter.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agostat: further group percentage fixes
Sitsofe Wheeler [Tue, 27 Jun 2017 06:21:32 +0000 (07:21 +0100)]
stat: further group percentage fixes

db84b73bd7b0c3b718596fbeb6a5f940b05a6735 forgot to fix the group
percentage calculation when using json or terse output. It also did its
caculation after the mean's base might have been changed to bits but the
aggregate bandwidth's unit is (k)bytes. Fix these.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoadd FD_PARSE debug print for size= option (which exists in offset=)
Tomohiro Kusumi [Wed, 28 Jun 2017 14:21:06 +0000 (17:21 +0300)]
add FD_PARSE debug print for size= option (which exists in offset=)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO/manpage: update percentage explanation using '%'
Tomohiro Kusumi [Wed, 28 Jun 2017 14:21:05 +0000 (17:21 +0300)]
HOWTO/manpage: update percentage explanation using '%'

82dbb8cb (HOWTO: add offset unit info for offset= option) added
how to specify offset= option in percentage format based on below post,
which was technically correct, but str_to_decimal() invoked via
option parser can convert '%' format to low-level representation.

This commit replaces 82dbb8cb using '%' syntax since this is
considered better for documentation. Note that size= option which
also supports percentage uses '%' in documentation.

in what units is offset option?
http://www.spinics.net/lists/fio/msg05969.html

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoblktrace: remove unused ioctl definitions
Jens Axboe [Wed, 28 Jun 2017 01:59:49 +0000 (19:59 -0600)]
blktrace: remove unused ioctl definitions

They can conflict with <linux/fs.h> definitions, if we have them.
Fio doesn't use them at all, so just kill them.

Fixes: https://github.com/axboe/fio/issues/391
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agomtd: add private rpmatch()
Jens Axboe [Wed, 28 Jun 2017 00:01:17 +0000 (18:01 -0600)]
mtd: add private rpmatch()

No point in doing elaborate checks for uclibc versions, when
we can just include this (very) small function ourselves.
Apparently this is also problematic on Android.

Fixes: https://github.com/axboe/fio/issues/390
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoUpdate API for file write hints
Jens Axboe [Tue, 27 Jun 2017 19:06:57 +0000 (13:06 -0600)]
Update API for file write hints

This is now queued up for 4.13 inclusion. Adjust the bits we need
to conform to what is queued up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman page: add offset unit info for offset= option
Tomohiro Kusumi [Tue, 27 Jun 2017 15:20:07 +0000 (18:20 +0300)]
man page: add offset unit info for offset= option

Apply below commit to fio(1) man page.
82dbb8cb (HOWTO: add offset unit info for offset= option)

Certain sections/options of the man page are not in sync with HOWTO,
but this one was identical, thus the changes are the same as above.

--
On 06/27/2017 08:56 AM, Elliott, Robert (Persistent Memory) wrote:
>>  HOWTO | 8 +++++---
>>  1 file changed, 5 insertions(+), 3 deletions(-)
>
> The manpage (fio.1) should be updated as well - that's more visible to users.
Agree, I usually catch that.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO: add offset unit info for offset= option
Tomohiro Kusumi [Tue, 27 Jun 2017 10:21:03 +0000 (13:21 +0300)]
HOWTO: add offset unit info for offset= option

Add "in bytes" and "A percentage can be specified by ...",
based on an email post titled "in what units is offset option?".

-- example of offset in percentage
 # ./fio --debug=parse --name=xxxxx --ioengine=sync --rw=rw --size=1m --unlink=1 --offset=-51 | grep "SET start_offset_percent"
 parse    12859 SET start_offset_percent 50

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agolib/ffz: remove dead store
Daniel Verkamp [Tue, 27 Jun 2017 13:24:32 +0000 (07:24 -0600)]
lib/ffz: remove dead store

word' is written in the last 'if' block but never read again.

This is harmless, but scan-build (clang-analyzer) flags it as a warning.

Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO: fix the v3 terse output definition
Ido Ben-Tsion [Tue, 27 Jun 2017 13:22:15 +0000 (07:22 -0600)]
HOWTO: fix the v3 terse output definition

There's a typo on the HOWTO file, read_clat_min and read_clat_max are
swapped.  and so are their write counterparts. Also - pu_minf instead of
cpu_minf.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: fix group percentage
Sitsofe Wheeler [Mon, 26 Jun 2017 23:04:40 +0000 (00:04 +0100)]
stat: fix group percentage

When af7f87cb59fb3fc29a7d9f56f03f7b42680a45f2 converted group stats to
bytes it broke the group percentage calculation because the mean is in
kilobytes but the aggregate bandwidth is in kilobytes. Fix this by
converting aggregate bandwidth to kilobytes when calculating the
thread's group percentage.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agostat: fix printf format specifier
Sitsofe Wheeler [Mon, 26 Jun 2017 23:38:59 +0000 (00:38 +0100)]
stat: fix printf format specifier

54c05828da413f6b4f9eec800a30624c453a4d90 introduced the printing of the
number of samples but on 64 bit macOS with clang the format specifier
generates warnings like:

stat.c:502:4: warning: format specifies type 'unsigned long' but the argument has type 'uint64_t'
      (aka 'unsigned long long') [-Wformat]
                        (&ts->bw_stat[ddir])->samples);
                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Fix this by using the PRIu64 macro.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoREADME: update Red Hat fio package URL
Sitsofe Wheeler [Fri, 23 Jun 2017 10:13:39 +0000 (11:13 +0100)]
README: update Red Hat fio package URL

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: note that crc32c will automatically use hw
Sitsofe Wheeler [Sun, 25 Jun 2017 22:09:07 +0000 (23:09 +0100)]
HOWTO: note that crc32c will automatically use hw

- Add note that crc32c checksumming will automatically use hardware
  acceleration if possible.
- Add note that crc32c-intel is just a synonym for crc32c.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoinit: update --crctest help syntax
Sitsofe Wheeler [Fri, 23 Jun 2017 05:40:54 +0000 (06:40 +0100)]
init: update --crctest help syntax

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO/examples: fix writetrim "typo"
Sitsofe Wheeler [Fri, 23 Jun 2017 06:31:03 +0000 (07:31 +0100)]
HOWTO/examples: fix writetrim "typo"

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: modernize output examples and descriptions
Sitsofe Wheeler [Fri, 23 Jun 2017 06:01:50 +0000 (07:01 +0100)]
HOWTO: modernize output examples and descriptions

- Update output examples to reflect what is produced by the current fio
- Add hidden comments saying how to generate output for fio examples.
- Update output descriptions to match examples.
- Add missing thread lifecycle states.

3 years agoHOWTO: Reword Log File Formats and add reference
Sitsofe Wheeler [Fri, 23 Jun 2017 23:17:03 +0000 (00:17 +0100)]
HOWTO: Reword Log File Formats and add reference

- Add reference to Log File Formats in log_avg_msec
- Reword Log File Formats for improved clarity.
- In Log File Formats note that IOPS will always be 1 when using the
  default of individual I/O logging.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: reword iodepth and submit distribution text
Sitsofe Wheeler [Wed, 21 Jun 2017 06:29:54 +0000 (07:29 +0100)]
HOWTO: reword iodepth and submit distribution text

Reword the explanation text so phrasing of iodepth and submit
distributions is more similar. Also draw attention to the fact the value
represents the end-point in iodepth distributions and the start-point in
submit distributions.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: reorder client/server phrasing
Sitsofe Wheeler [Fri, 23 Jun 2017 22:34:51 +0000 (23:34 +0100)]
HOWTO: reorder client/server phrasing

Reorder sentences so that backend is in the same position as server and the
frontend is in the same position as client.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add some markup
Sitsofe Wheeler [Fri, 23 Jun 2017 22:31:52 +0000 (23:31 +0100)]
HOWTO: add some markup

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add rate example
Sitsofe Wheeler [Fri, 23 Jun 2017 22:27:23 +0000 (23:27 +0100)]
HOWTO: add rate example

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add --output-format=terse as another way to get minimal output
Sitsofe Wheeler [Fri, 23 Jun 2017 06:37:24 +0000 (07:37 +0100)]
HOWTO: add --output-format=terse as another way to get minimal output

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: reword HDFS description
Sitsofe Wheeler [Fri, 23 Jun 2017 06:36:42 +0000 (07:36 +0100)]
HOWTO: reword HDFS description

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: make filesize syntax show it can take a typed range
Sitsofe Wheeler [Fri, 23 Jun 2017 06:35:47 +0000 (07:35 +0100)]
HOWTO: make filesize syntax show it can take a typed range

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: Fix some capitalisation
Sitsofe Wheeler [Fri, 23 Jun 2017 06:35:14 +0000 (07:35 +0100)]
HOWTO: Fix some capitalisation

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: description rewording/fixes
Sitsofe Wheeler [Fri, 23 Jun 2017 06:27:31 +0000 (07:27 +0100)]
HOWTO: description rewording/fixes

Reword descriptions to make them easier to understand, fix
typos/grammar, fix formatting and reduce duplication.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: minor internal/reordering/formatting changes
Sitsofe Wheeler [Fri, 23 Jun 2017 06:08:54 +0000 (07:08 +0100)]
HOWTO: minor internal/reordering/formatting changes

3 years agoHOWTO: general consistency
Sitsofe Wheeler [Fri, 23 Jun 2017 05:53:07 +0000 (06:53 +0100)]
HOWTO: general consistency

- Prefer built in to builtin
- Link to man page when mentioning syscalls.
- Standardize on behavior (grumble) rather than behaviour.
- Prefer hostname to host name.
- Prefer filename to file name.
- Prefer timestamps to time stamps.
- Prefer bit mask to bitmask.
- Prefer zeros to zeroes.
- Reorder comparison operators so they work ;-)
- Fix up an unwanted apostrophe too.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: update command line option descriptions
Sitsofe Wheeler [Fri, 23 Jun 2017 05:36:45 +0000 (06:36 +0100)]
HOWTO: update command line option descriptions

- Make it clearer what --help will print (since page is ambiguous in
  the context of the HOWTO).
- Update the wording and syntax for --crctest so it's clear it doesn't
  always need a parameter.
- Use KiB rather than kb to refer to powers of two byte sizes and update
  the maximum number of pools.
- Make the --idle-prof description more verbose (and clear up confusion
  that might make people try and pass it literal option
  'system,percpu').

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: update directory and filename option descriptions
Sitsofe Wheeler [Thu, 22 Jun 2017 06:22:38 +0000 (07:22 +0100)]
HOWTO: update directory and filename option descriptions

- Minor grammar changes.
- Explictly note that '\' and ':' need escaping.
- Turn reference to filename_format into link.
- Split escaping into two paragraphs and add another Windows example.
- Make it clearer '-' can be a filename
- Markup stdin/stdout.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: update time specification
Sitsofe Wheeler [Thu, 22 Jun 2017 06:20:34 +0000 (07:20 +0100)]
HOWTO: update time specification

Add that time may take a suffix for day and list all time suffixes in
the same format.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: escape =
Sitsofe Wheeler [Thu, 22 Jun 2017 05:49:51 +0000 (06:49 +0100)]
HOWTO: escape =

We often don't want a space to be put before an equals sign. Workaround
Sphinx doing this by escaping the equals with a '\'.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: grammar/spelling changes
Sitsofe Wheeler [Thu, 22 Jun 2017 05:15:24 +0000 (06:15 +0100)]
HOWTO: grammar/spelling changes

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: state default time unit
Sitsofe Wheeler [Thu, 22 Jun 2017 05:55:24 +0000 (06:55 +0100)]
HOWTO: state default time unit

Where an fio option takes a time, state how the value will be
interpreted if the unit is left off.

Also remove explanation of time units in startdelay as it's clear time
can take different units already.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add defaults
Sitsofe Wheeler [Fri, 23 Jun 2017 22:57:19 +0000 (23:57 +0100)]
HOWTO: add defaults

- Update defaults.
- Linkify reference to end_fsync.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agostat: fix alignment of the iops stats
Jens Axboe [Mon, 26 Jun 2017 14:50:49 +0000 (08:50 -0600)]
stat: fix alignment of the iops stats

Fixes: 54c05828da4 ("stat: Print number of samples in bw and iops stats")
Reported-by: Elliott, Robert (Persistent Memory) <elliott@hpe.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoAndroid: fix missing sysmacros.h include
Jens Axboe [Sun, 25 Jun 2017 15:51:13 +0000 (09:51 -0600)]
Android: fix missing sysmacros.h include

Fix from Elliott Hughes and Sitsofe Wheeler.

Fixes: https://github.com/axboe/fio/issues/367
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Add iops stat and sample number information to terse format
Andreas Herrmann [Fri, 23 Jun 2017 22:27:38 +0000 (00:27 +0200)]
stat: Add iops stat and sample number information to terse format

This is inserted beyond bandwidth information. It required creation
of a new terse version (v5).

Updated man page and HOWTO accordingly. Also added hints when fields
are specific to certain terse versions.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Merge show_thread_status_terse_* functions
Andreas Herrmann [Fri, 23 Jun 2017 22:27:37 +0000 (00:27 +0200)]
stat: Merge show_thread_status_terse_* functions

Merge show_thread_status_terse_v2 and show_thread_status_terse_v3_v4
into new function show_thread_status_terse_all.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Print number of samples in bw and iops stats
Andreas Herrmann [Fri, 23 Jun 2017 22:27:36 +0000 (00:27 +0200)]
stat: Print number of samples in bw and iops stats

This adds the actual number of samples for bw and iops stats printed
in normal and json fio output. Example for normal output:

   bw (  MiB/s): min=  120, max= 3304, per=0.10%, avg=2659.34, stdev=1085.25, samples=  109
   iops        : min=102776, max=830416, avg=675639.05, stdev=280619.99, samples=   21

Note: This example was created using options
"--write_bw_log=... --log_avg_msec=100 --runtime=11s" (ie. no iops log
written). That is why number of samples differs for IOPs and BW.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Print one-line iops stat
Andreas Herrmann [Fri, 23 Jun 2017 22:27:35 +0000 (00:27 +0200)]
stat: Print one-line iops stat

This adds basic iops statistics (likewise to what we already have for
bandwidth) to normal and json fio output. Example for normal output:

   bw (  KiB/s): min=42192, max=162068, per=0.10%, avg=93236.21, stdev=22180.02
   iops : min=21090, max=81020, avg=46606.84, stdev=11088.40
    lat (usec) : 2=55.13%, 4=21.11%, 10=0.36%, 20=0.03%, 50=23.14%

Added handling for iops_stat to client/server and bump server version.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agosmalloc: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:08:01 +0000 (01:08 +0300)]
smalloc: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoverify: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:08:00 +0000 (01:08 +0300)]
verify: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoclient: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:59 +0000 (01:07 +0300)]
client: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoinit: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:58 +0000 (01:07 +0300)]
init: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

(threads is a pointer to struct thread_data)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoio_u: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:57 +0000 (01:07 +0300)]
io_u: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoserver: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:56 +0000 (01:07 +0300)]
server: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMakefile: use fmt(1) rather than tr(1) on NetBSD/etc
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:55 +0000 (01:07 +0300)]
Makefile: use fmt(1) rather than tr(1) on NetBSD/etc

e291cff1 (Use fmt -w WIDTH option instead of -WIDTH) changed the
command option to a common one to suppress an error, but it only
fixed the unknown option error without being functional on NetBSD.

This change is taken from NetBSD's pkgsrc. It may work against
other platforms that don't work with fmt(1), but only enabled for
NetBSD at the moment. FreeBSD/DragonFlyBSD/OpenBSD work with fmt(1).

(This actually works on Linux too, but the existing one should be
kept for Linux (and other platforms) provided it has been used on
various distros for years)

https://github.com/NetBSD/pkgsrc/blob/trunk/benchmarks/fio/patches/patch-Makefile

> Convert the fmt(1) command to a tr(1) one (the fmt(1) old syntax
> command is not supported on all Unix systems).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/time-test: cleanups
Jens Axboe [Fri, 23 Jun 2017 01:29:35 +0000 (19:29 -0600)]
t/time-test: cleanups

Kill dead code. It's in git history now, so can easily be resurrected.

Fixup the worst style offenses.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoiolog: ensure proper flushing of compressed logs
Jens Axboe [Thu, 22 Jun 2017 23:15:27 +0000 (17:15 -0600)]
iolog: ensure proper flushing of compressed logs

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoiolog: punt freeing of data back to original thread
Jens Axboe [Thu, 22 Jun 2017 22:20:36 +0000 (16:20 -0600)]
iolog: punt freeing of data back to original thread

This is a bit of a hack, since we can't touch the memory passed in.
But we should, at most have 1-2 entries deferred, so having an
array of 8 should suffice. Add a single-trigger warning if that
logic fails.

This should fix corruption on using log compression, when we have
to do runtime compressing.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoiolog: get work items out of shared memory pool
Jens Axboe [Thu, 22 Jun 2017 21:50:01 +0000 (15:50 -0600)]
iolog: get work items out of shared memory pool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoclient/server: bool conversion
Jens Axboe [Thu, 22 Jun 2017 20:10:23 +0000 (14:10 -0600)]
client/server: bool conversion

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoarch: tsc_reliable can be a bool
Jens Axboe [Thu, 22 Jun 2017 20:12:29 +0000 (14:12 -0600)]
arch: tsc_reliable can be a bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agocrc32c: use bool
Jens Axboe [Thu, 22 Jun 2017 20:03:14 +0000 (14:03 -0600)]
crc32c: use bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>