fio.git
3 years agofio: make gauss a duplicate of normal for file_service_type
Sitsofe Wheeler [Wed, 28 Jun 2017 21:53:06 +0000 (22:53 +0100)]
fio: make gauss a duplicate of normal for file_service_type

To be consistent with random_distribution add a normal parameter to
file_service_type and state that gauss is an alias of it.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodoc: fix random_distribution Gaussian parameter name
Sitsofe Wheeler [Wed, 28 Jun 2017 21:30:24 +0000 (22:30 +0100)]
doc: fix random_distribution Gaussian parameter name

random_distribution's parameter for a Gaussian distribution is called
"normal". Note this differs to file_service_type where the Gaussian
distribution really is selected with the "gauss" parameter.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agostat: further group percentage fixes
Sitsofe Wheeler [Tue, 27 Jun 2017 06:21:32 +0000 (07:21 +0100)]
stat: further group percentage fixes

db84b73bd7b0c3b718596fbeb6a5f940b05a6735 forgot to fix the group
percentage calculation when using json or terse output. It also did its
caculation after the mean's base might have been changed to bits but the
aggregate bandwidth's unit is (k)bytes. Fix these.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoadd FD_PARSE debug print for size= option (which exists in offset=)
Tomohiro Kusumi [Wed, 28 Jun 2017 14:21:06 +0000 (17:21 +0300)]
add FD_PARSE debug print for size= option (which exists in offset=)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO/manpage: update percentage explanation using '%'
Tomohiro Kusumi [Wed, 28 Jun 2017 14:21:05 +0000 (17:21 +0300)]
HOWTO/manpage: update percentage explanation using '%'

82dbb8cb (HOWTO: add offset unit info for offset= option) added
how to specify offset= option in percentage format based on below post,
which was technically correct, but str_to_decimal() invoked via
option parser can convert '%' format to low-level representation.

This commit replaces 82dbb8cb using '%' syntax since this is
considered better for documentation. Note that size= option which
also supports percentage uses '%' in documentation.

in what units is offset option?
http://www.spinics.net/lists/fio/msg05969.html

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoblktrace: remove unused ioctl definitions
Jens Axboe [Wed, 28 Jun 2017 01:59:49 +0000 (19:59 -0600)]
blktrace: remove unused ioctl definitions

They can conflict with <linux/fs.h> definitions, if we have them.
Fio doesn't use them at all, so just kill them.

Fixes: https://github.com/axboe/fio/issues/391
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agomtd: add private rpmatch()
Jens Axboe [Wed, 28 Jun 2017 00:01:17 +0000 (18:01 -0600)]
mtd: add private rpmatch()

No point in doing elaborate checks for uclibc versions, when
we can just include this (very) small function ourselves.
Apparently this is also problematic on Android.

Fixes: https://github.com/axboe/fio/issues/390
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoUpdate API for file write hints
Jens Axboe [Tue, 27 Jun 2017 19:06:57 +0000 (13:06 -0600)]
Update API for file write hints

This is now queued up for 4.13 inclusion. Adjust the bits we need
to conform to what is queued up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoman page: add offset unit info for offset= option
Tomohiro Kusumi [Tue, 27 Jun 2017 15:20:07 +0000 (18:20 +0300)]
man page: add offset unit info for offset= option

Apply below commit to fio(1) man page.
82dbb8cb (HOWTO: add offset unit info for offset= option)

Certain sections/options of the man page are not in sync with HOWTO,
but this one was identical, thus the changes are the same as above.

--
On 06/27/2017 08:56 AM, Elliott, Robert (Persistent Memory) wrote:
>>  HOWTO | 8 +++++---
>>  1 file changed, 5 insertions(+), 3 deletions(-)
>
> The manpage (fio.1) should be updated as well - that's more visible to users.
Agree, I usually catch that.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO: add offset unit info for offset= option
Tomohiro Kusumi [Tue, 27 Jun 2017 10:21:03 +0000 (13:21 +0300)]
HOWTO: add offset unit info for offset= option

Add "in bytes" and "A percentage can be specified by ...",
based on an email post titled "in what units is offset option?".

-- example of offset in percentage
 # ./fio --debug=parse --name=xxxxx --ioengine=sync --rw=rw --size=1m --unlink=1 --offset=-51 | grep "SET start_offset_percent"
 parse    12859 SET start_offset_percent 50

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agolib/ffz: remove dead store
Daniel Verkamp [Tue, 27 Jun 2017 13:24:32 +0000 (07:24 -0600)]
lib/ffz: remove dead store

word' is written in the last 'if' block but never read again.

This is harmless, but scan-build (clang-analyzer) flags it as a warning.

Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoHOWTO: fix the v3 terse output definition
Ido Ben-Tsion [Tue, 27 Jun 2017 13:22:15 +0000 (07:22 -0600)]
HOWTO: fix the v3 terse output definition

There's a typo on the HOWTO file, read_clat_min and read_clat_max are
swapped.  and so are their write counterparts. Also - pu_minf instead of
cpu_minf.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: fix group percentage
Sitsofe Wheeler [Mon, 26 Jun 2017 23:04:40 +0000 (00:04 +0100)]
stat: fix group percentage

When af7f87cb59fb3fc29a7d9f56f03f7b42680a45f2 converted group stats to
bytes it broke the group percentage calculation because the mean is in
kilobytes but the aggregate bandwidth is in kilobytes. Fix this by
converting aggregate bandwidth to kilobytes when calculating the
thread's group percentage.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agostat: fix printf format specifier
Sitsofe Wheeler [Mon, 26 Jun 2017 23:38:59 +0000 (00:38 +0100)]
stat: fix printf format specifier

54c05828da413f6b4f9eec800a30624c453a4d90 introduced the printing of the
number of samples but on 64 bit macOS with clang the format specifier
generates warnings like:

stat.c:502:4: warning: format specifies type 'unsigned long' but the argument has type 'uint64_t'
      (aka 'unsigned long long') [-Wformat]
                        (&ts->bw_stat[ddir])->samples);
                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Fix this by using the PRIu64 macro.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoREADME: update Red Hat fio package URL
Sitsofe Wheeler [Fri, 23 Jun 2017 10:13:39 +0000 (11:13 +0100)]
README: update Red Hat fio package URL

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: note that crc32c will automatically use hw
Sitsofe Wheeler [Sun, 25 Jun 2017 22:09:07 +0000 (23:09 +0100)]
HOWTO: note that crc32c will automatically use hw

- Add note that crc32c checksumming will automatically use hardware
  acceleration if possible.
- Add note that crc32c-intel is just a synonym for crc32c.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoinit: update --crctest help syntax
Sitsofe Wheeler [Fri, 23 Jun 2017 05:40:54 +0000 (06:40 +0100)]
init: update --crctest help syntax

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO/examples: fix writetrim "typo"
Sitsofe Wheeler [Fri, 23 Jun 2017 06:31:03 +0000 (07:31 +0100)]
HOWTO/examples: fix writetrim "typo"

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: modernize output examples and descriptions
Sitsofe Wheeler [Fri, 23 Jun 2017 06:01:50 +0000 (07:01 +0100)]
HOWTO: modernize output examples and descriptions

- Update output examples to reflect what is produced by the current fio
- Add hidden comments saying how to generate output for fio examples.
- Update output descriptions to match examples.
- Add missing thread lifecycle states.

3 years agoHOWTO: Reword Log File Formats and add reference
Sitsofe Wheeler [Fri, 23 Jun 2017 23:17:03 +0000 (00:17 +0100)]
HOWTO: Reword Log File Formats and add reference

- Add reference to Log File Formats in log_avg_msec
- Reword Log File Formats for improved clarity.
- In Log File Formats note that IOPS will always be 1 when using the
  default of individual I/O logging.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: reword iodepth and submit distribution text
Sitsofe Wheeler [Wed, 21 Jun 2017 06:29:54 +0000 (07:29 +0100)]
HOWTO: reword iodepth and submit distribution text

Reword the explanation text so phrasing of iodepth and submit
distributions is more similar. Also draw attention to the fact the value
represents the end-point in iodepth distributions and the start-point in
submit distributions.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: reorder client/server phrasing
Sitsofe Wheeler [Fri, 23 Jun 2017 22:34:51 +0000 (23:34 +0100)]
HOWTO: reorder client/server phrasing

Reorder sentences so that backend is in the same position as server and the
frontend is in the same position as client.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add some markup
Sitsofe Wheeler [Fri, 23 Jun 2017 22:31:52 +0000 (23:31 +0100)]
HOWTO: add some markup

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add rate example
Sitsofe Wheeler [Fri, 23 Jun 2017 22:27:23 +0000 (23:27 +0100)]
HOWTO: add rate example

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add --output-format=terse as another way to get minimal output
Sitsofe Wheeler [Fri, 23 Jun 2017 06:37:24 +0000 (07:37 +0100)]
HOWTO: add --output-format=terse as another way to get minimal output

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: reword HDFS description
Sitsofe Wheeler [Fri, 23 Jun 2017 06:36:42 +0000 (07:36 +0100)]
HOWTO: reword HDFS description

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: make filesize syntax show it can take a typed range
Sitsofe Wheeler [Fri, 23 Jun 2017 06:35:47 +0000 (07:35 +0100)]
HOWTO: make filesize syntax show it can take a typed range

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: Fix some capitalisation
Sitsofe Wheeler [Fri, 23 Jun 2017 06:35:14 +0000 (07:35 +0100)]
HOWTO: Fix some capitalisation

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: description rewording/fixes
Sitsofe Wheeler [Fri, 23 Jun 2017 06:27:31 +0000 (07:27 +0100)]
HOWTO: description rewording/fixes

Reword descriptions to make them easier to understand, fix
typos/grammar, fix formatting and reduce duplication.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: minor internal/reordering/formatting changes
Sitsofe Wheeler [Fri, 23 Jun 2017 06:08:54 +0000 (07:08 +0100)]
HOWTO: minor internal/reordering/formatting changes

3 years agoHOWTO: general consistency
Sitsofe Wheeler [Fri, 23 Jun 2017 05:53:07 +0000 (06:53 +0100)]
HOWTO: general consistency

- Prefer built in to builtin
- Link to man page when mentioning syscalls.
- Standardize on behavior (grumble) rather than behaviour.
- Prefer hostname to host name.
- Prefer filename to file name.
- Prefer timestamps to time stamps.
- Prefer bit mask to bitmask.
- Prefer zeros to zeroes.
- Reorder comparison operators so they work ;-)
- Fix up an unwanted apostrophe too.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: update command line option descriptions
Sitsofe Wheeler [Fri, 23 Jun 2017 05:36:45 +0000 (06:36 +0100)]
HOWTO: update command line option descriptions

- Make it clearer what --help will print (since page is ambiguous in
  the context of the HOWTO).
- Update the wording and syntax for --crctest so it's clear it doesn't
  always need a parameter.
- Use KiB rather than kb to refer to powers of two byte sizes and update
  the maximum number of pools.
- Make the --idle-prof description more verbose (and clear up confusion
  that might make people try and pass it literal option
  'system,percpu').

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: update directory and filename option descriptions
Sitsofe Wheeler [Thu, 22 Jun 2017 06:22:38 +0000 (07:22 +0100)]
HOWTO: update directory and filename option descriptions

- Minor grammar changes.
- Explictly note that '\' and ':' need escaping.
- Turn reference to filename_format into link.
- Split escaping into two paragraphs and add another Windows example.
- Make it clearer '-' can be a filename
- Markup stdin/stdout.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: update time specification
Sitsofe Wheeler [Thu, 22 Jun 2017 06:20:34 +0000 (07:20 +0100)]
HOWTO: update time specification

Add that time may take a suffix for day and list all time suffixes in
the same format.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: escape =
Sitsofe Wheeler [Thu, 22 Jun 2017 05:49:51 +0000 (06:49 +0100)]
HOWTO: escape =

We often don't want a space to be put before an equals sign. Workaround
Sphinx doing this by escaping the equals with a '\'.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: grammar/spelling changes
Sitsofe Wheeler [Thu, 22 Jun 2017 05:15:24 +0000 (06:15 +0100)]
HOWTO: grammar/spelling changes

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: state default time unit
Sitsofe Wheeler [Thu, 22 Jun 2017 05:55:24 +0000 (06:55 +0100)]
HOWTO: state default time unit

Where an fio option takes a time, state how the value will be
interpreted if the unit is left off.

Also remove explanation of time units in startdelay as it's clear time
can take different units already.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoHOWTO: add defaults
Sitsofe Wheeler [Fri, 23 Jun 2017 22:57:19 +0000 (23:57 +0100)]
HOWTO: add defaults

- Update defaults.
- Linkify reference to end_fsync.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agostat: fix alignment of the iops stats
Jens Axboe [Mon, 26 Jun 2017 14:50:49 +0000 (08:50 -0600)]
stat: fix alignment of the iops stats

Fixes: 54c05828da4 ("stat: Print number of samples in bw and iops stats")
Reported-by: Elliott, Robert (Persistent Memory) <elliott@hpe.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoAndroid: fix missing sysmacros.h include
Jens Axboe [Sun, 25 Jun 2017 15:51:13 +0000 (09:51 -0600)]
Android: fix missing sysmacros.h include

Fix from Elliott Hughes and Sitsofe Wheeler.

Fixes: https://github.com/axboe/fio/issues/367
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Add iops stat and sample number information to terse format
Andreas Herrmann [Fri, 23 Jun 2017 22:27:38 +0000 (00:27 +0200)]
stat: Add iops stat and sample number information to terse format

This is inserted beyond bandwidth information. It required creation
of a new terse version (v5).

Updated man page and HOWTO accordingly. Also added hints when fields
are specific to certain terse versions.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Merge show_thread_status_terse_* functions
Andreas Herrmann [Fri, 23 Jun 2017 22:27:37 +0000 (00:27 +0200)]
stat: Merge show_thread_status_terse_* functions

Merge show_thread_status_terse_v2 and show_thread_status_terse_v3_v4
into new function show_thread_status_terse_all.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Print number of samples in bw and iops stats
Andreas Herrmann [Fri, 23 Jun 2017 22:27:36 +0000 (00:27 +0200)]
stat: Print number of samples in bw and iops stats

This adds the actual number of samples for bw and iops stats printed
in normal and json fio output. Example for normal output:

   bw (  MiB/s): min=  120, max= 3304, per=0.10%, avg=2659.34, stdev=1085.25, samples=  109
   iops        : min=102776, max=830416, avg=675639.05, stdev=280619.99, samples=   21

Note: This example was created using options
"--write_bw_log=... --log_avg_msec=100 --runtime=11s" (ie. no iops log
written). That is why number of samples differs for IOPs and BW.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: Print one-line iops stat
Andreas Herrmann [Fri, 23 Jun 2017 22:27:35 +0000 (00:27 +0200)]
stat: Print one-line iops stat

This adds basic iops statistics (likewise to what we already have for
bandwidth) to normal and json fio output. Example for normal output:

   bw (  KiB/s): min=42192, max=162068, per=0.10%, avg=93236.21, stdev=22180.02
   iops : min=21090, max=81020, avg=46606.84, stdev=11088.40
    lat (usec) : 2=55.13%, 4=21.11%, 10=0.36%, 20=0.03%, 50=23.14%

Added handling for iops_stat to client/server and bump server version.

Signed-off-by: Andreas Herrmann <aherrmann@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agosmalloc: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:08:01 +0000 (01:08 +0300)]
smalloc: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoverify: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:08:00 +0000 (01:08 +0300)]
verify: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoclient: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:59 +0000 (01:07 +0300)]
client: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoinit: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:58 +0000 (01:07 +0300)]
init: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

(threads is a pointer to struct thread_data)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoio_u: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:57 +0000 (01:07 +0300)]
io_u: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoserver: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:56 +0000 (01:07 +0300)]
server: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMakefile: use fmt(1) rather than tr(1) on NetBSD/etc
Tomohiro Kusumi [Fri, 23 Jun 2017 22:07:55 +0000 (01:07 +0300)]
Makefile: use fmt(1) rather than tr(1) on NetBSD/etc

e291cff1 (Use fmt -w WIDTH option instead of -WIDTH) changed the
command option to a common one to suppress an error, but it only
fixed the unknown option error without being functional on NetBSD.

This change is taken from NetBSD's pkgsrc. It may work against
other platforms that don't work with fmt(1), but only enabled for
NetBSD at the moment. FreeBSD/DragonFlyBSD/OpenBSD work with fmt(1).

(This actually works on Linux too, but the existing one should be
kept for Linux (and other platforms) provided it has been used on
various distros for years)

https://github.com/NetBSD/pkgsrc/blob/trunk/benchmarks/fio/patches/patch-Makefile

> Convert the fmt(1) command to a tr(1) one (the fmt(1) old syntax
> command is not supported on all Unix systems).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agot/time-test: cleanups
Jens Axboe [Fri, 23 Jun 2017 01:29:35 +0000 (19:29 -0600)]
t/time-test: cleanups

Kill dead code. It's in git history now, so can easily be resurrected.

Fixup the worst style offenses.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoiolog: ensure proper flushing of compressed logs
Jens Axboe [Thu, 22 Jun 2017 23:15:27 +0000 (17:15 -0600)]
iolog: ensure proper flushing of compressed logs

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoiolog: punt freeing of data back to original thread
Jens Axboe [Thu, 22 Jun 2017 22:20:36 +0000 (16:20 -0600)]
iolog: punt freeing of data back to original thread

This is a bit of a hack, since we can't touch the memory passed in.
But we should, at most have 1-2 entries deferred, so having an
array of 8 should suffice. Add a single-trigger warning if that
logic fails.

This should fix corruption on using log compression, when we have
to do runtime compressing.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoiolog: get work items out of shared memory pool
Jens Axboe [Thu, 22 Jun 2017 21:50:01 +0000 (15:50 -0600)]
iolog: get work items out of shared memory pool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoclient/server: bool conversion
Jens Axboe [Thu, 22 Jun 2017 20:10:23 +0000 (14:10 -0600)]
client/server: bool conversion

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoarch: tsc_reliable can be a bool
Jens Axboe [Thu, 22 Jun 2017 20:12:29 +0000 (14:12 -0600)]
arch: tsc_reliable can be a bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agocrc32c: use bool
Jens Axboe [Thu, 22 Jun 2017 20:03:14 +0000 (14:03 -0600)]
crc32c: use bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'nsec'
Jens Axboe [Thu, 22 Jun 2017 15:15:14 +0000 (09:15 -0600)]
Merge branch 'nsec'

Pull in the work from Vincent on enabling nanosecond resolution.

3 years agoFixup some style issues
Jens Axboe [Thu, 22 Jun 2017 15:15:03 +0000 (09:15 -0600)]
Fixup some style issues

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'nanosecond-2stage' of https://github.com/vincentkfu/fio into nsec
Jens Axboe [Thu, 22 Jun 2017 15:05:28 +0000 (09:05 -0600)]
Merge branch 'nanosecond-2stage' of https://github.com/vincentkfu/fio into nsec

Pull the nsec branch from Vincent.

3 years agoserver: bump server version for the change to FIO_IO_U_PLAT_GROUP_NR
Vincent Fu [Wed, 21 Jun 2017 14:42:10 +0000 (10:42 -0400)]
server: bump server version for the change to FIO_IO_U_PLAT_GROUP_NR

3 years agogettime: drop tv_valid->last_cycles and tv_valid->last_tv_valid
Vincent Fu [Tue, 20 Jun 2017 16:12:56 +0000 (12:12 -0400)]
gettime: drop tv_valid->last_cycles and tv_valid->last_tv_valid

The two variables were set on every call to fio_gettime that uses the CPU
clock. However, these values were never read anywhere. So we should drop
them to reduce the amount of work done in the fast path.

3 years agogettime: for better accuracy calculate cycles_per_msec instead of cycles_per_usec
Vincent Fu [Mon, 19 Jun 2017 19:12:36 +0000 (15:12 -0400)]
gettime: for better accuracy calculate cycles_per_msec instead of cycles_per_usec

fio (before the nsec changes) would finish about 9 sec early for a 16 hour job
when relying on the CPU clock. This is because fio calculates cycles_per_usec
to carry out the clock ticks to time conversion. cycles_per_usec only provides
4 significant digits. Changing this to cycles_per_msec provides 7 significant
digits and makes the actual job run time more closely match wall time.

3 years agonanosecond: update t/time-test.c to include experiments using seqlock for conversion
Vincent Fu [Mon, 1 May 2017 18:55:21 +0000 (14:55 -0400)]
nanosecond: update t/time-test.c to include experiments using seqlock for conversion

3 years agolib/seqlock: #include "types.h" for bool type
Vincent Fu [Thu, 27 Apr 2017 19:09:13 +0000 (15:09 -0400)]
lib/seqlock: #include "types.h" for bool type

3 years agonanosecond: add test program t/time-test for experimenting with cpu clock ticks to...
Vincent Fu [Tue, 11 Apr 2017 19:57:15 +0000 (15:57 -0400)]
nanosecond: add test program t/time-test for experimenting with cpu clock ticks to nsec conversion

3 years agonanosecond: fix up conversion of ticks to nsec by doing the conversion in 2 stages
Vincent Fu [Tue, 11 Apr 2017 17:04:08 +0000 (13:04 -0400)]
nanosecond: fix up conversion of ticks to nsec by doing the conversion in 2 stages

The ticks to nsec conversion is challenging because we care about small
differences between very large numbers. The solution used here is to do the
conversion in two stages.

The first stage carves out large chunks of time. For this patch each chunk is
approximately an hour long. The number of ticks per hour is calculated. When a
conversion is requested, an hour worth of time is accumulated each time this
number of ticks has elapsed.

The remaining ticks are handled in the second stage with the standard
multiplication and shift conversion.

This strategy deals with the multiplication-and-shift overflow problem by
limiting the scale of this operation to at most one hour. This way we can have
reasonable accuracy with this portion of the conversion. At the same time the
first stage allows the ticks to nsec conversion to be valid for long periods
of time.

3 years agonanosecond: fiologparser_hist set default --group_nr to 29 to match stat.h FIO_IO_U_P...
Vincent Fu [Wed, 22 Mar 2017 15:01:16 +0000 (11:01 -0400)]
nanosecond: fiologparser_hist set default --group_nr to 29 to match stat.h FIO_IO_U_PLAT_GROUP_NR

3 years agonanosecond: alter gfio to accommodate nanosecond timing
Vincent Fu [Tue, 21 Mar 2017 18:58:50 +0000 (14:58 -0400)]
nanosecond: alter gfio to accommodate nanosecond timing

Some of the plot labels need to be changed because they might
have values with nsec units.

3 years agonanosecond: reconcile terse output with nanosecond timing for latencies
Vincent Fu [Tue, 21 Mar 2017 15:22:08 +0000 (11:22 -0400)]
nanosecond: reconcile terse output with nanosecond timing for latencies

The terse output format is unchanged and latencies are still reported
as usec. Completion latencies are tracked with nsec precision in io_u_lat_n.
Put these latencies in the first microsecond latency bucket which is where they
would have fallen before the change to nsec timing.

3 years agonanosecond: update completion latency recording and normal, json output to use nanose...
Vincent Fu [Thu, 16 Mar 2017 16:11:46 +0000 (12:11 -0400)]
nanosecond: update completion latency recording and normal, json output to use nanoseconds

A new set of nsec bins (0-1, 2-3, 4-9, 10-19, ...) is created for tracking latencies.
The number of io_u_plat[] latency bins is increased by about 50% to accommodate the
greater resolution while retaining the same max latency value of ~9 seconds.

Example output

test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-2.20-39-g83b3
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=14.6GiB/s,w=0KiB/s][r=3832k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=17621: Thu Jun  1 15:09:38 2017
   read: IOPS=3784k, BW=14.4GiB/s (15.5GB/s)(144GiB/10000msec)
    clat (nsec): min=15, max=1685.2k, avg=26.92, stdev=433.62
     lat (nsec): min=46, max=5989.9k, avg=67.31, stdev=1659.58
    clat percentiles (nsec):
     |  1.00th=[   18],  5.00th=[   18], 10.00th=[   19], 20.00th=[   19],
     | 30.00th=[   22], 40.00th=[   22], 50.00th=[   25], 60.00th=[   26],
     | 70.00th=[   26], 80.00th=[   27], 90.00th=[   36], 95.00th=[   41],
     | 99.00th=[   46], 99.50th=[   50], 99.90th=[  104], 99.95th=[  139],
     | 99.99th=[ 7840]
   bw (  MiB/s): min=13393, max=15647, per=0.10%, avg=14784.86, stdev=558.40
    lat (nsec) : 20=20.87%, 50=78.57%, 100=0.43%, 250=0.10%, 500=0.01%
    lat (nsec) : 750=0.01%, 1000=0.01%
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%
  cpu          : usr=99.72%, sys=0.12%, ctx=321, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=37840231,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=14.4GiB/s (15.5GB/s), 14.4GiB/s-14.4GiB/s (15.5GB/s-15.5GB/s), io=144GiB (155GB), run=10000-10000msec

3 years agonanosecond: initial commit changing timeval to timespec
Vincent Fu [Tue, 14 Mar 2017 18:41:26 +0000 (14:41 -0400)]
nanosecond: initial commit changing timeval to timespec

struct timeval has usec resolution whereas struct timespec has nsec resolution.
This commit changes several dozen instances of struct timeval to struct timespec.
All of the output is unchanged.

The ticks to nsec conversion in this patch is broken but fixed in a later patch.

3 years agoEnsure that thread_stat alignment is correct
Omri Mor [Mon, 19 Jun 2017 22:41:51 +0000 (16:41 -0600)]
Ensure that thread_stat alignment is correct

clang seems to align this weirdly, and then fault when it
isn't 128-bit aligned. Align it explicitly, so it does the
right thing.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoUpdate write_hint mechanism to latest API
Jens Axboe [Mon, 19 Jun 2017 16:31:11 +0000 (10:31 -0600)]
Update write_hint mechanism to latest API

Kill the fadvise stream stuff, we never used it. Also dump the
pvsync2 options.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoMerge branch 'dev/doc/pkg' of https://github.com/bobsaintcool/fio
Jens Axboe [Sun, 18 Jun 2017 16:32:11 +0000 (10:32 -0600)]
Merge branch 'dev/doc/pkg' of https://github.com/bobsaintcool/fio

3 years ago<README/pkg: Add reference to Arch Linux package>
Quentin Bourgeois [Sat, 17 Jun 2017 10:38:37 +0000 (12:38 +0200)]
<README/pkg: Add reference to Arch Linux package>

3 years agoFio 2.21 fio-2.21
Jens Axboe [Thu, 15 Jun 2017 18:25:03 +0000 (12:25 -0600)]
Fio 2.21

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agosync: add support for write life time hint
Jens Axboe [Wed, 14 Jun 2017 15:55:40 +0000 (09:55 -0600)]
sync: add support for write life time hint

Not documented yet, as the interface may change. This matches
what I currently have implemented on the kernel side.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agobinject: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:11 +0000 (23:02 +0300)]
binject: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosplice: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:10 +0000 (23:02 +0300)]
splice: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: don't use void* for pointer arithmetic (gcc)
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:09 +0000 (23:02 +0300)]
sg: don't use void* for pointer arithmetic (gcc)

I think gcc extension should be avoided when it can be done by
just changing a pointer type.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: drop redundant void* cast
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:08 +0000 (23:02 +0300)]
sg: drop redundant void* cast

No need to cast to void*.
This is the only void* cast for memcpy(3) in the entire fio.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: drop unneeded strdup from ->errdetails() handler
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:07 +0000 (23:02 +0300)]
sg: drop unneeded strdup from ->errdetails() handler

*ret as well as strdup aren't really necessary, but only make the code
complicated as shown in the previous commit. No functional change.

This commit includes reverting of the previous commit.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: add missing free(msg); in ->errdetails() handler
Tomohiro Kusumi [Mon, 12 Jun 2017 20:02:06 +0000 (23:02 +0300)]
sg: add missing free(msg); in ->errdetails() handler

msg gets malloc'd unconditionally, so we need to free msg if we're
not using msg for ret (strdup isn't really needed here to begin with).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoman: Update buffer_pattern entry in man pages
Stephen Bates [Thu, 8 Jun 2017 18:10:10 +0000 (12:10 -0600)]
man: Update buffer_pattern entry in man pages

Update the fio man page to reflect the new usage of buffer_pattern.

Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooslib/strndup: cleanup and remember to include for aux programs
Jens Axboe [Thu, 8 Jun 2017 16:07:55 +0000 (10:07 -0600)]
oslib/strndup: cleanup and remember to include for aux programs

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoconfigure: remember to initialize 'strndup' to "no"
Jens Axboe [Thu, 8 Jun 2017 15:58:40 +0000 (09:58 -0600)]
configure: remember to initialize 'strndup' to "no"

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoAdd strndup() function, if we don't have it
Jens Axboe [Thu, 8 Jun 2017 15:37:52 +0000 (09:37 -0600)]
Add strndup() function, if we don't have it

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agopattern: Add support for files in buffer_pattern argument.
Stephen Bates [Thu, 8 Jun 2017 11:46:46 +0000 (05:46 -0600)]
pattern: Add support for files in buffer_pattern argument.

It is useful to be able to initialize the buffer contents using an
input file (e.g. when testing certain pathological patterns for
compression). Add support to the buffer_pattern input argument for
reading data from a file via enclosing the filename in ``''``.

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-By: Muli Ben-Yehuda <muli@lightbitslabs.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoHOWTO: mention some details of ignore_error= option
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:49 +0000 (23:18 +0300)]
HOWTO: mention some details of ignore_error= option

ignore_error= and continue_on_error= are related in their
implementation, and easier to understand how they differ by
mentioning to it.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agouse ARRAY_SIZE() for ignore_error_nr[etype]
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:48 +0000 (23:18 +0300)]
use ARRAY_SIZE() for ignore_error_nr[etype]

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agouse enum error_type_bit for ignore_error index
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:47 +0000 (23:18 +0300)]
use enum error_type_bit for ignore_error index

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agodon't leave ignore_error_nr[etype] with 4 on blank input or error
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:46 +0000 (23:18 +0300)]
don't leave ignore_error_nr[etype] with 4 on blank input or error

This isn't a bug since td_non_fatal_error() checks errnos to ignore
by td->o.ignore_error[etype] being NULL or not, but it's still strange
to leave td->o.ignore_error_nr[etype] with 4 (or multiple of 4 on
error) when there aren't any.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofix wrong malloc size for ignore_error buffer
Tomohiro Kusumi [Wed, 7 Jun 2017 20:18:45 +0000 (23:18 +0300)]
fix wrong malloc size for ignore_error buffer

This is supposed to be allocating the initial buffer which consists
of 4 integers to store errno values to ignore.
Also explicitly zero clear the buffer.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoblktrace: ensure that dev loop doesn't truncate name
Jens Axboe [Wed, 7 Jun 2017 16:46:40 +0000 (10:46 -0600)]
blktrace: ensure that dev loop doesn't truncate name

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agodiskutil: ensure we have enough room to not write past end
Jens Axboe [Wed, 7 Jun 2017 16:45:46 +0000 (10:45 -0600)]
diskutil: ensure we have enough room to not write past end

gcc 7.1 triggers this warning:

diskutil.c: In function ‘find_block_dir’:
diskutil.c:371:22: warning: ‘__builtin___sprintf_chk’ may write a terminating nul past the end of the destination [-Wformat-overflow=]
   sprintf(full_path, "%s/%s", path, dir->d_name);
                      ^~~~~~~
In file included from /usr/include/stdio.h:936:0,
                 from diskutil.c:1:

Ensure that it's 256+1 in length.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agoFix up some style
Jens Axboe [Wed, 7 Jun 2017 14:52:47 +0000 (08:52 -0600)]
Fix up some style

The offset-as-percent was a little different than what we usually
do, clean it up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agofileset: fix double addition of file offset
Jens Axboe [Wed, 7 Jun 2017 14:48:05 +0000 (08:48 -0600)]
fileset: fix double addition of file offset

The offset-as-percentage patch inadvertently screwed up the regular
offset path, fix that up.

Fixes: 89978a6b26f8 ("allow a percent value for the offset parameter")
Fixes: https://github.com/axboe/fio/issues/382
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 years agostat: correct json 'io_bytes' output
Jens Axboe [Mon, 5 Jun 2017 20:05:43 +0000 (14:05 -0600)]
stat: correct json 'io_bytes' output

Since the dawn of time, this has outputted kilobytes, not bytes.
Correct this oversight, and also add a 'io_kbytes' key that shows
the old output.

Fixes: 378
Signed-off-by: Jens Axboe <axboe@kernel.dk>