fio.git
3 years agosteadystate: reject job if steadystate options are not consistent within reporting...
Vincent Fu [Fri, 28 Oct 2016 19:19:57 +0000 (15:19 -0400)]
steadystate: reject job if steadystate options are not consistent within reporting group

3 years agosteadystate: bug fixes
Vincent Fu [Thu, 27 Oct 2016 20:36:43 +0000 (16:36 -0400)]
steadystate: bug fixes

1) Fix steadystate option parsing callback bug
2) Re-insert missing multiplier in slope calculation
3) Revert 'attained' in the JSON output to be either 0 or 1

3 years agosteadystate: kill ->last_in_group and ->ramp_time_over
Jens Axboe [Wed, 17 Aug 2016 05:23:31 +0000 (23:23 -0600)]
steadystate: kill ->last_in_group and ->ramp_time_over

Also move to state flags. Rename ->mode to ->state, to better
reflect what it does.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosteadystate: kill off ->attained
Jens Axboe [Wed, 17 Aug 2016 05:17:26 +0000 (23:17 -0600)]
steadystate: kill off ->attained

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosteadystate: eliminate some steadystate_data members
Jens Axboe [Wed, 17 Aug 2016 05:13:48 +0000 (23:13 -0600)]
steadystate: eliminate some steadystate_data members

Working towards a goal of killing members we don't really need,
and splitting steadystate_data into a runtime part and an
output part. The latter needs to be wire safe.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosteadystate: rename 'steadystate' to 'steadystate_enabled'
Jens Axboe [Sun, 14 Aug 2016 02:34:41 +0000 (20:34 -0600)]
steadystate: rename 'steadystate' to 'steadystate_enabled'

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosteadystate: more cleanups
Jens Axboe [Sat, 13 Aug 2016 15:04:16 +0000 (09:04 -0600)]
steadystate: more cleanups

- Get rid of the worst ?: constructs, they only help to obfuscate
  the readability.
- Cleanup the most egregious line lengths

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosteadystate: cleanups
Jens Axboe [Sat, 13 Aug 2016 14:39:57 +0000 (08:39 -0600)]
steadystate: cleanups

- Make this more localized
- Get rid of ->evaluate() pointer, towards making the steadystate_data
  appropriate for wire transfer.
- Move more things to steadystate.[ch]

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosteadystate: move into its own header
Jens Axboe [Fri, 12 Aug 2016 20:18:24 +0000 (14:18 -0600)]
steadystate: move into its own header

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoTODO: man page missing
Jens Axboe [Fri, 12 Aug 2016 03:53:19 +0000 (21:53 -0600)]
TODO: man page missing

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosteady-state: convert options across the wire
Jens Axboe [Fri, 12 Aug 2016 03:50:59 +0000 (21:50 -0600)]
steady-state: convert options across the wire

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd local TODO for steady-state work
Jens Axboe [Fri, 29 Jul 2016 21:29:47 +0000 (15:29 -0600)]
Add local TODO for steady-state work

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoClear up white space errors
Vincent Fu [Fri, 29 Jul 2016 21:07:06 +0000 (17:07 -0400)]
Clear up white space errors

3 years agoUpdate test script for new JSON steadystate['criterion'] reporting
Vincent Fu [Wed, 13 Jul 2016 19:33:42 +0000 (15:33 -0400)]
Update test script for new JSON steadystate['criterion'] reporting

3 years agoSplit helper thread debug logging away from steadystate debug logging
Vincent Fu [Wed, 13 Jul 2016 18:50:13 +0000 (14:50 -0400)]
Split helper thread debug logging away from steadystate debug logging

3 years agoFix bug where measurements were not printed in the correct order when steady state...
Vincent Fu [Wed, 13 Jul 2016 18:38:57 +0000 (14:38 -0400)]
Fix bug where measurements were not printed in the correct order when steady state was not attained

ss->head points to the beginning of the buffer up through the point where the buffer is filled for the first time. afterwards, when a new element is added, ss->head is advanced to point to the second element in the buffer. if steady state is attained upon adding a new element, ss->head is not advanced so it actually does point to the head of the buffer.

3 years agoJSON output, code formatting changes
Vincent Fu [Wed, 13 Jul 2016 18:00:59 +0000 (14:00 -0400)]
JSON output, code formatting changes

1. Change 'criterion' to match the specified limit. It will always be a string.
If the specified limit was a percentage, 'criterion' will also be a percentage (e.g., "0.0123%").
If the specified limit was not a percentage, 'criterion' will be a number (e.g., "12345").

2. Move JSON mean IOPS, BW to the top of the steadystate['data'] section

3 years agoSteady state detection: enhance reporting of results, change memory allocation point
Vincent Fu [Thu, 30 Jun 2016 18:06:30 +0000 (14:06 -0400)]
Steady state detection: enhance reporting of results, change memory allocation point

1. Always record and report both BW and IOPS for steady state window (useful when multiple block sizes are used)
2. Always report both slope and maximum deviation even though one is always unused
3. Report mean BW and mean IOPS in JSON output
4. Allocate memory for ring buffers in steadystate_setup() instead of add_job(). The benefit is that only one set of buffers is now allocated for eaceh group. Previously each thread had its own set of buffers enough though only one set in each reporting group was actually used.
5. Update test script to accommodate new output

3 years agoChange steadystate reporting:
Vincent Fu [Thu, 23 Jun 2016 19:28:25 +0000 (15:28 -0400)]
Change steadystate reporting:

1. Remove superfluous member 'limit'
2. Report criterion as (maximum deviation / mean) or (slope / mean) when the limit is specified as a percentage

3 years agoAdd locking around reads of td->io_blocks and io_byes.
Vincent Fu [Thu, 23 Jun 2016 15:32:04 +0000 (11:32 -0400)]
Add locking around reads of td->io_blocks and io_byes.

This is present in add_bw/iops/lat_sample but absent from show_thread_status. Not certain if this is really necessary.

3 years agoFix typo, restore unintended HOWTO deletion
Vincent Fu [Sun, 19 Jun 2016 03:09:39 +0000 (23:09 -0400)]
Fix typo, restore unintended HOWTO deletion

3 years agoAllow fio to terminate jobs when steady state is attained
Vincent Fu [Sun, 19 Jun 2016 02:59:32 +0000 (22:59 -0400)]
Allow fio to terminate jobs when steady state is attained

KNOWN ISSUES
Will not work over a network connection

3 years agofio: inherit IO engine flags to 'td'
Jens Axboe [Mon, 15 Aug 2016 15:42:37 +0000 (09:42 -0600)]
fio: inherit IO engine flags to 'td'

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: move thread_options
Jens Axboe [Mon, 15 Aug 2016 15:39:35 +0000 (09:39 -0600)]
fio: move thread_options

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: remove dependency of 'o' being first in 'td'
Jens Axboe [Mon, 15 Aug 2016 15:31:25 +0000 (09:31 -0600)]
options: remove dependency of 'o' being first in 'td'

- We currently assume that &td->o == td, but it's unfortunate that we
  have to store 3-4K of option data in the first part of 'td'.

- Kill td_var_offset(), just use offsetof() instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoOnly enable atomic io_u flag setting/clearing if we need it
Jens Axboe [Mon, 15 Aug 2016 03:31:16 +0000 (21:31 -0600)]
Only enable atomic io_u flag setting/clearing if we need it

Make it dependent on td_async_processing(), like we do for
other cases.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: silence warning on pointer cast on 32-bit compiles
Jens Axboe [Mon, 8 Aug 2016 17:32:34 +0000 (11:32 -0600)]
iolog: silence warning on pointer cast on 32-bit compiles

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoCheck if sysfs ioscheduler entry is "none"
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:31 +0000 (02:27 +0900)]
Check if sysfs ioscheduler entry is "none"

Block devices without elevator (bio-based-dm/loop/etc) have "none"
for sysfs ioscheduler tunable, and write to this entry never fail
from the way elv_iosched_store() is implemented.

This commit checks if the entry is "none" type so as not to show
an irrelevant error message which assumes the entry has a list of
available I/O schedulers.

 # cat /sys/block/dm-0/queue/scheduler
 none
 # echo deadline > /sys/block/dm-0/queue/scheduler ; echo $?
 0
 # cat /sys/block/dm-0/queue/scheduler
 none
 # echo aaa > /sys/block/dm-0/queue/scheduler ; echo $?
 0
 # cat /sys/block/dm-0/queue/scheduler
 none

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse a pointer to const char* for I/O engine name (in response to aa2b823c)
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:30 +0000 (02:27 +0900)]
Use a pointer to const char* for I/O engine name (in response to aa2b823c)

aa2b823c also needed to change ioengine_ops::name size to 64,
or make it a pointer to a readonly string which seems to be better
in this case as no one is to (or should be able to) modify it.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing FIO_NET_CMD entry to fio_server_ops[]
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:29 +0000 (02:27 +0900)]
Add missing FIO_NET_CMD entry to fio_server_ops[]

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMake local const string array static
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:28 +0000 (02:27 +0900)]
Make local const string array static

like it is in io_ddir_name().

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoChange --output-format argument from optional to required
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:27 +0000 (02:27 +0900)]
Change --output-format argument from optional to required

Not sure if there were any reason d10983d7 chose to add if(!optarg)
conditional, but this is what other fio options (or in general) do.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse char* for pid_file path
Tomohiro Kusumi [Mon, 8 Aug 2016 17:27:26 +0000 (02:27 +0900)]
Use char* for pid_file path

since no function expects this to be void*.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix spelling error
Jens Axboe [Mon, 8 Aug 2016 16:25:31 +0000 (10:25 -0600)]
Fix spelling error

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd 'unlink_each_loop' option
mrturtledev [Mon, 8 Aug 2016 15:57:14 +0000 (09:57 -0600)]
Add 'unlink_each_loop' option

If this option is set, fio will unlink the files after each loop
of a job has completed.

Original patch by Martin, various tweaks and updates from me.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoioengines: fixup td_io_unlink_file() error propagation
Jens Axboe [Mon, 8 Aug 2016 15:46:06 +0000 (09:46 -0600)]
ioengines: fixup td_io_unlink_file() error propagation

Return 0 for success, error number on error. gluster already did this,
but pmemblk did not.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: hist_sum() should return unsigned long
Jens Axboe [Mon, 8 Aug 2016 13:48:33 +0000 (07:48 -0600)]
iolog: hist_sum() should return unsigned long

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agostat: fixups to histogram logging
Jens Axboe [Sun, 7 Aug 2016 21:18:38 +0000 (15:18 -0600)]
stat: fixups to histogram logging

1) Use correct size for the 'dst' passin to __add_log_sample()
2) Various style fixups

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: style updates
Jens Axboe [Sun, 7 Aug 2016 21:11:32 +0000 (15:11 -0600)]
iolog: style updates

Fixup some slightly off coding style from the histogram logging commit.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: bump protocol version
Jens Axboe [Sun, 7 Aug 2016 21:08:51 +0000 (15:08 -0600)]
server: bump protocol version

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'histograms-PR' of https://github.com/cronburg/fio
Jens Axboe [Sun, 7 Aug 2016 21:07:53 +0000 (15:07 -0600)]
Merge branch 'histograms-PR' of https://github.com/cronburg/fio

3 years agoThis commit / feature adds completion latency histogram output to fio, piggybacking
Karl Cronburg [Fri, 5 Aug 2016 18:45:00 +0000 (14:45 -0400)]
This commit / feature adds completion latency histogram output to fio, piggybacking
on the existing histograms recorded by stat.c and adding the following command
line options:

- log_hist_msec operates like log_avg_msec does, but produces files
  named *_clat_hist*
- write_hist_log has the same semantics as write_lat_log but for histograms.
- log_hist_coarseness changes how many bins are logged.

The more substantial contribution is a post-processor for digesting these log files,
located in fio/tools/hist/fiologparser_hist.py, which computes min/max/avg/percentile
statistics.

To compare the accuracy of the statistics produced by fiologparser_hist.py,
a reference implementation is located here:

https://github.com/cronburg/fio/blob/histograms/tools/hist/fiologparser_numpy.py

for computing the same statistics from the *_clat.* files fio already produces.
A comparison can be viewed here:

https://cronburg.com/fio/demo/latency.html

The reference implementation is excluded from this commit, as I'm still working
with Mark (markhpc) on getting it into the branch where he's working on fiologparser.
Also I'd eventually like to make FIO_IO_U_PLAT_GROUP_NR in stat.h
configurable (allowing for a larger maximum histogram latency value than 17
seconds), but haven't added it to keep the changes to fio minimal for now.

A more detailed description of the motivation behind this feature, and explanation
of the design process and algorithms used in the post-processor can be found here:

https://cronburg.com/fio/cloud-latency-problem-measurement/

3 years agoRevert "filesetup: ensure that we catch a file flagged for extend"
Jens Axboe [Fri, 5 Aug 2016 01:40:15 +0000 (19:40 -0600)]
Revert "filesetup: ensure that we catch a file flagged for extend"

This reverts commit bcb7260a54438ab38a5928279f7e5a3a465ed1bb.

Don't think this was a functionally sound change, let's revert
to the previous behavior of create_on_open.

3 years agotravis: don't enable rbd
Jens Axboe [Wed, 3 Aug 2016 16:18:42 +0000 (10:18 -0600)]
travis: don't enable rbd

Apparently it fails with an older version of the rbd library. Need
to figure out how to check and test for that.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agotravis: add rbd/zlib
Jens Axboe [Wed, 3 Aug 2016 16:14:10 +0000 (10:14 -0600)]
travis: add rbd/zlib

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofilesetup: ensure that we catch a file flagged for extend
Jens Axboe [Wed, 3 Aug 2016 15:53:34 +0000 (09:53 -0600)]
filesetup: ensure that we catch a file flagged for extend

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: prevent early entry from skewing entire logging run
Jens Axboe [Wed, 3 Aug 2016 15:11:56 +0000 (09:11 -0600)]
iolog: prevent early entry from skewing entire logging run

If we go out of ramp time, our wakeup trigger could cause us to enter
too early in add_log_sample(). If that happens, the time diff could be
negative (so huge unsigned). Since we carry that forward, we end up
logging every entry and not just over the average window.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofix typo in HOWTO
Jan Fajerski [Wed, 3 Aug 2016 11:35:27 +0000 (13:35 +0200)]
fix typo in HOWTO

3 years agofix typo in HOWTO
Jan Fajerski [Wed, 3 Aug 2016 11:22:16 +0000 (13:22 +0200)]
fix typo in HOWTO

3 years agoengines/rbd: fix compile without blkin support
Jens Axboe [Tue, 2 Aug 2016 21:23:43 +0000 (15:23 -0600)]
engines/rbd: fix compile without blkin support

If we don't have rbd-blkin support, compile fails with:

engines/rbd.c:21:26: error: field ‘info’ has incomplete type
  struct blkin_trace_info info;
                          ^~~~
Makefile:313: recipe for target 'engines/rbd.o' failed

Hide 'info' behind the ifdef.

Fixes: a4c4c3460bd5 ("Add support for blkin tracing in rbd engine")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'wip-traceinfo' of https://github.com/vears91/fio
Jens Axboe [Tue, 2 Aug 2016 21:05:59 +0000 (15:05 -0600)]
Merge branch 'wip-traceinfo' of https://github.com/vears91/fio

3 years agoRevert "filesetup: ensure that we align file starting offset"
Jens Axboe [Tue, 2 Aug 2016 15:07:07 +0000 (09:07 -0600)]
Revert "filesetup: ensure that we align file starting offset"

There are valid reasons to use an offset that isn't a multiple
of the block size. We previously made this change to fix an
issue with a job file, but that job file had an offset that
wasn't a multiple of the hardware block size. As long as the
offset is a multiple of the hardware block size for unbuffered
IO, then it should be fine.

This reverts commit c465cd14ca958dfdb87bb1998add722d62ae3691.

3 years agobackend: do_verify() cleanup
Jens Axboe [Mon, 1 Aug 2016 19:46:17 +0000 (13:46 -0600)]
backend: do_verify() cleanup

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: do_dry_run(): get_io_u() can return an error pointer
Jens Axboe [Fri, 29 Jul 2016 15:59:38 +0000 (09:59 -0600)]
backend: do_dry_run(): get_io_u() can return an error pointer

Don't just check for NULL.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse larger local buffer for I/O engine name
Tomohiro Kusumi [Fri, 29 Jul 2016 15:06:00 +0000 (00:06 +0900)]
Use larger local buffer for I/O engine name

16 bytes may not be large enough in the future if a new I/O engine
with a name longer than strlen(name)=15 is added.

The longest one right now seems to be "fusion-aw-sync" which is 14(+1).

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoNull terminate before (or after) strncpy(3)
Tomohiro Kusumi [Fri, 29 Jul 2016 15:05:59 +0000 (00:05 +0900)]
Null terminate before (or after) strncpy(3)

These three strncpy() calls copy at most sizeof(buffer)-1 bytes,
but buffer isn't explicitly 0 cleared, so 0 terminate the last byte.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMake switch_ioscheduler() return 0 if FIO_HAVE_IOSCHED_SWITCH is undefined
Tomohiro Kusumi [Fri, 29 Jul 2016 15:05:58 +0000 (00:05 +0900)]
Make switch_ioscheduler() return 0 if FIO_HAVE_IOSCHED_SWITCH is undefined

Defining FIO_HAVE_IOSCHED_SWITCH currently equals saying it's Linux,
as switch_ioscheduler() only works on Linux kernel with "scheduler"
sysfs entry (though read/write to sysfs obviously compiles on others).

This commit makes the function return 0 if FIO_HAVE_IOSCHED_SWITCH
is undefined (i.e. if not Linux). This is essentially the same as
{diskutil,cgroup,blktrace}.c being compiled only on Linux.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse in-place path separator "/" for Linux specific code
Tomohiro Kusumi [Fri, 29 Jul 2016 15:05:57 +0000 (00:05 +0900)]
Use in-place path separator "/" for Linux specific code

diskutil,cgroup,blktrace related code fully depend on Linux kernel,
so sprintf variants can use in-place path separator "/" instead of
FIO_OS_PATH_SEPARATOR.

In fact these Linux specific files are mix of two types within the
same file depending on who wrote them, where they were originally
using "/".

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogettime: remove unneeded 'ret' in {utime,mtime}_since()
Jens Axboe [Fri, 29 Jul 2016 14:35:28 +0000 (08:35 -0600)]
gettime: remove unneeded 'ret' in {utime,mtime}_since()

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix overflow caused by signed long division by unsigned long.
YukiKita [Fri, 29 Jul 2016 00:25:17 +0000 (09:25 +0900)]
Fix overflow caused by signed long division by unsigned long.
The over flow seems to occurr when the value of 'log_avg_msec' option is relatively large.

3 years agoFix memory leak in _fio_rbd_connect()
Jevon Qiao [Thu, 28 Jul 2016 07:26:38 +0000 (15:26 +0800)]
Fix memory leak in _fio_rbd_connect()

When running fio with rbd engine and specifying clustername, a piece of
memory will be allocated in _fio_rbd_connect(), but it never gets freed.

Signed-off-by: Jevon Qiao <scaleqiao@gmail.com>
3 years agoFix segmentation fault while specifying clustername with rbd engine
Jevon Qiao [Thu, 28 Jul 2016 07:22:40 +0000 (15:22 +0800)]
Fix segmentation fault while specifying clustername with rbd engine

Fio with rbd engine will panic if clustername option is specified but clientname
is not. In that case, o->client_name will be a NULL pointer, strlen(o->client_name) in _fio_rbd_connect() will lead to the segmentation fault.

Signed-off-by: Jevon Qiao <scaleqiao@gmail.com>
3 years agoexamples/backwards-read.fio: add size
Jens Axboe [Wed, 27 Jul 2016 14:33:21 +0000 (08:33 -0600)]
examples/backwards-read.fio: add size

Ensure that it runs standalone.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoChange arch_i386 to arch_x86
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:20 +0000 (22:37 +0900)]
Change arch_i386 to arch_x86

fio_arch_strings[arch_i386] is "x86" (not "i386"),
so change arch_i386 to arch_x86.

Other than arch_i386, "i386" is only used in __i386__.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing archs in fio_arch_strings[]
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:19 +0000 (22:37 +0900)]
Add missing archs in fio_arch_strings[]

Add MIPS and AArch64 after PARISC.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRename exists_and_not_file() to exists_and_not_regfile()
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:18 +0000 (22:37 +0900)]
Rename exists_and_not_file() to exists_and_not_regfile()

Renaming makes the purpose of this function more clear IMO.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix stat(2) related bugs introduced by changes made for Windows
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:17 +0000 (22:37 +0900)]
Fix stat(2) related bugs introduced by changes made for Windows

Non-Windows could have regular files that start with "\\\\.\\",
so the conditionals added by ecc314ba and 3892182a in 2011-2012
introduced corner case bugs, though in reality I doubt anyone
would want to create such a file.

This commit brings back the original conditional that was there
before ecc314ba for non-Windows platforms. It also adds WIN32
guard for a change made by 3892182a.

--
 # uname
 Linux
 # cat ./test1.c
 #include <stdio.h>
 #include <string.h>
 #include <sys/types.h>
 #include <sys/stat.h>
 /* copied from fio/init.c */
 static int exists_and_not_file(const char *filename)
 {
  struct stat sb;
  if (lstat(filename, &sb) == -1)
  return 0;
  /* \\.\ is the device namespace in Windows, where every file
   * is a device node */
  if (S_ISREG(sb.st_mode) && strncmp(filename, "\\\\.\\", 4) != 0)
  return 0;
  return 1;
 }
 int main(int argc, char **argv) {
  printf("exists_and_not_file(\"%s\") = %d\n",
  argv[1], exists_and_not_file(argv[1]));
  return 0;
 }
 # gcc -Wall -g ./test1.c -o test1
 # ls /dev/sda
 /dev/sda
 # ./test1 /dev/sda
 exists_and_not_file("/dev/sda") = 1  /* the existing blkdev isn't a file */
 # touch xxxxx
 # ./test1 ./xxxxx
 exists_and_not_file("./xxxxx") = 0  /* the existing regfile is not not a file */
 # touch "\\\\.\\xxxxx"
 # ./test1 "\\\\.\\xxxxx"
 exists_and_not_file("\\.\xxxxx") = 1  /* XXX */
 # stat "\\\\.\\xxxxx" | head -2
   File: `\\\\.\\xxxxx'
   Size: 0               Blocks: 0          IO Block: 4096   regular empty file

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDon't malloc/memcpy ioengine_ops on td initialization
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:16 +0000 (22:37 +0900)]
Don't malloc/memcpy ioengine_ops on td initialization

The only reason load_ioengine() needs to malloc extra space for
struct ioengine_ops and memcpy from the original ops is because td
has per-thread/process ioengine private data which is a part of
this structure.

If data (and dlhandle) are separated from struct ioengine_ops,
load_ioengine() no longer needs to malloc, as the rest of structure
fields such as function pointers and string literal are static.
Td only needs a pointer to the original ops. Also avoid memcpying
a list_head from the original ops.

This commit moves/renames data and dlhandle to struct thread_data
as thread_data::io_ops_data and thread_data::io_ops_dlhandle.
engines/*.c which access ioengine private data by dereferencing
td->io_ops->... are modified accordingly, but nothing more than that.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing --cmdhelp type string for FIO_OPT_UNSUPPORTED
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:15 +0000 (22:37 +0900)]
Add missing --cmdhelp type string for FIO_OPT_UNSUPPORTED

Without "unsupported" at index 12 (FIO_OPT_UNSUPPORTED) of typehelp[],
unsupported commands print irrelevant binary at typehelp[12].

 # fio --cmdhelp=numa_cpu_nodes
       numa_cpu_nodes: Build fio with libnuma-dev(el) to enable this option
                 type: ÏÿF
              default: no default

With this commit.
 # fio --cmdhelp=numa_cpu_nodes
       numa_cpu_nodes: Build fio with libnuma-dev(el) to enable this option
                 type: unsupported
              default: no default

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse correct I/O engine name "cpuio" instead of "cpu"
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:14 +0000 (22:37 +0900)]
Use correct I/O engine name "cpuio" instead of "cpu"

"cpuio" is the name of the engine/option, though the filename is cpu.c.

 # grep "ioengine =" engines/cpu.c -A1
 static struct ioengine_ops ioengine = {
  .name           = "cpuio",

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoIgnore exit_io_done= option if no I/O threads are configured
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:13 +0000 (22:37 +0900)]
Ignore exit_io_done= option if no I/O threads are configured

The cpuio engine option exit_io_done= should be effective only if
there is at least one real I/O thread configured.

The existing fio_cpuio_queue() with exit_io_done= option enabled
lets cpuio threads exit even if cpuio threads are the only threads
configured.

Since this option is supposed to mean "exit when I/O threads are done",
cpuio threads exiting (but only after the first spin cycle is done)
when no threads have done any I/O is a bit confusing.

> [cpu] exit_on_io_done=bool Detect when IO threads are done, then exit.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMention cpuio never finishes without real I/O in documentation
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:12 +0000 (22:37 +0900)]
Mention cpuio never finishes without real I/O in documentation

reap_threads() terminates cpuio threads only when there is
at least one real I/O thread that is exiting.

> if (*nr_running == cputhreads && !pending && realthreads)
> fio_terminate_threads(TERMINATE_ALL);

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMention default values for readwrite=/ioengine=/mem= in documentation
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:11 +0000 (22:37 +0900)]
Mention default values for readwrite=/ioengine=/mem= in documentation

HOWTO/fio(1) mention defaults for many of the available options.
This commit adds defaults for the following options.

 readwrite=
 ioengine=
 mem=

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse sizeof(char*) instead of sizeof(void*)
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:10 +0000 (22:37 +0900)]
Use sizeof(char*) instead of sizeof(void*)

Conform to the exact data type though both pointers are
normally expected to be the same size on various archs.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse default CPU_COUNT() function in DragonFlyBSD
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:08 +0000 (22:37 +0900)]
Use default CPU_COUNT() function in DragonFlyBSD

os/os.h has a default CPU_COUNT() inline function for those that
are expected to have CPU_COUNT() macro, but not in certain versions.
This seems to have been added for RHEL5 (or older?) and variants,
but use this in os/os-dragonfly.h since it's essentially the same
as its own fio_cpu_count() implementation.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix typos in log_err() message
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:07 +0000 (22:37 +0900)]
Fix typos in log_err() message

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMake return value type of fio_getaffinity() consistent
Tomohiro Kusumi [Wed, 27 Jul 2016 13:37:06 +0000 (22:37 +0900)]
Make return value type of fio_getaffinity() consistent

Return type of fio_getaffinity() isn't consistent among supported OS.
Windows and DragonFlyBSD return void while FreeBSD version is int.
The default version for those that don't support is do{}while(0).

Linux version is a macro for sched_getaffinity(2) which returns 0
on success and -1 otherwise, so others should basically follow that.

Note that I haven't compiled this on Windows, but it shouldn't fail.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix memory leak in _fio_setup_rbd_data()
Jevon Qiao [Wed, 27 Jul 2016 07:36:36 +0000 (15:36 +0800)]
Fix memory leak in _fio_setup_rbd_data()

Signed-off-by: Jevon Qiao <scaleqiao@gmail.com>
3 years agoAdd sample job file showing how to read backwards
Jens Axboe [Tue, 26 Jul 2016 20:50:02 +0000 (14:50 -0600)]
Add sample job file showing how to read backwards

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRevert "Revert "fio: Simplify forking of processes""
Jens Axboe [Tue, 26 Jul 2016 14:17:42 +0000 (08:17 -0600)]
Revert "Revert "fio: Simplify forking of processes""

This reverts commit cb92ecf8118b1a8b13909ab18d8a646e24ac79f2.

3 years agoAdd support for blkin tracing in rbd engine
vears91 [Mon, 27 Jun 2016 13:38:34 +0000 (16:38 +0300)]
Add support for blkin tracing in rbd engine

Add configure support to disable or enable fio rbd engine with blkin, and use rbd_aio_write_traced and rbd_aio_read_traced to pass the trace information to librbd if fio was compiled with blkin.

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>
3 years agoFio 2.13 fio-2.13
Jens Axboe [Fri, 22 Jul 2016 19:43:56 +0000 (13:43 -0600)]
Fio 2.13

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: fix averaged latency logging
Jens Axboe [Fri, 22 Jul 2016 19:19:31 +0000 (13:19 -0600)]
log: fix averaged latency logging

We inadvertently broke that in the previous commit, fix it up.

Fixes: d454a205f81e ("log: fix missing entries")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: fix missing entries
Jens Axboe [Fri, 22 Jul 2016 18:34:53 +0000 (12:34 -0600)]
log: fix missing entries

If the timing is just off, we could miss adding entries to
an averaged iops or bandwidth log. Allow a bit of slack for
that, similarly to what we do elsewhere.

Fixes: a47591e4923f ("Improve logging accuracy")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix "exitall_on_error" option
YukiKita [Fri, 22 Jul 2016 09:18:14 +0000 (18:18 +0900)]
Fix "exitall_on_error" option
"exitall_on_error" option should be enabled without any argument.

3 years agodrifting in output of interval-averaged values was eventually causing IOP samples...
Karl Cronburg [Wed, 20 Jul 2016 20:21:55 +0000 (16:21 -0400)]
drifting in output of interval-averaged values was eventually causing IOP samples to be dropped.

3 years agoAdd missing header inclusion for Android from 1c764dbe
Tomohiro Kusumi [Tue, 19 Jul 2016 22:30:42 +0000 (07:30 +0900)]
Add missing header inclusion for Android from 1c764dbe

1c764dbe (which just got committed) was missing below for Android,
sorry about that.
 #include "./os-linux-syscall.h"

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd os/os-linux-syscall.h to separate syscall NR from arch headers
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:47 +0000 (01:33 +0900)]
Add os/os-linux-syscall.h to separate syscall NR from arch headers

Linux syscall NR should be defined under os/ with ifdef(arch)
switch rather than arch/ where headers are included not only by
Linux but also all other supported OS that don't need these NR.
(It'll get worse and worse if someone wants to add something
similar under arch/ for other supported OS)

Not sure if you like the way I separated these by adding
os/os-linux-syscall.h, but doing something like
 #if FIO_OS is (Linux or Android)
 ...
 #endif
in arch/arch-*.h doesn't work because arch headers are to be
included prior to os/os-*.h.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoChange ARCH_X86_64_h to ARCH_X86_64_H
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:46 +0000 (01:33 +0900)]
Change ARCH_X86_64_h to ARCH_X86_64_H

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd ioprio_set() support for DragonFlyBSD
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:45 +0000 (01:33 +0900)]
Add ioprio_set() support for DragonFlyBSD

It basically has the same interface as Linux kernel's ioprio_set(2),
but needed to workaround a compile issue by defining it as a macro
as mentioned in a comment (it can be called without syscall(NR, ...)).

This commit is based on the last two commits which made I/O priority
option more generic for non Linux environment.

 # uname
 DragonFly
 # fio --cmdhelp | grep prioclass -B1
 prio                    : Set job IO priority value
 prioclass               : Your platform does not support IO priority classes

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMake I/O priority option generic for non-Linux environment [2/2]
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:44 +0000 (01:33 +0900)]
Make I/O priority option generic for non-Linux environment [2/2]

There is a kernel that supports I/O priority with a syscall similar
to Linux (e.g. DragonFlyBSD's ioprio_set(2) which apparently seems
to have been inspired by ioprio_set(2) in Linux kernel), however
the idea of class within the syscall may not exist depending on the
I/O scheduler and its design, so "prioclass" option should be an
optional one for "prio" option.

This commit adds FIO_HAVE_IOPRIO_CLASS to separate "prioclass" from
"prio" on compile-time for those that do support priority itself,
but not priority classes.

If the platform supports I/O priority, it defines FIO_HAVE_IOPRIO
like it did before. If the platform supports I/O priority classes,
it defines FIO_HAVE_IOPRIO_CLASS in addition to above.
If FIO_HAVE_IOPRIO_CLASS is enabled, FIO_HAVE_IOPRIO must also be
enabled since FIO_HAVE_IOPRIO defines io_prioset(). This is also
checked on compile-time.

Linux (incl Android) has both of these macros enabled, so there is
no functional changes made by this commit.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMake I/O priority option generic for non-Linux environment [1/2]
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:43 +0000 (01:33 +0900)]
Make I/O priority option generic for non-Linux environment [1/2]

There is a kernel that supports I/O priority with a syscall similar
to Linux (e.g. DragonFlyBSD's ioprio_set(2) which apparently seems
to have been inspired by ioprio_set(2) in Linux kernel), however
min/max value of the priority may differ, so use OS specific macros
instead of hardcoded values that are designed to work on Linux.

This commit adds IOPRIO_MIN|MAX_PRIO and IOPRIO_MIN|MAX_PRIO_CLASS
to Linux (incl Android), but no functional changes.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd CPU affinity support for DragonFlyBSD
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:42 +0000 (01:33 +0900)]
Add CPU affinity support for DragonFlyBSD

Confirmed that the cpu affinity works using cpuio engine.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix wrong cpuio option name in documentation
Tomohiro Kusumi [Tue, 19 Jul 2016 16:33:41 +0000 (01:33 +0900)]
Fix wrong cpuio option name in documentation

The name of an option for cpuio engine is "cpuchunks",
though the code internally uses "cpucycle" for variables.

(If the option name "cpuchunks" is the incorrect one,
I'll resubmit a patch that corrects the option name)

 # fio --enghelp=cpuio
 cpuload                 : Use this percentage of CPU
   cpuchunks             : Length of the CPU burn cycles (usecs)
   exit_on_io_done         : Exit when IO threads finish

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoplot: indicate that the pattern is a glob
Sitsofe Wheeler [Sun, 17 Jul 2016 07:49:12 +0000 (08:49 +0100)]
plot: indicate that the pattern is a glob

Rename the word regex to glob in fio2gnuplot because its pattern parameter was
incorrectly called a regex when it was always really a glob.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoplot: add gnuplot 5 support
Sitsofe Wheeler [Sun, 17 Jul 2016 07:35:14 +0000 (08:35 +0100)]
plot: add gnuplot 5 support

gnuplot 5 no longer supports using old-style ($1, $2 etc.) gnuplot 4 parameter
substituion causing fio2gnuplot to break on newer Linux distributions.

Fix this by using new-style arg variables everywhere and conditionally convert
old-style substituions to new-style variables. Add some casting to the average
variable so it is treated as a number or string appropriately and bail out with
an error message when too few parameters are passed in.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agopthread: bump min stack size
Jens Axboe [Thu, 14 Jul 2016 17:36:12 +0000 (10:36 -0700)]
pthread: bump min stack size

For some reason fio fails to create a thread with the stack size
set to the minimum. For now, let's just bump it to 2x the min size,
that seems to work...

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd os_trim() support for FreeBSD
Tomohiro Kusumi [Thu, 14 Jul 2016 12:27:43 +0000 (21:27 +0900)]
Add os_trim() support for FreeBSD

It has the same interface as Linux kernel with a different ioctl name.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd os_trim() support for DragonFlyBSD
Tomohiro Kusumi [Thu, 14 Jul 2016 12:27:42 +0000 (21:27 +0900)]
Add os_trim() support for DragonFlyBSD

It has the same interface as Linux kernel with a different ioctl name.

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'fix_verify' of https://github.com/charles-jacobsen/fio
Jens Axboe [Wed, 13 Jul 2016 16:13:46 +0000 (09:13 -0700)]
Merge branch 'fix_verify' of https://github.com/charles-jacobsen/fio