fio.git
19 months agoMerge branch 'master' of https://github.com/kelleymh/fio
Jens Axboe [Fri, 17 Aug 2018 23:29:11 +0000 (17:29 -0600)]
Merge branch 'master' of https://github.com/kelleymh/fio

* 'master' of https://github.com/kelleymh/fio:
  Remove unused code in lib/axmap.c

19 months agohttp: fix compile-time warnings
Tomohiro Kusumi [Fri, 17 Aug 2018 22:22:00 +0000 (15:22 -0700)]
http: fix compile-time warnings

Fix following warnings on RHEL6 and variants.

engines/http.c: In function 'fio_http_setup':
engines/http.c:611: warning: call to '_curl_easy_setopt_err_seek_cb' declared with attribute warning: curl_easy_setopt expects a curl_seek_callback argument for this option
engines/http.c: In function 'fio_http_queue':
engines/http.c:549: warning: call to '_curl_easy_setopt_err_curl_off_t' declared with attribute warning: curl_easy_setopt expects a curl_off_t argument for this option

For engines/http.c:611, pass &_http_seek instead of cast, since
lack of cast isn't fio's issue. See below comments for details.

"Re: [PATCH] http: fix compile-time warnings"
https://www.spinics.net/lists/fio/msg07246.html

Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agoRemove unused code in lib/axmap.c
Michael Kelley [Fri, 17 Aug 2018 21:49:49 +0000 (14:49 -0700)]
Remove unused code in lib/axmap.c

With the new axmap_next_free() implementation, the "first_free" cache
is no longer used.  Remove all references to it.  Also axmap_clear()
is never referenced, so remove it and related code.

19 months agoMerge branch 'ddn-ime'
Jens Axboe [Fri, 17 Aug 2018 19:39:40 +0000 (13:39 -0600)]
Merge branch 'ddn-ime'

Pull in DDN IME engine

* ddn-ime:
  Sync man page with fio for IME
  engines/ime: various code and style cleanups
  Add support for DDN's Infinite Memory Engine

19 months agoSync man page with fio for IME
Jens Axboe [Fri, 17 Aug 2018 19:39:12 +0000 (13:39 -0600)]
Sync man page with fio for IME

Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agoengines/ime: various code and style cleanups
Jens Axboe [Fri, 17 Aug 2018 19:38:59 +0000 (13:38 -0600)]
engines/ime: various code and style cleanups

Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agoMerge branch 'ime-support' of https://github.com/DDNStorage/fio-public into ddn-ime
Jens Axboe [Fri, 17 Aug 2018 19:34:21 +0000 (13:34 -0600)]
Merge branch 'ime-support' of https://github.com/DDNStorage/fio-public into ddn-ime

* 'ime-support' of https://github.com/DDNStorage/fio-public:
  Add support for DDN's Infinite Memory Engine

19 months agoMerge branch 'master' of https://github.com/kelleymh/fio
Jens Axboe [Fri, 17 Aug 2018 19:27:25 +0000 (13:27 -0600)]
Merge branch 'master' of https://github.com/kelleymh/fio

* 'master' of https://github.com/kelleymh/fio:
  Add tests specifically for axmap_next_free()
  Reimplement axmap_next_free() to prevent distribution skew

19 months agoAdd support for DDN's Infinite Memory Engine
Gaëtan Bossu [Wed, 1 Aug 2018 14:27:37 +0000 (16:27 +0200)]
Add support for DDN's Infinite Memory Engine

Created 3 engines in engines/ime.c:
ime_psync, ime_psyncv, and ime_aio

Signed-off-by: Gaëtan Bossu <gbossu@ddn.com>
19 months agoAdd tests specifically for axmap_next_free()
Michael Kelley [Fri, 17 Aug 2018 19:17:00 +0000 (12:17 -0700)]
Add tests specifically for axmap_next_free()

Add tests for axmap_next_free() to ensure the new search algorithm
works correctly.  Since the new algorithm always looks forward for
the next free bit, it's possible to test for specific results given
a known condition of the map.  Test maps that are mostly empty and
mostly full. Test wrap-around. Test maps of particular sizes that
have unused bits in middle levels as well as in level 0.

19 months agoReimplement axmap_next_free() to prevent distribution skew
Michael Kelley [Fri, 17 Aug 2018 19:02:58 +0000 (12:02 -0700)]
Reimplement axmap_next_free() to prevent distribution skew

New algorithm starts the search at the specified bit, and only
looks forward (higher index) for a free bit, wrapping around the
end of the map if necessary.  This avoids a distribution skew in
the current algorithm where the low order 5 or 6 bits of the
replacement address tend toward zero.

The first_free cached value is not updated or used. But performance
of the new algorithm is marginally faster than the old algorithm
when measured on a 200 million entry map that is half full,
averaged across searches starting with each of the 200 million
entries.

A separate commit adds new tests to t/axmap.c to confirm the
correct behavior in a variety of situations and edge cases.

19 months agoconfigure: avoid pkg-config usage for http engine
David Disseldorp [Thu, 16 Aug 2018 16:56:02 +0000 (18:56 +0200)]
configure: avoid pkg-config usage for http engine

Signed-off-by: David Disseldorp <ddiss@suse.de>
19 months agoMerge branch 'wip-http-swift' of https://github.com/l-mb/fio
Jens Axboe [Thu, 16 Aug 2018 16:38:41 +0000 (10:38 -0600)]
Merge branch 'wip-http-swift' of https://github.com/l-mb/fio

* 'wip-http-swift' of https://github.com/l-mb/fio:
  engines/http: Add support for Swift storage backend

19 months agoengines/http: Add support for Swift storage backend
Lars Marowsky-Bree [Thu, 16 Aug 2018 11:06:47 +0000 (13:06 +0200)]
engines/http: Add support for Swift storage backend

This also converts two configuration variables to use "posval" instead
of string parsing or integers.

19 months agoengines/http: fix use of uninitialized variable
Jens Axboe [Thu, 16 Aug 2018 15:30:21 +0000 (09:30 -0600)]
engines/http: fix use of uninitialized variable

Fixes: c2f6a13ddc64 ("engines/http: Add support for WebDAV and S3")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agoMerge branch 'http_older_openssl' of https://github.com/ddiss/fio
Jens Axboe [Thu, 16 Aug 2018 15:20:43 +0000 (09:20 -0600)]
Merge branch 'http_older_openssl' of https://github.com/ddiss/fio

* 'http_older_openssl' of https://github.com/ddiss/fio:
  engines/http: support openssl < 1.1.0
  configure: use pkg-config to detect libcurl & openssl

19 months agoengines/http: support openssl < 1.1.0
David Disseldorp [Thu, 16 Aug 2018 14:53:49 +0000 (16:53 +0200)]
engines/http: support openssl < 1.1.0

openssl versions prior to 1.1.0 do not use an opaque pointer for
HMAC_CTX.

Signed-off-by: David Disseldorp <ddiss@suse.de>
19 months agoconfigure: use pkg-config to detect libcurl & openssl
David Disseldorp [Thu, 16 Aug 2018 14:37:50 +0000 (16:37 +0200)]
configure: use pkg-config to detect libcurl & openssl

Signed-off-by: David Disseldorp <ddiss@suse.de>
19 months agoFix compilation on centos 7
Manish Dusane [Wed, 15 Aug 2018 18:06:14 +0000 (12:06 -0600)]
Fix compilation on centos 7

"log.h:12: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘log_err’"

Fixes: https://github.com/axboe/fio/issues/651
Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agoMerge branch 'wip-http-engine' of https://github.com/l-mb/fio
Jens Axboe [Tue, 14 Aug 2018 16:09:47 +0000 (10:09 -0600)]
Merge branch 'wip-http-engine' of https://github.com/l-mb/fio

* 'wip-http-engine' of https://github.com/l-mb/fio:
  engines/http: Add support for WebDAV and S3

19 months agoengines/http: Add support for WebDAV and S3
Lars Marowsky-Bree [Fri, 10 Aug 2018 08:19:32 +0000 (10:19 +0200)]
engines/http: Add support for WebDAV and S3

The http engine allows fio to read/write objects to WebDAV and S3
storage systems via libcurl.

Writes are mapped to PUT, reads to GET, and TRIM to DELETE of individual
objects within the bucket/path.

19 months agoMerge branch 'fio-man-page' of https://github.com/bvanassche/fio
Jens Axboe [Tue, 14 Aug 2018 03:05:33 +0000 (21:05 -0600)]
Merge branch 'fio-man-page' of https://github.com/bvanassche/fio

* 'fio-man-page' of https://github.com/bvanassche/fio:
  Improve zone support documentation

19 months agoRemove old zone gen from options
Jens Axboe [Tue, 14 Aug 2018 02:54:46 +0000 (20:54 -0600)]
Remove old zone gen from options

Forgot to remove this in the previous commit...

Fixes: 077761273931 ("Fix double free of zone cache data")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agoFix double free of zone cache data
Jens Axboe [Tue, 14 Aug 2018 02:21:05 +0000 (20:21 -0600)]
Fix double free of zone cache data

We can't set it up in the parsing callback, as that happens before
we fork off the thread. Fixes a segfault with:

./fio --ioengine=libaio --randrepeat=0 --norandommap --thread --direct=1 --name=pre_test --rw=randwrite --bssplit=4k/67:8k/10:16k/7:32k/3:64k/13 --random_distribution=zoned:50/5:30/15:20/80 --iodepth=32 --runtime=60 --time_based --numjobs=2 --filename=/dev/nvme0n1p9 --group_reporting=1

Fixes: https://github.com/axboe/fio/issues/650
Fixes: e0a04ac15f61 ("Add support for zones of random IO, with varying frequency of access")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agoImprove zone support documentation
Bart Van Assche [Fri, 10 Aug 2018 00:02:59 +0000 (17:02 -0700)]
Improve zone support documentation

Improve the documentation of the zonesize, zoneskip and zonerange job
options such that this documentation becomes easier to comprehend.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
19 months agoasprintf: fix indentation
Jens Axboe [Mon, 13 Aug 2018 17:18:09 +0000 (11:18 -0600)]
asprintf: fix indentation

Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agogclient: bump output time buf size
Jens Axboe [Mon, 13 Aug 2018 15:47:45 +0000 (09:47 -0600)]
gclient: bump output time buf size

We could potentially write a few bytes too many, just bump the
timebuf size up into a safe region.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agoMerge branch 'minor_fixes' of https://github.com/sitsofe/fio
Jens Axboe [Sun, 12 Aug 2018 21:51:34 +0000 (15:51 -0600)]
Merge branch 'minor_fixes' of https://github.com/sitsofe/fio

* 'minor_fixes' of https://github.com/sitsofe/fio:
  man: fix missing/too many backslashes
  doc: rewording and add reference to --aux-path
  doc: update Log File Formats and write_iops_log sections
  minor fio.service cleanups

19 months agoman: fix missing/too many backslashes
Sitsofe Wheeler [Sun, 12 Aug 2018 19:58:56 +0000 (20:58 +0100)]
man: fix missing/too many backslashes

When viewing the man page some backslashes were AWOL whereas in other
places there were too many so fix them all.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
19 months agodoc: rewording and add reference to --aux-path
Sitsofe Wheeler [Sun, 12 Aug 2018 19:38:18 +0000 (20:38 +0100)]
doc: rewording and add reference to --aux-path

- Reword --aux-path definition
- Reword "filename semantic" text in directory definition
- Add reference to --aux-path from --filename definition

Closes: https://github.com/axboe/fio/issues/530
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
19 months agodoc: update Log File Formats and write_iops_log sections
Sitsofe Wheeler [Sat, 2 Sep 2017 11:14:19 +0000 (12:14 +0100)]
doc: update Log File Formats and write_iops_log sections

Make it clearer when various log entries will be always set to 0, move
"IOPS value is 1 without windowed logging" warning to write_iops_log
option description, change offset definition to position, add some
formatting.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
19 months agominor fio.service cleanups
Sitsofe Wheeler [Thu, 14 Sep 2017 02:08:35 +0000 (03:08 +0100)]
minor fio.service cleanups

- Remove PIDFile line because the service Type=simple and nothing will
  write the pidfile (we aren't launching fio with -daemonize=pathtopid)
- Change some capitalisation
- [Install] section

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
19 months agotravis: include new xcode 9.4
Jens Axboe [Sun, 12 Aug 2018 16:55:53 +0000 (10:55 -0600)]
travis: include new xcode 9.4

Signed-off-by: Jens Axboe <axboe@kernel.dk>
19 months agoMerge branch 'armv6' of https://github.com/sitsofe/fio
Jens Axboe [Sun, 12 Aug 2018 15:09:16 +0000 (09:09 -0600)]
Merge branch 'armv6' of https://github.com/sitsofe/fio

* 'armv6' of https://github.com/sitsofe/fio:
  arch: fix build breakage on armv6

19 months agoarch: fix build breakage on armv6
Sitsofe Wheeler [Sun, 12 Aug 2018 08:42:03 +0000 (09:42 +0100)]
arch: fix build breakage on armv6

The FreeBSD folks found fio didn't build on their armv6 systems
(https://svnweb.freebsd.org/ports/head/benchmarks/fio/Makefile?revision=432897&view=markup
) but Mikaël Urankar fixed the issue in
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228042 so copy the fix
to mainline.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
20 months agoeta: clean up ETA string printing
Jens Axboe [Fri, 10 Aug 2018 15:33:44 +0000 (09:33 -0600)]
eta: clean up ETA string printing

No need to have a separate case for trims, we just need to look
at if we have reads, writes, trims.

This also eliminates printing read or write strings if we are not
doing that particular type of IO.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoMerge branch 'realloc-io_u-buffers' of https://github.com/aclamk/fio
Jens Axboe [Thu, 9 Aug 2018 15:31:26 +0000 (09:31 -0600)]
Merge branch 'realloc-io_u-buffers' of https://github.com/aclamk/fio

* 'realloc-io_u-buffers' of https://github.com/aclamk/fio:
  iolog replay: Realloc io_u buffers to adapt to operation size.
  iolog replay: Treat 'open' on file that is scheduled to close as cancel of 'close' operation.

20 months agoxxhash: fix function definition and declaration mismatch
Udit agarwal [Thu, 9 Aug 2018 15:20:12 +0000 (09:20 -0600)]
xxhash: fix function definition and declaration mismatch

During the declaration of, let's say XXH32_init parameter type is
'unsigned int' while during the definition it is 'uint32_t'. Now, on
most desktop os, they are are usually same, however on some embedded
platforms, for example RTEMS which uses newlib as it's standard c
library, they both are different.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoiolog replay: Realloc io_u buffers to adapt to operation size.
Adam Kupczyk [Thu, 9 Aug 2018 14:05:22 +0000 (16:05 +0200)]
iolog replay: Realloc io_u buffers to adapt to operation size.

When iolog is read in chunked mode, some prefetching is done. Based on this size of buffers are determined.
Further operations may need larger buffers. If so, io_u buffers are reallocated.
Example that cause problem when read_iolog_chunked=1:
fio version 2 iolog
rbd_data.0 add
rbd_data.0 open
rbd_data.0 write 0 4096 (x20)
rbd_data.0 write 0 4096000 (x10)
rbd_data.0 close

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
20 months agoiolog replay: Treat 'open' on file that is scheduled to close as cancel of 'close...
Adam Kupczyk [Thu, 9 Aug 2018 10:53:29 +0000 (12:53 +0200)]
iolog replay: Treat 'open' on file that is scheduled to close as cancel of 'close' operation.

Problem occurs when processing generated iolog files, such as this:
fio version 2 iolog
rbd_data.0 add
rbd_data.0 open
rbd_data.0 write 0 4096
rbd_data.0 close
rbd_data.0 open
rbd_data.0 write 0 4096
rbd_data.0 close

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
20 months agolibpmem: fix type print
Jens Axboe [Tue, 7 Aug 2018 14:09:47 +0000 (08:09 -0600)]
libpmem: fix type print

Fixes: 5fff95436922 ("Add support for >= 4G block sizes")
Fixes: https://github.com/axboe/fio/issues/639
Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoclient: support --status-interval option in client/server mode
Tomohiro Kusumi [Sun, 5 Aug 2018 16:30:14 +0000 (09:30 -0700)]
client: support --status-interval option in client/server mode

This fixes "FIO client/server periodic stats" for non JSON output
(the reporter is using JSON output).
https://www.spinics.net/lists/fio/msg07204.html

Mark -L/--status-interval option with FIO_CLIENT_FLAG, so that the
command line string is sent to the server side.

The client can now control the server to periodically show status,
and each status is sent to the client via fio_server_text_output()
(FIO_NET_CMD_TEXT socket event) in log_info_buf().

Reported-by: Mark Beierl <Mark.Beierl@dell.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoiolog: move the chunked items-to-fetch logic into separate function
Jens Axboe [Fri, 3 Aug 2018 20:40:17 +0000 (14:40 -0600)]
iolog: move the chunked items-to-fetch logic into separate function

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoiolog: fix potential div-by-zero
Jens Axboe [Fri, 3 Aug 2018 20:35:57 +0000 (14:35 -0600)]
iolog: fix potential div-by-zero

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoMerge branch 'chunked-iolog-reading' of https://github.com/aclamk/fio
Jens Axboe [Fri, 3 Aug 2018 15:17:06 +0000 (09:17 -0600)]
Merge branch 'chunked-iolog-reading' of https://github.com/aclamk/fio

* 'chunked-iolog-reading' of https://github.com/aclamk/fio:
  iolog: Added new option description to HOWTO
  iolog: Added option read_iolog_chunked. Used to avoid reading large iologs at once.  Allows iologs to be infinite, generated.

20 months agoMerge branch 'read_iolog-from-unix-socket' of https://github.com/aclamk/fio
Jens Axboe [Fri, 3 Aug 2018 15:16:10 +0000 (09:16 -0600)]
Merge branch 'read_iolog-from-unix-socket' of https://github.com/aclamk/fio

* 'read_iolog-from-unix-socket' of https://github.com/aclamk/fio:
  iolog: allow to read_iolog from unix socket

20 months agoMerge branch 'windows-s_issock' of https://github.com/aclamk/fio
Jens Axboe [Fri, 3 Aug 2018 15:15:30 +0000 (09:15 -0600)]
Merge branch 'windows-s_issock' of https://github.com/aclamk/fio

* 'windows-s_issock' of https://github.com/aclamk/fio:
  platforms/windows: Add S_ISSOCK macro.

20 months agoiolog: allow to read_iolog from unix socket
Adam Kupczyk [Mon, 30 Jul 2018 10:02:55 +0000 (12:02 +0200)]
iolog: allow to read_iolog from unix socket

Checks is file provided via --read_iolog parameter is a unix socket.
In such case connect to it and fetch data from there.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
20 months agoplatforms/windows: Add S_ISSOCK macro.
Adam Kupczyk [Fri, 3 Aug 2018 10:48:28 +0000 (12:48 +0200)]
platforms/windows: Add S_ISSOCK macro.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
20 months agoiolog: Added new option description to HOWTO
Adam Kupczyk [Fri, 3 Aug 2018 10:41:59 +0000 (12:41 +0200)]
iolog: Added new option description to HOWTO

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
20 months agoMerge branch 'szaydel/solaris-Wincompatible-pointer-types' of https://github.com...
Jens Axboe [Thu, 2 Aug 2018 22:20:24 +0000 (16:20 -0600)]
Merge branch 'szaydel/solaris-Wincompatible-pointer-types' of https://github.com/szaydel/fio

* 'szaydel/solaris-Wincompatible-pointer-types' of https://github.com/szaydel/fio:
  Fix incompatible pointer types warning on Solaris with gcc 6.4

20 months agoFix incompatible pointer types warning on Solaris with gcc 6.4
Sam Zaydel [Thu, 2 Aug 2018 22:01:27 +0000 (15:01 -0700)]
Fix incompatible pointer types warning on Solaris with gcc 6.4

20 months agoiolog: Added option read_iolog_chunked. Used to avoid reading large iologs at once.
Adam Kupczyk [Wed, 1 Aug 2018 11:45:27 +0000 (13:45 +0200)]
iolog: Added option read_iolog_chunked. Used to avoid reading large iologs at once.
Allows iologs to be infinite, generated.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
20 months agoMerge branch 'fio-histo-fix' of https://github.com/parallel-fs-utils/fio
Jens Axboe [Mon, 30 Jul 2018 14:24:20 +0000 (08:24 -0600)]
Merge branch 'fio-histo-fix' of https://github.com/parallel-fs-utils/fio

* 'fio-histo-fix' of https://github.com/parallel-fs-utils/fio:
  clean up argparse usage

20 months agoclean up argparse usage
Ben England [Sun, 29 Jul 2018 17:14:59 +0000 (13:14 -0400)]
clean up argparse usage

20 months agoMerge branch 'sgunmap2' of https://github.com/vincentkfu/fio
Jens Axboe [Thu, 26 Jul 2018 17:47:28 +0000 (11:47 -0600)]
Merge branch 'sgunmap2' of https://github.com/vincentkfu/fio

* 'sgunmap2' of https://github.com/vincentkfu/fio:
  docs: update HOWTO and manpage for sg trim support
  testing: add test scripts for sg ioengine
  engines/sg: move asserts and debug statements behind a debug flag
  engines/sg: add cmdp and dxferp for trims to sg error string
  engines/sg: support trim operations via the UNMAP command
  stat: add IO submit and complete depths to JSON output
  ioengines: have ioengines with commit do own io accounting for trims

20 months agodocs: update HOWTO and manpage for sg trim support
Vincent Fu [Thu, 26 Jul 2018 16:27:21 +0000 (09:27 -0700)]
docs: update HOWTO and manpage for sg trim support

Also fix HOWTO sg_write_mode formatting

20 months agotesting: add test scripts for sg ioengine
Vincent Fu [Fri, 20 Jul 2018 16:54:49 +0000 (09:54 -0700)]
testing: add test scripts for sg ioengine

t/sgunmap-perf.py carries out basic performance testing using
the sg ioengine

t/sgunmap-test.py checks that the IO depths reported for sg
trim workloads are sensible

20 months agoengines/sg: move asserts and debug statements behind a debug flag
Vincent Fu [Wed, 20 Jun 2018 19:47:16 +0000 (12:47 -0700)]
engines/sg: move asserts and debug statements behind a debug flag

The sg ioengine modifications include some debug statements and
integrity checks that could hurt performance. Put these behind
a debug flag.

20 months agoengines/sg: add cmdp and dxferp for trims to sg error string
Vincent Fu [Wed, 25 Jul 2018 20:00:26 +0000 (13:00 -0700)]
engines/sg: add cmdp and dxferp for trims to sg error string

20 months agoengines/sg: support trim operations via the UNMAP command
Vincent Fu [Thu, 24 May 2018 21:00:36 +0000 (17:00 -0400)]
engines/sg: support trim operations via the UNMAP command

Multiple trim ranges can be submitted with a single UNMAP command
for character devices (e.g., /dev/sg1) using iodepth_batch. For
example, with iodepth_batch=4, each UNMAP command will include
four ranges. IOPS will still be reported as if four IO operations
were carried out.

Note that currently sg only supports a maximum of 16 commands
in flight per file. So be sure to maintain (iodepth/iodepth_batch) <= 16.

20 months agostat: add IO submit and complete depths to JSON output
Vincent Fu [Tue, 19 Jun 2018 19:25:58 +0000 (12:25 -0700)]
stat: add IO submit and complete depths to JSON output

The IO submit and IO complete depths were missing from the JSON
output. This patch adds JSON objects for these distributions.

20 months agoioengines: have ioengines with commit do own io accounting for trims
Vincent Fu [Wed, 25 Jul 2018 19:37:06 +0000 (12:37 -0700)]
ioengines: have ioengines with commit do own io accounting for trims

ioengines.c:td_io_queue has a special case where it calls
io_u_mark_submit() and io_u_mark_complete() when the io_u
is a trim operation or when the ioengine does not have a
commit function. This patch makes these two calls the
responsibility of the ioengine for trim io_u's when the
ioengine does have a commit function.

This only applies to libaio because it is the only ioengine
supporting trims with a commit function. All other ioengines that
support trim operations do not have a commit function. These
ioengines are: falloc, glusterfs_async, mmap, mtd, posixaio,
rados, rbd, splice, sync, psync, pvsync, pvsync2.

20 months agoMerge branch 'fio-histo-log-pctiles' of https://github.com/parallel-fs-utils/fio
Jens Axboe [Wed, 25 Jul 2018 20:42:57 +0000 (14:42 -0600)]
Merge branch 'fio-histo-log-pctiles' of https://github.com/parallel-fs-utils/fio

* 'fio-histo-log-pctiles' of https://github.com/parallel-fs-utils/fio:
  design document for tools/hist/fio-histo-log-pctiles.py
  switch to argparse module for CLI parsing
  use interpolation for more accurate percentile calculation
  get latency percentiles over time from fio histo logs

20 months agodesign document for tools/hist/fio-histo-log-pctiles.py
Ben England [Wed, 25 Jul 2018 19:40:04 +0000 (15:40 -0400)]
design document for tools/hist/fio-histo-log-pctiles.py

20 months agoMerge branch 'fio-c++-engine' of https://github.com/tchaikov/fio
Jens Axboe [Wed, 25 Jul 2018 14:41:38 +0000 (08:41 -0600)]
Merge branch 'fio-c++-engine' of https://github.com/tchaikov/fio

* 'fio-c++-engine' of https://github.com/tchaikov/fio:
  replace typeof with __typeof__

20 months agoreplace typeof with __typeof__
Kefu Chai [Wed, 25 Jul 2018 14:02:09 +0000 (22:02 +0800)]
replace typeof with __typeof__

so we are able to extend fio with C++ .

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
20 months agoinit: unify 't' time period
Jens Axboe [Tue, 24 Jul 2018 21:23:28 +0000 (15:23 -0600)]
init: unify 't' time period

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agolibpmem: update print statement for bs now being ULL
Jens Axboe [Mon, 23 Jul 2018 16:12:54 +0000 (10:12 -0600)]
libpmem: update print statement for bs now being ULL

Fixes: 5fff95436922 ("Add support for >= 4G block sizes")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoparse: mark another fall-through switch case
Jens Axboe [Mon, 23 Jul 2018 16:12:40 +0000 (10:12 -0600)]
parse: mark another fall-through switch case

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoparse: mark fall-through switch case
Jens Axboe [Mon, 23 Jul 2018 16:10:12 +0000 (10:10 -0600)]
parse: mark fall-through switch case

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoUse stdlib.h instead of malloc.h
Jens Axboe [Mon, 23 Jul 2018 16:05:49 +0000 (10:05 -0600)]
Use stdlib.h instead of malloc.h

A few more conversions.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoAdd support for >= 4G block sizes
Jeff Furlong [Mon, 23 Jul 2018 15:15:40 +0000 (09:15 -0600)]
Add support for >= 4G block sizes

For trims, it's useful to be able to support larger than (or equal
to) 4GB. This extends the block sizes support for that.

Change from Jeff, various little fixups from me.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoFio 3.8 fio-3.8
Jens Axboe [Mon, 23 Jul 2018 14:55:32 +0000 (08:55 -0600)]
Fio 3.8

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agogfio: cleanup includes
Jens Axboe [Mon, 23 Jul 2018 14:29:14 +0000 (08:29 -0600)]
gfio: cleanup includes

Use stdlib.h, not malloc.h. And ensure that we pull in the
right header for basename(3).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoswitch to argparse module for CLI parsing
Ben England [Fri, 20 Jul 2018 17:49:37 +0000 (13:49 -0400)]
switch to argparse module for CLI parsing

20 months agouse interpolation for more accurate percentile calculation
Ben England [Thu, 19 Jul 2018 22:05:52 +0000 (18:05 -0400)]
use interpolation for more accurate percentile calculation

20 months agoget latency percentiles over time from fio histo logs
Ben England [Tue, 17 Jul 2018 18:40:01 +0000 (14:40 -0400)]
get latency percentiles over time from fio histo logs

20 months agoaxmap: optimize ulog64 usage in axmap_handler()
Jens Axboe [Thu, 12 Jul 2018 14:33:14 +0000 (08:33 -0600)]
axmap: optimize ulog64 usage in axmap_handler()

We can do this incrementally, no need to compute from scratch
for every iteration.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agot/axmap: add longer overlap test case
Jens Axboe [Wed, 11 Jul 2018 21:32:31 +0000 (15:32 -0600)]
t/axmap: add longer overlap test case

Also fix bad test stop termination.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agot/axmap: add regression case for recent overlap failure case
Jens Axboe [Wed, 11 Jul 2018 21:17:32 +0000 (15:17 -0600)]
t/axmap: add regression case for recent overlap failure case

Catches the issue that bd71edcedc3d fixes.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoaxmap: fix continued sequential bit setting
Jens Axboe [Wed, 11 Jul 2018 21:16:58 +0000 (15:16 -0600)]
axmap: fix continued sequential bit setting

We need to remember to clear ->set_bits if we don't set new bits,
or the caller will think we re-set the previously set bits.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoaxmap: a few more cleanups
Jens Axboe [Wed, 11 Jul 2018 19:59:32 +0000 (13:59 -0600)]
axmap: a few more cleanups

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoaxmap: remove unused 'data' argument to topdown handler
Jens Axboe [Wed, 11 Jul 2018 19:48:32 +0000 (13:48 -0600)]
axmap: remove unused 'data' argument to topdown handler

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agofio: should_fsync() returns bool
Jens Axboe [Wed, 11 Jul 2018 19:26:19 +0000 (13:26 -0600)]
fio: should_fsync() returns bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
20 months agoaxmap: code cleanups
Jens Axboe [Wed, 11 Jul 2018 18:25:43 +0000 (12:25 -0600)]
axmap: code cleanups

- Move the const bit masks to the start of the file
- Get rid of a useless variable in axmap_handler_topdown()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agot/axmap: add zero return overlap cases
Jens Axboe [Wed, 11 Jul 2018 15:11:15 +0000 (09:11 -0600)]
t/axmap: add zero return overlap cases

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoaxmap: clean up 'no bits to set' case
Jens Axboe [Wed, 11 Jul 2018 14:34:06 +0000 (08:34 -0600)]
axmap: clean up 'no bits to set' case

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agot/axmap: don't print 'pass' on failure
Jens Axboe [Wed, 11 Jul 2018 14:32:37 +0000 (08:32 -0600)]
t/axmap: don't print 'pass' on failure

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agot/axmap: a few more overlap cases
Jens Axboe [Wed, 11 Jul 2018 14:29:22 +0000 (08:29 -0600)]
t/axmap: a few more overlap cases

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agot/axmap: clean up overlap tests
Jens Axboe [Wed, 11 Jul 2018 03:51:16 +0000 (21:51 -0600)]
t/axmap: clean up overlap tests

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoMakefile: lib/axmap no longer needs hweight
Jens Axboe [Tue, 10 Jul 2018 23:02:25 +0000 (17:02 -0600)]
Makefile: lib/axmap no longer needs hweight

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agot/axmap: add a few more overlap test cases
Jens Axboe [Tue, 10 Jul 2018 23:02:03 +0000 (17:02 -0600)]
t/axmap: add a few more overlap test cases

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoaxmap: ensure that overlaps are handled strictly sequential
Jens Axboe [Tue, 10 Jul 2018 22:52:16 +0000 (16:52 -0600)]
axmap: ensure that overlaps are handled strictly sequential

We must terminate at the first overlapping bit, we can't
just mask off non-sequential ranges.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agot/axmap: add overlap test cases
Jens Axboe [Tue, 10 Jul 2018 22:51:24 +0000 (16:51 -0600)]
t/axmap: add overlap test cases

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoio_u: ensure we generate the full length of block sizes
Jens Axboe [Tue, 10 Jul 2018 21:10:17 +0000 (15:10 -0600)]
io_u: ensure we generate the full length of block sizes

Since we round down, we can miss the last entry. This ensures that
if we do:

bsrange=4k-16k

we actually get an even split of 4,8,12,16k ios.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoio_u: fix negative offset due to wrap
Jens Axboe [Tue, 10 Jul 2018 19:06:38 +0000 (13:06 -0600)]
io_u: fix negative offset due to wrap

If we do wrap, the math is off and we end up wrapping a 64-bit
value. Instead reset to the initial offset.

Reported-by: Bart Van Assche <Bart.VanAssche@wdc.com>
Fixes: 4c8be5b1569f ("Fix bug with zone and zone skipping and io_limit")
Fixes: 224b3093cc21 ("Fix zoning issue with seq-io and randommap issue")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoblktrace: just ignore zero byte traces
Jens Axboe [Wed, 4 Jul 2018 20:59:18 +0000 (14:59 -0600)]
blktrace: just ignore zero byte traces

Apparently they do happen. Log a warning (once) and just discard
them.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agoFix compilation without cgroups
Jens Axboe [Fri, 29 Jun 2018 14:00:29 +0000 (08:00 -0600)]
Fix compilation without cgroups

Fixes: c3dc516ad75e ("fio: work with cgroup2 as well")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
21 months agofio: add job_runtime to the thread json output
Josef Bacik [Fri, 29 Jun 2018 13:49:03 +0000 (09:49 -0400)]
fio: add job_runtime to the thread json output

I discovered that the 'elapsed' value of each job is the overall time
the whole fio run took, not the per-job elapsed time, despite it being
in the per-job section of the json output.  Rather than break everybody
by fixing this, just add a 'job_runtime' value to the job section and
include the time the thread ran in milliseconds.  This is handy for jobs
that want to compare runtime of two threads in different cgroups.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>