Jens Axboe [Sat, 25 Aug 2018 00:22:13 +0000 (18:22 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Fix the typehelp[] array
Bart Van Assche [Fri, 24 Aug 2018 23:48:11 +0000 (16:48 -0700)]
Fix the typehelp[] array
This patch fixes Coverity ID 183335 (Out-of-bounds access of typehelp[] array).
Fixes:
5fff95436922 ("Add support for >= 4G block sizes")
Fixes:
c47537ee91cd ("parse: add support for soft deprecated options")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Fri, 24 Aug 2018 18:59:45 +0000 (12:59 -0600)]
Merge branch 'zbd'
* zbd:
Add scripts for testing the fio zoned block device support code
Add support for resetting zones periodically
Make it possible to limit the number of open zones
Collect and show zone reset statistics
Add support for zoned block devices
Add two assert statements in mark_random_map()
Pass offset and buffer length explicitly to mark_random_map()
Introduce the io_u.post_submit callback function pointer
Add the zonemode job option
Add the 'zbd' debug level
configure: Add <linux/blkzoned.h> test
Bart Van Assche [Fri, 24 Aug 2018 18:31:31 +0000 (11:31 -0700)]
Add scripts for testing the fio zoned block device support code
These scripts can be run e.g. as follows:
t/zbd/run-tests-against-zoned-nullb &&
t/zbd/run-tests-against-regular-nullb &&
t/zbd/test-zbd-support /dev/disk/by-id/scsi-SHGST_HSH721414AL52M0_VEG007HG &&
echo All tests passed
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:30 +0000 (11:31 -0700)]
Add support for resetting zones periodically
Filesystems that support zoned block devices typically perform garbage
collection if device usage exceeds a certain threshold. Add two command
line options that allow to simulate this behavior.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:29 +0000 (11:31 -0700)]
Make it possible to limit the number of open zones
When running a random I/O test across an entire drive many more zones
will be open than in a typical application workload. Hence add a command
line option that allows to limit the number of open zones.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:28 +0000 (11:31 -0700)]
Collect and show zone reset statistics
Show how many zone resets have been performed in the I/O statistics.
An example:
[ ... ]
write: IOPS=17.0k, BW=66.5MiB/s (69.8MB/s)(1024MiB/15387msec); 4 zone resets
[ ... ]
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:27 +0000 (11:31 -0700)]
Add support for zoned block devices
This patch adds support for zoned block devices as follows:
- After the file size has been determined, check whether the file name
refers to a block device. If so, check whether that block device is a
host-managed block device. If that is the case, read the zone
information using the BLKREPORTZONE ioctl. That ioctl is supported by
the Linux kernel since kernel version v4.10.
- After all command-line options have been processed and all job files
have been read, verify whether these refer to a zoned block device
and also whether the specified options are compatible with a zoned
block device. Complain if that is not the case.
- After each get_next_block() call, verify whether the block is
appropriate for a zoned block device. When writing data to a
sequential zone, adjust the write offset to the zone write pointer.
When reading from a sequential zone, avoid reading past the write
pointer.
- After I/O submission, update the variable that represents the write
pointer.
- When writing data and with data verification enabled, reset a zone
before writing any data into a zone. Otherwise reset a zone before
issuing a write if that zone is full.
- Translate trim into zone resets. Zoned block devices namely do not
have to support any of the SCSI commands that are used by the kernel
to implement the discard ioctl (UNMAP / WRITE SAME).
This work started from a patch from Masato Suzuki <masato.suzuki@wdc.com>.
Some of the ideas in this patch come from Phillip Chen
<phillip.a.chen@seagate.com>.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:26 +0000 (11:31 -0700)]
Add two assert statements in mark_random_map()
Add two assert statements that verify whether mark_random_map() is
used correctly.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:25 +0000 (11:31 -0700)]
Pass offset and buffer length explicitly to mark_random_map()
This patch does not change any functionality. The changes introduced by
this patch will be used by the zoned block device code.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:24 +0000 (11:31 -0700)]
Introduce the io_u.post_submit callback function pointer
This patch does not change any functionality. The code introduced by
this patch will be used by the zoned block device code.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:23 +0000 (11:31 -0700)]
Add the zonemode job option
Fio's zone support makes fio perform I/O inside a zone before it skips
to the next zone. That behavior is the opposite of the behavior needed
for zoned block devices, namely to consider all zones when performing
random I/O. Hence introduce a new job option that allows users to
choose between fio's traditional zone mode and the behavior needed for
zoned block devices. This patch makes fio behave identically with
--zonemode=none and --zonemode=zbd. A later patch will implement new
behavior for --zonemode=zbd.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:22 +0000 (11:31 -0700)]
Add the 'zbd' debug level
This debug level will be used in the next patch.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:21 +0000 (11:31 -0700)]
configure: Add <linux/blkzoned.h> test
Add a test for <linux/blkzoned.h> because that header file will be used
by the zoned block device code.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 19:58:21 +0000 (13:58 -0600)]
t/axmap: print explicit overlap ranges tested
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 18:22:58 +0000 (12:22 -0600)]
t/axmap: add regression test for overlap case resulting in 0 settable bits
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 18:22:10 +0000 (12:22 -0600)]
axmap: return early of an overlap results in 0 settable bits
Reported-by: Bart Van Assche <bart.vanassche@wdc.com>
Fixes:
15a4f49 ("lib/axmap: Simplify axmap_set_fn()")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 17:05:34 +0000 (11:05 -0600)]
Merge branch '
20180823-terse-remote-fix' of https://github.com/mcgrof/fio
* '
20180823-terse-remote-fix' of https://github.com/mcgrof/fio:
init: add semantics for all types of backends running
client: respect terse output on client <--> backend relationship
Luis Chamberlain [Tue, 21 Aug 2018 18:15:39 +0000 (12:15 -0600)]
init: add semantics for all types of backends running
Commit
67a176fc7 ("Fix segfault with client/server and minimal output")
addressed a segfault but it wasn't very clear *why* the fix was needed.
The reason it was needed is that the disk util and its respective
semaphore are only initialized when setup_disk_util() is called, this
happens upon fio_backend() calls. That is, either we have a dedicated
backend or have initiated a backend for the localhost due to some local
work. And show_thread_status() is currently called from a complete
client setup -- handle_ts() calls show_thread_status(), and a client
does not collect any local disk data, it receives this from the backend.
As such, the semaphore won't be setup in a client setup and this is why
we segfault here.
We can enable show_thread_status() then only if any type of backend is
running, however there is are no semantics currently which enable us to
query for this. Add such semantics and replace the previous check with
a check for if fio_backend() was ever called. This will make it clearer.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Luis Chamberlain [Tue, 21 Aug 2018 13:24:36 +0000 (07:24 -0600)]
client: respect terse output on client <--> backend relationship
You end up with different results if you run this terse output
on a local system which also runs its own backend Vs running a
client to connect to a remote server which is running fio as a
backend only. The reason is the client ops handle printing of
threads / disk utils separately. The terse output created *by*
the backend is the right and expected output, so just use that,
and we can piggy back off of the fact that the server will send
its own output via FIO_NET_CMD_TEXT.
Another solution is to address getting the disk util data sent
to be cached locally, and then upon handle_ts() print that, but that
would require significant re-architecturing.
Terse output is supposed to be just that, terse. This implies
that it will not be clear from what backend data came from, but
for this the best strategy is to *extend* the terse version with
yet another field, the remote hostname/ip address.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Jens Axboe [Thu, 23 Aug 2018 16:29:12 +0000 (10:29 -0600)]
parse: fix bssplit option
Ensure that we copy option sets for FIO_OPT_STR_ULL like we do
for FIO_OPT_STR, in case only one data direction has been defined.
Fixes:
5fff95436922 ("Add support for >= 4G block sizes")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 02:13:45 +0000 (20:13 -0600)]
Merge branch 'axmap' of https://github.com/bvanassche/fio
* 'axmap' of https://github.com/bvanassche/fio:
lib/axmap: Optimize __axmap_set()
lib/axmap: Simplify axmap_set_fn()
lib/axmap: Make axmap_new() more robust
lib/axmap: Inline ulog64()
lib/axmap: Add more documentation
Bart Van Assche [Wed, 22 Aug 2018 20:22:12 +0000 (13:22 -0700)]
lib/axmap: Optimize __axmap_set()
Since it is guaranteed that nr_bits <= BLOCKS_PER_UNIT and since
axmap_set_fn() needs to be called only once to set that number of
bits, remove the loop from __axmap_set().
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 22 Aug 2018 17:43:46 +0000 (10:43 -0700)]
lib/axmap: Simplify axmap_set_fn()
Instead of handling ffz(~overlap) == bit as a special case, handle
this case in the same way as ffz(~overlap) != bit. This patch does
not change any functionality but removes one if-instruction from the
hot path.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 22 Aug 2018 16:00:15 +0000 (09:00 -0700)]
lib/axmap: Make axmap_new() more robust
Return NULL instead of triggering a segmentation fault if calloc() fails.
Note: calling free(NULL) is safe so it is not necessary to check the
free() argument before calling free().
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 22 Aug 2018 20:35:19 +0000 (13:35 -0700)]
lib/axmap: Inline ulog64()
Since the function ulog64() only has one caller and since it can be
replaced by a single closed-form expression, inline that function.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 22 Aug 2018 15:58:00 +0000 (08:58 -0700)]
lib/axmap: Add more documentation
Make the axmap code easier to follow by documenting the semantics of
the structures and functions in the lib/axmap.c source file.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jeff Furlong [Mon, 20 Aug 2018 23:25:29 +0000 (23:25 +0000)]
io_u: residiual size should be unsigned long long
If we target a number of bytes of IO and keep subtracting a
xfer_buflen, we need to ensure the residual length is of the
same size.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 20 Aug 2018 14:34:13 +0000 (08:34 -0600)]
Update HOWTO for read_iolog change
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 20 Aug 2018 14:32:29 +0000 (08:32 -0600)]
Merge branch 'multiple-read_iolog' of https://github.com/aclamk/fio
* 'multiple-read_iolog' of https://github.com/aclamk/fio:
iolog: Now --read_iolog can contain multiple replay files, separated by ':'.
Adam Kupczyk [Fri, 17 Aug 2018 13:30:58 +0000 (15:30 +0200)]
iolog: Now --read_iolog can contain multiple replay files, separated by ':'.
Useful when --numjobs is more then 1.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Jens Axboe [Sat, 18 Aug 2018 01:39:07 +0000 (19:39 -0600)]
Move steady state unit test to t/
That's where we keep the other tests.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 17 Aug 2018 23:29:11 +0000 (17:29 -0600)]
Merge branch 'master' of https://github.com/kelleymh/fio
* 'master' of https://github.com/kelleymh/fio:
Remove unused code in lib/axmap.c
Tomohiro Kusumi [Fri, 17 Aug 2018 22:22:00 +0000 (15:22 -0700)]
http: fix compile-time warnings
Fix following warnings on RHEL6 and variants.
engines/http.c: In function 'fio_http_setup':
engines/http.c:611: warning: call to '_curl_easy_setopt_err_seek_cb' declared with attribute warning: curl_easy_setopt expects a curl_seek_callback argument for this option
engines/http.c: In function 'fio_http_queue':
engines/http.c:549: warning: call to '_curl_easy_setopt_err_curl_off_t' declared with attribute warning: curl_easy_setopt expects a curl_off_t argument for this option
For engines/http.c:611, pass &_http_seek instead of cast, since
lack of cast isn't fio's issue. See below comments for details.
"Re: [PATCH] http: fix compile-time warnings"
https://www.spinics.net/lists/fio/msg07246.html
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Michael Kelley [Fri, 17 Aug 2018 21:49:49 +0000 (14:49 -0700)]
Remove unused code in lib/axmap.c
With the new axmap_next_free() implementation, the "first_free" cache
is no longer used. Remove all references to it. Also axmap_clear()
is never referenced, so remove it and related code.
Jens Axboe [Fri, 17 Aug 2018 19:39:40 +0000 (13:39 -0600)]
Merge branch 'ddn-ime'
Pull in DDN IME engine
* ddn-ime:
Sync man page with fio for IME
engines/ime: various code and style cleanups
Add support for DDN's Infinite Memory Engine
Jens Axboe [Fri, 17 Aug 2018 19:39:12 +0000 (13:39 -0600)]
Sync man page with fio for IME
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 17 Aug 2018 19:38:59 +0000 (13:38 -0600)]
engines/ime: various code and style cleanups
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 17 Aug 2018 19:34:21 +0000 (13:34 -0600)]
Merge branch 'ime-support' of https://github.com/DDNStorage/fio-public into ddn-ime
* 'ime-support' of https://github.com/DDNStorage/fio-public:
Add support for DDN's Infinite Memory Engine
Jens Axboe [Fri, 17 Aug 2018 19:27:25 +0000 (13:27 -0600)]
Merge branch 'master' of https://github.com/kelleymh/fio
* 'master' of https://github.com/kelleymh/fio:
Add tests specifically for axmap_next_free()
Reimplement axmap_next_free() to prevent distribution skew
Gaëtan Bossu [Wed, 1 Aug 2018 14:27:37 +0000 (16:27 +0200)]
Add support for DDN's Infinite Memory Engine
Created 3 engines in engines/ime.c:
ime_psync, ime_psyncv, and ime_aio
Signed-off-by: Gaëtan Bossu <gbossu@ddn.com>
Michael Kelley [Fri, 17 Aug 2018 19:17:00 +0000 (12:17 -0700)]
Add tests specifically for axmap_next_free()
Add tests for axmap_next_free() to ensure the new search algorithm
works correctly. Since the new algorithm always looks forward for
the next free bit, it's possible to test for specific results given
a known condition of the map. Test maps that are mostly empty and
mostly full. Test wrap-around. Test maps of particular sizes that
have unused bits in middle levels as well as in level 0.
Michael Kelley [Fri, 17 Aug 2018 19:02:58 +0000 (12:02 -0700)]
Reimplement axmap_next_free() to prevent distribution skew
New algorithm starts the search at the specified bit, and only
looks forward (higher index) for a free bit, wrapping around the
end of the map if necessary. This avoids a distribution skew in
the current algorithm where the low order 5 or 6 bits of the
replacement address tend toward zero.
The first_free cached value is not updated or used. But performance
of the new algorithm is marginally faster than the old algorithm
when measured on a 200 million entry map that is half full,
averaged across searches starting with each of the 200 million
entries.
A separate commit adds new tests to t/axmap.c to confirm the
correct behavior in a variety of situations and edge cases.
David Disseldorp [Thu, 16 Aug 2018 16:56:02 +0000 (18:56 +0200)]
configure: avoid pkg-config usage for http engine
Signed-off-by: David Disseldorp <ddiss@suse.de>
Jens Axboe [Thu, 16 Aug 2018 16:38:41 +0000 (10:38 -0600)]
Merge branch 'wip-http-swift' of https://github.com/l-mb/fio
* 'wip-http-swift' of https://github.com/l-mb/fio:
engines/http: Add support for Swift storage backend
Lars Marowsky-Bree [Thu, 16 Aug 2018 11:06:47 +0000 (13:06 +0200)]
engines/http: Add support for Swift storage backend
This also converts two configuration variables to use "posval" instead
of string parsing or integers.
Jens Axboe [Thu, 16 Aug 2018 15:30:21 +0000 (09:30 -0600)]
engines/http: fix use of uninitialized variable
Fixes:
c2f6a13ddc64 ("engines/http: Add support for WebDAV and S3")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 16 Aug 2018 15:20:43 +0000 (09:20 -0600)]
Merge branch 'http_older_openssl' of https://github.com/ddiss/fio
* 'http_older_openssl' of https://github.com/ddiss/fio:
engines/http: support openssl < 1.1.0
configure: use pkg-config to detect libcurl & openssl
David Disseldorp [Thu, 16 Aug 2018 14:53:49 +0000 (16:53 +0200)]
engines/http: support openssl < 1.1.0
openssl versions prior to 1.1.0 do not use an opaque pointer for
HMAC_CTX.
Signed-off-by: David Disseldorp <ddiss@suse.de>
David Disseldorp [Thu, 16 Aug 2018 14:37:50 +0000 (16:37 +0200)]
configure: use pkg-config to detect libcurl & openssl
Signed-off-by: David Disseldorp <ddiss@suse.de>
Manish Dusane [Wed, 15 Aug 2018 18:06:14 +0000 (12:06 -0600)]
Fix compilation on centos 7
"log.h:12: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘log_err’"
Fixes: https://github.com/axboe/fio/issues/651
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 14 Aug 2018 16:09:47 +0000 (10:09 -0600)]
Merge branch 'wip-http-engine' of https://github.com/l-mb/fio
* 'wip-http-engine' of https://github.com/l-mb/fio:
engines/http: Add support for WebDAV and S3
Lars Marowsky-Bree [Fri, 10 Aug 2018 08:19:32 +0000 (10:19 +0200)]
engines/http: Add support for WebDAV and S3
The http engine allows fio to read/write objects to WebDAV and S3
storage systems via libcurl.
Writes are mapped to PUT, reads to GET, and TRIM to DELETE of individual
objects within the bucket/path.
Jens Axboe [Tue, 14 Aug 2018 03:05:33 +0000 (21:05 -0600)]
Merge branch 'fio-man-page' of https://github.com/bvanassche/fio
* 'fio-man-page' of https://github.com/bvanassche/fio:
Improve zone support documentation
Jens Axboe [Tue, 14 Aug 2018 02:54:46 +0000 (20:54 -0600)]
Remove old zone gen from options
Forgot to remove this in the previous commit...
Fixes:
077761273931 ("Fix double free of zone cache data")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 14 Aug 2018 02:21:05 +0000 (20:21 -0600)]
Fix double free of zone cache data
We can't set it up in the parsing callback, as that happens before
we fork off the thread. Fixes a segfault with:
./fio --ioengine=libaio --randrepeat=0 --norandommap --thread --direct=1 --name=pre_test --rw=randwrite --bssplit=4k/67:8k/10:16k/7:32k/3:64k/13 --random_distribution=zoned:50/5:30/15:20/80 --iodepth=32 --runtime=60 --time_based --numjobs=2 --filename=/dev/nvme0n1p9 --group_reporting=1
Fixes: https://github.com/axboe/fio/issues/650
Fixes:
e0a04ac15f61 ("Add support for zones of random IO, with varying frequency of access")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 10 Aug 2018 00:02:59 +0000 (17:02 -0700)]
Improve zone support documentation
Improve the documentation of the zonesize, zoneskip and zonerange job
options such that this documentation becomes easier to comprehend.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Mon, 13 Aug 2018 17:18:09 +0000 (11:18 -0600)]
asprintf: fix indentation
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 13 Aug 2018 15:47:45 +0000 (09:47 -0600)]
gclient: bump output time buf size
We could potentially write a few bytes too many, just bump the
timebuf size up into a safe region.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 12 Aug 2018 21:51:34 +0000 (15:51 -0600)]
Merge branch 'minor_fixes' of https://github.com/sitsofe/fio
* 'minor_fixes' of https://github.com/sitsofe/fio:
man: fix missing/too many backslashes
doc: rewording and add reference to --aux-path
doc: update Log File Formats and write_iops_log sections
minor fio.service cleanups
Sitsofe Wheeler [Sun, 12 Aug 2018 19:58:56 +0000 (20:58 +0100)]
man: fix missing/too many backslashes
When viewing the man page some backslashes were AWOL whereas in other
places there were too many so fix them all.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 12 Aug 2018 19:38:18 +0000 (20:38 +0100)]
doc: rewording and add reference to --aux-path
- Reword --aux-path definition
- Reword "filename semantic" text in directory definition
- Add reference to --aux-path from --filename definition
Closes: https://github.com/axboe/fio/issues/530
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sat, 2 Sep 2017 11:14:19 +0000 (12:14 +0100)]
doc: update Log File Formats and write_iops_log sections
Make it clearer when various log entries will be always set to 0, move
"IOPS value is 1 without windowed logging" warning to write_iops_log
option description, change offset definition to position, add some
formatting.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Thu, 14 Sep 2017 02:08:35 +0000 (03:08 +0100)]
minor fio.service cleanups
- Remove PIDFile line because the service Type=simple and nothing will
write the pidfile (we aren't launching fio with -daemonize=pathtopid)
- Change some capitalisation
- [Install] section
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Sun, 12 Aug 2018 16:55:53 +0000 (10:55 -0600)]
travis: include new xcode 9.4
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 12 Aug 2018 15:09:16 +0000 (09:09 -0600)]
Merge branch 'armv6' of https://github.com/sitsofe/fio
* 'armv6' of https://github.com/sitsofe/fio:
arch: fix build breakage on armv6
Sitsofe Wheeler [Sun, 12 Aug 2018 08:42:03 +0000 (09:42 +0100)]
arch: fix build breakage on armv6
The FreeBSD folks found fio didn't build on their armv6 systems
(https://svnweb.freebsd.org/ports/head/benchmarks/fio/Makefile?revision=432897&view=markup
) but Mikaël Urankar fixed the issue in
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228042 so copy the fix
to mainline.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Fri, 10 Aug 2018 15:33:44 +0000 (09:33 -0600)]
eta: clean up ETA string printing
No need to have a separate case for trims, we just need to look
at if we have reads, writes, trims.
This also eliminates printing read or write strings if we are not
doing that particular type of IO.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 9 Aug 2018 15:31:26 +0000 (09:31 -0600)]
Merge branch 'realloc-io_u-buffers' of https://github.com/aclamk/fio
* 'realloc-io_u-buffers' of https://github.com/aclamk/fio:
iolog replay: Realloc io_u buffers to adapt to operation size.
iolog replay: Treat 'open' on file that is scheduled to close as cancel of 'close' operation.
Udit agarwal [Thu, 9 Aug 2018 15:20:12 +0000 (09:20 -0600)]
xxhash: fix function definition and declaration mismatch
During the declaration of, let's say XXH32_init parameter type is
'unsigned int' while during the definition it is 'uint32_t'. Now, on
most desktop os, they are are usually same, however on some embedded
platforms, for example RTEMS which uses newlib as it's standard c
library, they both are different.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Adam Kupczyk [Thu, 9 Aug 2018 14:05:22 +0000 (16:05 +0200)]
iolog replay: Realloc io_u buffers to adapt to operation size.
When iolog is read in chunked mode, some prefetching is done. Based on this size of buffers are determined.
Further operations may need larger buffers. If so, io_u buffers are reallocated.
Example that cause problem when read_iolog_chunked=1:
fio version 2 iolog
rbd_data.0 add
rbd_data.0 open
rbd_data.0 write 0 4096 (x20)
rbd_data.0 write 0
4096000 (x10)
rbd_data.0 close
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Adam Kupczyk [Thu, 9 Aug 2018 10:53:29 +0000 (12:53 +0200)]
iolog replay: Treat 'open' on file that is scheduled to close as cancel of 'close' operation.
Problem occurs when processing generated iolog files, such as this:
fio version 2 iolog
rbd_data.0 add
rbd_data.0 open
rbd_data.0 write 0 4096
rbd_data.0 close
rbd_data.0 open
rbd_data.0 write 0 4096
rbd_data.0 close
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Jens Axboe [Tue, 7 Aug 2018 14:09:47 +0000 (08:09 -0600)]
libpmem: fix type print
Fixes:
5fff95436922 ("Add support for >= 4G block sizes")
Fixes: https://github.com/axboe/fio/issues/639
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Tomohiro Kusumi [Sun, 5 Aug 2018 16:30:14 +0000 (09:30 -0700)]
client: support --status-interval option in client/server mode
This fixes "FIO client/server periodic stats" for non JSON output
(the reporter is using JSON output).
https://www.spinics.net/lists/fio/msg07204.html
Mark -L/--status-interval option with FIO_CLIENT_FLAG, so that the
command line string is sent to the server side.
The client can now control the server to periodically show status,
and each status is sent to the client via fio_server_text_output()
(FIO_NET_CMD_TEXT socket event) in log_info_buf().
Reported-by: Mark Beierl <Mark.Beierl@dell.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 3 Aug 2018 20:40:17 +0000 (14:40 -0600)]
iolog: move the chunked items-to-fetch logic into separate function
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 3 Aug 2018 20:35:57 +0000 (14:35 -0600)]
iolog: fix potential div-by-zero
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 3 Aug 2018 15:17:06 +0000 (09:17 -0600)]
Merge branch 'chunked-iolog-reading' of https://github.com/aclamk/fio
* 'chunked-iolog-reading' of https://github.com/aclamk/fio:
iolog: Added new option description to HOWTO
iolog: Added option read_iolog_chunked. Used to avoid reading large iologs at once. Allows iologs to be infinite, generated.
Jens Axboe [Fri, 3 Aug 2018 15:16:10 +0000 (09:16 -0600)]
Merge branch 'read_iolog-from-unix-socket' of https://github.com/aclamk/fio
* 'read_iolog-from-unix-socket' of https://github.com/aclamk/fio:
iolog: allow to read_iolog from unix socket
Jens Axboe [Fri, 3 Aug 2018 15:15:30 +0000 (09:15 -0600)]
Merge branch 'windows-s_issock' of https://github.com/aclamk/fio
* 'windows-s_issock' of https://github.com/aclamk/fio:
platforms/windows: Add S_ISSOCK macro.
Adam Kupczyk [Mon, 30 Jul 2018 10:02:55 +0000 (12:02 +0200)]
iolog: allow to read_iolog from unix socket
Checks is file provided via --read_iolog parameter is a unix socket.
In such case connect to it and fetch data from there.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Adam Kupczyk [Fri, 3 Aug 2018 10:48:28 +0000 (12:48 +0200)]
platforms/windows: Add S_ISSOCK macro.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Adam Kupczyk [Fri, 3 Aug 2018 10:41:59 +0000 (12:41 +0200)]
iolog: Added new option description to HOWTO
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Jens Axboe [Thu, 2 Aug 2018 22:20:24 +0000 (16:20 -0600)]
Merge branch 'szaydel/solaris-Wincompatible-pointer-types' of https://github.com/szaydel/fio
* 'szaydel/solaris-Wincompatible-pointer-types' of https://github.com/szaydel/fio:
Fix incompatible pointer types warning on Solaris with gcc 6.4
Sam Zaydel [Thu, 2 Aug 2018 22:01:27 +0000 (15:01 -0700)]
Fix incompatible pointer types warning on Solaris with gcc 6.4
Adam Kupczyk [Wed, 1 Aug 2018 11:45:27 +0000 (13:45 +0200)]
iolog: Added option read_iolog_chunked. Used to avoid reading large iologs at once.
Allows iologs to be infinite, generated.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Jens Axboe [Mon, 30 Jul 2018 14:24:20 +0000 (08:24 -0600)]
Merge branch 'fio-histo-fix' of https://github.com/parallel-fs-utils/fio
* 'fio-histo-fix' of https://github.com/parallel-fs-utils/fio:
clean up argparse usage
Ben England [Sun, 29 Jul 2018 17:14:59 +0000 (13:14 -0400)]
clean up argparse usage
Jens Axboe [Thu, 26 Jul 2018 17:47:28 +0000 (11:47 -0600)]
Merge branch 'sgunmap2' of https://github.com/vincentkfu/fio
* 'sgunmap2' of https://github.com/vincentkfu/fio:
docs: update HOWTO and manpage for sg trim support
testing: add test scripts for sg ioengine
engines/sg: move asserts and debug statements behind a debug flag
engines/sg: add cmdp and dxferp for trims to sg error string
engines/sg: support trim operations via the UNMAP command
stat: add IO submit and complete depths to JSON output
ioengines: have ioengines with commit do own io accounting for trims
Vincent Fu [Thu, 26 Jul 2018 16:27:21 +0000 (09:27 -0700)]
docs: update HOWTO and manpage for sg trim support
Also fix HOWTO sg_write_mode formatting
Vincent Fu [Fri, 20 Jul 2018 16:54:49 +0000 (09:54 -0700)]
testing: add test scripts for sg ioengine
t/sgunmap-perf.py carries out basic performance testing using
the sg ioengine
t/sgunmap-test.py checks that the IO depths reported for sg
trim workloads are sensible
Vincent Fu [Wed, 20 Jun 2018 19:47:16 +0000 (12:47 -0700)]
engines/sg: move asserts and debug statements behind a debug flag
The sg ioengine modifications include some debug statements and
integrity checks that could hurt performance. Put these behind
a debug flag.
Vincent Fu [Wed, 25 Jul 2018 20:00:26 +0000 (13:00 -0700)]
engines/sg: add cmdp and dxferp for trims to sg error string
Vincent Fu [Thu, 24 May 2018 21:00:36 +0000 (17:00 -0400)]
engines/sg: support trim operations via the UNMAP command
Multiple trim ranges can be submitted with a single UNMAP command
for character devices (e.g., /dev/sg1) using iodepth_batch. For
example, with iodepth_batch=4, each UNMAP command will include
four ranges. IOPS will still be reported as if four IO operations
were carried out.
Note that currently sg only supports a maximum of 16 commands
in flight per file. So be sure to maintain (iodepth/iodepth_batch) <= 16.
Vincent Fu [Tue, 19 Jun 2018 19:25:58 +0000 (12:25 -0700)]
stat: add IO submit and complete depths to JSON output
The IO submit and IO complete depths were missing from the JSON
output. This patch adds JSON objects for these distributions.
Vincent Fu [Wed, 25 Jul 2018 19:37:06 +0000 (12:37 -0700)]
ioengines: have ioengines with commit do own io accounting for trims
ioengines.c:td_io_queue has a special case where it calls
io_u_mark_submit() and io_u_mark_complete() when the io_u
is a trim operation or when the ioengine does not have a
commit function. This patch makes these two calls the
responsibility of the ioengine for trim io_u's when the
ioengine does have a commit function.
This only applies to libaio because it is the only ioengine
supporting trims with a commit function. All other ioengines that
support trim operations do not have a commit function. These
ioengines are: falloc, glusterfs_async, mmap, mtd, posixaio,
rados, rbd, splice, sync, psync, pvsync, pvsync2.
Jens Axboe [Wed, 25 Jul 2018 20:42:57 +0000 (14:42 -0600)]
Merge branch 'fio-histo-log-pctiles' of https://github.com/parallel-fs-utils/fio
* 'fio-histo-log-pctiles' of https://github.com/parallel-fs-utils/fio:
design document for tools/hist/fio-histo-log-pctiles.py
switch to argparse module for CLI parsing
use interpolation for more accurate percentile calculation
get latency percentiles over time from fio histo logs
Ben England [Wed, 25 Jul 2018 19:40:04 +0000 (15:40 -0400)]
design document for tools/hist/fio-histo-log-pctiles.py
Jens Axboe [Wed, 25 Jul 2018 14:41:38 +0000 (08:41 -0600)]
Merge branch 'fio-c++-engine' of https://github.com/tchaikov/fio
* 'fio-c++-engine' of https://github.com/tchaikov/fio:
replace typeof with __typeof__
Kefu Chai [Wed, 25 Jul 2018 14:02:09 +0000 (22:02 +0800)]
replace typeof with __typeof__
so we are able to extend fio with C++ .
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Jens Axboe [Tue, 24 Jul 2018 21:23:28 +0000 (15:23 -0600)]
init: unify 't' time period
Signed-off-by: Jens Axboe <axboe@kernel.dk>