Jens Axboe [Mon, 17 Sep 2018 03:43:56 +0000 (21:43 -0600)]
axmap: use 64-bit index for the handlers
Most important one here is axmap_handler_topdown(), in which the
first iteration could end up truncating the index calculation.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 17 Sep 2018 03:30:39 +0000 (21:30 -0600)]
axmap: use 64-bit type for number of bits
For huge devices, we can overflow on a 32-bit build of fio.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 16 Sep 2018 16:13:23 +0000 (10:13 -0600)]
Merge branch 'windows_static' of https://github.com/sitsofe/fio
* 'windows_static' of https://github.com/sitsofe/fio:
build: change where we set -static for Windows
Sitsofe Wheeler [Sat, 15 Sep 2018 05:58:33 +0000 (06:58 +0100)]
build: change where we set -static for Windows
The Makefile forced static Windows builds by adding -static to the
CFLAGS but this is long after the configure tests had run. This could
lead to problems (such as https://github.com/axboe/fio/issues/683 "Fio
fails to link libcurl on windows (undefined references to
__imp_curl_easy_cleanup)") where the availability of a dynamic version
of library causes a feature to be enabled but when it comes to final
linking time we find that there's no static version of the library
available and fall over.
Fix the above by asking for a static build via the "Forcing some known
good options on Windows" section of the configure file which occurs
prior to configure tests. This also means configure will correctly
display that we are doing a static build.
Fixes https://github.com/axboe/fio/issues/683
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Wed, 12 Sep 2018 20:33:04 +0000 (14:33 -0600)]
Merge branch 'histo-log-dup-timestamp' of https://github.com/parallel-fs-utils/fio
* 'histo-log-dup-timestamp' of https://github.com/parallel-fs-utils/fio:
filter out records with duplicate timestamps
Ben England [Wed, 12 Sep 2018 17:14:50 +0000 (13:14 -0400)]
filter out records with duplicate timestamps
fio should not be outputting records with duplicate timestamps
into the histogram log, but because it does do this,
this script was getting divide-by-zero errors.
This workaround will filter out the duplicates,
which allows the script to operate on existing fio logs
Jens Axboe [Wed, 12 Sep 2018 14:50:16 +0000 (08:50 -0600)]
windows: make win_to_posix_error() more resilient
If we pass in ERROR_SUCCESS, we should get a zero back.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Sep 2018 22:54:39 +0000 (16:54 -0600)]
windows: fix the most egregious posix.c style errors
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Sep 2018 22:29:25 +0000 (16:29 -0600)]
windows: handle ERROR_NOT_READY
The windows-to-posix error translation doesn't include
ERROR_NOT_READY, add that.
Also add a default case that catches if we don't know what
an error is, and return EIO. That seems much safer than just
returning the Windows error directly, since the caller expects
an errno.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Sep 2018 15:08:07 +0000 (09:08 -0600)]
Fio 3.10
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Sep 2018 14:27:41 +0000 (08:27 -0600)]
Add regression test for recent offload locking bug
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 11 Sep 2018 14:25:57 +0000 (08:25 -0600)]
Merge branch 'offload-flags-fix' of https://github.com/vincentkfu/fio
* 'offload-flags-fix' of https://github.com/vincentkfu/fio:
rate_submit: synchronize accesses to io_u_queue->nr
Vincent Fu [Tue, 11 Sep 2018 13:01:56 +0000 (09:01 -0400)]
rate_submit: synchronize accesses to io_u_queue->nr
Accesses to io_u_queue->nr are not properly synchronized in offload
submission mode. put_io_u locks td but the parent td flags reflecting
the need to lock are not propogated to child threads when the child
threads are intialized.
The main thread accesses io_u_queue->nr via io_u_qpop() as it prepares
io_u's for handing off to the worker threads. The worker threads access
io_u_queue->nr via io_u_qpush() as they complete io_u's. When these
accesses are not protected by locks, io_u_qpop() will return NULL when
it means to provide a valid io_u pointer. This occurs in offload
submission mode with iodepth > 1.
Fixes:
26b3a18 ("Make td_io_u_lock/unlock() explicit")
Jens Axboe [Mon, 10 Sep 2018 15:56:11 +0000 (09:56 -0600)]
Merge branch 'no-unittest-dep' of https://github.com/parallel-fs-utils/fio
* 'no-unittest-dep' of https://github.com/parallel-fs-utils/fio:
remove dependency on unittest2 module
Jens Axboe [Sun, 9 Sep 2018 23:42:10 +0000 (17:42 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
num2str(): Avoid an out-of-bounds array access
Micro-optimize num2str()
iolog: Ensure that sockaddr_un.sun_path is '\0'-terminated
Jens Axboe [Sat, 8 Sep 2018 14:06:57 +0000 (08:06 -0600)]
Kill fusion atomic write engine
Never really went anywhere, kill it with fire.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 8 Sep 2018 14:09:35 +0000 (08:09 -0600)]
Rename example job files (*.job -> *.fio)
Make it consistent.
Sugested-by: Rebecca Cran <rebecca@bluestop.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 7 Sep 2018 21:59:51 +0000 (15:59 -0600)]
client: cleanup output types
- Only print hostname= etc info for the NORMAL output, not for
minimal/terse.
- Only print disk util header for NORMAL output.
- Cleanup json finish check
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 7 Sep 2018 20:25:30 +0000 (14:25 -0600)]
client: switch to per-client buffer
Even with the local storage, for some operations we still end up
having to do split dumps. Include the buffer in the actual client,
and flush when the client is done.
Hopefully this fixes the remaining buffer output split for
client/server.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 7 Sep 2018 19:52:18 +0000 (13:52 -0600)]
client: use temp buffer for single output flush for json/disk util
Similar to previous fixes, ensure that we log everything locally
before flushing it out.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 7 Sep 2018 19:06:32 +0000 (13:06 -0600)]
Revert "client: respect terse output on client <--> backend relationship"
This breaks a standard use case, which is a server without arguments
and a client with the --minimal option.
This reverts commit
1d1b65dc17c39f43dbe9d66062ca61fc3e90ddfc.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 7 Sep 2018 18:57:52 +0000 (12:57 -0600)]
log: remember to free output buffer when done
We need to ensure we call buf_output_free() after using the
temp storage, otherwise we leaked the memory that we allocated
while filling the buffer.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 7 Sep 2018 18:19:43 +0000 (12:19 -0600)]
init: use __log_buf() if we know buf != NULL
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 7 Sep 2018 18:09:24 +0000 (12:09 -0600)]
log: use __log_buf() if we know buf != NULL
This apparently causes complaints on cygwin:
CC stat.o
In file included from os/os-windows.h:18:0,
from os/os.h:54,
from thread_options.h:5,
from fio.h:18,
from init.c:19:
init.c: In function ‘add_job’:
os/../log.h:24:12: error: the comparison will always evaluate as ‘true’ for the address of ‘out’ will never be NULL [-Werror=address]
if ((buf) != NULL) \
^
os/../log.h:24:12: note: in definition of macro ‘log_buf’
if ((buf) != NULL) \
which is utterly stupid, but hey...
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 7 Sep 2018 18:04:14 +0000 (12:04 -0600)]
client: use temp buffer for client text output
Don't split it into pieces, log it into a local buffer and then
flush it all at once.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 7 Sep 2018 18:02:39 +0000 (12:02 -0600)]
Collect startup output before logging it
We currently do the startup logging in separate chunks. This is
inefficient in terms of network trafic for client/server, and
it also causes output to be potentially mingled together:
<x1> Starting <x1> Starting 1 process1 process
Fix this by logging to a buffer, then transmitting that buffer
in one piece.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Ben England [Fri, 7 Sep 2018 16:43:21 +0000 (12:43 -0400)]
remove dependency on unittest2 module
The dependency of fio-histo-log-pctiles.py on unittest2 module was causing heartburn
for pbench community so I removed the dependency.
If unittest2 python package is installed and unittest2 module can be imported,
that's fine, we'll use it if requested,
but if it's not, that's fine too, you can still use the program.
Bart Van Assche [Fri, 7 Sep 2018 14:19:34 +0000 (07:19 -0700)]
num2str(): Avoid an out-of-bounds array access
This patch fixes Coverity ID 169307.
Fixes:
1ec3d69b0ed8 ("Implement a better num2str()")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Sun, 26 Aug 2018 15:54:19 +0000 (08:54 -0700)]
Micro-optimize num2str()
Instead of selecting an element from the thousand[] array several times,
perform the choice between 1000 and 1024 once. This patch does not change
any functionality.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Bart Van Assche [Sun, 26 Aug 2018 15:38:28 +0000 (08:38 -0700)]
iolog: Ensure that sockaddr_un.sun_path is '\0'-terminated
This patch fixes Coverity ID 183494.
Fixes:
2f8f4821ef61 ("iolog: allow to read_iolog from unix socket")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Thu, 6 Sep 2018 15:07:55 +0000 (09:07 -0600)]
Fio 3.9
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 6 Sep 2018 01:25:39 +0000 (19:25 -0600)]
Merge branch 'master' of https://github.com/damien-lemoal/fio
* 'master' of https://github.com/damien-lemoal/fio:
zbd: Use bytes unit
zbd: Fix zbd_zone_idx()
zbd: Remove inexistent functions declaration
zbd: Improve read randomness
Damien Le Moal [Wed, 29 Aug 2018 01:45:22 +0000 (10:45 +0900)]
zbd: Use bytes unit
Simplify zbd.c code by using byte unit for the zone information values
instead of 512B sector count. Doing so, many 9 bits shift operations
are removed and code readability improved.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Damien Le Moal [Wed, 29 Aug 2018 01:29:02 +0000 (10:29 +0900)]
zbd: Fix zbd_zone_idx()
For a zone size that is not a power of 2 number of sectors,
f->zbd_info->zone_size_log2 is set to -1. So use bit shift in
zbd_zone_idx() only if zone_size_log2 is not negative.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Damien Le Moal [Tue, 28 Aug 2018 05:09:13 +0000 (14:09 +0900)]
zbd: Remove inexistent functions declaration
The functions zbd_do_trim() and zbd_update_wp() are not implemented.
Remove there declarations from zbd.h.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Damien Le Moal [Tue, 28 Aug 2018 04:47:23 +0000 (13:47 +0900)]
zbd: Improve read randomness
With zonemode=zbd, for random read operations with read_beyond_wp=0,
the zbd code will always adjust an I/O offset so that the I/O hits a
non empty zone. However, the adjustment always sets the I/O offset to
the start of the zone, resulting in a high device read cache hit rate
if the device has few zones written.
Improve randomness of read I/Os by adjusting the I/O offset to a random
value within the range of written data of the chosen zone. Also ensure
that the modified I/O does not cross over the zone wp position by
adjusting its size. Doing so, the search for a new zone is now done only
if the target zone cannot accomodate an I/O of at least min_bs bytes.
The comments have also been changed to better explain the code.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Rebecca Cran [Wed, 5 Sep 2018 00:14:23 +0000 (18:14 -0600)]
Windows: update download URL and add missing examples
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 5 Sep 2018 19:33:43 +0000 (13:33 -0600)]
init: move log name generation into helper
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 5 Sep 2018 19:29:58 +0000 (13:29 -0600)]
Merge branch 'sz/log-names-need-help' of https://github.com/szaydel/fio
* 'sz/log-names-need-help' of https://github.com/szaydel/fio:
Log files names start with _ when write_XX_log= keys in config file have empty value(s)
Jens Axboe [Wed, 5 Sep 2018 15:29:00 +0000 (09:29 -0600)]
Document oddity with --status-interval and --output-format=json
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Damien Le Moal [Wed, 5 Sep 2018 08:02:37 +0000 (17:02 +0900)]
zbd: Fix test scripts
Fix handling of the path to the test-zbd-support script to avoid an
error when the tests are run directly when in fio/t/zbd using the
command "./run-against...".
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Sam Zaydel [Tue, 4 Sep 2018 21:28:29 +0000 (14:28 -0700)]
Log files names start with _ when write_XX_log= keys in config file have empty value(s)
Vincent Fu [Mon, 27 Aug 2018 20:40:11 +0000 (16:40 -0400)]
engines/sg: improve error handling
The Linux sg driver accepts only 16 SCSI commands in flight
at a time per file descriptor. fio does not exit gracefully
when it attempts to queue up 17 or more trim commands via
the sg ioengine. This patch improves error handling in the
sg ioengine commit function to achieve a graceful exit if
fio attempts to queue up too many trim commands. The key to
this is calling clear_io_u on each io_u in the 17th trim
command. With this patch fio no longer loops forever
waiting for the IOs in the (not succesffully submitted)
17th trim command to complete.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jeff Furlong [Fri, 31 Aug 2018 18:26:05 +0000 (18:26 +0000)]
client: fix nr_samples
In convert_iolog_gz() a 32 bit variable is created to store struct cmd_iolog_pdu's 64 bit member.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 30 Aug 2018 14:22:21 +0000 (08:22 -0600)]
Merge branch 'configure-help' of https://github.com/hahnjo/fio
* 'configure-help' of https://github.com/hahnjo/fio:
configure: Document more switches to disable features
Jonas Hahnfeld [Thu, 30 Aug 2018 12:25:45 +0000 (14:25 +0200)]
configure: Document more switches to disable features
This helps a lot when building on one machine and running on another
that doesn't have all libraries installed. The switches are all there,
list them in ./configure --help so that future users don't have to
search the shell script.
Jens Axboe [Sat, 25 Aug 2018 16:22:31 +0000 (10:22 -0600)]
Make td_io_u_lock/unlock() explicit
Hopefully this will make coverity a little happier, it currently
thinks they are unbalanced.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Tomohiro Kusumi [Fri, 24 Aug 2018 16:42:10 +0000 (09:42 -0700)]
client: suppress non JSON default outputs on --output-format=json/json+
Suppress "hostname=...", "Disk stats (read/write)" and thread status
if json/json+ is specified, as these are regular messages.
JSON parsers can't parse the output with these messages, and JSON
spec doesn't support comment either.
--
# ./fio ./fio.cfg --output-format=json > out
# python3 -c "import json; json.load(open('out'))"
# ./fio ./fio.cfg --client=localhost --output-format=json > out
# python3 -c "import json; json.load(open('out'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/lib/python3.7/json/__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 25 Aug 2018 00:22:13 +0000 (18:22 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Fix the typehelp[] array
Bart Van Assche [Fri, 24 Aug 2018 23:48:11 +0000 (16:48 -0700)]
Fix the typehelp[] array
This patch fixes Coverity ID 183335 (Out-of-bounds access of typehelp[] array).
Fixes:
5fff95436922 ("Add support for >= 4G block sizes")
Fixes:
c47537ee91cd ("parse: add support for soft deprecated options")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Jens Axboe [Fri, 24 Aug 2018 18:59:45 +0000 (12:59 -0600)]
Merge branch 'zbd'
* zbd:
Add scripts for testing the fio zoned block device support code
Add support for resetting zones periodically
Make it possible to limit the number of open zones
Collect and show zone reset statistics
Add support for zoned block devices
Add two assert statements in mark_random_map()
Pass offset and buffer length explicitly to mark_random_map()
Introduce the io_u.post_submit callback function pointer
Add the zonemode job option
Add the 'zbd' debug level
configure: Add <linux/blkzoned.h> test
Bart Van Assche [Fri, 24 Aug 2018 18:31:31 +0000 (11:31 -0700)]
Add scripts for testing the fio zoned block device support code
These scripts can be run e.g. as follows:
t/zbd/run-tests-against-zoned-nullb &&
t/zbd/run-tests-against-regular-nullb &&
t/zbd/test-zbd-support /dev/disk/by-id/scsi-SHGST_HSH721414AL52M0_VEG007HG &&
echo All tests passed
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:30 +0000 (11:31 -0700)]
Add support for resetting zones periodically
Filesystems that support zoned block devices typically perform garbage
collection if device usage exceeds a certain threshold. Add two command
line options that allow to simulate this behavior.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:29 +0000 (11:31 -0700)]
Make it possible to limit the number of open zones
When running a random I/O test across an entire drive many more zones
will be open than in a typical application workload. Hence add a command
line option that allows to limit the number of open zones.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:28 +0000 (11:31 -0700)]
Collect and show zone reset statistics
Show how many zone resets have been performed in the I/O statistics.
An example:
[ ... ]
write: IOPS=17.0k, BW=66.5MiB/s (69.8MB/s)(1024MiB/15387msec); 4 zone resets
[ ... ]
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:27 +0000 (11:31 -0700)]
Add support for zoned block devices
This patch adds support for zoned block devices as follows:
- After the file size has been determined, check whether the file name
refers to a block device. If so, check whether that block device is a
host-managed block device. If that is the case, read the zone
information using the BLKREPORTZONE ioctl. That ioctl is supported by
the Linux kernel since kernel version v4.10.
- After all command-line options have been processed and all job files
have been read, verify whether these refer to a zoned block device
and also whether the specified options are compatible with a zoned
block device. Complain if that is not the case.
- After each get_next_block() call, verify whether the block is
appropriate for a zoned block device. When writing data to a
sequential zone, adjust the write offset to the zone write pointer.
When reading from a sequential zone, avoid reading past the write
pointer.
- After I/O submission, update the variable that represents the write
pointer.
- When writing data and with data verification enabled, reset a zone
before writing any data into a zone. Otherwise reset a zone before
issuing a write if that zone is full.
- Translate trim into zone resets. Zoned block devices namely do not
have to support any of the SCSI commands that are used by the kernel
to implement the discard ioctl (UNMAP / WRITE SAME).
This work started from a patch from Masato Suzuki <masato.suzuki@wdc.com>.
Some of the ideas in this patch come from Phillip Chen
<phillip.a.chen@seagate.com>.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:26 +0000 (11:31 -0700)]
Add two assert statements in mark_random_map()
Add two assert statements that verify whether mark_random_map() is
used correctly.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:25 +0000 (11:31 -0700)]
Pass offset and buffer length explicitly to mark_random_map()
This patch does not change any functionality. The changes introduced by
this patch will be used by the zoned block device code.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:24 +0000 (11:31 -0700)]
Introduce the io_u.post_submit callback function pointer
This patch does not change any functionality. The code introduced by
this patch will be used by the zoned block device code.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:23 +0000 (11:31 -0700)]
Add the zonemode job option
Fio's zone support makes fio perform I/O inside a zone before it skips
to the next zone. That behavior is the opposite of the behavior needed
for zoned block devices, namely to consider all zones when performing
random I/O. Hence introduce a new job option that allows users to
choose between fio's traditional zone mode and the behavior needed for
zoned block devices. This patch makes fio behave identically with
--zonemode=none and --zonemode=zbd. A later patch will implement new
behavior for --zonemode=zbd.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:22 +0000 (11:31 -0700)]
Add the 'zbd' debug level
This debug level will be used in the next patch.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 24 Aug 2018 18:31:21 +0000 (11:31 -0700)]
configure: Add <linux/blkzoned.h> test
Add a test for <linux/blkzoned.h> because that header file will be used
by the zoned block device code.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 19:58:21 +0000 (13:58 -0600)]
t/axmap: print explicit overlap ranges tested
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 18:22:58 +0000 (12:22 -0600)]
t/axmap: add regression test for overlap case resulting in 0 settable bits
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 18:22:10 +0000 (12:22 -0600)]
axmap: return early of an overlap results in 0 settable bits
Reported-by: Bart Van Assche <bart.vanassche@wdc.com>
Fixes:
15a4f49 ("lib/axmap: Simplify axmap_set_fn()")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 17:05:34 +0000 (11:05 -0600)]
Merge branch '
20180823-terse-remote-fix' of https://github.com/mcgrof/fio
* '
20180823-terse-remote-fix' of https://github.com/mcgrof/fio:
init: add semantics for all types of backends running
client: respect terse output on client <--> backend relationship
Luis Chamberlain [Tue, 21 Aug 2018 18:15:39 +0000 (12:15 -0600)]
init: add semantics for all types of backends running
Commit
67a176fc7 ("Fix segfault with client/server and minimal output")
addressed a segfault but it wasn't very clear *why* the fix was needed.
The reason it was needed is that the disk util and its respective
semaphore are only initialized when setup_disk_util() is called, this
happens upon fio_backend() calls. That is, either we have a dedicated
backend or have initiated a backend for the localhost due to some local
work. And show_thread_status() is currently called from a complete
client setup -- handle_ts() calls show_thread_status(), and a client
does not collect any local disk data, it receives this from the backend.
As such, the semaphore won't be setup in a client setup and this is why
we segfault here.
We can enable show_thread_status() then only if any type of backend is
running, however there is are no semantics currently which enable us to
query for this. Add such semantics and replace the previous check with
a check for if fio_backend() was ever called. This will make it clearer.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Luis Chamberlain [Tue, 21 Aug 2018 13:24:36 +0000 (07:24 -0600)]
client: respect terse output on client <--> backend relationship
You end up with different results if you run this terse output
on a local system which also runs its own backend Vs running a
client to connect to a remote server which is running fio as a
backend only. The reason is the client ops handle printing of
threads / disk utils separately. The terse output created *by*
the backend is the right and expected output, so just use that,
and we can piggy back off of the fact that the server will send
its own output via FIO_NET_CMD_TEXT.
Another solution is to address getting the disk util data sent
to be cached locally, and then upon handle_ts() print that, but that
would require significant re-architecturing.
Terse output is supposed to be just that, terse. This implies
that it will not be clear from what backend data came from, but
for this the best strategy is to *extend* the terse version with
yet another field, the remote hostname/ip address.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Jens Axboe [Thu, 23 Aug 2018 16:29:12 +0000 (10:29 -0600)]
parse: fix bssplit option
Ensure that we copy option sets for FIO_OPT_STR_ULL like we do
for FIO_OPT_STR, in case only one data direction has been defined.
Fixes:
5fff95436922 ("Add support for >= 4G block sizes")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 23 Aug 2018 02:13:45 +0000 (20:13 -0600)]
Merge branch 'axmap' of https://github.com/bvanassche/fio
* 'axmap' of https://github.com/bvanassche/fio:
lib/axmap: Optimize __axmap_set()
lib/axmap: Simplify axmap_set_fn()
lib/axmap: Make axmap_new() more robust
lib/axmap: Inline ulog64()
lib/axmap: Add more documentation
Bart Van Assche [Wed, 22 Aug 2018 20:22:12 +0000 (13:22 -0700)]
lib/axmap: Optimize __axmap_set()
Since it is guaranteed that nr_bits <= BLOCKS_PER_UNIT and since
axmap_set_fn() needs to be called only once to set that number of
bits, remove the loop from __axmap_set().
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 22 Aug 2018 17:43:46 +0000 (10:43 -0700)]
lib/axmap: Simplify axmap_set_fn()
Instead of handling ffz(~overlap) == bit as a special case, handle
this case in the same way as ffz(~overlap) != bit. This patch does
not change any functionality but removes one if-instruction from the
hot path.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 22 Aug 2018 16:00:15 +0000 (09:00 -0700)]
lib/axmap: Make axmap_new() more robust
Return NULL instead of triggering a segmentation fault if calloc() fails.
Note: calling free(NULL) is safe so it is not necessary to check the
free() argument before calling free().
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 22 Aug 2018 20:35:19 +0000 (13:35 -0700)]
lib/axmap: Inline ulog64()
Since the function ulog64() only has one caller and since it can be
replaced by a single closed-form expression, inline that function.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 22 Aug 2018 15:58:00 +0000 (08:58 -0700)]
lib/axmap: Add more documentation
Make the axmap code easier to follow by documenting the semantics of
the structures and functions in the lib/axmap.c source file.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jeff Furlong [Mon, 20 Aug 2018 23:25:29 +0000 (23:25 +0000)]
io_u: residiual size should be unsigned long long
If we target a number of bytes of IO and keep subtracting a
xfer_buflen, we need to ensure the residual length is of the
same size.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 20 Aug 2018 14:34:13 +0000 (08:34 -0600)]
Update HOWTO for read_iolog change
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 20 Aug 2018 14:32:29 +0000 (08:32 -0600)]
Merge branch 'multiple-read_iolog' of https://github.com/aclamk/fio
* 'multiple-read_iolog' of https://github.com/aclamk/fio:
iolog: Now --read_iolog can contain multiple replay files, separated by ':'.
Adam Kupczyk [Fri, 17 Aug 2018 13:30:58 +0000 (15:30 +0200)]
iolog: Now --read_iolog can contain multiple replay files, separated by ':'.
Useful when --numjobs is more then 1.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Jens Axboe [Sat, 18 Aug 2018 01:39:07 +0000 (19:39 -0600)]
Move steady state unit test to t/
That's where we keep the other tests.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 17 Aug 2018 23:29:11 +0000 (17:29 -0600)]
Merge branch 'master' of https://github.com/kelleymh/fio
* 'master' of https://github.com/kelleymh/fio:
Remove unused code in lib/axmap.c
Tomohiro Kusumi [Fri, 17 Aug 2018 22:22:00 +0000 (15:22 -0700)]
http: fix compile-time warnings
Fix following warnings on RHEL6 and variants.
engines/http.c: In function 'fio_http_setup':
engines/http.c:611: warning: call to '_curl_easy_setopt_err_seek_cb' declared with attribute warning: curl_easy_setopt expects a curl_seek_callback argument for this option
engines/http.c: In function 'fio_http_queue':
engines/http.c:549: warning: call to '_curl_easy_setopt_err_curl_off_t' declared with attribute warning: curl_easy_setopt expects a curl_off_t argument for this option
For engines/http.c:611, pass &_http_seek instead of cast, since
lack of cast isn't fio's issue. See below comments for details.
"Re: [PATCH] http: fix compile-time warnings"
https://www.spinics.net/lists/fio/msg07246.html
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Michael Kelley [Fri, 17 Aug 2018 21:49:49 +0000 (14:49 -0700)]
Remove unused code in lib/axmap.c
With the new axmap_next_free() implementation, the "first_free" cache
is no longer used. Remove all references to it. Also axmap_clear()
is never referenced, so remove it and related code.
Jens Axboe [Fri, 17 Aug 2018 19:39:40 +0000 (13:39 -0600)]
Merge branch 'ddn-ime'
Pull in DDN IME engine
* ddn-ime:
Sync man page with fio for IME
engines/ime: various code and style cleanups
Add support for DDN's Infinite Memory Engine
Jens Axboe [Fri, 17 Aug 2018 19:39:12 +0000 (13:39 -0600)]
Sync man page with fio for IME
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 17 Aug 2018 19:38:59 +0000 (13:38 -0600)]
engines/ime: various code and style cleanups
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 17 Aug 2018 19:34:21 +0000 (13:34 -0600)]
Merge branch 'ime-support' of https://github.com/DDNStorage/fio-public into ddn-ime
* 'ime-support' of https://github.com/DDNStorage/fio-public:
Add support for DDN's Infinite Memory Engine
Jens Axboe [Fri, 17 Aug 2018 19:27:25 +0000 (13:27 -0600)]
Merge branch 'master' of https://github.com/kelleymh/fio
* 'master' of https://github.com/kelleymh/fio:
Add tests specifically for axmap_next_free()
Reimplement axmap_next_free() to prevent distribution skew
Gaëtan Bossu [Wed, 1 Aug 2018 14:27:37 +0000 (16:27 +0200)]
Add support for DDN's Infinite Memory Engine
Created 3 engines in engines/ime.c:
ime_psync, ime_psyncv, and ime_aio
Signed-off-by: Gaëtan Bossu <gbossu@ddn.com>
Michael Kelley [Fri, 17 Aug 2018 19:17:00 +0000 (12:17 -0700)]
Add tests specifically for axmap_next_free()
Add tests for axmap_next_free() to ensure the new search algorithm
works correctly. Since the new algorithm always looks forward for
the next free bit, it's possible to test for specific results given
a known condition of the map. Test maps that are mostly empty and
mostly full. Test wrap-around. Test maps of particular sizes that
have unused bits in middle levels as well as in level 0.
Michael Kelley [Fri, 17 Aug 2018 19:02:58 +0000 (12:02 -0700)]
Reimplement axmap_next_free() to prevent distribution skew
New algorithm starts the search at the specified bit, and only
looks forward (higher index) for a free bit, wrapping around the
end of the map if necessary. This avoids a distribution skew in
the current algorithm where the low order 5 or 6 bits of the
replacement address tend toward zero.
The first_free cached value is not updated or used. But performance
of the new algorithm is marginally faster than the old algorithm
when measured on a 200 million entry map that is half full,
averaged across searches starting with each of the 200 million
entries.
A separate commit adds new tests to t/axmap.c to confirm the
correct behavior in a variety of situations and edge cases.
David Disseldorp [Thu, 16 Aug 2018 16:56:02 +0000 (18:56 +0200)]
configure: avoid pkg-config usage for http engine
Signed-off-by: David Disseldorp <ddiss@suse.de>
Jens Axboe [Thu, 16 Aug 2018 16:38:41 +0000 (10:38 -0600)]
Merge branch 'wip-http-swift' of https://github.com/l-mb/fio
* 'wip-http-swift' of https://github.com/l-mb/fio:
engines/http: Add support for Swift storage backend
Lars Marowsky-Bree [Thu, 16 Aug 2018 11:06:47 +0000 (13:06 +0200)]
engines/http: Add support for Swift storage backend
This also converts two configuration variables to use "posval" instead
of string parsing or integers.
Jens Axboe [Thu, 16 Aug 2018 15:30:21 +0000 (09:30 -0600)]
engines/http: fix use of uninitialized variable
Fixes:
c2f6a13ddc64 ("engines/http: Add support for WebDAV and S3")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 16 Aug 2018 15:20:43 +0000 (09:20 -0600)]
Merge branch 'http_older_openssl' of https://github.com/ddiss/fio
* 'http_older_openssl' of https://github.com/ddiss/fio:
engines/http: support openssl < 1.1.0
configure: use pkg-config to detect libcurl & openssl
David Disseldorp [Thu, 16 Aug 2018 14:53:49 +0000 (16:53 +0200)]
engines/http: support openssl < 1.1.0
openssl versions prior to 1.1.0 do not use an opaque pointer for
HMAC_CTX.
Signed-off-by: David Disseldorp <ddiss@suse.de>
David Disseldorp [Thu, 16 Aug 2018 14:37:50 +0000 (16:37 +0200)]
configure: use pkg-config to detect libcurl & openssl
Signed-off-by: David Disseldorp <ddiss@suse.de>
Manish Dusane [Wed, 15 Aug 2018 18:06:14 +0000 (12:06 -0600)]
Fix compilation on centos 7
"log.h:12: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘log_err’"
Fixes: https://github.com/axboe/fio/issues/651
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 14 Aug 2018 16:09:47 +0000 (10:09 -0600)]
Merge branch 'wip-http-engine' of https://github.com/l-mb/fio
* 'wip-http-engine' of https://github.com/l-mb/fio:
engines/http: Add support for WebDAV and S3