fio.git
23 months agoeta: fix run_str_condensed overflow with maximum jobs
Sitsofe Wheeler [Sat, 23 Dec 2017 08:01:01 +0000 (08:01 +0000)]
eta: fix run_str_condensed overflow with maximum jobs

When the maximum number of jobs all have a different state to their
previous job, run_str didn't have space for the terminating null. Fix
this by making run_str one character larger.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoeta: adjust truncation case
Sitsofe Wheeler [Fri, 22 Dec 2017 11:10:35 +0000 (11:10 +0000)]
eta: adjust truncation case

c076de85fa4901683f6ce23ecdc071c17801ffad ("eta: fix buffer overflow in
ETA output") made adjustments to cope with snprintf truncation. However
since we're replacing the null added by snprintf, left only needs to be
one smaller than the output buffer. Make this change and add a comment
about what we're doing.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoMerge branch 'eta_overflow' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 21 Dec 2017 15:22:39 +0000 (08:22 -0700)]
Merge branch 'eta_overflow' of https://github.com/sitsofe/fio

23 months agoeta: fix buffer overflow in ETA output
Sitsofe Wheeler [Thu, 21 Dec 2017 12:23:36 +0000 (12:23 +0000)]
eta: fix buffer overflow in ETA output

When 900 or more jobs all have different states it is possible for the
space required to display the ETA line to be larger than the ETA output
buffer. fio initially truncates what it puts in the output buffer but
incorrectly advances where it thinks the end of the buffer is - when
truncating snprintf returns the number of characters that _would_ have
been converted if there had been enough space...

This patch fixes the post truncation calculation and adjusts the "left"
variable so there is room for the carriage return and a null terminator.

The following script reproduces the problem when fio has been compiled
with -fstack-protector or -fsanitize=address:

rw[0]='read'; rw[1]='write'; \
for i in {1..1000}; do \
 echo -e "[job$i]\nrw=${rw[$((i % 2))]}\n" \
         "ramp_time=$(((1000 - i) / 350 * 5))\n" \
         "runtime=$((60 - (i % 3) * 2))"; \
done | \
./fio --eta-newline=1s --group_reporting --ioengine=null --size=1g \
 --time_based --bs=512 --thread --rate_iops=123456 -

Fixes: https://github.com/axboe/fio/issues/500 ("Large number of threads
result in Seg faults")
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
23 months agoFio 3.3 fio-3.3
Jens Axboe [Tue, 19 Dec 2017 20:16:36 +0000 (13:16 -0700)]
Fio 3.3

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agobackend: tweaks to missed rate thinktime
Jens Axboe [Tue, 19 Dec 2017 17:17:41 +0000 (10:17 -0700)]
backend: tweaks to missed rate thinktime

shimrot says:

"May I suggest a tweak to the backend.c::handle_thinktime calculation of
"over"? That is, adjust based on the rate per byte similar to what
you've done. That is, if thinktime is greater than one op worth of
bytes, then subtract one op worth of bytes. But, if think time is
greater than one ops worth, instead add the missing the number of bytes
a full op would have exceeded the think time."

See: https://github.com/axboe/fio/issues/497#issuecomment-352816955

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoioengines: clear out ->td_ops_dlhandle if we close it
Jens Axboe [Fri, 15 Dec 2017 20:35:56 +0000 (13:35 -0700)]
ioengines: clear out ->td_ops_dlhandle if we close it

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoinit: fix missing dlhandle reference put
Jens Axboe [Fri, 15 Dec 2017 20:34:22 +0000 (13:34 -0700)]
init: fix missing dlhandle reference put

If we keep using the same engine, put the dlhandle reference that
we now hold twice.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoparse: don't check for < 0 on an unsigned type
Jens Axboe [Fri, 15 Dec 2017 16:13:28 +0000 (09:13 -0700)]
parse: don't check for < 0 on an unsigned type

We do strict errors on some builds, this breaks them. Remove the
<= 0 check, just check for a value that's too large.

Fixes: c26438ad ("parse: dump option type when using --debug=parse")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoioengines: improve "is this the same IO engine" check
Jens Axboe [Fri, 15 Dec 2017 16:08:26 +0000 (09:08 -0700)]
ioengines: improve "is this the same IO engine" check

We can't just compare the name, that assumes that name and filename
match for an IO engine. While that's generally the case for the
engines that fio ships with, it's not a requirement, and it's
definitely not true for external engines.

Fixup the check by re-loading the engine and checking the OPS
instead. That should be bullet proof.

Fixes: 800334d ("Correctly detect whether ioengine_load can exit early")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoparse: dump option type when using --debug=parse
Jens Axboe [Fri, 15 Dec 2017 15:41:03 +0000 (08:41 -0700)]
parse: dump option type when using --debug=parse

Currently we do things like:

parse    8731  __handle_option=dummy, type=10, ptr=1

for the debug parsing, and then you have to look up what that
option type is. Add names to them so we get the below instead:

parse    9170  __handle_option=dummy, type=OPT_STR_SET, ptr=1

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoAllow configurable ETA intervals
Jens Axboe [Thu, 14 Dec 2017 18:51:41 +0000 (11:51 -0700)]
Allow configurable ETA intervals

By default, fio prints ETA output every second. For some client/server
setups, it's desirable to allow a much longer interval, to avoid
spending too much time getting and printing ETA time.

Takes a normal time input, allowing usec/msec/sec etc postfixes.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoclient: respect --eta=never for networked connections
Jens Axboe [Thu, 14 Dec 2017 17:58:03 +0000 (10:58 -0700)]
client: respect --eta=never for networked connections

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoserver: convert more memset to on-stack initialization
Jens Axboe [Thu, 14 Dec 2017 16:47:35 +0000 (09:47 -0700)]
server: convert more memset to on-stack initialization

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoserver: cleanup iolog pdu prep
Jens Axboe [Thu, 14 Dec 2017 16:25:10 +0000 (09:25 -0700)]
server: cleanup iolog pdu prep

We don't have to memset() the whole thing, we just need to ensure
that the members we don't assign are cleared. Remove the memset
and initialize the pdu on the stack.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoFix Windows local time
Jeff Furlong [Thu, 14 Dec 2017 01:19:09 +0000 (01:19 +0000)]
Fix Windows local time

The fio summary output shows when the job completed, but references
system time on Windows.  On Linux, local time (such as with time zone
specifics) is referenced.  This patch converts Windows system time to
local time.

Before:
fio-3.2-72-ge80d-dirty
Starting 1 thread
Jobs: 1 (f=0): [f(1)][100.0%][r=69.6MiB/s,w=0KiB/s][r=17.8k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1360: Wed Dec 13 23:21:16 2017 <--System Time

After:
fio-3.2-72-ge80d-dirty
Starting 1 thread
Jobs: 1 (f=0): [f(1)][100.0%][r=73.2MiB/s,w=0KiB/s][r=18.7k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=8088: Wed Dec 13 15:22:07 2017 <--Local Time

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months ago.gitignore: ignore tags files and additional output binaries
Robert Elliott [Mon, 4 Dec 2017 20:53:16 +0000 (14:53 -0600)]
.gitignore: ignore tags files and additional output binaries

Ignore tags files used by emacs and ctags/vim.

Ignore the additional output binaries alongside fio, such as gfio
and all the programs in t/.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoserver: initialize first iolog header properly
Jens Axboe [Wed, 13 Dec 2017 15:44:34 +0000 (08:44 -0700)]
server: initialize first iolog header properly

We're sending potential stack garbage over the wire, init the
whole pdu to zeroes before filling it in. This shuts up valgrind
about using uninitialized memory.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agofio: kill td->nr_normal_files
Jens Axboe [Fri, 8 Dec 2017 19:50:28 +0000 (12:50 -0700)]
fio: kill td->nr_normal_files

It's unused.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agoRemove old emails
Jens Axboe [Fri, 8 Dec 2017 19:38:27 +0000 (12:38 -0700)]
Remove old emails

Just use my private email, it's the one I prefer using anyway.
Remove the old fusionio/oracle emails, they just cause
confusion and people tend to just write to all of them at once
when reporting an issue.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
23 months agorate: fix bad math
Jens Axboe [Fri, 8 Dec 2017 18:59:38 +0000 (11:59 -0700)]
rate: fix bad math

To figure out how much data we missed when doing a thinktime sleep,
we're currently dividing by the time slept. This is wrong, it should
be multiplied by the time slept and divided by 1000000 to go from
usec to a second base.

Additionally, don't ever subtract more than a block of data, and
adjust down depending on sleep.

Fixes: 1aa39b0c ("rate: ensure IO issue restarts right after sleep")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agorate: ensure IO issue restarts right after sleep
Jens Axboe [Thu, 7 Dec 2017 16:06:04 +0000 (09:06 -0700)]
rate: ensure IO issue restarts right after sleep

If we have rate_ignore_thinktime set, we account the number of bytes
we would have done in the period of the sleep. However, we need
to subtract one block, otherwise we end up doing a rated sleep
right after the thinktime sleep.

Fixes: 1a9bf814 ("Add option to ignore thinktime for rated IO")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: rate cleanup and spelling error
Jens Axboe [Wed, 6 Dec 2017 19:30:20 +0000 (12:30 -0700)]
io_u: rate cleanup and spelling error

Fixes: 50a8ce86 ("Implement new Rate Control")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoAdd option to ignore thinktime for rated IO
Jens Axboe [Wed, 6 Dec 2017 19:27:07 +0000 (12:27 -0700)]
Add option to ignore thinktime for rated IO

By default, fio will ignore thinktime when calculating the next
time to issue and IO, if rated IO is specified. This leads to
fio entering a catch-up type of mode after doing the specified
sleep. For some workloads, that may not be useful. If someone
asks for a specific amount of IOPS and sets a thinktime, they
may want to exclude the sleep time.

Fixes: https://github.com/axboe/fio/issues/497
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'wip-cleanup' of https://github.com/ZVampirEM77/fio
Jens Axboe [Sun, 3 Dec 2017 17:11:53 +0000 (10:11 -0700)]
Merge branch 'wip-cleanup' of https://github.com/ZVampirEM77/fio

2 years agoconfigure: fix typos
Enming Zhang [Sun, 3 Dec 2017 09:50:10 +0000 (17:50 +0800)]
configure: fix typos

check for rbd_invaidate_cache() --> check for rbd_invalidate_cache()

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>
2 years agosteadystate: style cleanup
Jens Axboe [Sat, 2 Dec 2017 23:29:44 +0000 (16:29 -0700)]
steadystate: style cleanup

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agosteadystate: add free helper
Jens Axboe [Sat, 2 Dec 2017 16:18:46 +0000 (09:18 -0700)]
steadystate: add free helper

Also fix a style issue.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agothread_options: drop fadvise_stream from thread_options
Vincent Fu [Fri, 1 Dec 2017 21:52:08 +0000 (16:52 -0500)]
thread_options: drop fadvise_stream from thread_options

ae8e559e got rid of the fadvise_stream option but did
not remove the corresponding member from thread_options

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agomemcpy: add hybrid
Jens Axboe [Fri, 1 Dec 2017 21:49:42 +0000 (14:49 -0700)]
memcpy: add hybrid

Use builtin memcpy for < 64 bytes, use simple variant for larger
ones. libc doesn't seem to like to use SSE, the hand rolled simple
variant uses that for larger copies.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agomemcpy: free buffer in case of failure
Jens Axboe [Fri, 1 Dec 2017 20:56:09 +0000 (13:56 -0700)]
memcpy: free buffer in case of failure

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agomemcpy: use malloc
Jens Axboe [Fri, 1 Dec 2017 18:30:49 +0000 (11:30 -0700)]
memcpy: use malloc

We don't have posix_memalign() everywhere, let's just use malloc.

Also free memory when done.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofio_time: should include time.h
Jens Axboe [Fri, 1 Dec 2017 18:21:58 +0000 (11:21 -0700)]
fio_time: should include time.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoAdd basic memcpy test
Jens Axboe [Fri, 1 Dec 2017 18:13:23 +0000 (11:13 -0700)]
Add basic memcpy test

Research is ongoing in how to improve (and make deterministic) the
memcpy speed, since it's especially applicable to the persistent
memory engines.

Not documented yet.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: don't account io issue blocks for verify backlog
Jens Axboe [Fri, 1 Dec 2017 04:48:12 +0000 (21:48 -0700)]
io_u: don't account io issue blocks for verify backlog

We don't account the bytes, we should not account the blocks
either.

Fixes: ae2fafc8 ("verify: verify bytes should not add to this_io_bytes")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooptions: warn if we exceed the supported number of split entries
Jens Axboe [Fri, 1 Dec 2017 04:29:32 +0000 (21:29 -0700)]
options: warn if we exceed the supported number of split entries

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoDocumentation: add note about how many bssplit and zones fio supports
Jens Axboe [Fri, 1 Dec 2017 04:28:15 +0000 (21:28 -0700)]
Documentation: add note about how many bssplit and zones fio supports

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoioengine: don't account verify bytes
Jens Axboe [Fri, 1 Dec 2017 03:24:54 +0000 (20:24 -0700)]
ioengine: don't account verify bytes

If we do that, we can terminate early for a readwrite+verify
backlog workload.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoclient: fix use-after-free for client timeout
Jens Axboe [Fri, 1 Dec 2017 02:50:34 +0000 (19:50 -0700)]
client: fix use-after-free for client timeout

We free the reply, but if it times out, we dereference it afterwards.
Store the opcode before calling into handle_cmd_timeout().

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoengines/dev-dax: fix leak of 'sfile' in error case
Jens Axboe [Fri, 1 Dec 2017 02:48:25 +0000 (19:48 -0700)]
engines/dev-dax: fix leak of 'sfile' in error case

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agot/verify-state: fix leak in error case
Jens Axboe [Fri, 1 Dec 2017 02:47:42 +0000 (19:47 -0700)]
t/verify-state: fix leak in error case

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agobackend: make it clear that we passed 'fd' to the new thread
Jens Axboe [Fri, 1 Dec 2017 02:45:05 +0000 (19:45 -0700)]
backend: make it clear that we passed 'fd' to the new thread

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agogettime-thread: fix failure to check setaffinity return value
Jens Axboe [Fri, 1 Dec 2017 02:41:22 +0000 (19:41 -0700)]
gettime-thread: fix failure to check setaffinity return value

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoDocumentation cleanup
Jens Axboe [Fri, 1 Dec 2017 02:17:38 +0000 (19:17 -0700)]
Documentation cleanup

- Space at end-of-line for the significant_figures change
- > 80 char line section in HOWTO

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooptions: don't overrun bssplit array
Jens Axboe [Thu, 30 Nov 2017 23:49:23 +0000 (16:49 -0700)]
options: don't overrun bssplit array

Since we now have a different number of splits we support for
zones and block size splits, make sure we pass in the right
max.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoBump support of zones to 256 max
Jens Axboe [Thu, 30 Nov 2017 23:47:27 +0000 (16:47 -0700)]
Bump support of zones to 256 max

This moves it from 64 to 256.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agosteadystate: make flags conform to usual fio standard
Jens Axboe [Thu, 30 Nov 2017 23:43:18 +0000 (16:43 -0700)]
steadystate: make flags conform to usual fio standard

__VAL are usually bit shift values, with VAL being the proper
mask. For some reason, the steadystate code uses both as
masks, fix that up.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoserver: process connection list before executing trigger
Jens Axboe [Thu, 30 Nov 2017 20:28:48 +0000 (13:28 -0700)]
server: process connection list before executing trigger

This shrinks the window a bit for allowing the backend to notice
that jobs have gone away, and notify the client.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoclient: ignore a client timeout, if the last thing we saw as a trigger
Jens Axboe [Thu, 30 Nov 2017 20:27:40 +0000 (13:27 -0700)]
client: ignore a client timeout, if the last thing we saw as a trigger

If triggers are used and are fatal to the fio backend (killing power,
etc), then we don't have time to process the stop/quit commands. We
expect the client to go away, but the backend can't let us know.

If the last thing we saw was a trigger command, don't exit in error
if the client times out and goes away.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofio: add check rate flag
Jens Axboe [Thu, 30 Nov 2017 17:59:25 +0000 (10:59 -0700)]
fio: add check rate flag

One of the things we do a lot for each IO, is check if we should be
checking the rate. Add a specific flag for that case, so we can answer
that question very cheaply.

This is good for more than a 5% speedup for a null engine test case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: speed up small_content_scramble()
Jens Axboe [Thu, 30 Nov 2017 16:18:37 +0000 (09:18 -0700)]
io_u: speed up small_content_scramble()

This is a hot path for write workloads, since we don't want to send the
same buffers to the device again and again. The idea is to defeat basic
dedupe/compression, but slightly modifying the buffer for each write.
small_content_scramble() does this by filling in the io_u offset into a
random spot in each 512b chunk of an io buffer, and filling in the start
time (sec,nsec) at the end of each 512b chunk.

With this change, we still do those two things, but we generate a random
cacheline within each 512b chunk, and fill the offset at the beginning
of the cacheline, and the time at the end of it.  This means that
instead of potentially dirtying 2 cachelines for each 512b chunk in an
IO buffer, we dirty just 1.

The results should still be random enough that small_content_scramble()
fullfils the promise to defeat basic dedupe and compression, but it is
lighter to run.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: cleanup check_get_trim()
Jens Axboe [Thu, 30 Nov 2017 14:03:33 +0000 (07:03 -0700)]
io_u: cleanup check_get_trim()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: tweak small content buffer scramble
Jens Axboe [Thu, 30 Nov 2017 13:50:36 +0000 (06:50 -0700)]
io_u: tweak small content buffer scramble

We currently generate a 'random' offset in a 512b chunk to fill in
the offset. Since we don't want the later time scramble to
overwrite it, we check and adjust for that. Instead just ensure that
we generate a random offset in the first half of the 512b chunk,
then we know we never overlap.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: use nsec value for buffer scramble
Jens Axboe [Thu, 30 Nov 2017 13:26:21 +0000 (06:26 -0700)]
io_u: use nsec value for buffer scramble

Just use the nanosecond value directly, it's pointless to shift
it down and lose 10 bits with of scrambling data.

Fixes: d5d3795c ("io_u: don't do expensive int divide for buffer scramble")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooptions: correct parser type for max_latency
Jens Axboe [Thu, 30 Nov 2017 05:11:09 +0000 (22:11 -0700)]
options: correct parser type for max_latency

It's a 64-bit type now, use the proper time conversion type.

Fixes: c3a32714 ("Change latency targets to be in nsec values internally")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoverify: convert hdr time to sec+nsec
Jens Axboe [Thu, 30 Nov 2017 05:00:35 +0000 (22:00 -0700)]
verify: convert hdr time to sec+nsec

Use the internal values instead of converting.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoverify: kill unneeded forward declaration
Jens Axboe [Thu, 30 Nov 2017 04:58:28 +0000 (21:58 -0700)]
verify: kill unneeded forward declaration

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoChange latency targets to be in nsec values internally
Jens Axboe [Thu, 30 Nov 2017 04:52:46 +0000 (21:52 -0700)]
Change latency targets to be in nsec values internally

Since all of our timekeeping is in nsec now, it's easier to convert
these at init time and not have to do it at runtime.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooptions: make max_latency a 64-bit variable
Jens Axboe [Thu, 30 Nov 2017 04:47:43 +0000 (21:47 -0700)]
options: make max_latency a 64-bit variable

Also removes one pad in the thread options. This should bump
the protocol version, but we just did that, so let it slide.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooptions: make it clear that max_latency is in usecs
Jens Axboe [Thu, 30 Nov 2017 04:30:21 +0000 (21:30 -0700)]
options: make it clear that max_latency is in usecs

The other latency options include this in their long name.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: do nsec -> usec converison in one spot in account_io_completion()
Jens Axboe [Thu, 30 Nov 2017 04:25:32 +0000 (21:25 -0700)]
io_u: do nsec -> usec converison in one spot in account_io_completion()

Should not matter for runtime, but it's cleaner.

What we should really do is convert the internal values to nsec, so
we don't have to do this conversion.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: don't do expensive int divide for buffer scramble
Jens Axboe [Thu, 30 Nov 2017 04:01:36 +0000 (21:01 -0700)]
io_u: don't do expensive int divide for buffer scramble

We don't need the conversion from nsec to usec to be exact, so
just shift by 10 instead.

Fixes: 8b6a404cd ("nanosecond: initial commit changing timeval to timespec")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoUnify max split zone support
Jens Axboe [Thu, 30 Nov 2017 03:51:09 +0000 (20:51 -0700)]
Unify max split zone support

We have two different defines, use the one from the networking
protocol. Add a compile time check that bssplit and zones
have the same max, so we don't inadvertently add an overflow
condition if we change one of them.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoio_u: cleanup and simplify __get_next_rand_offset_zoned_abs()
Jens Axboe [Thu, 30 Nov 2017 03:45:33 +0000 (20:45 -0700)]
io_u: cleanup and simplify __get_next_rand_offset_zoned_abs()

We can drop various variables, it's easier to read this way too.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoexamples/rand-zones.fio: add zoned_abs example
Jens Axboe [Thu, 30 Nov 2017 02:29:25 +0000 (19:29 -0700)]
examples/rand-zones.fio: add zoned_abs example

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoAdd support for absolute random zones
Jens Axboe [Thu, 30 Nov 2017 02:25:59 +0000 (19:25 -0700)]
Add support for absolute random zones

We currently support random_distribution=zoned, which allows the user
to specify a percentage of access to a zoned define as a percentage
of the file/device size. This commit adds support for zoned_abs,
which works exactly like zoned, except you give the zone size in
an absolute value.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agooptions: don't quicksort zoned distribution series
Jens Axboe [Wed, 29 Nov 2017 19:37:30 +0000 (12:37 -0700)]
options: don't quicksort zoned distribution series

If we do that, we mess up the distribution.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoRevert "Avoid irrelevant "offset extend ends" error message for chrdev"
Tomohiro Kusumi [Wed, 29 Nov 2017 09:29:22 +0000 (11:29 +0200)]
Revert "Avoid irrelevant "offset extend ends" error message for chrdev"

This reverts commit 0f34169ac64f009b193bed1c8fb6692844bf99c2.

> Commit 0f34169a broke it, moving a check from a non-character
> device path up a level to where it now triggers on a character
> device (for which the size is left set to -1ULL).
>
> This gets it working again:
>         git revert 0f34169a
>         make
>
> Tomohiro, could you help prepare a better fix?

Sorry, I'll just revert it at the moment.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoinit: did_arg cleanup
Jens Axboe [Tue, 28 Nov 2017 23:00:22 +0000 (16:00 -0700)]
init: did_arg cleanup

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoinit: remove dead code
Jens Axboe [Tue, 28 Nov 2017 21:02:11 +0000 (14:02 -0700)]
init: remove dead code

We know did_arg == 0 here, so kill the other part.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoinit: check and fail status-interval settings that are too small
Jens Axboe [Tue, 28 Nov 2017 17:11:53 +0000 (10:11 -0700)]
init: check and fail status-interval settings that are too small

Smallest possible value is 1ms, so fail any setting that yields
a value less than that.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agodocs: Add documention for RDMA ioengine options.
Stephen Bates [Thu, 23 Nov 2017 19:21:56 +0000 (12:21 -0700)]
docs: Add documention for RDMA ioengine options.

Add documentation to HOWTO and the man page for the RDMA ioengine
options as these were missing.

Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'cleanup' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 23 Nov 2017 02:58:21 +0000 (19:58 -0700)]
Merge branch 'cleanup' of https://github.com/sitsofe/fio

2 years agodoc: reword buffer_compress_percentage, buffer_compress_chunk, dedupe_percentage
Sitsofe Wheeler [Wed, 22 Nov 2017 22:58:10 +0000 (22:58 +0000)]
doc: reword buffer_compress_percentage, buffer_compress_chunk, dedupe_percentage

Make the documentation for these options better reflect how buffer
compression and dedupe tuning can be used since commit
1de80624466405bccdbc4607d71cd249320da3f1 ("Default buffer_compress_chunk
to 512").

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoHOWTO: fix up broken formatting in logging options
Sitsofe Wheeler [Wed, 22 Nov 2017 22:38:18 +0000 (22:38 +0000)]
HOWTO: fix up broken formatting in logging options

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoadd significant_figures parameter
Jeff Furlong [Wed, 22 Nov 2017 18:38:33 +0000 (11:38 -0700)]
add significant_figures parameter

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agordma: Add bind option
Stephen Bates [Wed, 22 Nov 2017 18:24:56 +0000 (11:24 -0700)]
rdma: Add bind option

In certain configurations it can be useful to bind a rdma_cm to a
particular network interface. For example in multi-path or loopback.

Add a bindname option that the local rdma_cm will try and bind too.

The bind code is based off that used in rping [1].

[1] https://github.com/linux-rdma/rdma-core/blob/ \
    master/librdmacm/examples/rping.c

Signed-off-by: Stephen Bates <sbates@raithlin.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'libpmem'
Jens Axboe [Fri, 17 Nov 2017 16:21:19 +0000 (09:21 -0700)]
Merge branch 'libpmem'

Merge in libpmem engine, with fixes.

2 years agoexamples/libpmem.fio: clean up example
Jens Axboe [Fri, 17 Nov 2017 16:20:52 +0000 (09:20 -0700)]
examples/libpmem.fio: clean up example

Spelling errors, typos, etc.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agolibpmem: move mmap alignment to init time
Jens Axboe [Fri, 17 Nov 2017 16:19:26 +0000 (09:19 -0700)]
libpmem: move mmap alignment to init time

Also use the generally available page_size, don't need to roll our
own in there.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agolibpmem: code cleanups
Jens Axboe [Fri, 17 Nov 2017 16:16:44 +0000 (09:16 -0700)]
libpmem: code cleanups

No functional changes in this patch.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'add-libpmem-engine' of https://github.com/tishizaki/fio into libpmem
Jens Axboe [Fri, 17 Nov 2017 16:03:23 +0000 (09:03 -0700)]
Merge branch 'add-libpmem-engine' of https://github.com/tishizaki/fio into libpmem

2 years agofio: add libpmem engine
Teruaki Ishizaki [Fri, 17 Nov 2017 02:54:26 +0000 (11:54 +0900)]
fio: add libpmem engine

Adding an ioengine that access with the libpmem as memory
through a memory mmaped file on DAX filesystem.

It's very similar to the mmap engine and the dev-dax engine.

Signed-off-by: Teruaki Ishizaki <ishizaki.teruaki@lab.ntt.co.jp>
Signed-off-by: Takashi Menjo <menjo.takashi@lab.ntt.co.jp>
2 years agoioengines: remove pointless list initializations
Jens Axboe [Fri, 17 Nov 2017 03:03:15 +0000 (20:03 -0700)]
ioengines: remove pointless list initializations

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoos: make fio_cpu_isset() return a bool
Jens Axboe [Thu, 16 Nov 2017 21:16:41 +0000 (14:16 -0700)]
os: make fio_cpu_isset() return a bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoos-windows: fix cpumask operations
Robert Elliott [Tue, 14 Nov 2017 02:22:09 +0000 (20:22 -0600)]
os-windows: fix cpumask operations

Fix cpumask manipulation on (64-bit) Windows systems.

cpus_allowed=nn values greater than 32 does not work, due to
* the compiler not promoting expressions like "1 << cpu" to
  64 bits
* the clear function clearing the wrong bit (using "cpu - 1"),
* the clear function using XOR to clear (which only works if the
  bit was previously set)
* the check function returning a 64-bit value through a 32-bit
  return value

Example problems (from extra debug prints):
* setting CPU 32 really sets bit 0 (CPU 0)
Set mask of 0000000000000000 to add 0000000000000001 (32)

* setting CPU 63 really sets bit 31 (CPU 31)
Set mask of 0000000000000000 to add 0000000080000000 (63)

* clearing CPU 0 really clears bit 63 (CPU 63)
Clear mask of 0000000055555555 to remove 8000000000000000 (0)

* clearing CPU 2 really clears bit 2 (CPU 2)
Clear mask of 0000000055555555 to remove 0000000000000002 (2)

* checking claims CPU 32 is not in a mask of CPUs {32,34,36,38}
for 0000005500000000 check if bit 32 is set ==> FALSE

Tested with x86_64-w64-mingw32-gcc 6.4.0 from cygwin on a system
with 64 CPU cores (all fitting in one Windows processor group).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years ago.gitignore: ignore .exe files (for Windows)
Robert Elliott [Mon, 13 Nov 2017 21:56:59 +0000 (15:56 -0600)]
.gitignore: ignore .exe files (for Windows)

Make git ignore .exe files, which are generated by Windows
cygwin builds.

Examples:
        new file:   fio.exe
        new file:   t/axmap.exe
        new file:   t/fio-dedupe.exe
        new file:   t/fio-genzipf.exe
        new file:   t/fio-verify-state.exe
        new file:   t/gen-rand.exe
        new file:   t/ieee754.exe
        new file:   t/lfsr-test.exe
        new file:   t/stest.exe

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoman page: fix bad case for 'pre-reading file' state
Jens Axboe [Wed, 15 Nov 2017 16:53:14 +0000 (09:53 -0700)]
man page: fix bad case for 'pre-reading file' state

Fixes: https://github.com/axboe/fio/issues/491
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'fix-libhdfs' of https://github.com/follitude/fio
Jens Axboe [Mon, 6 Nov 2017 16:11:07 +0000 (09:11 -0700)]
Merge branch 'fix-libhdfs' of https://github.com/follitude/fio

2 years agoMakefile: tiny fix of libhdfs
follitude [Mon, 6 Nov 2017 12:28:20 +0000 (20:28 +0800)]
Makefile: tiny fix of libhdfs

2 years agoFio 3.2 fio-3.2
Jens Axboe [Fri, 3 Nov 2017 21:23:49 +0000 (15:23 -0600)]
Fio 3.2

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agoMerge branch 'json_bw_bytes' of https://github.com/sitsofe/fio
Jens Axboe [Fri, 3 Nov 2017 18:37:03 +0000 (12:37 -0600)]
Merge branch 'json_bw_bytes' of https://github.com/sitsofe/fio

2 years agosolaris: add get_fs_free_size() implementation
Tomohiro Kusumi [Fri, 3 Nov 2017 16:46:03 +0000 (18:46 +0200)]
solaris: add get_fs_free_size() implementation

Copied from os/os-freebsd.h. POSIX statvfs works on Solaris
(actually compiled/tested on illumos) as well.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agosolaris: add os_phys_mem() implementation
Tomohiro Kusumi [Fri, 3 Nov 2017 16:46:02 +0000 (18:46 +0200)]
solaris: add os_phys_mem() implementation

Copied from os/os-linux.h. The standard sysconf(3C) names work on
Solaris (actually compiled/tested on illumos) as well.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agosolaris: #include <pthread.h>
Tomohiro Kusumi [Fri, 3 Nov 2017 16:46:01 +0000 (18:46 +0200)]
solaris: #include <pthread.h>

os/os-solaris.h header uses pthread, but doesn't include <pthread.h>
which should be to be able to include this header alone.
Needed for unittesting of OS specific stuff.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agostat: add bw_bytes JSON key
Sitsofe Wheeler [Fri, 3 Nov 2017 07:23:55 +0000 (07:23 +0000)]
stat: add bw_bytes JSON key

Fixes: https://github.com/axboe/fio/issues/422
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 years agoio_u_queue: convert rings to bool
Jens Axboe [Thu, 2 Nov 2017 18:26:39 +0000 (12:26 -0600)]
io_u_queue: convert rings to bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofilesetup: allocate 'r' locally in fallocate_file()
Jens Axboe [Thu, 2 Nov 2017 15:09:31 +0000 (09:09 -0600)]
filesetup: allocate 'r' locally in fallocate_file()

For platforms that don't have posix or linux fallocate, we trigger
a warning on 'r' being unused.

Fixes: c0aabe37db1b ("filesetup: don't print non-debug error on native fallocate failure")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofilesetup: recurse_dir() can use bool
Jens Axboe [Thu, 2 Nov 2017 15:07:11 +0000 (09:07 -0600)]
filesetup: recurse_dir() can use bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 years agofilesetup: create_work_dirs() can return bool
Jens Axboe [Thu, 2 Nov 2017 15:06:14 +0000 (09:06 -0600)]
filesetup: create_work_dirs() can return bool

Signed-off-by: Jens Axboe <axboe@kernel.dk>