Jens Axboe [Fri, 11 Dec 2015 22:13:41 +0000 (15:13 -0700)]
server: comment sk_entry struct and remove hole
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 11 Dec 2015 20:25:24 +0000 (13:25 -0700)]
Fixup and improve per-thread data
- Get rid of backend_data, just use sk_out
- Add reference counting to sk_out and proper assign/drop helpers
- Ensure we pass it to other threads created
- Fix leaks of sk_out
This should get us closer to the new model actually working as
intended.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 11 Dec 2015 00:46:05 +0000 (17:46 -0700)]
server: cleanup and proper error returns
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 10 Dec 2015 23:07:32 +0000 (16:07 -0700)]
server: make the io log transmit use the new infrastructure
Add support for vectored commands, and pass them on the backend
pipe as well.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 10 Dec 2015 22:32:15 +0000 (15:32 -0700)]
server: rewrite message handling
Currently we don't do any synchronization when we send out
network messages, this can cause a variety of issues. Instead
of adding serialization for transmits, let the messages queue
up in the network backend, and let the backend send them out.
Work in progress, not everything has been tested with this.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 10 Dec 2015 22:31:02 +0000 (15:31 -0700)]
log: remove log_local_buf()
Have the server return -1 if it can't log, then we can handle
it internally.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 10 Dec 2015 21:26:44 +0000 (14:26 -0700)]
Add backend related data
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 10 Dec 2015 21:22:43 +0000 (14:22 -0700)]
mutex: fixup fio_mutex_down_timeout()
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 10 Dec 2015 18:15:52 +0000 (11:15 -0700)]
Add exitall_on_error option
Like exitall, but only if a job exits due to erroring out.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 9 Dec 2015 19:44:09 +0000 (12:44 -0700)]
eta: use bool for forcing jobs_eta return
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 9 Dec 2015 19:40:52 +0000 (12:40 -0700)]
server: ensure we send the right sized jobs_eta when faking it
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 9 Dec 2015 19:37:05 +0000 (12:37 -0700)]
server: always end back ETA reply
Otherwise the client will time out waiting for a response. If we don't
have a local jobs_eta, then just send a zero filled one.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 9 Dec 2015 18:23:15 +0000 (11:23 -0700)]
thread_options: kill converted cpu masks
We can't currently convert them in cconv.c, so let's just disable
them as they eat a lot of space in the thread options.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 9 Dec 2015 16:16:13 +0000 (09:16 -0700)]
iolog: don't copy log samples for compression
For larger settings of log_compression, the allocation and copying of
the log will adversely affect performance and logged latencies. Instead
just reference the log in the compression code, while resetting the
current log to a smaller size. We pay a slightly higher cost in memory
until compression has completd, but it saves us from allocating and
copying tens (or hundreds) of megabytes inline.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 9 Dec 2015 16:08:10 +0000 (09:08 -0700)]
eta: fix comparison reversal for time based jobs
If time_based=1 and runtime=x, then we know the job will take x
time to run. Fix a bug introduced in a previous commit.
Fixes: ("Fix ETA calculations")
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 9 Dec 2015 15:37:50 +0000 (08:37 -0700)]
Fix ramp time for io_submit_mode=offload
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 9 Dec 2015 15:28:00 +0000 (08:28 -0700)]
backend: terminate loop if we didn't do IO
The below commit modified how we clear stats, and that can sometimes
cause fio to hang around when ramp + runtime has been exceeded. The
below job file showed the issue:
[test-job]
bs=4k
rw=read
direct=1
runtime=6s
ramp_time=5s
ioengine=null
iodepth=32
size=100g
Where it would do the ramp time, then count down the 6s of runtime, then
fio would just sit there busy looping.
Fixes:
ac28d9053679 ("Fixup bw/iops logging for short runs")
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 22:45:12 +0000 (15:45 -0700)]
options: add log_compression_cpus option
We generally don't want online log compression to interfere with the
IO jobs. Add an option that allows us to contain the log compression
to a specific CPU, or set of CPUs.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 22:31:22 +0000 (15:31 -0700)]
options: check for dryrun in cpu mask setting
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 22:20:54 +0000 (15:20 -0700)]
iolog: cleanup up data unref'ing
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 22:17:27 +0000 (15:17 -0700)]
iolog: cleanup log writing
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 20:04:59 +0000 (13:04 -0700)]
iolog: fix potential duplicate definition of compress init/exit
Move the code inside the #else part of the zlib check.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 19:51:23 +0000 (12:51 -0700)]
iolog: fix build with zlib not being installed
backend.o: In function `thread_main':
/home/axboe/git/fio/backend.c:1553: undefined reference to `iolog_compress_init'
/home/axboe/git/fio/backend.c:1656: undefined reference to `iolog_compress_exit'
collect2: error: ld returned 1 exit status
make: *** [fio] Error 1
Provide dummy init/exit functions.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 19:43:40 +0000 (12:43 -0700)]
iolog: cleanup data wait
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 19:42:26 +0000 (12:42 -0700)]
iolog: remember to destroy cond/lock when done
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 19:37:30 +0000 (12:37 -0700)]
workqueue: make workqueue_exit() safe for multiple exit calls
This might happen if we fail init.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 19:36:27 +0000 (12:36 -0700)]
rate-submit: clean up init/exit helpers
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 19:31:38 +0000 (12:31 -0700)]
iolog: reinstate log compression nice level
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 19:30:24 +0000 (12:30 -0700)]
workqueue/rate-submit: update comments
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 19:25:07 +0000 (12:25 -0700)]
workqueue: ensure that workqueue_enqueue() can't fail
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 18:35:39 +0000 (11:35 -0700)]
iolog: replace tp usage with workqueue
Now that workqueue.c is generic enough to handle any kind of
work offloading, convert the online log compression to workqueue
usage and kill off lib/tp.*
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 15:56:30 +0000 (08:56 -0700)]
workqueue: add nice support
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 15:51:43 +0000 (08:51 -0700)]
backend: move rated submit code to its own file
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 15:44:56 +0000 (08:44 -0700)]
workqueue: move last bits of end accounting to caller
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 15:36:01 +0000 (08:36 -0700)]
workqueue: move private accounting to caller
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 15:32:46 +0000 (08:32 -0700)]
workqueue: move private exit code to caller
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 15:28:53 +0000 (08:28 -0700)]
workqueue: move init worker private code to the caller
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 15:23:24 +0000 (08:23 -0700)]
workqueue: move 'td' private data to the workqueue user
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 05:35:31 +0000 (22:35 -0700)]
workqueue: remove knowledge of td queue state
Add pre sleep checker and worker ops, so we can push this to the
owner of the workqueue.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 05:20:02 +0000 (22:20 -0700)]
workqueue: add a workqueue_work type
Don't pass in a struct io_u, that's specific to the use case of
using this for queueing IO. Use a generic work type, so we can reuse
this helper code for other purposes.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 05:13:22 +0000 (22:13 -0700)]
workqueue: don't use ioengine return codes
Use true/false as return, signaling ok/busy.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 05:09:24 +0000 (22:09 -0700)]
workqueue: remove knowledge of io issue (and others) stats
This doesn't belong in the workqueue code, let the caller deal with
it instead.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 04:56:34 +0000 (21:56 -0700)]
iolog: ensure we always store compressed, if log_store_compressed == 1
Before this commit, we depend on log_compression= always being set.
But we might want the logs store compressed, even if we don't
continually compress chunks at runtime.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 8 Dec 2015 03:24:51 +0000 (20:24 -0700)]
Fix latency logging if disable_slat and disable_clat is set
We can still do total latencies.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 7 Dec 2015 21:30:34 +0000 (14:30 -0700)]
Fix stat summing for unified_rw_reporting
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 7 Dec 2015 17:45:57 +0000 (10:45 -0700)]
verify: fix header verification version check
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 7 Dec 2015 17:43:45 +0000 (10:43 -0700)]
workqueue: grab sw->lock for flag manipulation
For consistency, should not matter in the shutdown path.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 7 Dec 2015 17:42:24 +0000 (10:42 -0700)]
init: have set_debug() check for NULL optarg
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 7 Dec 2015 17:40:18 +0000 (10:40 -0700)]
t/genzip: cast division to double
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 7 Dec 2015 17:37:30 +0000 (10:37 -0700)]
options: don't throw away bssplit() return value
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 7 Dec 2015 17:34:25 +0000 (10:34 -0700)]
crc/test: don't throw away results
We don't want the compiler thinking that it can toss them.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 4 Dec 2015 20:15:36 +0000 (13:15 -0700)]
workqueue: fix potential ABBA deadlock in stats summing
Signed-off-by: Jens Axboe <axboe@fb.com>
Li Zhong [Fri, 4 Dec 2015 05:50:06 +0000 (13:50 +0800)]
powerpc: enable cpu clock for powerpc64
This patch tries to enable cpu clock for powerpc64, the code is copied from
mftb() in kernel source.
The two instructions after mfspr are added in the kernel to solve an erratum on
Cell and fsl booke CPUs. On those CPUs, 64 bits mftb is not atomic, so it is
possible that the low order 32 bits are already reset to 0x00000000 but the
high order bits are not yet incremented by one.
Don't know how to tell whether it will be running on those CPUs or not, I just
keep the above fix for all ppc64 CPUs. Even if we have some method to check
whether we are on those CPUs or not at run time, I think the check won't cost
less than the two added instructions. Maybe we could use the similar fix up
code kernel uses to dynamically patch the instructions with nops if not needed.
But that would add much more complexity.
Signed-off-by: Li Zhong <zhong@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 4 Dec 2015 17:20:57 +0000 (10:20 -0700)]
workqueue: properly account ->cur_depth
Also handle edge cases better.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 4 Dec 2015 02:58:23 +0000 (19:58 -0700)]
io_u: make io_u_quiesce() return how many IOs it completed
Ditto for io_u_queued_complete().
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 3 Dec 2015 22:08:08 +0000 (15:08 -0700)]
posix_fadvise() returns positive error values
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 3 Dec 2015 21:52:27 +0000 (14:52 -0700)]
filesetup: properly propagate error value from invalidate
The IO engine ops ->invalidate() will return the error value, and
so does posix_fadvise().
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 1 Dec 2015 16:25:08 +0000 (09:25 -0700)]
configure: add --disable-lex
On some centos 5 the lex -o option is broken. Add a way to just
disable lex/yacc for math support.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 1 Dec 2015 16:11:47 +0000 (09:11 -0700)]
workqueue: bool
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 1 Dec 2015 16:08:47 +0000 (09:08 -0700)]
mutex: use bool
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 1 Dec 2015 16:00:00 +0000 (09:00 -0700)]
Add 'bool' type
Let's cleanup and modernize the code a bit, add a boolean type that
we can use for all the 0/1 function returns.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 1 Dec 2015 02:46:20 +0000 (19:46 -0700)]
Makefile: missing conversion, and oslib/ dep cleanup
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 1 Dec 2015 02:40:58 +0000 (19:40 -0700)]
Move conditional lib files to oslib/
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 1 Dec 2015 02:31:20 +0000 (19:31 -0700)]
Makefile: re-arrange lib/
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 1 Dec 2015 02:21:08 +0000 (19:21 -0700)]
Move memalign to lib/
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 1 Dec 2015 02:18:07 +0000 (19:18 -0700)]
stat: move header declarations from iolog.h
The functions are in stat.c, move the to the right header.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 30 Nov 2015 23:06:22 +0000 (16:06 -0700)]
stat: use DDIR_RWDIR_CNT instead of hardwired '3'
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 30 Nov 2015 21:44:06 +0000 (14:44 -0700)]
Clear rusage stat when resetting all stats
This fixes an issue with bad rusage reporting, particularly when
the runtime to ramptime ratio is low.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 30 Nov 2015 21:28:54 +0000 (14:28 -0700)]
io_u: don't use xfer_buflen before we have set it up
Signed-off-by: Jens Axboe <axboe@fb.com>
Andrey Kuzmin [Mon, 30 Nov 2015 10:52:05 +0000 (13:52 +0300)]
io_size vs. time_based discrepancy
I'm witnessing an annoying discrepancy between the outcome of the same
job when being run as io_size-based vs. time_based. In the former
mode, the job does exactly what I want it to do, writing the
prescribed amount of data randomly w/o being concerned whether that
total bytes written is in any way related to the target file size.
On the contrary, in the latter mode that same job, after writing the
file's size worth of bytes, resets random generator and essentially
restarts the just completed loop. The offending code is below, and the
suggested fix brings back home the io_size-like behavior when running
time-based. Nonetheless, I'm in doubt regarding whether the do_io loop
break-out below was intended to support the designed behavior (looks
unlikely to me, as looping like that is produced by the 'loops'
option, although time_based definition under HOWTO is rather unclear
in this regard) or is a bug worth fixing.
Regards,
Andrey
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 26 Nov 2015 04:24:01 +0000 (21:24 -0700)]
Makefile: generate .d dep file for init.c
Could be cleaner, we're duplicating the general .c rule here. But
as a quickie, let's hope it fixes up the rebuild issues when
init.c dependencies have changed.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 25 Nov 2015 04:13:34 +0000 (21:13 -0700)]
Fio 2.2.12
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 23 Nov 2015 22:04:56 +0000 (15:04 -0700)]
A few min/max cleanups
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 23 Nov 2015 19:28:11 +0000 (12:28 -0700)]
options: set 'refill_buffers' for compress_percentage != 0
Only do it if the user hasn't set it.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 23 Nov 2015 17:33:34 +0000 (10:33 -0700)]
io_u: properly split buffer prep for compression
Matthew reports:
I noticed that sometimes using --buffer_compress_percentage would not
result in the desired compression. After running some tests I found
that only some block sizes are affected.
512b, 4K, 8K, 16K, and 32K seem to work as expected.
64K, 128K, 256K, 512K, and 1024K exhibited the bug. I have not tested
beyond 1024K.
Here is the test script I used only changing block size for each test.
for i in {0..100}; do
fio --name=test --rw=write --bs=128k --ioengine=libaio --direct=1
--iodepth=32 --size=512m --refill_buffers
--buffer_compress_percentage=$i --filename=testfile.$i --eta=never
--output=/dev/null
done
for i in {0..100}; do
gzip -v testfile.$i &>> gzip.txt
done
rm *.gz
Below are compression results for 4K and then 128K. For 128K, you can
see compression stops matching at around 50%.
bs=4k
testfile.0: 0.3% -- replaced with testfile.0.gz
testfile.1: 1.2% -- replaced with testfile.1.gz
testfile.2: 2.2% -- replaced with testfile.2.gz
testfile.3: 3.2% -- replaced with testfile.3.gz
testfile.4: 4.1% -- replaced with testfile.4.gz
testfile.5: 5.1% -- replaced with testfile.5.gz
testfile.6: 6.1% -- replaced with testfile.6.gz
testfile.7: 7.1% -- replaced with testfile.7.gz
testfile.8: 8.1% -- replaced with testfile.8.gz
testfile.9: 9.1% -- replaced with testfile.9.gz
testfile.10: 10.1% -- replaced with testfile.10.gz
testfile.11: 11.1% -- replaced with testfile.11.gz
testfile.12: 12.1% -- replaced with testfile.12.gz
testfile.13: 13.1% -- replaced with testfile.13.gz
testfile.14: 14.1% -- replaced with testfile.14.gz
testfile.15: 15.1% -- replaced with testfile.15.gz
testfile.16: 16.0% -- replaced with testfile.16.gz
testfile.17: 17.0% -- replaced with testfile.17.gz
testfile.18: 18.0% -- replaced with testfile.18.gz
testfile.19: 19.0% -- replaced with testfile.19.gz
testfile.20: 20.0% -- replaced with testfile.20.gz
testfile.21: 21.0% -- replaced with testfile.21.gz
testfile.22: 22.0% -- replaced with testfile.22.gz
testfile.23: 23.0% -- replaced with testfile.23.gz
testfile.24: 24.0% -- replaced with testfile.24.gz
testfile.25: 25.0% -- replaced with testfile.25.gz
testfile.26: 25.9% -- replaced with testfile.26.gz
testfile.27: 26.9% -- replaced with testfile.27.gz
testfile.28: 27.9% -- replaced with testfile.28.gz
testfile.29: 28.9% -- replaced with testfile.29.gz
testfile.30: 29.9% -- replaced with testfile.30.gz
testfile.31: 30.9% -- replaced with testfile.31.gz
testfile.32: 31.9% -- replaced with testfile.32.gz
testfile.33: 32.9% -- replaced with testfile.33.gz
testfile.34: 33.9% -- replaced with testfile.34.gz
testfile.35: 34.9% -- replaced with testfile.35.gz
testfile.36: 35.9% -- replaced with testfile.36.gz
testfile.37: 36.9% -- replaced with testfile.37.gz
testfile.38: 37.9% -- replaced with testfile.38.gz
testfile.39: 38.9% -- replaced with testfile.39.gz
testfile.40: 39.9% -- replaced with testfile.40.gz
testfile.41: 40.8% -- replaced with testfile.41.gz
testfile.42: 41.8% -- replaced with testfile.42.gz
testfile.43: 42.8% -- replaced with testfile.43.gz
testfile.44: 43.8% -- replaced with testfile.44.gz
testfile.45: 44.8% -- replaced with testfile.45.gz
testfile.46: 45.8% -- replaced with testfile.46.gz
testfile.47: 46.8% -- replaced with testfile.47.gz
testfile.48: 47.8% -- replaced with testfile.48.gz
testfile.49: 48.8% -- replaced with testfile.49.gz
testfile.50: 49.8% -- replaced with testfile.50.gz
testfile.51: 50.3% -- replaced with testfile.51.gz
testfile.52: 51.2% -- replaced with testfile.52.gz
testfile.53: 52.3% -- replaced with testfile.53.gz
testfile.54: 53.2% -- replaced with testfile.54.gz
testfile.55: 54.2% -- replaced with testfile.55.gz
testfile.56: 55.3% -- replaced with testfile.56.gz
testfile.57: 56.1% -- replaced with testfile.57.gz
testfile.58: 57.2% -- replaced with testfile.58.gz
testfile.59: 58.2% -- replaced with testfile.59.gz
testfile.60: 59.2% -- replaced with testfile.60.gz
testfile.61: 60.2% -- replaced with testfile.61.gz
testfile.62: 61.2% -- replaced with testfile.62.gz
testfile.63: 62.2% -- replaced with testfile.63.gz
testfile.64: 63.2% -- replaced with testfile.64.gz
testfile.65: 64.1% -- replaced with testfile.65.gz
testfile.66: 65.2% -- replaced with testfile.66.gz
testfile.67: 66.2% -- replaced with testfile.67.gz
testfile.68: 67.2% -- replaced with testfile.68.gz
testfile.69: 68.2% -- replaced with testfile.69.gz
testfile.70: 69.2% -- replaced with testfile.70.gz
testfile.71: 70.2% -- replaced with testfile.71.gz
testfile.72: 71.2% -- replaced with testfile.72.gz
testfile.73: 72.2% -- replaced with testfile.73.gz
testfile.74: 73.2% -- replaced with testfile.74.gz
testfile.75: 74.3% -- replaced with testfile.75.gz
testfile.76: 75.2% -- replaced with testfile.76.gz
testfile.77: 76.1% -- replaced with testfile.77.gz
testfile.78: 77.2% -- replaced with testfile.78.gz
testfile.79: 78.2% -- replaced with testfile.79.gz
testfile.80: 79.1% -- replaced with testfile.80.gz
testfile.81: 80.1% -- replaced with testfile.81.gz
testfile.82: 81.2% -- replaced with testfile.82.gz
testfile.83: 82.2% -- replaced with testfile.83.gz
testfile.84: 83.1% -- replaced with testfile.84.gz
testfile.85: 84.2% -- replaced with testfile.85.gz
testfile.86: 85.1% -- replaced with testfile.86.gz
testfile.87: 86.1% -- replaced with testfile.87.gz
testfile.88: 87.2% -- replaced with testfile.88.gz
testfile.89: 88.2% -- replaced with testfile.89.gz
testfile.90: 89.2% -- replaced with testfile.90.gz
testfile.91: 90.1% -- replaced with testfile.91.gz
testfile.92: 91.1% -- replaced with testfile.92.gz
testfile.93: 92.0% -- replaced with testfile.93.gz
testfile.94: 93.2% -- replaced with testfile.94.gz
testfile.95: 94.1% -- replaced with testfile.95.gz
testfile.96: 95.0% -- replaced with testfile.96.gz
testfile.97: 96.4% -- replaced with testfile.97.gz
testfile.98: 97.3% -- replaced with testfile.98.gz
testfile.99: 98.5% -- replaced with testfile.99.gz
testfile.100: 99.9% -- replaced with testfile.100.gz
bs=128k
testfile.0: 0.3% -- replaced with testfile.0.gz
testfile.1: 1.3% -- replaced with testfile.1.gz
testfile.2: 2.3% -- replaced with testfile.2.gz
testfile.3: 3.3% -- replaced with testfile.3.gz
testfile.4: 4.2% -- replaced with testfile.4.gz
testfile.5: 5.2% -- replaced with testfile.5.gz
testfile.6: 6.2% -- replaced with testfile.6.gz
testfile.7: 7.2% -- replaced with testfile.7.gz
testfile.8: 8.2% -- replaced with testfile.8.gz
testfile.9: 9.2% -- replaced with testfile.9.gz
testfile.10: 10.2% -- replaced with testfile.10.gz
testfile.11: 11.2% -- replaced with testfile.11.gz
testfile.12: 12.2% -- replaced with testfile.12.gz
testfile.13: 13.2% -- replaced with testfile.13.gz
testfile.14: 14.2% -- replaced with testfile.14.gz
testfile.15: 15.2% -- replaced with testfile.15.gz
testfile.16: 16.1% -- replaced with testfile.16.gz
testfile.17: 17.1% -- replaced with testfile.17.gz
testfile.18: 18.1% -- replaced with testfile.18.gz
testfile.19: 19.1% -- replaced with testfile.19.gz
testfile.20: 20.1% -- replaced with testfile.20.gz
testfile.21: 21.1% -- replaced with testfile.21.gz
testfile.22: 22.1% -- replaced with testfile.22.gz
testfile.23: 23.1% -- replaced with testfile.23.gz
testfile.24: 24.1% -- replaced with testfile.24.gz
testfile.25: 25.1% -- replaced with testfile.25.gz
testfile.26: 26.1% -- replaced with testfile.26.gz
testfile.27: 27.1% -- replaced with testfile.27.gz
testfile.28: 28.1% -- replaced with testfile.28.gz
testfile.29: 29.1% -- replaced with testfile.29.gz
testfile.30: 30.0% -- replaced with testfile.30.gz
testfile.31: 31.0% -- replaced with testfile.31.gz
testfile.32: 32.0% -- replaced with testfile.32.gz
testfile.33: 33.0% -- replaced with testfile.33.gz
testfile.34: 34.0% -- replaced with testfile.34.gz
testfile.35: 35.0% -- replaced with testfile.35.gz
testfile.36: 36.0% -- replaced with testfile.36.gz
testfile.37: 37.0% -- replaced with testfile.37.gz
testfile.38: 38.0% -- replaced with testfile.38.gz
testfile.39: 39.0% -- replaced with testfile.39.gz
testfile.40: 40.0% -- replaced with testfile.40.gz
testfile.41: 41.0% -- replaced with testfile.41.gz
testfile.42: 42.0% -- replaced with testfile.42.gz
testfile.43: 43.0% -- replaced with testfile.43.gz
testfile.44: 44.0% -- replaced with testfile.44.gz
testfile.45: 45.0% -- replaced with testfile.45.gz
testfile.46: 46.0% -- replaced with testfile.46.gz
testfile.47: 47.0% -- replaced with testfile.47.gz
testfile.48: 47.9% -- replaced with testfile.48.gz
testfile.49: 48.9% -- replaced with testfile.49.gz
testfile.50: 49.9% -- replaced with testfile.50.gz
testfile.51: 48.9% -- replaced with testfile.51.gz
testfile.52: 47.9% -- replaced with testfile.52.gz
testfile.53: 47.0% -- replaced with testfile.53.gz
testfile.54: 46.0% -- replaced with testfile.54.gz
testfile.55: 45.0% -- replaced with testfile.55.gz
testfile.56: 44.0% -- replaced with testfile.56.gz
testfile.57: 43.0% -- replaced with testfile.57.gz
testfile.58: 42.0% -- replaced with testfile.58.gz
testfile.59: 41.0% -- replaced with testfile.59.gz
testfile.60: 40.0% -- replaced with testfile.60.gz
testfile.61: 39.0% -- replaced with testfile.61.gz
testfile.62: 38.0% -- replaced with testfile.62.gz
testfile.63: 37.0% -- replaced with testfile.63.gz
testfile.64: 36.0% -- replaced with testfile.64.gz
testfile.65: 35.0% -- replaced with testfile.65.gz
testfile.66: 34.0% -- replaced with testfile.66.gz
testfile.67: 34.0% -- replaced with testfile.67.gz
testfile.68: 36.0% -- replaced with testfile.68.gz
testfile.69: 38.0% -- replaced with testfile.69.gz
testfile.70: 40.0% -- replaced with testfile.70.gz
testfile.71: 42.0% -- replaced with testfile.71.gz
testfile.72: 43.9% -- replaced with testfile.72.gz
testfile.73: 45.9% -- replaced with testfile.73.gz
testfile.74: 47.9% -- replaced with testfile.74.gz
testfile.75: 49.9% -- replaced with testfile.75.gz
testfile.76: 47.9% -- replaced with testfile.76.gz
testfile.77: 45.9% -- replaced with testfile.77.gz
testfile.78: 43.9% -- replaced with testfile.78.gz
testfile.79: 41.9% -- replaced with testfile.79.gz
testfile.80: 40.0% -- replaced with testfile.80.gz
testfile.81: 42.9% -- replaced with testfile.81.gz
testfile.82: 45.9% -- replaced with testfile.82.gz
testfile.83: 48.9% -- replaced with testfile.83.gz
testfile.84: 47.9% -- replaced with testfile.84.gz
testfile.85: 44.9% -- replaced with testfile.85.gz
testfile.86: 43.9% -- replaced with testfile.86.gz
testfile.87: 47.9% -- replaced with testfile.87.gz
testfile.88: 87.4% -- replaced with testfile.88.gz
testfile.89: 88.4% -- replaced with testfile.89.gz
testfile.90: 89.4% -- replaced with testfile.90.gz
testfile.91: 90.4% -- replaced with testfile.91.gz
testfile.92: 91.4% -- replaced with testfile.92.gz
testfile.93: 92.4% -- replaced with testfile.93.gz
testfile.94: 93.4% -- replaced with testfile.94.gz
testfile.95: 94.4% -- replaced with testfile.95.gz
testfile.96: 95.4% -- replaced with testfile.96.gz
testfile.97: 96.4% -- replaced with testfile.97.gz
testfile.98: 97.4% -- replaced with testfile.98.gz
testfile.99: 98.4% -- replaced with testfile.99.gz
testfile.100: 99.9% -- replaced with testfile.100.gz
The issue is that once we get above 32-64k, we slide out of the gzip
window for compression. Fixup the buffer_compress_chunk so that it
switches random states, hence doing the compression evenly over a
more limited range that normal compression will catch.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 21 Nov 2015 00:26:37 +0000 (17:26 -0700)]
gettime: get rid of the (unecessary) 10x scaling factor
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 21 Nov 2015 00:20:06 +0000 (17:20 -0700)]
gettime: make cpuclock calibration more tolerant
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 21 Nov 2015 00:01:16 +0000 (17:01 -0700)]
gettime: print first clock offset for CPU clock
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Rosenboom [Thu, 19 Nov 2015 20:36:18 +0000 (13:36 -0700)]
Fix ETA calculations
There are a couple of issues in eta.c:
- The comparison for time- vs. rate-based ETA calculation is done the
wrong way round.
- The elapsed ramp time should only be subtracted when the thread has
indeed entered the ramp phase.
- Currently jobs that are stonewalled are not taken into account for ETA
calculation. This patch at least fixes this for the case that there is
only a single job per stonewall group.
---
This patch tries to fix some issues I have found, this may
also be related to the ones mentioned earlier in
http://www.spinics.net/lists/fio/msg04187.html
The part regarding stonewalled jobs may require some further work, but
at least it gives proper values now for my standard fio runs containing
just a sequence of singular jobs.
Also, I'm still wondering about the intention behind the way done_secs
is meant to work. It cumulates time spent in finished jobs globally, but
is being used in per-job ETA calculations. This seems to break at
least for the case of stonewalled jobs, too.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 19 Nov 2015 18:26:21 +0000 (11:26 -0700)]
Don't output json headers for fio backend
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 16 Nov 2015 21:00:41 +0000 (14:00 -0700)]
Fix group_id != groupid mixup
Also make sure we don't overflow 'groupid'
Signed-off-by: Jens Axboe <axboe@fb.com>
Casey Bodley [Wed, 4 Nov 2015 16:03:11 +0000 (11:03 -0500)]
iolog: fix error when compiled as c++
Given the main.cc source file:
#include <fio.h>
int main() { return 0; }
And the gcc command line:
gcc main.cc -Ifio
In file included from fio/stat.h:4:0,
from fio/thread_options.h:7,
from fio/fio.h:18,
from main.cc:1:
fio/iolog.h: In function 'io_sample* __get_sample(void*, int,
uint64_t)':
fio/iolog.h:125:53: warning: pointer of type 'void *' used in arithmetic
[-Wpointer-arith]
return samples + sample * __log_entry_sz(log_offset);
^
fio/iolog.h:125:17: error: invalid conversion from 'void*' to
'io_sample*' [-fpermissive]
return samples + sample * __log_entry_sz(log_offset);
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Vincent Fu [Wed, 11 Nov 2015 18:26:37 +0000 (11:26 -0700)]
json output fixups
- diskutil json+ fixup
- idletime missed a conversion for the multiple output formats
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 11 Nov 2015 01:13:40 +0000 (18:13 -0700)]
Fix verification error with bsrange and split read/write phase
If you have separate phases for the verify writes and the verify reads,
and fio ends up having to align to the verify interval header size,
then we can end up trimming the buflen on the read side (where
do_verify=1 is set), but not on the write side. This causes a
verification failure. Fix this by always aligning, regardless of
whether do_verify is true or not.
Example job files:
-------------------------------------------------------------------------------
[global]
thread=1
blocksize_range=4k-256k
ioengine=libaio
verify_interval=512
iodepth=64
loops=1
verify_pattern=0x03715998
size=100MB
offset=0
verify_dump=1
filename=/tmp/disk.img
[write-phase]
rw=randwrite
fill_device=1
do_verify=0
-------------------------------------------------------------------------------
[global]
thread=1
blocksize_range=4k-256k
ioengine=libaio
verify_interval=512
iodepth=64
loops=1
verify_pattern=0x03715998
size=100MB
offset=0
verify_dump=1
filename=/tmp/disk.img
[verify-phase]
stonewall
rw=randread
create_serialize=0
do_verify=1
-------------------------------------------------------------------------------
Reported-by: Kevin Vajk <kmv@datera.io>
Fixes:
a9f70b1f5087 ("Make sure io_u->buflen is aligned to the verify_interval")
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 10 Nov 2015 02:38:15 +0000 (19:38 -0700)]
smalloc: get rid of global lock
We don't dynamically add pools anymore, so we don't need global
read/write locks.
Also drop the needless wrappers around pool locking.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 10 Nov 2015 01:45:35 +0000 (18:45 -0700)]
Clarify spread/lambda of poisson
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 9 Nov 2015 23:37:33 +0000 (16:37 -0700)]
Rename rate_poisson to rate_process
This will more accurately follow what other options could adopt,
for instance if we want to do a dynamic IO depth.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 9 Nov 2015 23:05:19 +0000 (16:05 -0700)]
Provide some consistency in rate_* options
ratemin and ratecycle are the only ones without an underscore.
Change them to rate_min and rate_cycle, but retain the old
names as an alias for compat reasons.
Signed-off-by: Jens Axboe <axboe@fb.com>
Song Liu [Mon, 9 Nov 2015 22:32:14 +0000 (14:32 -0800)]
In fio.1 and HOWTO, add link to Poisson process in wikipedia
Signed-off-by: Jens Axboe <axboe@fb.com>
Song Liu [Mon, 9 Nov 2015 22:32:13 +0000 (14:32 -0800)]
add example using --rate_poisson
Signed-off-by: Jens Axboe <axboe@fb.com>
Song Liu [Mon, 9 Nov 2015 22:32:12 +0000 (14:32 -0800)]
make sure __rand_0_1 does not return 0.0
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 9 Nov 2015 22:07:14 +0000 (15:07 -0700)]
Add poisson rate selection to FD_RATE output
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 9 Nov 2015 19:54:37 +0000 (12:54 -0700)]
Fixups for poisson rate
- Change name to poisson_rate, we could add other uses for this
functionality (like depth, etc).
- Add "rate" as option parent.
- Ensure we convert the option for client/server, bump protocol
version.
- Add a separate rand state for poisson, so it becomes randomly
controllable (through randrepeat).
- Add random 0..1 helper.
Signed-off-by: Jens Axboe <axboe@fb.com>
Song Liu [Mon, 9 Nov 2015 19:34:56 +0000 (12:34 -0700)]
Enable request flow under Poisson process
To better mimic real world wordload, where incoming requests follow
Poisson process, this patch enables poisson process request flow
for rate limited benchmarks.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sun, 8 Nov 2015 00:33:38 +0000 (17:33 -0700)]
smalloc: only clear the bitmap, not the whole pool
This reduces startup time, and the memset() isn't useful nor
needed on the full range.
Signed-off-by: Jens Axboe <axboe@fb.com>
Martin Steigerwald [Sun, 8 Nov 2015 00:30:22 +0000 (17:30 -0700)]
Fix spelling error in engines/rdma.c
> lintian -IE fio_2.2.10-1_amd64.changes
I: fio: spelling-error-in-binary usr/bin/fio
Recieve Receive
Signed-off-by: Jens Axboe <axboe@fb.com>
Vincent Fu [Fri, 6 Nov 2015 15:46:44 +0000 (08:46 -0700)]
Add latency bin output to the json output format
We add a new format for this, json+, as it generates a lot of data.
As of now, the only difference between json and json+ is the full
dump of all latency bins.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 5 Nov 2015 16:05:15 +0000 (09:05 -0700)]
Fio 2.2.11
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 31 Oct 2015 01:34:21 +0000 (19:34 -0600)]
output_buffer: only realloc once, and memset just what we need
No need looping around a realloc(), just alloc what we need upfront.
Additionally, don't memset the parts we'll be copying into anyway.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 30 Oct 2015 02:52:23 +0000 (11:52 +0900)]
Bump the client ETA timeout from 5s to 30s
5 seconds is low enough that we sometimes falsely trigger the timeout.
It's meant to catch clients that have disappeared, so 5 seconds is
arguably way too low. Bump it to 30 seconds instead.
Signed-off-by: Jens Axboe <axboe@fb.com>