Jens Axboe [Wed, 17 Dec 2014 02:43:55 +0000 (19:43 -0700)]
gettime: improve gettimeofday() offload support
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 16 Dec 2014 22:37:25 +0000 (15:37 -0700)]
gettime: limit warning on CPU clock
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 16 Dec 2014 22:13:45 +0000 (15:13 -0700)]
gettime: don't attempt to fixup what looks like a backwards clock
It could just be a wrap. The code is buggy, kill it, we'll deal
with the wrap later.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 16 Dec 2014 17:46:24 +0000 (10:46 -0700)]
engines/mmap: clear partial mmap flag on close
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 16 Dec 2014 17:35:07 +0000 (10:35 -0700)]
engines/mmap: fix segfault on large devices and 32-bit archs
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 16 Dec 2014 02:54:57 +0000 (19:54 -0700)]
backend: use monotonic clock for ETA, if we have it
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 15 Dec 2014 20:45:53 +0000 (13:45 -0700)]
backend: fix off-by-one in nsec calculation
If it's equal to 10^9 we need to bump seconds and subtract 10^9
from nsec, not only if it's larger than.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 15 Dec 2014 16:38:43 +0000 (09:38 -0700)]
file: unionize lfsr/randommap
We only use one of them, add file flags to distinguish between the
two.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 15 Dec 2014 15:26:11 +0000 (08:26 -0700)]
Add helpers for getting/setting file engine data
Since it's a uint64_t, we need casting to get/set the engine data
or some platforms will complain. Encapsulate that in helper macros.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 15 Dec 2014 02:17:27 +0000 (19:17 -0700)]
file: move mmap related data to engines/mmap.c where it belongs
We should not keep this in the generic file structure, it belongs
in file engine data for the specific engine.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 15 Dec 2014 02:03:12 +0000 (19:03 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Mon, 15 Dec 2014 02:01:24 +0000 (19:01 -0700)]
Fix read/write mix and different levels of randomness
Stephen reports:
When using fio configuration below:
[global]
ioengine=libaio
direct=1
runtime=600
bs=32k
iodepth=8
rw=randrw
rwmixread=80
percentage_random=100,0
[drive1]
filename=/dev/sda
I am expecting to see 80% reads, 20% writes where all reads are random
and all writes are sequential. I captured a bus trace of traffic to the
disk and the bus trace reflected as much with one issue. The write
commands are essentially random. Each write begins at a new random LBA.
If 2 or more writes occur in a row, the LBA's are sequential based on
the block size BUT I feel the heart of this feature would be to emulate
a large file write during random access. With that in mind would it be
possible for sequential reads or writes within mixed sequential/random
workload to remember the last LBA accessed? In this scenario the writes
would still only take up 20% of the workload but when a write did occur
it should be the next sequential step from the last write.
-----
This is due to the fact that fio tracks last start/end on a unified
basis, not per read/write/trim direction. Fix that.
Signed-off-by: Jens Axboe <axboe@fb.com>
Robert Elliott [Sat, 13 Dec 2014 04:50:30 +0000 (21:50 -0700)]
eta.c: check malloc return code
Check the malloc return code in get_jobs_eta.
Signed-off-by: Robert Elliott <elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 12 Dec 2014 18:34:28 +0000 (11:34 -0700)]
backend: fix should_check_rate() typo in wait_for_completions()
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 12 Dec 2014 15:49:12 +0000 (08:49 -0700)]
backend: abstract out waiting for completions
We do it both in do_verify() and in do_io(), so it's duplicated code.
And we want to potentially add a completor thread, so lets move this
to a helper.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 11 Dec 2014 02:21:37 +0000 (19:21 -0700)]
Update size/io_size descriptions
Hopefully this is clearer. Also add an alias for io_limit to io_size,
since that is probably more descriptive of what this option does.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 10 Dec 2014 21:01:36 +0000 (22:01 +0100)]
btrace2fio: add support for collapsing close entries
Signed-off-by: Jens Axboe <axboe@fb.com>
Fabrice Bacchella [Wed, 10 Dec 2014 16:39:48 +0000 (17:39 +0100)]
Improved support for libhdfs
Jens Axboe [Wed, 10 Dec 2014 15:12:56 +0000 (08:12 -0700)]
gluster: wire up sync/datasync opcodes
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 9 Dec 2014 21:12:47 +0000 (14:12 -0700)]
Fix server/client set_options conversion
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 9 Dec 2014 21:10:05 +0000 (14:10 -0700)]
Use fio_option_is_set() for ioprio setting
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 9 Dec 2014 20:59:58 +0000 (13:59 -0700)]
Always scramble buffers, if scramble_buffers is set
This makes scramble_buffers and zero_buffers cooperate nicely.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 9 Dec 2014 20:17:33 +0000 (13:17 -0700)]
Get rid of _set variables
We can just check the option set mask now.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 9 Dec 2014 19:58:15 +0000 (12:58 -0700)]
options: add support for checking if an option has been set
We often just do a non-zero check, but an option can be set
and be set to zero, so we can't currently detect that. For
complicated options, we add a second variable to mark if it
has been set or not. This allows us to directly check if a
given option was set by the user or not.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sun, 7 Dec 2014 01:47:43 +0000 (18:47 -0700)]
Update date and email in man page
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 6 Dec 2014 16:53:46 +0000 (09:53 -0700)]
options: strip off =optval when matching
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 6 Dec 2014 16:17:52 +0000 (09:17 -0700)]
options: show closest match for unknown job option
Just like we did for the command line options.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 5 Dec 2014 18:04:19 +0000 (11:04 -0700)]
lib/rand: cleanup __fill_random_buffer()
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 5 Dec 2014 17:48:01 +0000 (10:48 -0700)]
lib/rand: fix bug with non uint64_t aligned random buffer fill
Now that we honor the compression percentage, we can easily
get buffer fills that are not aligned to uint64_t. Make
sure that __fill_random_buf() handles this correctly.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 5 Dec 2014 16:35:40 +0000 (09:35 -0700)]
parse: make suggestions for unknown options
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 5 Dec 2014 16:35:19 +0000 (09:35 -0700)]
sync: use io_u_log_error() for more detailed error messages
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 4 Dec 2014 23:49:53 +0000 (16:49 -0700)]
verify: cleanup
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 4 Dec 2014 22:40:32 +0000 (15:40 -0700)]
Use specified compression/pattern for verify buffers too
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 4 Dec 2014 15:27:21 +0000 (08:27 -0700)]
Don't clear 'refill_buffers' unconditionally for pattern fill
If we have asked for a specific buffer compression, then we need
the refill option (if set).
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 4 Dec 2014 02:55:33 +0000 (19:55 -0700)]
Use specified buffer_pattern (if given) for all io_u fills
For compression, we use a fixed '0' pattern. But if the user
specified a pattern to use in the job file, then we should
use that instead. It could slightly skew the compression ratio
for long patterns, but that is to be expected.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 4 Dec 2014 02:25:13 +0000 (19:25 -0700)]
Remove 'td' parameter from fill_pattern()
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 3 Dec 2014 17:10:59 +0000 (10:10 -0700)]
Fix a few spelling errors
Signed-off-by: Jens Axboe <axboe@fb.com>
Elliott, Robert (Server Storage) [Tue, 2 Dec 2014 20:54:32 +0000 (13:54 -0700)]
fio: remove deprecated --latency-log from manpage
The --latency-log argument is no longer accepted; fio returns
an error message saying it is deprecated. So, remove it from
the manpage.
Signed-off-by: Robert Elliott <elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 1 Dec 2014 16:44:02 +0000 (09:44 -0700)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Mon, 1 Dec 2014 16:42:48 +0000 (09:42 -0700)]
Bump disk util stats from 32 to 64-bit values
Robert reports:
A long run can result in wrapping in the disk stats
(ios, merge, ticks, in_queue).
In this example, there were 212,245,615,134 read IOs
(that is from uint64_t total_io_u[3]):
read : io=101207GB, bw=491309KB/s, iops=982618, runt=216000016msec
cpu : usr=0.09%, sys=4.33%, ctx=
152784302066, majf=0, minf=2928
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.5%, 16=0.5%, 32=1.0%, >=64=1.0%
submit : 0=0.0%, 4=18.0%, 8=6.6%, 16=8.5%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=15.0%, 32=0.1%, 64=0.1%, >=64=0.1%
issued : total=r=
212245615134/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=96
Run status group 0 (all jobs):
READ: io=101207GB, aggrb=491309KB/s, minb=491309KB/s, maxb=491309KB/s, mint=216000016msec, maxt=216000016msec
All the disk stats, however, wrap at ios=4,294,967,295 (32 bit).
In this case it caught sdi falling a bit behind the others,
but all of them have wrapped multiple times.
Disk stats (read/write):
sdb: ios=
796157304/0, merge=0/0, ticks=
722039638/0, in_queue=
735284999, util=100.00%
sdc: ios=
725235994/0, merge=0/0, ticks=
725893305/0, in_queue=
738790349, util=100.00%
sdd: ios=
722370867/0, merge=0/0, ticks=
724540972/0, in_queue=
738926102, util=100.00%
sde: ios=
722490025/0, merge=0/0, ticks=
724775894/0, in_queue=
745581851, util=100.00%
sdf: ios=
719665329/0, merge=0/0, ticks=
724281840/0, in_queue=
740201203, util=100.00%
sdg: ios=
721386377/0, merge=0/0, ticks=
725529483/0, in_queue=
740729310, util=100.00%
sdh: ios=
720461888/0, merge=0/0, ticks=
724759222/0, in_queue=
741212131, util=100.00%
sdi: ios=
3280272329/0, merge=0/0, ticks=
3337080572/0, in_queue=
3340290775, util=81.57%
sdj: ios=
717395167/0, merge=0/0, ticks=
723762993/0, in_queue=
739136654, util=100.00%
sdk: ios=
824804416/0, merge=0/0, ticks=
722973383/0, in_queue=
734915061, util=100.00%
sdl: ios=
787317365/0, merge=0/0, ticks=
720963594/0, in_queue=
736506238, util=100.00%
sdm: ios=
791112845/0, merge=0/0, ticks=
723371287/0, in_queue=
745590636, util=100.00%
sdn: ios=
786009870/0, merge=0/0, ticks=
721111447/0, in_queue=
733786474, util=100.00%
sdo: ios=
787226466/0, merge=0/0, ticks=
722200707/0, in_queue=
737673635, util=100.00%
sdp: ios=
789062478/0, merge=0/0, ticks=
723648481/0, in_queue=
737820918, util=100.00%
sdq: ios=
786121713/0, merge=0/0, ticks=
722203635/0, in_queue=
752578258, util=100.00%
These structures are the reason:
struct disk_util_stats {
uint32_t ios[2];
uint32_t merges[2];
uint64_t sectors[2];
uint32_t ticks[2];
uint32_t io_ticks;
uint32_t time_in_queue;
uint64_t msec;
};
struct disk_util_agg {
uint32_t ios[2];
uint32_t merges[2];
uint64_t sectors[2];
uint32_t ticks[2];
uint32_t io_ticks;
uint32_t time_in_queue;
uint32_t slavecount;
fio_fp64_t max_util;
};
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 25 Nov 2014 23:02:52 +0000 (16:02 -0700)]
Make gfio build with --build-static
We can't build the gtk app statically, so filter out the static part
for just that. Fix up the Makefile so that fio doesn't include
all the gtk libs as well.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 24 Nov 2014 15:07:03 +0000 (08:07 -0700)]
Fixup t/ test apps for lfsr_next change()
Reported-by: Bruce Cran <bruce@cran.org.uk>
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 24 Nov 2014 01:41:11 +0000 (18:41 -0700)]
lfsr: don't pass in last value to lfsr_next()
It's cached in the 'fl' struct. This means we can move the max
block calculation outside if the lfsr part, too.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 21 Nov 2014 21:33:33 +0000 (14:33 -0700)]
iolog: account for wasted time in ipo stall
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 21 Nov 2014 20:49:00 +0000 (13:49 -0700)]
blktrace: correct setting of iodepth
We want to check for == 1, not != 1. Also set iodepth_low, like
we would have done for a "normal" job file.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 19 Nov 2014 16:34:01 +0000 (09:34 -0700)]
server: fix missing ETA in some cases
If we get an ETA request from the client, we pass in 'force' to
ignore any time constraints. But we fail to pass the force through,
so we can fail sending the requested ETA and the client just looks
like nothing is running even though it is.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 19 Nov 2014 16:21:16 +0000 (09:21 -0700)]
trigger: always send trigger, not just when remote command is set
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 19 Nov 2014 16:09:01 +0000 (09:09 -0700)]
client: defer local trigger execute until after state is received
Don't run it immediately, we have to wait until we receive the
data from the server.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 19 Nov 2014 16:06:42 +0000 (09:06 -0700)]
HOWTO: add description and examples of verify triggers
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 19 Nov 2014 15:36:12 +0000 (08:36 -0700)]
trigger: enable separate remote and local trigger
Change the options so that we have the trigger file and command
separate, and remote and local trigger command separate as well.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 19 Nov 2014 03:14:23 +0000 (20:14 -0700)]
Kill off -Wshadow again
Too many warnings, even for system headers.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 19 Nov 2014 02:16:06 +0000 (19:16 -0700)]
verify: allow empty/no trigger file
If we use a trigger timeout, we don't need the file.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 12 Nov 2014 18:11:20 +0000 (11:11 -0700)]
Fix some shadow warnings for some gcc variants
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 12 Nov 2014 03:51:37 +0000 (20:51 -0700)]
verify: verify_state_gen_name() should check size
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 11 Nov 2014 18:10:35 +0000 (11:10 -0700)]
Merge branch 'verify-trigger'
Jens Axboe [Tue, 11 Nov 2014 18:09:59 +0000 (11:09 -0700)]
Fio 2.1.14
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 11 Nov 2014 17:24:00 +0000 (10:24 -0700)]
client: replace duplicate code with read_data()
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 11 Nov 2014 03:34:00 +0000 (20:34 -0700)]
Add support for verify triggers and verify state saving
This allows you to (for instance) instantly terminate a verify
write workload, and then later that everything was written
correctly up to that very point. This can be useful for testing
powercut scenarios, which is often problematic on storage
devices.
The trigger part is a file based notification scheme, similar
to what is provided for the status dumps. When triggered,
fio will exit immediately and write the verify state safely
to disk. A trigger can be accompanied by a trigger command.
Say you wanted to test powercut safety, the trigger could be
something that immediately cut power to the machine.
The verify state is either saved locally (if run locally),
or saved over the network if run in client/server mode.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 11 Nov 2014 16:39:11 +0000 (09:39 -0700)]
t/btrace2fio: don't dec inflight for requeue
We don't inc it again on issue, so we'll end up with an assert
trigger on inflight going negative.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 11 Nov 2014 15:09:04 +0000 (08:09 -0700)]
t/btrace2fio: fix divide-by-zero in rate calculation
If usec is less than 1000, it'd crash.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 11 Nov 2014 15:07:09 +0000 (08:07 -0700)]
Makefile: add -Wshadow
Signed-off-by: Jens Axboe <axboe@fb.com>
Stephen M. Cameron [Tue, 11 Nov 2014 03:31:26 +0000 (20:31 -0700)]
Fixup some of the time (usec) based conversions
Modified by Jens to fixup some of the mtime/utime confusion,
and add a specific ->is_time to options to get rid of future
issues in this area.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 10 Nov 2014 22:40:24 +0000 (15:40 -0700)]
Get rid if ddir_trim() macro
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 10 Nov 2014 21:08:50 +0000 (14:08 -0700)]
Fix crash on threads being reaped before they are created
If we do that, then ->rusage_sem is never allocated.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 10 Nov 2014 18:18:06 +0000 (11:18 -0700)]
Add option for statically build fio
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 10 Nov 2014 15:52:23 +0000 (08:52 -0700)]
t/btrace2fio: cap depth if we don't see completion traces
Signed-off-by: Jens Axboe <axboe@fb.com>
Vasily Tarasov [Mon, 10 Nov 2014 03:22:24 +0000 (20:22 -0700)]
Fix for a race when fio prints I/O statistics periodically
Below is the demonstration for the latest code in git:
Job-file:
[job]
filename=/dev/sda
rw=randread
blocksize=4096
direct=1
time_based=1
runtime=300
numjobs=16
Command line:
fio --output-format=json --status-interval=10 /tmp/job.file | tee /tmp/fio.out
After parsing json and printing per-job "read->runtime" and "read->iops" in two
columns, one can see this:
runtime iops
9854 4762
19861 4724
29867 4711
39873 4705
49880 4701
59886 4698
69894 4696
79900 4694
89907 4693
99913 4692
109919 4692
119926 4691
129933 4691
139940 4691
149946 4691
159953 4690
169959 4690
179966 4690
189972 4690
199978 4690
209984 4690
219990 4690
229996 4690
240002 4690
250009 4690
260015 4690
270021 4690
280027 4690
290033 4690
600043 2344 <--- You can see a spurious jump in runtime here
300001 4690
The problem is two-fold, IMHO:
1) td->ts.runtime[DDIR_READ/DDIR_WRITE/DDIR_TRIM] are set in microseconds
instead of milliseconds in thread_main() function. The values are later
converted to milliseconds (after exiting from the loop in thread_main()),
but it is too late for the out-of-band access by the helper_thread
(the one that prints periodic statistics).
2) The helper_thread in __show_running_run_stats() temporarily adds to
td->ts.runtime[DDIR_READ/DDIR_WRITE/DDIR_TRIM]
the time that has passed since the beginning of the loop (td->start).
However, the same addition happens in thread_main() by the job
thread and td->start is not reset atomically with this addition.
As a result, the runtime can be added twice. (That's why in the output
we see 600043, which is approximately twice larger than the
previous reading).
My proposal is to make the job threads updating both td->ts.runtime and
td->start atomically by using an already existing stat_mutex.
__show_running_run_stats() already takes this mutex where needed.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Sat, 8 Nov 2014 01:47:41 +0000 (18:47 -0700)]
client/server: percentile_precision wasn't net converted
We'd get 0 precision on the percentile dumps, which was confusing
and lost info.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 7 Nov 2014 22:41:24 +0000 (15:41 -0700)]
iolog: fix intermittent crash on exit with verify enabled
Commit
9342d5f807795 added flist_first_entry(), but the
conversion io iolog hist list pruning was wrong. Fix it.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 7 Nov 2014 17:36:33 +0000 (10:36 -0700)]
Add time_since_genesis()
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 6 Nov 2014 22:19:31 +0000 (15:19 -0700)]
smalloc: add zeroing scalloc() variant
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 6 Nov 2014 16:21:10 +0000 (09:21 -0700)]
Make fio -Wshadow clean
Found a few issues, actually.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 6 Nov 2014 01:39:23 +0000 (18:39 -0700)]
Get rid of __ prefix for internal frand state
It made (some) sense when we had both OS and frand in anon unions,
but now lets just kill it.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 6 Nov 2014 01:34:02 +0000 (18:34 -0700)]
Remove use of OS provided random functions
We added the internal random generator a long time ago, and kept
the OS variant around as an opt-in feature with using use_os_rand=1.
We defaulted to using the fio provided one, and I doubt that
anyone has used the option.
The time has come to kill it.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 5 Nov 2014 02:53:04 +0000 (19:53 -0700)]
Add basic DragonFly support
Saw the build failing in their logs, and it hurt my feelings.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 30 Oct 2014 23:16:32 +0000 (17:16 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Thu, 30 Oct 2014 16:13:33 +0000 (10:13 -0600)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Thu, 30 Oct 2014 16:11:32 +0000 (10:11 -0600)]
configure: disable lex/yacc on Solaris
It currently breaks, disable it until we can figure it out:
LEX lex.yy.c
"exp/expression-parser.l":line 166: Warning: Non-portable Character Class
YACC y.tab.c
CC lex.yy.o
exp/expression-parser.l:43:37: error: conflicting types for ‘yyunput’
lex.yy.c:31:7: note: previous declaration of ‘yyunput’ was here
exp/expression-parser.l:44:46: error: macro "input" passed 1 arguments, but takes just 0
exp/expression-parser.l: In function ‘yylex’:
exp/expression-parser.l:61:1: warning: label ‘yyfussy’ defined but not used
exp/expression-parser.l:54:22: warning: unused variable ‘yyprevious’
exp/expression-parser.l: At top level:
exp/expression-parser.l:297:1: warning: missing braces around initializer
exp/expression-parser.l:297:1: warning: (near initialization for ‘yycrank[0]’)
exp/expression-parser.l:376:1: warning: missing braces around initializer
exp/expression-parser.l:376:1: warning: (near initialization for ‘yysvec[0]’)
exp/expression-parser.l:714:6: error: conflicting types for ‘yyunput’
exp/expression-parser.l:43:37: note: previous declaration of ‘yyunput’ was here
make: *** [lex.yy.o] Error 1
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Thu, 30 Oct 2014 14:40:47 +0000 (08:40 -0600)]
mmap: set 'fd' to -1
This looks like a typo, using '1' makes no sense. This fixes Solaris
anon allocs.
Reported-by: Sébastien Bouchex Bellomié <sbouchex@infovista.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 29 Oct 2014 14:30:07 +0000 (08:30 -0600)]
engines: rename option specific pad from 'td' to 'pad'
Except the net engine, they all just use it as padding to avoid
having a zero offset member. The latter makes the parser think
the offset isn't set, so not valid.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 29 Oct 2014 14:26:28 +0000 (08:26 -0600)]
engines/rbd: sort IO by start time before waiting on it
Instead of waiting on the first random IO we find, wait on the
oldest one. For higher queue depth, this should yield better
performance and lower latencies.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 29 Oct 2014 14:23:46 +0000 (08:23 -0600)]
Merge branch 'optimize-busy-poll' of https://github.com/ketor/fio
Ketor writes:
Origin code use rbd_aio_is_complete to check if the io is complete.But
this function use mutex lock and unlock to get the complete result, this
will cause the thread context switch and lock wait latency. And also
because the lock and context switch latency is in the getevents thread,
will decrease the getevents speed and increase the test latency.
This patch go back to use an varible io_complete to check if the io is
complete. This variable is updated in the aiocb callback function and
will not influence the getevents speed.
In our practice, we run fio test jobs=1 iodepth=1 bs=4k busy_poll=1.
Before this patch: IOPS is 30930 After this patch: IOPS is 37300. And
also after this patch we get avg 25us latency, is nearly equal to the
systemtap test result of the librbd randwrite process.
So I think this patch is realy needed.
Ketor Meng [Wed, 29 Oct 2014 15:57:02 +0000 (15:57 +0000)]
use io_complete var to avoid the rbd_aio_is_complete lock and context switch
Signed-off-by: Ketor Meng <d.ketor@gmail.com>
Ketor Meng [Wed, 29 Oct 2014 15:09:57 +0000 (15:09 +0000)]
fix rbd_option parse error
Signed-off-by: Ketor Meng <d.ketor@gmail.com>
Jens Axboe [Wed, 29 Oct 2014 14:22:45 +0000 (08:22 -0600)]
Merge branch 'fix-rbd-options-parse' of https://github.com/ketor/fio
Ketor writes:
There is two bug in the rbd_options parse:
1. The memeber struct thread_data *td of struct rbd_options is filled in
the cmdline parse process.So it is usable and cannot be deleted.
2.The busy_poll offset of struct rbd_options is not the "busy_poll" but
"client_name"
After fix these two bugs, the fio test then can run.
Jens Axboe [Wed, 29 Oct 2014 02:49:56 +0000 (20:49 -0600)]
engines/glusterfs_async: remove no-op prep function
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 29 Oct 2014 02:48:33 +0000 (20:48 -0600)]
engines/glusterfs_async: cleanups
- Style issues
- Unnecessary casts
- Unnecessary variables inits
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Wed, 29 Oct 2014 02:40:21 +0000 (20:40 -0600)]
engines/glfs_async: add trim support
If available.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 28 Oct 2014 16:02:22 +0000 (10:02 -0600)]
engines/rbd: add option to busy poll on event completion
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 28 Oct 2014 15:19:55 +0000 (09:19 -0600)]
engines/rbd: remove unused rbd_options->td
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 28 Oct 2014 15:03:11 +0000 (09:03 -0600)]
engines/rbd: use rbd_aio_is_complete() and add TRIM support
Using fri->io_complete is potentially racy, since we have no
locking around checking and waiting. Use the API provided
function to check for completion instead.
Add support for TRIM through rbd_aio_discard().
Cleanup the queue interface and completion creation.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Tue, 28 Oct 2014 15:00:06 +0000 (09:00 -0600)]
engines/rbd: add support for rbd_invalidate_cache()
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Mon, 27 Oct 2014 16:53:52 +0000 (10:53 -0600)]
rbd: various fixes and cleanups
- Don't use usleep() when no completions are available, call into
rbd to wait for one.
- Check for errors on reads/writes/syncs instead of just assuming
that everything went fine.
- Fix leak of completions when read/write/sync fails.
- Style cleanups, and remove unnecessary casts.
Signed-off-by: Jens Axboe <axboe@fb.com>
Gwendal Grignou [Sat, 25 Oct 2014 23:04:32 +0000 (16:04 -0700)]
fio: Fix padding properly
Completely fix padding:
- use anonymous union for padding.
- move existing padding in thread_stat.
- add alignment checks.
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
Brian Fulton [Fri, 24 Oct 2014 20:47:34 +0000 (14:47 -0600)]
Loop Iteration Offset Bug
Reset file descriptor offset value to the initial value when file is
closed and io state is cleared. This fixes the bug where previous offset
values persist between loop iterations.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 24 Oct 2014 20:47:02 +0000 (14:47 -0600)]
Fix fp alignment
Commit
a5bbbd8a324a fixed alignment earlier up, but then screwed
it up further down. Fix it.
Signed-off-by: Jens Axboe <axboe@fb.com>
Gwendal Grignou [Fri, 24 Oct 2014 20:42:37 +0000 (13:42 -0700)]
fio: fix alignement to prevent bus error on ARM
Add a filler field to be ensure 64bit alignment.
Otherwise, we would trigger SIGBUS error in sum_stat()
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 24 Oct 2014 05:47:01 +0000 (23:47 -0600)]
diskutil: get rid of disk_util_start_exit()
No need to wrap this in a function, just set the exit variable.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 24 Oct 2014 05:16:50 +0000 (23:16 -0600)]
Consolidate disk util, eta, and status check thread
We don't need two, we can just have one do everything.
Signed-off-by: Jens Axboe <axboe@fb.com>
Jens Axboe [Fri, 24 Oct 2014 05:04:37 +0000 (23:04 -0600)]
stat: avoid hang with race in stat thread exit and rusage update
Signed-off-by: Jens Axboe <axboe@fb.com>