fio: Introduce new constant thinkcycles option The thinkcycles parameter allows to set a number of cycles to spin between requests to model real-world applications more realistically The thinktime parameter family can be used to model an application processing the data to be able to model real-world applications more closely. Unfortunately this is currently set per constant time and therefore is affected by CPU frequency settings or task migration to a CPU with different capacity. The new thinkcycles parameter closes that gap and allows specifying a constant number of cycles instead, such that CPU capacity is taken into account. Signed-off-by: Christian Loehle <christian.loehle@arm.com>
Record job start time to fix time pain points Add a new key in the json per-job output, job_start, that records the job start time obtained via a call to clock_gettime using the clock_id specified by the new job_start_clock_id option. This allows times of fio jobs and log entries to be compared/ordered against each other and against other system events recorded against the same clock_id. Add a note to the documentation for group_reporting about how there are several per-job values for which only the first job's value is recorded in the json output format when group_reporting is enabled. Fixes #1544 Signed-off-by: Nick Neumann nick@pcpartpicker.com
Support for alternate epochs in fio log files Add options log_alternate_epoch and log_alternate_epoch_clock_id. This is similar to the log_unix_epoch option. This resolves the issue raised in Issue #1314 log_alternate_epoch, if true, causes log files to use the same epoch used used by the clock_id parameter to the unix clock_gettime function, where clock_id is specified by the log_alternate_epoch_clock_id option. This is particularly useful as it allows us to specify a clock id like CLOCK_MONOTONIC_RAW, which is natural for synchronizing log files between processes. The current log_unix_epoch is problematic for that purpose because that clock is not monotonic or continuous. It turns out that log_unix_epoch is actually equivalent to log_alternate_epoch with log_alternate_epoch_clock_id set to CLOCK_REALTIME=0. Since this is the default value of the log_alternate_epoch_clock_id option anyways, we treat log_alternate_epoch and log_unix_epoch as equivalent in functionality, retaining the latter to avoid breaking existing clients. Signed-off-by: Nick Neumann <nick@pcpartpicker.com>
fio: reset more counters when ramp time has elapsed Currently, fio --name=test --ramp_time=2 --time_based=1 --runtime=2s --size=1M will produce an output line like IO depths : 1=184.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% 184.7% appears because the denominator used in the IO depth calculation is reset at the conclusion of ramp_time but the numerator is not. Latency and other IO depth statistics also include data from IOs completed during ramp time. This patch resets these counters when ramp time is over. Tested-by: Sebastien Boisvert <sboisvert@gydle.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
time: ensure that offload mode switches parent out of ramp If we have worker threads submitting IO for the jobs AND using ramp time, then we need to ensure that the parent job is switched out of ramp mode. This normally happens as part of IO submission, but since we have worker threads doing that for us, it never happens for the parent. Fixes: https://github.com/axboe/fio/issues/459 Signed-off-by: Jens Axboe <axboe@kernel.dk>
time: fix overflow in timespec_add_msec Commit 8b6a404cdd2c40715885e562416c3db039912773 ("nanosecond: initial commit changing timeval to timespec") introduced a change to timespec_add_msec that decreased the time to (unsigned) overflow of adj_nsec leading to strange values in the write_*_log files when using log_avg_msec. The following fio job reproduces the problem: ./fio --ioengine=null --size=2G --time_based --runtime=11s \ --ramp_time=5s --write_iops_log=overflow --log_avg_msec=5000 \ --name=overflow cat overflow_iops.1.log shows the following: 5000, 997853, 0, 0 10640, 141323, 0, 0 Compiling fio with clang's undefined behaviour sanitizer with unsigned wraparound detection turned on by using CC=clang ./configure --extra-cflags="-fsanitize=undefined,integer" and running the previous job produces a lot of false positives but also flags up this: time.c:11:35: runtime error: unsigned integer overflow: 1000000 * 5000 cannot be represented in type 'unsigned int' Fix this by forcing the type of the integer literal to ensure that a 64 bit multiplication takes place. Further, force the result variables to be uint64_t allowing us to remove UL from other literals. Finally adjust some spacing while we're here. Fixes https://github.com/axboe/fio/issues/426 (which was also reported on the mailing list - http://www.spinics.net/lists/fio/msg06190.html ). Reported-by: Joseph R Gruher <joseph.r.gruher@intel.com> Suggested-by: Vincent Fu <vincent.fu@wdc.com> Suggested-by: Jeff Furlong <jeff.furlong@wdc.com> Reviewed-by: Vincent Fu <vincent.fu@wdc.com> Tested-by: Joseph R Gruher <joseph.r.gruher@intel.com> Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
nanosecond: initial commit changing timeval to timespec struct timeval has usec resolution whereas struct timespec has nsec resolution. This commit changes several dozen instances of struct timeval to struct timespec. All of the output is unchanged. The ticks to nsec conversion in this patch is broken but fixed in a later patch.
time: fix overflow in timeval_add_msec() There is an issue in FIO when using the detailed bandwidth and iops logging with averaging over a period of time > 1 second. It seems that usecs overflows which later causes negative time diff values resulting in skewed toward 0 results. I have attached a potential fix that should prevent usecs from going beyond 1000000. [Modified by Jens to put the adj_sec in the branch.] Signed-off-by: Jens Axboe <axboe@fb.com>
Fix ramp time breakage A previous commit inadvertently used ramp_time_over() instead of in_ramp_time(), which breaks some jobs with ramp time. This is because the former function has side effects, where we really just want to check if we're in ramp time or not. Fixes: a47591e4923f ("Improve logging accuracy") Signed-off-by: Jens Axboe <axboe@fb.com>
Improve logging accuracy If we do windowed logging (over a period of time), then we can take the logging out of the IO worker fast path, and just have our helper thread do it. This means we'll reduce the overhead for the IO worker (which is important), and that we'll be able to provide better accuracy of when we log. Before this change, we'd log when we had exceeded the window of time at a completion even. This means that if you had a logging time of 500 msec, you'd always exceed that and even tend to skew it to being much later. With this change, logging should be very accurate time wise: $ ./fio --inflate-log=fsrandW4-8-iopslog_iops.1.log.fz | head -n10 500, 470, 0, 0 1000, 492, 0, 0 1500, 484, 0, 0 2000, 492, 0, 0 2500, 454, 0, 0 3000, 462, 0, 0 3500, 480, 0, 0 4000, 462, 0, 0 4500, 472, 0, 0 5000, 496, 0, 0 and not exhibit any drift at all. Signed-off-by: Jens Axboe <axboe@fb.com>
Improve rate limiting If we have pending IO, we should commit it before going to sleep, not just wait for IO that has already been issued. Also improve the delay functions to return the time spent, so that users don't have to track tha separately if they care about the precision. Signed-off-by: Jens Axboe <axboe@fb.com>
fio: allow milliseconds on all time specifiers This patch allows all time specifiers to be specified down to milliseconds. Default will stay seconds for compatibility with old configs. It also adds documentation to the existing time units day, hour and minute. Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com> Signed-off-by: Jens Axboe <axboe@fb.com>