summaryrefslogtreecommitdiff
path: root/stat.h
AgeCommit message (Collapse)Author
2020-02-04fio: better distinguish between high and low priorityVincent Fu
Change the variable names for the low priority latency variables to explicity distinguish them from the high priority latency variables. Also rename the JSON objects to do the same. This reduces the mental effort required to read the code and output. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-04fio: report percentiles for slat, clat, latVincent Fu
Currently, fio can report percentiles for either completion latencies or total latencies. This patch adds the ability to simultaneously report percentiles for submission, completion, and total latencies. - simplify the JSON lat percentile reporting code - parsing the JSON output will be unaffected, although now setting lat_percentiles=1 no longer suppresses clat percentiles - this may break parsing of the normal output since more latency percentile blocks may be displayed - to maintain backward compatibility in the terse output, total latencies are reported if lat_percentiles=1 even when clat percentiles have also been computed - change the gtod_reduce callback to only change option values when gtod_reduce=1 - for the new cmdprio_percentage latencies, if lat_percentiles=1, *total* latency percentiles will be tracked. Otherwise, *completion* latency percentiles will be tracked. - also make sure we skip the high/low priority latency code if we are producing output for sync latencies. - make add_slat_sample use an unsigned long long for the time to be consistent with other functions Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-04fio: groundwork for adding slat, lat percentilesVincent Fu
Expand the data structure used for tabulating completion latency percentiles so that we will have the ability to tabulate percentiles for submission and total latency as well. This should not change fio's behavior. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-01-22stat: ensure we align correctlyJens Axboe
Fixes: b2a432bfbb6d ("Per-command priority: Priority logging and libaio/io_uring cmdprio_percentage") Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-01-22Per-command priority: Priority logging and libaio/io_uring cmdprio_percentagePhillip Chen
Add cmdprio_percentage option to libaio and io_uring engines to set ioprio on a per-command basis. Add tracking of high priority commands to be displayed separately in human readable and JSON outputs.
2020-01-15Moved diskutil reporting functions to stat.cLeaflessMelospiza
2019-08-15stat: ensure that struct jobs_eta packs nicelyJens Axboe
It has two holes in it, and some weird mid-struct packing. Let's clean it up. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-08-14Verify the absence of holes in struct jobs_eta at compile timeBart Van Assche
This patch verifies the correctness of the previous patch. Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-08-14Refine packed annotations in stat.hBart Van Assche
Instead of declaring the whole structure packed, only declare non-aligned members packed. This patch is an alternative way to fix the following gcc 9 compiler warnings: eta.c: In function 'calc_thread_status': eta.c:510:7: error: taking address of packed member of 'struct jobs_eta' may result in an unaligned pointer value [-Werror=address-of-packed-member] 510 | je->rate); | ~~^~~~~~ eta.c:522:66: error: taking address of packed member of 'struct jobs_eta' may result in an unaligned pointer value [-Werror=address-of-packed-member] 522 | calc_rate(unified_rw_rep, disp_time, io_bytes, disp_io_bytes, je->rate); | ~~^~~~~~ eta.c:523:64: error: taking address of packed member of 'struct jobs_eta' may result in an unaligned pointer value [-Werror=address-of-packed-member] 523 | calc_iops(unified_rw_rep, disp_time, io_iops, disp_io_iops, je->iops); | Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-12-14Add cache hit statsJens Axboe
With the aioring engine, we can get notified if a buffered read was a cache hit or if it hit media. Add that to the output stats for normal and json output. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-10-04Bool conversionsJens Axboe
Just various low hanging fruits in terms of local or global variables that are only 0/1, switch them to boolean. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-08-24Collect and show zone reset statisticsBart Van Assche
Show how many zone resets have been performed in the I/O statistics. An example: [ ... ] write: IOPS=17.0k, BW=66.5MiB/s (69.8MB/s)(1024MiB/15387msec); 4 zone resets [ ... ] Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-07-23Add support for >= 4G block sizesJeff Furlong
For trims, it's useful to be able to support larger than (or equal to) 4GB. This extends the block sizes support for that. Change from Jeff, various little fixups from me. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-06-05Remove show_run_stats() because it has no callersBart Van Assche
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2018-03-15Rename fio_mutex into fio_semBart Van Assche
Since fio_mutex implements a semaphore, rename fio_mutex into fio_sem. Rename those fio_sem instances that have 'mutex' embedded in their name. Rename the mutex.h and .c source files into fio_sem.h and .c respectively. Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2018-03-01Fix overflow of counters incremented on each I/O operationAlexander Larin
- In the thread_stat struct: uint32_t counters are updated to uint64_t. - In the io_u_plat_entry struct: unsigned int counters are updated to uint64_t. It fixes overflow of these counters. Signed-off-by: Alexander Larin <znahar@yandex-team.ru>
2018-01-25stat: ensure that we align ts->sync_stat appropriatelyJens Axboe
clang correctly complains that we don't know if the alignment is correct when the structure is packed, so ensure that we align it to an 8-byte boundary. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-01-25Track fsync/fdatasync/sync_file_range issue countJens Axboe
Also fix an issue where we didn't initialize the sync minimum latency correctly, so it was always reported as 0. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-01-24Add suppor for logging fsync (and friends) latenciesJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-11-22add significant_figures parameterJeff Furlong
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-12Fix more overflows in percentile calculation for WindowsAndrzej Jakowski
The previous patch caught one place where we could potentially overflow for large number of IOs, this patches up a few more. Signed-off-by: Andrzej Jakowski <andrzej.jakowski@intel.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-09stat: update description of clat accounting in stat.hVincent Fu
The description in stat.h of the data structure used for storing completion latency percentiles had not been updated to reflect the transition to nanosecond timing. This patch updates the description to be consistent with the nanosecond changes. There have been a few instances on github and the mailing list where this had caused confusion. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-09-13Add support for doing total latency percentilesJens Axboe
By default, fio does completion latency percentiles. Sometimes what you want is total IO latency percentiles, including the submission part as well. If that's your thing, then set lat_percentiles=1 and get that instead of the default completion latency percentiles. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-06-22Fixup some style issuesJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-06-21nanosecond: alter gfio to accommodate nanosecond timingVincent Fu
Some of the plot labels need to be changed because they might have values with nsec units.
2017-06-21nanosecond: update completion latency recording and normal, json output to ↵Vincent Fu
use nanoseconds A new set of nsec bins (0-1, 2-3, 4-9, 10-19, ...) is created for tracking latencies. The number of io_u_plat[] latency bins is increased by about 50% to accommodate the greater resolution while retaining the same max latency value of ~9 seconds. Example output test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1 fio-2.20-39-g83b3 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=14.6GiB/s,w=0KiB/s][r=3832k,w=0 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=17621: Thu Jun 1 15:09:38 2017 read: IOPS=3784k, BW=14.4GiB/s (15.5GB/s)(144GiB/10000msec) clat (nsec): min=15, max=1685.2k, avg=26.92, stdev=433.62 lat (nsec): min=46, max=5989.9k, avg=67.31, stdev=1659.58 clat percentiles (nsec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 26], | 70.00th=[ 26], 80.00th=[ 27], 90.00th=[ 36], 95.00th=[ 41], | 99.00th=[ 46], 99.50th=[ 50], 99.90th=[ 104], 99.95th=[ 139], | 99.99th=[ 7840] bw ( MiB/s): min=13393, max=15647, per=0.10%, avg=14784.86, stdev=558.40 lat (nsec) : 20=20.87%, 50=78.57%, 100=0.43%, 250=0.10%, 500=0.01% lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=99.72%, sys=0.12%, ctx=321, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=37840231,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=14.4GiB/s (15.5GB/s), 14.4GiB/s-14.4GiB/s (15.5GB/s-15.5GB/s), io=144GiB (155GB), run=10000-10000msec
2017-05-22fio: fix some struct alignment issuesSitsofe Wheeler
Fix unaligned/misaligned accesses related to struct thread_stat and struct jobs_eta seen when running a build produced by CC=~/clang-3.9/build/bin/clang ./configure --disable-optimizations \ --extra-cflags="-D__compiler_offsetof=__builtin_offsetof \ -fsanitize=undefined" and add to the compile time asserts to make these problems more visible. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2017-01-02Convert group_run_stats to use bytes instead of KiB/KBRobert Elliott
In group_run_stats, convert these fields: * io_kb (bytes transferred) * agg (aggregate average bandwidth) * min_bw * max_bw from tracking either KiB or KB depending on kb_base to simply tracking bytes. Rename io_kb to iobytes to match this new meaning. This will simplify computing quantities and bandwidth rates with both IEC binary and SI decimal unit prefixes (e.g., GiB/s and GB/s) in future patches. Signed-off-by: Jens Axboe <axboe@fb.com>
2016-12-13Merge git://git.kernel.dk/fio into steady-stateVincent Fu
2016-12-07steadystate: fix alignment in stat.hVincent Fu
This was not problematic in my Linux and OSX testing, but Windows clients did not successfully receive thread_stat without this padding.
2016-12-07steadystate: ensure that pointers in thread_stat occupy the same amount of ↵Vincent Fu
storage in 32- and 64-bit platforms
2016-12-07steadystate: instead of including ss_sum_y in thread_stat record whether ↵Vincent Fu
ss_sum_y is nonzero in ss_state via __FIO_SS_BUFFER_FULL
2016-12-07steadystate: implement transmission of steadystate data over the wire in ↵Vincent Fu
client/server mode 1) Pare down thread_stat steadystate members to only those that need to go over the wire 2) Plumb up sending/receiving of steadystate data including ring buffers
2016-11-27stat: Change access to io_sample unionSitsofe Wheeler
Compiling on certain platforms produces warnings like stat.c: In function ‘add_clat_sample’: stat.c:2241:28: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] because the size of a pointer is different to that of a long. Work around this by explicitly naming the union used in io_sample, forcing the "val" in the union to be a uint64_t, changing function parameters to accept a union (rather than an unsigned long) and adding helper functions to wrap values into a union. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2016-11-27Silence compiler warningsSitsofe Wheeer
- Mark __parse_jobs_ini and fmt_desc[] as static as they aren't used outside their modules. - Cast the shifted trim state enum to an unsigned int to silence spurious Wshift-sign-overflow warning. Signed-off-by: Sitsofe Wheeer <sitsofe@yahoo.com>
2016-11-27fio: Fix (unsigned) integer overflow issuesSitsofe Wheeler
This patch tries to address issues found when compiling fio with clang's undefined behaviour sanitizer on 32 bit Linux. While this generated many false positives it did identify a number of subtle issues when rates/amounts/time calculations went above 32 bit values. The following job was used: [global] time_based runtime=180m bs=256k rw=randrw norandommap rate=512M ioengine=null direct=1 iodepth=8 thread numjobs=10 ramp_time=20m [null] size=1P It should fix https://github.com/axboe/fio/issues/270 because various 32 bit ETA variables were overflowing due to the microsecond arithmetic being done on them (even on 64 bit systems). This patch should also fix https://github.com/axboe/fio/issues/194 and https://github.com/axboe/fio/issues/266 because 64 bit Windows is a LLP64 platform so ints and longs still default to 32 bits. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2016-08-16Merge branch 'histogram-delta' of https://github.com/cronburg/fio into histogramJens Axboe
2016-08-16Make histogram samples non-cumulative by tracking a linked-listKarl Cronburg
of the most recent histogram and differencing it when we print to the log file(s). Linked list of pointers used to minimize runtime impact on recording side, instead choosing to do subtraction on the logging (when logs get printed to file) side. This helps cleanup fiologparser_hist.py, makes the log files easier to understand at a glance, reduces file size(s), and averts the possibility of integer overflow in the logs (for a long enough fio job).
2016-08-15Various cleanupsJens Axboe
- Killing unused declarations - Bool - Making code static Signed-off-by: Jens Axboe <axboe@fb.com>
2016-08-15Allow fio to terminate jobs when steady state is attainedVincent Fu
KNOWN ISSUES Will not work over a network connection
2016-05-06Improve logging accuracyJens Axboe
If we do windowed logging (over a period of time), then we can take the logging out of the IO worker fast path, and just have our helper thread do it. This means we'll reduce the overhead for the IO worker (which is important), and that we'll be able to provide better accuracy of when we log. Before this change, we'd log when we had exceeded the window of time at a completion even. This means that if you had a logging time of 500 msec, you'd always exceed that and even tend to skew it to being much later. With this change, logging should be very accurate time wise: $ ./fio --inflate-log=fsrandW4-8-iopslog_iops.1.log.fz | head -n10 500, 470, 0, 0 1000, 492, 0, 0 1500, 484, 0, 0 2000, 492, 0, 0 2500, 454, 0, 0 3000, 462, 0, 0 3500, 480, 0, 0 4000, 462, 0, 0 4500, 472, 0, 0 5000, 496, 0, 0 and not exhibit any drift at all. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-12-15client/server: pass back job options to clientJens Axboe
Also fill in a few more missing pieces from the server based json output, like time stamp. Now the json output should match the local, on all accounts. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-12-09eta: use bool for forcing jobs_eta returnJens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-12-07Fix stat summing for unified_rw_reportingJens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-11-30stat: move header declarations from iolog.hJens Axboe
The functions are in stat.c, move the to the right header. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-11-30stat: use DDIR_RWDIR_CNT instead of hardwired '3'Jens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-11-30Clear rusage stat when resetting all statsJens Axboe
This fixes an issue with bad rusage reporting, particularly when the runtime to ramptime ratio is low. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-10-27Enable the use of multiple output formatsJens Axboe
You can now do ./fio --output-format=normal,json jobfile.fio and get both types of output. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-04-20Don't export state variableDan Ehrenberg
There was no reason to export this extra variable. It was just a typo and it is never used. Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org> Signed-off-by: Jens Axboe <axboe@fb.com>
2015-04-14Collect a block-wise histogram of trim and write errorsDan Ehrenberg
This patch collects information on how many trims or writes each trim block sized unit can undergo before its first error. It also records the way in which an error occurred, if one did. Output is provided with a histogram which can be output either in normal or json output. Terse output is omitted, since it seemed like that would require a version rev bump. Signed-off-by: Dan Ehrenberg <dehrenberg@chromium.org> Signed-off-by: Jens Axboe <axboe@fb.com>