From: Karl Cronburg <kcronbur@redhat.com>
Date: Fri, 5 Aug 2016 18:45:00 +0000 (-0400)
Subject: This commit / feature adds completion latency histogram output to fio, piggybacking
X-Git-Tag: fio-2.14~57^2
X-Git-Url: https://git.kernel.dk/?p=fio.git;a=commitdiff_plain;h=1e613c9c23932006263dd8334007865f32891a0c;hp=1e613c9c23932006263dd8334007865f32891a0c

This commit / feature adds completion latency histogram output to fio, piggybacking
on the existing histograms recorded by stat.c and adding the following command
line options:

- log_hist_msec operates like log_avg_msec does, but produces files
  named *_clat_hist*
- write_hist_log has the same semantics as write_lat_log but for histograms.
- log_hist_coarseness changes how many bins are logged.

The more substantial contribution is a post-processor for digesting these log files,
located in fio/tools/hist/fiologparser_hist.py, which computes min/max/avg/percentile
statistics.

To compare the accuracy of the statistics produced by fiologparser_hist.py,
a reference implementation is located here:

https://github.com/cronburg/fio/blob/histograms/tools/hist/fiologparser_numpy.py

for computing the same statistics from the *_clat.* files fio already produces.
A comparison can be viewed here:

https://cronburg.com/fio/demo/latency.html

The reference implementation is excluded from this commit, as I'm still working
with Mark (markhpc) on getting it into the branch where he's working on fiologparser.
Also I'd eventually like to make FIO_IO_U_PLAT_GROUP_NR in stat.h
configurable (allowing for a larger maximum histogram latency value than 17
seconds), but haven't added it to keep the changes to fio minimal for now.

A more detailed description of the motivation behind this feature, and explanation
of the design process and algorithms used in the post-processor can be found here:

https://cronburg.com/fio/cloud-latency-problem-measurement/
---