stat: report clat stats on a per priority granularity

stat: report clat stats on a per priority granularity

Convert the stat code to report clat stats on a per priority granularity,
rather than simply supporting high/low priority.

This is made possible by using the new clat_prio_stat array (per ddir),
together with the clat_prio_stat index which is saved in each io_u.

The per priority samples are only printed when there are samples for more
than one priority in the clat_prio_stat array. If there are only samples
for one priority, that means that all I/Os where submitted using the same
priority, so no need to print.

For example, running the following fio command:
fio --name=test --filename=/dev/sdc --direct=1 --runtime=60 --rw=randread \
    --ioengine=io_uring --ioscheduler=mq-deadline --iodepth=32 --bs=32k \
    --prioclass=2 --prio=7 --cmdprio_bssplit=32k/20/3/0:32k/10/1/4

Now results in the following output:
test: (groupid=0, jobs=1): err= 0: pid=465655: Tue Feb  1 02:24:47 2022
  read: IOPS=146, BW=4695KiB/s (4808kB/s)(276MiB/60239msec)
    slat (usec): min=18, max=335, avg=62.87, stdev=22.59
    clat (msec): min=2, max=2135, avg=217.97, stdev=287.26
     lat (msec): min=2, max=2135, avg=218.03, stdev=287.26
    clat prio 2/7 (msec): min=3, max=606, avg=106.57, stdev=86.64
    clat prio 3/0 (msec): min=10, max=2135, avg=664.94, stdev=339.42
    clat prio 1/4 (msec): min=2, max=300, avg=52.29, stdev=42.52
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[   14], 10.00th=[   19], 20.00th=[   33],
     | 30.00th=[   52], 40.00th=[   77], 50.00th=[  108], 60.00th=[  144],
     | 70.00th=[  192], 80.00th=[  300], 90.00th=[  684], 95.00th=[  911],
     | 99.00th=[ 1234], 99.50th=[ 1318], 99.90th=[ 1687], 99.95th=[ 1770],
     | 99.99th=[ 2140]
    clat prio 2/7 (69.25% of IOs) percentiles (msec):
     |  1.00th=[    7],  5.00th=[   13], 10.00th=[   17], 20.00th=[   28],
     | 30.00th=[   44], 40.00th=[   64], 50.00th=[   85], 60.00th=[  111],
     | 70.00th=[  140], 80.00th=[  174], 90.00th=[  226], 95.00th=[  279],
     | 99.00th=[  368], 99.50th=[  418], 99.90th=[  502], 99.95th=[  567],
     | 99.99th=[  609]
    clat prio 3/0 (20.91% of IOs) percentiles (msec):
     |  1.00th=[   44],  5.00th=[  138], 10.00th=[  205], 20.00th=[  347],
     | 30.00th=[  464], 40.00th=[  558], 50.00th=[  659], 60.00th=[  760],
     | 70.00th=[  860], 80.00th=[  961], 90.00th=[ 1099], 95.00th=[ 1217],
     | 99.00th=[ 1485], 99.50th=[ 1687], 99.90th=[ 1871], 99.95th=[ 2140],
     | 99.99th=[ 2140]
    clat prio 1/4 (9.84% of IOs) percentiles (msec):
     |  1.00th=[    7],  5.00th=[   10], 10.00th=[   13], 20.00th=[   18],
     | 30.00th=[   24], 40.00th=[   30], 50.00th=[   39], 60.00th=[   51],
     | 70.00th=[   63], 80.00th=[   84], 90.00th=[  114], 95.00th=[  136],
     | 99.00th=[  188], 99.50th=[  197], 99.90th=[  300], 99.95th=[  300],
     | 99.99th=[  300]
   bw (  KiB/s): min= 3456, max= 5888, per=100.00%, avg=4697.60, stdev=472.38, samples=120
   iops        : min=  108, max=  184, avg=146.80, stdev=14.76, samples=120
  lat (msec)   : 4=0.11%, 10=2.57%, 20=8.67%, 50=18.21%, 100=18.34%
  lat (msec)   : 250=28.87%, 500=9.41%, 750=5.22%, 1000=5.09%, 2000=3.50%
  lat (msec)   : >=2000=0.01%
  cpu          : usr=0.16%, sys=0.97%, ctx=17715, majf=0, minf=262
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=8839,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Link: https://lore.kernel.org/r/20220203192814.18552-15-Niklas.Cassel@wdc.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>