tools/accounting/delaytop: add delaytop to record top-n task delay
Problem
=======
The "getdelays" can only display the latency of a single task
by specifying a PID, but it has the following limitations:
1. single-task perspective: only supports querying the latency (CPU, I/O,
memory, etc.) of an individual task via PID and cannot provide a global
analysis of high-latency processes across the system.
2. lack of High-Latency process awareness: when the overall system
latency is high (e.g., a spike in CPU latency), there is no way to
quickly identify the top N processes contributing to the highest
latency.
3. poor interactivity: It lacks dynamic sorting and refresh
capabilities (similar to top), making it difficult to monitor latency
changes in real time.
Solution
========
To address these limitations, we introduce the "delaytop" with the
following capabilities:
1. system view: monitors latency metrics (CPU, I/O, memory, IRQ, etc.)
for all system processes
2. supports field-based sorting (e.g., default sort by CPU latency in
descending order)
3. dynamic interactive interface: focus on specific processes with
--pid; limit displayed entries with --processes 20; control monitoring
duration with --iterations;
Use case
========
bash# ./delaytop
Top 20 processes (sorted by CPU delay):
PID TGID COMMAND CPU(ms) IO(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) IRQ(ms)
---------------------------------------------------------------------------------------------
26 26 kworker/1:0H 5.55 0.00 0.00 0.00 0.00 0.00 0.00 0.00
32 32 kworker/2:0H-kb 2.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00
38 38 kworker/3:0H-ev 2.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00
84 84 kworker/R-vfio- 1.62 0.00 0.00 0.00 0.00 0.00 0.00 0.00
24 24 ksoftirqd/1 1.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19 19 idle_inject/0 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00
16 16 rcu_exp_par_gp_ 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11 11 kworker/0:1 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00
22 22 idle_inject/1 0.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00
3 3 pool_workqueue_ 0.74 0.00 0.00 0.00 0.00 0.00 0.00 0.00
81 81 scsi_eh_1 0.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00
30 30 ksoftirqd/2 0.42 0.00 0.00 0.00 0.00 0.00 0.00 0.00
36 36 ksoftirqd/3 0.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
9 9 kworker/0:0-eve 0.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
8 8 kworker/R-netns 0.34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
76 76 kworker/1:1-pm 0.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
21 21 cpuhp/1 0.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
4 4 kworker/R-rcu_g 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12 12 kworker/u16:0-i 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1 1 init 0.18 0.00 0.00 0.00 0.00 0.00 0.08 0.00
Link: https://lkml.kernel.org/r/20250619211843633h05gWrBDMFkEH6xAVm_5y@zte.com.cn
Co-developed-by: Fan Yu <fan.yu9@zte.com.cn>
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Signed-off-by: Yaxin Wang <wang.yaxin@zte.com.cn>
Cc: Balbir Singh <bsingharora@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Peilin He <he.peilin@zte.com.cn>
Cc: Qiang Tu <tu.qiang35@zte.com.cn>
Cc: wangyong <wang.yong12@zte.com.cn>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Yang Yang <yang.yang29@zte.com.cn>
Cc: ye xingchen <ye.xingchen@zte.com.cn>
Cc: Yunkai Zhang <zhang.yunkai@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>