Commit | Line | Data |
---|---|---|
d6a3b247 | 1 | ===================== |
88ebc08e BR |
2 | CFS Bandwidth Control |
3 | ===================== | |
4 | ||
5 | [ This document only discusses CPU bandwidth control for SCHED_NORMAL. | |
d6a3b247 | 6 | The SCHED_RT case is covered in Documentation/scheduler/sched-rt-group.rst ] |
88ebc08e BR |
7 | |
8 | CFS bandwidth control is a CONFIG_FAIR_GROUP_SCHED extension which allows the | |
9 | specification of the maximum CPU bandwidth available to a group or hierarchy. | |
10 | ||
11 | The bandwidth allowed for a group is specified using a quota and period. Within | |
12 | each given "period" (microseconds), a group is allowed to consume only up to | |
13 | "quota" microseconds of CPU time. When the CPU bandwidth consumption of a | |
14 | group exceeds this limit (for that period), the tasks belonging to its | |
15 | hierarchy will be throttled and are not allowed to run again until the next | |
16 | period. | |
17 | ||
18 | A group's unused runtime is globally tracked, being refreshed with quota units | |
19 | above at each period boundary. As threads consume this bandwidth it is | |
20 | transferred to cpu-local "silos" on a demand basis. The amount transferred | |
21 | within each of these updates is tunable and described as the "slice". | |
22 | ||
23 | Management | |
24 | ---------- | |
25 | Quota and period are managed within the cpu subsystem via cgroupfs. | |
26 | ||
27 | cpu.cfs_quota_us: the total available run-time within a period (in microseconds) | |
28 | cpu.cfs_period_us: the length of a period (in microseconds) | |
29 | cpu.stat: exports throttling statistics [explained further below] | |
30 | ||
d6a3b247 MCC |
31 | The default values are:: |
32 | ||
88ebc08e BR |
33 | cpu.cfs_period_us=100ms |
34 | cpu.cfs_quota=-1 | |
35 | ||
36 | A value of -1 for cpu.cfs_quota_us indicates that the group does not have any | |
37 | bandwidth restriction in place, such a group is described as an unconstrained | |
38 | bandwidth group. This represents the traditional work-conserving behavior for | |
39 | CFS. | |
40 | ||
41 | Writing any (valid) positive value(s) will enact the specified bandwidth limit. | |
42 | The minimum quota allowed for the quota or period is 1ms. There is also an | |
43 | upper bound on the period length of 1s. Additional restrictions exist when | |
44 | bandwidth limits are used in a hierarchical fashion, these are explained in | |
45 | more detail below. | |
46 | ||
47 | Writing any negative value to cpu.cfs_quota_us will remove the bandwidth limit | |
48 | and return the group to an unconstrained state once more. | |
49 | ||
50 | Any updates to a group's bandwidth specification will result in it becoming | |
51 | unthrottled if it is in a constrained state. | |
52 | ||
53 | System wide settings | |
54 | -------------------- | |
55 | For efficiency run-time is transferred between the global pool and CPU local | |
56 | "silos" in a batch fashion. This greatly reduces global accounting pressure | |
57 | on large systems. The amount transferred each time such an update is required | |
58 | is described as the "slice". | |
59 | ||
d6a3b247 MCC |
60 | This is tunable via procfs:: |
61 | ||
88ebc08e BR |
62 | /proc/sys/kernel/sched_cfs_bandwidth_slice_us (default=5ms) |
63 | ||
64 | Larger slice values will reduce transfer overheads, while smaller values allow | |
65 | for more fine-grained consumption. | |
66 | ||
67 | Statistics | |
68 | ---------- | |
69 | A group's bandwidth statistics are exported via 3 fields in cpu.stat. | |
70 | ||
71 | cpu.stat: | |
d6a3b247 | 72 | |
88ebc08e BR |
73 | - nr_periods: Number of enforcement intervals that have elapsed. |
74 | - nr_throttled: Number of times the group has been throttled/limited. | |
75 | - throttled_time: The total time duration (in nanoseconds) for which entities | |
76 | of the group have been throttled. | |
77 | ||
78 | This interface is read-only. | |
79 | ||
80 | Hierarchical considerations | |
81 | --------------------------- | |
82 | The interface enforces that an individual entity's bandwidth is always | |
83 | attainable, that is: max(c_i) <= C. However, over-subscription in the | |
84 | aggregate case is explicitly allowed to enable work-conserving semantics | |
d6a3b247 MCC |
85 | within a hierarchy: |
86 | ||
88ebc08e | 87 | e.g. \Sum (c_i) may exceed C |
d6a3b247 | 88 | |
88ebc08e BR |
89 | [ Where C is the parent's bandwidth, and c_i its children ] |
90 | ||
91 | ||
92 | There are two ways in which a group may become throttled: | |
d6a3b247 | 93 | |
88ebc08e BR |
94 | a. it fully consumes its own quota within a period |
95 | b. a parent's quota is fully consumed within its period | |
96 | ||
97 | In case b) above, even though the child may have runtime remaining it will not | |
98 | be allowed to until the parent's runtime is refreshed. | |
99 | ||
100 | Examples | |
101 | -------- | |
d6a3b247 | 102 | 1. Limit a group to 1 CPU worth of runtime:: |
88ebc08e BR |
103 | |
104 | If period is 250ms and quota is also 250ms, the group will get | |
105 | 1 CPU worth of runtime every 250ms. | |
106 | ||
107 | # echo 250000 > cpu.cfs_quota_us /* quota = 250ms */ | |
108 | # echo 250000 > cpu.cfs_period_us /* period = 250ms */ | |
109 | ||
d6a3b247 | 110 | 2. Limit a group to 2 CPUs worth of runtime on a multi-CPU machine |
88ebc08e | 111 | |
d6a3b247 MCC |
112 | With 500ms period and 1000ms quota, the group can get 2 CPUs worth of |
113 | runtime every 500ms:: | |
88ebc08e BR |
114 | |
115 | # echo 1000000 > cpu.cfs_quota_us /* quota = 1000ms */ | |
116 | # echo 500000 > cpu.cfs_period_us /* period = 500ms */ | |
117 | ||
118 | The larger period here allows for increased burst capacity. | |
119 | ||
120 | 3. Limit a group to 20% of 1 CPU. | |
121 | ||
d6a3b247 | 122 | With 50ms period, 10ms quota will be equivalent to 20% of 1 CPU:: |
88ebc08e BR |
123 | |
124 | # echo 10000 > cpu.cfs_quota_us /* quota = 10ms */ | |
125 | # echo 50000 > cpu.cfs_period_us /* period = 50ms */ | |
126 | ||
d6a3b247 MCC |
127 | By using a small period here we are ensuring a consistent latency |
128 | response at the expense of burst capacity. |