Commit | Line | Data |
---|---|---|
99c8b231 MCC |
1 | ============== |
2 | Control Groups | |
3 | ============== | |
ddbcc7e8 | 4 | |
45ce80fb | 5 | Written by Paul Menage <menage@google.com> based on |
da82c92f | 6 | Documentation/admin-guide/cgroup-v1/cpusets.rst |
ddbcc7e8 PM |
7 | |
8 | Original copyright statements from cpusets.txt: | |
99c8b231 | 9 | |
ddbcc7e8 | 10 | Portions Copyright (C) 2004 BULL SA. |
99c8b231 | 11 | |
ddbcc7e8 | 12 | Portions Copyright (c) 2004-2006 Silicon Graphics, Inc. |
99c8b231 | 13 | |
ddbcc7e8 | 14 | Modified by Paul Jackson <pj@sgi.com> |
99c8b231 | 15 | |
93e205a7 | 16 | Modified by Christoph Lameter <cl@linux.com> |
ddbcc7e8 | 17 | |
99c8b231 MCC |
18 | .. CONTENTS: |
19 | ||
20 | 1. Control Groups | |
21 | 1.1 What are cgroups ? | |
22 | 1.2 Why are cgroups needed ? | |
23 | 1.3 How are cgroups implemented ? | |
24 | 1.4 What does notify_on_release do ? | |
25 | 1.5 What does clone_children do ? | |
26 | 1.6 How do I use cgroups ? | |
27 | 2. Usage Examples and Syntax | |
28 | 2.1 Basic Usage | |
29 | 2.2 Attaching processes | |
30 | 2.3 Mounting hierarchies by name | |
31 | 3. Kernel API | |
32 | 3.1 Overview | |
33 | 3.2 Synchronization | |
34 | 3.3 Subsystem API | |
35 | 4. Extended attributes usage | |
36 | 5. Questions | |
ddbcc7e8 PM |
37 | |
38 | 1. Control Groups | |
d19e0583 | 39 | ================= |
ddbcc7e8 PM |
40 | |
41 | 1.1 What are cgroups ? | |
42 | ---------------------- | |
43 | ||
44 | Control Groups provide a mechanism for aggregating/partitioning sets of | |
45 | tasks, and all their future children, into hierarchical groups with | |
46 | specialized behaviour. | |
47 | ||
48 | Definitions: | |
49 | ||
50 | A *cgroup* associates a set of tasks with a set of parameters for one | |
51 | or more subsystems. | |
52 | ||
53 | A *subsystem* is a module that makes use of the task grouping | |
54 | facilities provided by cgroups to treat groups of tasks in | |
55 | particular ways. A subsystem is typically a "resource controller" that | |
56 | schedules a resource or applies per-cgroup limits, but it may be | |
57 | anything that wants to act on a group of processes, e.g. a | |
58 | virtualization subsystem. | |
59 | ||
60 | A *hierarchy* is a set of cgroups arranged in a tree, such that | |
61 | every task in the system is in exactly one of the cgroups in the | |
62 | hierarchy, and a set of subsystems; each subsystem has system-specific | |
63 | state attached to each cgroup in the hierarchy. Each hierarchy has | |
64 | an instance of the cgroup virtual filesystem associated with it. | |
65 | ||
caa790ba | 66 | At any one time there may be multiple active hierarchies of task |
ddbcc7e8 PM |
67 | cgroups. Each hierarchy is a partition of all tasks in the system. |
68 | ||
83b061fc | 69 | User-level code may create and destroy cgroups by name in an |
ddbcc7e8 | 70 | instance of the cgroup virtual file system, specify and query to |
83b061fc | 71 | which cgroup a task is assigned, and list the task PIDs assigned to |
ddbcc7e8 PM |
72 | a cgroup. Those creations and assignments only affect the hierarchy |
73 | associated with that instance of the cgroup file system. | |
74 | ||
75 | On their own, the only use for cgroups is for simple job | |
76 | tracking. The intention is that other subsystems hook into the generic | |
77 | cgroup support to provide new attributes for cgroups, such as | |
78 | accounting/limiting the resources which processes in a cgroup can | |
da82c92f | 79 | access. For example, cpusets (see Documentation/admin-guide/cgroup-v1/cpusets.rst) allow |
ddbcc7e8 PM |
80 | you to associate a set of CPUs and a set of memory nodes with the |
81 | tasks in each cgroup. | |
82 | ||
980660ca BS |
83 | .. _cgroups-why-needed: |
84 | ||
ddbcc7e8 PM |
85 | 1.2 Why are cgroups needed ? |
86 | ---------------------------- | |
87 | ||
88 | There are multiple efforts to provide process aggregations in the | |
83b061fc | 89 | Linux kernel, mainly for resource-tracking purposes. Such efforts |
ddbcc7e8 PM |
90 | include cpusets, CKRM/ResGroups, UserBeanCounters, and virtual server |
91 | namespaces. These all require the basic notion of a | |
92 | grouping/partitioning of processes, with newly forked processes ending | |
83b061fc | 93 | up in the same group (cgroup) as their parent process. |
ddbcc7e8 PM |
94 | |
95 | The kernel cgroup patch provides the minimum essential kernel | |
96 | mechanisms required to efficiently implement such groups. It has | |
97 | minimal impact on the system fast paths, and provides hooks for | |
98 | specific subsystems such as cpusets to provide additional behaviour as | |
99 | desired. | |
100 | ||
101 | Multiple hierarchy support is provided to allow for situations where | |
102 | the division of tasks into cgroups is distinctly different for | |
103 | different subsystems - having parallel hierarchies allows each | |
104 | hierarchy to be a natural division of tasks, without having to handle | |
105 | complex combinations of tasks that would be present if several | |
106 | unrelated subsystems needed to be forced into the same tree of | |
107 | cgroups. | |
108 | ||
109 | At one extreme, each resource controller or subsystem could be in a | |
110 | separate hierarchy; at the other extreme, all subsystems | |
111 | would be attached to the same hierarchy. | |
112 | ||
113 | As an example of a scenario (originally proposed by vatsa@in.ibm.com) | |
114 | that can benefit from multiple hierarchies, consider a large | |
115 | university server with various users - students, professors, system | |
116 | tasks etc. The resource planning for this server could be along the | |
99c8b231 | 117 | following lines:: |
ddbcc7e8 | 118 | |
6ad85239 | 119 | CPU : "Top cpuset" |
ddbcc7e8 PM |
120 | / \ |
121 | CPUSet1 CPUSet2 | |
6ad85239 GL |
122 | | | |
123 | (Professors) (Students) | |
ddbcc7e8 PM |
124 | |
125 | In addition (system tasks) are attached to topcpuset (so | |
126 | that they can run anywhere) with a limit of 20% | |
127 | ||
6ad85239 | 128 | Memory : Professors (50%), Students (30%), system (20%) |
ddbcc7e8 | 129 | |
6ad85239 | 130 | Disk : Professors (50%), Students (30%), system (20%) |
ddbcc7e8 PM |
131 | |
132 | Network : WWW browsing (20%), Network File System (60%), others (20%) | |
133 | / \ | |
6ad85239 | 134 | Professors (15%) students (5%) |
ddbcc7e8 | 135 | |
83b061fc MK |
136 | Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd goes |
137 | into the NFS network class. | |
ddbcc7e8 | 138 | |
caa790ba | 139 | At the same time Firefox/Lynx will share an appropriate CPU/Memory class |
ddbcc7e8 PM |
140 | depending on who launched it (prof/student). |
141 | ||
142 | With the ability to classify tasks differently for different resources | |
83b061fc | 143 | (by putting those resource subsystems in different hierarchies), |
ddbcc7e8 | 144 | the admin can easily set up a script which receives exec notifications |
99c8b231 | 145 | and depending on who is launching the browser he can:: |
ddbcc7e8 | 146 | |
f6e07d38 | 147 | # echo browser_pid > /sys/fs/cgroup/<restype>/<userclass>/tasks |
ddbcc7e8 PM |
148 | |
149 | With only a single hierarchy, he now would potentially have to create | |
150 | a separate cgroup for every browser launched and associate it with | |
67de0162 | 151 | appropriate network and other resource class. This may lead to |
ddbcc7e8 PM |
152 | proliferation of such cgroups. |
153 | ||
83b061fc | 154 | Also let's say that the administrator would like to give enhanced network |
ddbcc7e8 | 155 | access temporarily to a student's browser (since it is night and the user |
83b061fc MK |
156 | wants to do online gaming :)) OR give one of the student's simulation |
157 | apps enhanced CPU power. | |
ddbcc7e8 | 158 | |
83b061fc | 159 | With ability to write PIDs directly to resource classes, it's just a |
99c8b231 | 160 | matter of:: |
ddbcc7e8 | 161 | |
f6e07d38 | 162 | # echo pid > /sys/fs/cgroup/network/<new_class>/tasks |
ddbcc7e8 | 163 | (after some time) |
f6e07d38 | 164 | # echo pid > /sys/fs/cgroup/network/<orig_class>/tasks |
ddbcc7e8 | 165 | |
83b061fc | 166 | Without this ability, the administrator would have to split the cgroup into |
ddbcc7e8 PM |
167 | multiple separate ones and then associate the new cgroups with the |
168 | new resource classes. | |
169 | ||
170 | ||
171 | ||
172 | 1.3 How are cgroups implemented ? | |
173 | --------------------------------- | |
174 | ||
175 | Control Groups extends the kernel as follows: | |
176 | ||
177 | - Each task in the system has a reference-counted pointer to a | |
178 | css_set. | |
179 | ||
180 | - A css_set contains a set of reference-counted pointers to | |
181 | cgroup_subsys_state objects, one for each cgroup subsystem | |
182 | registered in the system. There is no direct link from a task to | |
183 | the cgroup of which it's a member in each hierarchy, but this | |
184 | can be determined by following pointers through the | |
185 | cgroup_subsys_state objects. This is because accessing the | |
186 | subsystem state is something that's expected to happen frequently | |
187 | and in performance-critical code, whereas operations that require a | |
188 | task's actual cgroup assignments (in particular, moving between | |
817929ec PM |
189 | cgroups) are less common. A linked list runs through the cg_list |
190 | field of each task_struct using the css_set, anchored at | |
191 | css_set->tasks. | |
ddbcc7e8 | 192 | |
83b061fc | 193 | - A cgroup hierarchy filesystem can be mounted for browsing and |
ddbcc7e8 PM |
194 | manipulation from user space. |
195 | ||
83b061fc | 196 | - You can list all the tasks (by PID) attached to any cgroup. |
ddbcc7e8 PM |
197 | |
198 | The implementation of cgroups requires a few, simple hooks | |
83b061fc | 199 | into the rest of the kernel, none in performance-critical paths: |
ddbcc7e8 PM |
200 | |
201 | - in init/main.c, to initialize the root cgroups and initial | |
202 | css_set at system boot. | |
203 | ||
204 | - in fork and exit, to attach and detach a task from its css_set. | |
205 | ||
83b061fc | 206 | In addition, a new file system of type "cgroup" may be mounted, to |
ddbcc7e8 PM |
207 | enable browsing and modifying the cgroups presently known to the |
208 | kernel. When mounting a cgroup hierarchy, you may specify a | |
209 | comma-separated list of subsystems to mount as the filesystem mount | |
210 | options. By default, mounting the cgroup filesystem attempts to | |
211 | mount a hierarchy containing all registered subsystems. | |
212 | ||
213 | If an active hierarchy with exactly the same set of subsystems already | |
214 | exists, it will be reused for the new mount. If no existing hierarchy | |
215 | matches, and any of the requested subsystems are in use in an existing | |
216 | hierarchy, the mount will fail with -EBUSY. Otherwise, a new hierarchy | |
217 | is activated, associated with the requested subsystems. | |
218 | ||
26d5bbe5 TH |
219 | It's not currently possible to bind a new subsystem to an active |
220 | cgroup hierarchy, or to unbind a subsystem from an active cgroup | |
221 | hierarchy. This may be possible in future, but is fraught with nasty | |
222 | error-recovery issues. | |
ddbcc7e8 PM |
223 | |
224 | When a cgroup filesystem is unmounted, if there are any | |
225 | child cgroups created below the top-level cgroup, that hierarchy | |
226 | will remain active even though unmounted; if there are no | |
227 | child cgroups then the hierarchy will be deactivated. | |
228 | ||
229 | No new system calls are added for cgroups - all support for | |
230 | querying and modifying cgroups is via this cgroup file system. | |
231 | ||
232 | Each task under /proc has an added file named 'cgroup' displaying, | |
233 | for each active hierarchy, the subsystem names and the cgroup name | |
234 | as the path relative to the root of the cgroup file system. | |
235 | ||
236 | Each cgroup is represented by a directory in the cgroup file system | |
237 | containing the following files describing that cgroup: | |
238 | ||
83b061fc MK |
239 | - tasks: list of tasks (by PID) attached to that cgroup. This list |
240 | is not guaranteed to be sorted. Writing a thread ID into this file | |
7823da36 | 241 | moves the thread into this cgroup. |
83b061fc MK |
242 | - cgroup.procs: list of thread group IDs in the cgroup. This list is |
243 | not guaranteed to be sorted or free of duplicate TGIDs, and userspace | |
7823da36 | 244 | should sort/uniquify the list if this property is required. |
83b061fc | 245 | Writing a thread group ID into this file moves all threads in that |
74a1166d | 246 | group into this cgroup. |
d19e0583 LZ |
247 | - notify_on_release flag: run the release agent on exit? |
248 | - release_agent: the path to use for release notifications (this file | |
249 | exists in the top cgroup only) | |
ddbcc7e8 PM |
250 | |
251 | Other subsystems such as cpusets may add additional files in each | |
d19e0583 | 252 | cgroup dir. |
ddbcc7e8 PM |
253 | |
254 | New cgroups are created using the mkdir system call or shell | |
255 | command. The properties of a cgroup, such as its flags, are | |
256 | modified by writing to the appropriate file in that cgroups | |
257 | directory, as listed above. | |
258 | ||
259 | The named hierarchical structure of nested cgroups allows partitioning | |
260 | a large system into nested, dynamically changeable, "soft-partitions". | |
261 | ||
262 | The attachment of each task, automatically inherited at fork by any | |
263 | children of that task, to a cgroup allows organizing the work load | |
264 | on a system into related sets of tasks. A task may be re-attached to | |
265 | any other cgroup, if allowed by the permissions on the necessary | |
266 | cgroup file system directories. | |
267 | ||
268 | When a task is moved from one cgroup to another, it gets a new | |
269 | css_set pointer - if there's an already existing css_set with the | |
83b061fc | 270 | desired collection of cgroups then that group is reused, otherwise a new |
b851ee79 LZ |
271 | css_set is allocated. The appropriate existing css_set is located by |
272 | looking into a hash table. | |
ddbcc7e8 | 273 | |
817929ec PM |
274 | To allow access from a cgroup to the css_sets (and hence tasks) |
275 | that comprise it, a set of cg_cgroup_link objects form a lattice; | |
276 | each cg_cgroup_link is linked into a list of cg_cgroup_links for | |
d19e0583 | 277 | a single cgroup on its cgrp_link_list field, and a list of |
817929ec PM |
278 | cg_cgroup_links for a single css_set on its cg_link_list. |
279 | ||
280 | Thus the set of tasks in a cgroup can be listed by iterating over | |
281 | each css_set that references the cgroup, and sub-iterating over | |
282 | each css_set's task set. | |
283 | ||
ddbcc7e8 PM |
284 | The use of a Linux virtual file system (vfs) to represent the |
285 | cgroup hierarchy provides for a familiar permission and name space | |
286 | for cgroups, with a minimum of additional kernel code. | |
287 | ||
288 | 1.4 What does notify_on_release do ? | |
289 | ------------------------------------ | |
290 | ||
ddbcc7e8 PM |
291 | If the notify_on_release flag is enabled (1) in a cgroup, then |
292 | whenever the last task in the cgroup leaves (exits or attaches to | |
293 | some other cgroup) and the last child cgroup of that cgroup | |
294 | is removed, then the kernel runs the command specified by the contents | |
295 | of the "release_agent" file in that hierarchy's root directory, | |
296 | supplying the pathname (relative to the mount point of the cgroup | |
297 | file system) of the abandoned cgroup. This enables automatic | |
298 | removal of abandoned cgroups. The default value of | |
299 | notify_on_release in the root cgroup at system boot is disabled | |
300 | (0). The default value of other cgroups at creation is the current | |
83b061fc | 301 | value of their parents' notify_on_release settings. The default value of |
ddbcc7e8 PM |
302 | a cgroup hierarchy's release_agent path is empty. |
303 | ||
97978e6d DL |
304 | 1.5 What does clone_children do ? |
305 | --------------------------------- | |
306 | ||
2260e7fc TH |
307 | This flag only affects the cpuset controller. If the clone_children |
308 | flag is enabled (1) in a cgroup, a new cpuset cgroup will copy its | |
309 | configuration from the parent during initialization. | |
97978e6d DL |
310 | |
311 | 1.6 How do I use cgroups ? | |
ddbcc7e8 PM |
312 | -------------------------- |
313 | ||
314 | To start a new job that is to be contained within a cgroup, using | |
99c8b231 | 315 | the "cpuset" cgroup subsystem, the steps are something like:: |
ddbcc7e8 | 316 | |
f6e07d38 JS |
317 | 1) mount -t tmpfs cgroup_root /sys/fs/cgroup |
318 | 2) mkdir /sys/fs/cgroup/cpuset | |
319 | 3) mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpuset | |
320 | 4) Create the new cgroup by doing mkdir's and write's (or echo's) in | |
845502d2 | 321 | the /sys/fs/cgroup/cpuset virtual file system. |
f6e07d38 | 322 | 5) Start a task that will be the "founding father" of the new job. |
83b061fc | 323 | 6) Attach that task to the new cgroup by writing its PID to the |
845502d2 | 324 | /sys/fs/cgroup/cpuset tasks file for that cgroup. |
f6e07d38 | 325 | 7) fork, exec or clone the job tasks from this founding father task. |
ddbcc7e8 PM |
326 | |
327 | For example, the following sequence of commands will setup a cgroup | |
328 | named "Charlie", containing just CPUs 2 and 3, and Memory Node 1, | |
99c8b231 | 329 | and then start a subshell 'sh' in that cgroup:: |
ddbcc7e8 | 330 | |
f6e07d38 JS |
331 | mount -t tmpfs cgroup_root /sys/fs/cgroup |
332 | mkdir /sys/fs/cgroup/cpuset | |
333 | mount -t cgroup cpuset -ocpuset /sys/fs/cgroup/cpuset | |
334 | cd /sys/fs/cgroup/cpuset | |
ddbcc7e8 PM |
335 | mkdir Charlie |
336 | cd Charlie | |
0f146a76 DG |
337 | /bin/echo 2-3 > cpuset.cpus |
338 | /bin/echo 1 > cpuset.mems | |
ddbcc7e8 PM |
339 | /bin/echo $$ > tasks |
340 | sh | |
341 | # The subshell 'sh' is now running in cgroup Charlie | |
342 | # The next line should display '/Charlie' | |
343 | cat /proc/self/cgroup | |
344 | ||
345 | 2. Usage Examples and Syntax | |
346 | ============================ | |
347 | ||
348 | 2.1 Basic Usage | |
349 | --------------- | |
350 | ||
83b061fc | 351 | Creating, modifying, using cgroups can be done through the cgroup |
ddbcc7e8 PM |
352 | virtual filesystem. |
353 | ||
99c8b231 MCC |
354 | To mount a cgroup hierarchy with all available subsystems, type:: |
355 | ||
356 | # mount -t cgroup xxx /sys/fs/cgroup | |
ddbcc7e8 PM |
357 | |
358 | The "xxx" is not interpreted by the cgroup code, but will appear in | |
359 | /proc/mounts so may be any useful identifying string that you like. | |
360 | ||
bb6405ea EM |
361 | Note: Some subsystems do not work without some user input first. For instance, |
362 | if cpusets are enabled the user will have to populate the cpus and mems files | |
363 | for each new cgroup created before that group can be used. | |
364 | ||
99c8b231 | 365 | As explained in section `1.2 Why are cgroups needed?` you should create |
f6e07d38 JS |
366 | different hierarchies of cgroups for each single resource or group of |
367 | resources you want to control. Therefore, you should mount a tmpfs on | |
368 | /sys/fs/cgroup and create directories for each cgroup resource or resource | |
99c8b231 | 369 | group:: |
f6e07d38 | 370 | |
99c8b231 MCC |
371 | # mount -t tmpfs cgroup_root /sys/fs/cgroup |
372 | # mkdir /sys/fs/cgroup/rg1 | |
f6e07d38 | 373 | |
595f4b69 | 374 | To mount a cgroup hierarchy with just the cpuset and memory |
99c8b231 MCC |
375 | subsystems, type:: |
376 | ||
377 | # mount -t cgroup -o cpuset,memory hier1 /sys/fs/cgroup/rg1 | |
ddbcc7e8 | 378 | |
9a8054aa DW |
379 | While remounting cgroups is currently supported, it is not recommend |
380 | to use it. Remounting allows changing bound subsystems and | |
381 | release_agent. Rebinding is hardly useful as it only works when the | |
382 | hierarchy is empty and release_agent itself should be replaced with | |
383 | conventional fsnotify. The support for remounting will be removed in | |
384 | the future. | |
b6719ec1 | 385 | |
99c8b231 MCC |
386 | To Specify a hierarchy's release_agent:: |
387 | ||
388 | # mount -t cgroup -o cpuset,release_agent="/sbin/cpuset_release_agent" \ | |
389 | xxx /sys/fs/cgroup/rg1 | |
b6719ec1 LZ |
390 | |
391 | Note that specifying 'release_agent' more than once will return failure. | |
ddbcc7e8 | 392 | |
26d5bbe5 TH |
393 | Note that changing the set of subsystems is currently only supported |
394 | when the hierarchy consists of a single (root) cgroup. Supporting | |
395 | the ability to arbitrarily bind/unbind subsystems from an existing | |
396 | cgroup hierarchy is intended to be implemented in the future. | |
ddbcc7e8 | 397 | |
f6e07d38 JS |
398 | Then under /sys/fs/cgroup/rg1 you can find a tree that corresponds to the |
399 | tree of the cgroups in the system. For instance, /sys/fs/cgroup/rg1 | |
ddbcc7e8 PM |
400 | is the cgroup that holds the whole system. |
401 | ||
99c8b231 MCC |
402 | If you want to change the value of release_agent:: |
403 | ||
404 | # echo "/sbin/new_release_agent" > /sys/fs/cgroup/rg1/release_agent | |
b6719ec1 LZ |
405 | |
406 | It can also be changed via remount. | |
407 | ||
99c8b231 MCC |
408 | If you want to create a new cgroup under /sys/fs/cgroup/rg1:: |
409 | ||
410 | # cd /sys/fs/cgroup/rg1 | |
411 | # mkdir my_cgroup | |
412 | ||
413 | Now you want to do something with this cgroup: | |
414 | ||
415 | # cd my_cgroup | |
ddbcc7e8 | 416 | |
99c8b231 | 417 | In this directory you can find several files:: |
ddbcc7e8 | 418 | |
99c8b231 MCC |
419 | # ls |
420 | cgroup.procs notify_on_release tasks | |
421 | (plus whatever files added by the attached subsystems) | |
ddbcc7e8 | 422 | |
99c8b231 MCC |
423 | Now attach your shell to this cgroup:: |
424 | ||
425 | # /bin/echo $$ > tasks | |
ddbcc7e8 PM |
426 | |
427 | You can also create cgroups inside your cgroup by using mkdir in this | |
99c8b231 MCC |
428 | directory:: |
429 | ||
430 | # mkdir my_sub_cs | |
431 | ||
432 | To remove a cgroup, just use rmdir:: | |
ddbcc7e8 | 433 | |
99c8b231 | 434 | # rmdir my_sub_cs |
ddbcc7e8 PM |
435 | |
436 | This will fail if the cgroup is in use (has cgroups inside, or | |
437 | has processes attached, or is held alive by other subsystem-specific | |
438 | reference). | |
439 | ||
440 | 2.2 Attaching processes | |
441 | ----------------------- | |
442 | ||
99c8b231 MCC |
443 | :: |
444 | ||
445 | # /bin/echo PID > tasks | |
ddbcc7e8 PM |
446 | |
447 | Note that it is PID, not PIDs. You can only attach ONE task at a time. | |
99c8b231 | 448 | If you have several tasks to attach, you have to do it one after another:: |
ddbcc7e8 | 449 | |
99c8b231 MCC |
450 | # /bin/echo PID1 > tasks |
451 | # /bin/echo PID2 > tasks | |
452 | ... | |
453 | # /bin/echo PIDn > tasks | |
ddbcc7e8 | 454 | |
99c8b231 | 455 | You can attach the current shell task by echoing 0:: |
bef67c5a | 456 | |
99c8b231 | 457 | # echo 0 > tasks |
bef67c5a | 458 | |
74a1166d | 459 | You can use the cgroup.procs file instead of the tasks file to move all |
83b061fc | 460 | threads in a threadgroup at once. Echoing the PID of any task in a |
74a1166d | 461 | threadgroup to cgroup.procs causes all tasks in that threadgroup to be |
1ae65ae9 | 462 | attached to the cgroup. Writing 0 to cgroup.procs moves all tasks |
74a1166d BB |
463 | in the writing task's threadgroup. |
464 | ||
bb6405ea EM |
465 | Note: Since every task is always a member of exactly one cgroup in each |
466 | mounted hierarchy, to remove a task from its current cgroup you must | |
467 | move it into a new cgroup (possibly the root cgroup) by writing to the | |
468 | new cgroup's tasks file. | |
469 | ||
5fe69d7e LZ |
470 | Note: Due to some restrictions enforced by some cgroup subsystems, moving |
471 | a process to another cgroup can fail. | |
bb6405ea | 472 | |
c6d57f33 PM |
473 | 2.3 Mounting hierarchies by name |
474 | -------------------------------- | |
475 | ||
476 | Passing the name=<x> option when mounting a cgroups hierarchy | |
477 | associates the given name with the hierarchy. This can be used when | |
478 | mounting a pre-existing hierarchy, in order to refer to it by name | |
479 | rather than by its set of active subsystems. Each hierarchy is either | |
480 | nameless, or has a unique name. | |
481 | ||
482 | The name should match [\w.-]+ | |
483 | ||
484 | When passing a name=<x> option for a new hierarchy, you need to | |
485 | specify subsystems manually; the legacy behaviour of mounting all | |
486 | subsystems when none are explicitly specified is not supported when | |
487 | you give a subsystem a name. | |
488 | ||
489 | The name of the subsystem appears as part of the hierarchy description | |
490 | in /proc/mounts and /proc/<pid>/cgroups. | |
491 | ||
492 | ||
ddbcc7e8 PM |
493 | 3. Kernel API |
494 | ============= | |
495 | ||
496 | 3.1 Overview | |
497 | ------------ | |
498 | ||
499 | Each kernel subsystem that wants to hook into the generic cgroup | |
500 | system needs to create a cgroup_subsys object. This contains | |
501 | various methods, which are callbacks from the cgroup system, along | |
83b061fc | 502 | with a subsystem ID which will be assigned by the cgroup system. |
ddbcc7e8 PM |
503 | |
504 | Other fields in the cgroup_subsys object include: | |
505 | ||
506 | - subsys_id: a unique array index for the subsystem, indicating which | |
d19e0583 | 507 | entry in cgroup->subsys[] this subsystem should be managing. |
ddbcc7e8 | 508 | |
d19e0583 LZ |
509 | - name: should be initialized to a unique subsystem name. Should be |
510 | no longer than MAX_CGROUP_TYPE_NAMELEN. | |
ddbcc7e8 | 511 | |
d19e0583 LZ |
512 | - early_init: indicate if the subsystem needs early initialization |
513 | at system boot. | |
ddbcc7e8 PM |
514 | |
515 | Each cgroup object created by the system has an array of pointers, | |
83b061fc | 516 | indexed by subsystem ID; this pointer is entirely managed by the |
ddbcc7e8 PM |
517 | subsystem; the generic cgroup code will never touch this pointer. |
518 | ||
519 | 3.2 Synchronization | |
520 | ------------------- | |
521 | ||
522 | There is a global mutex, cgroup_mutex, used by the cgroup | |
523 | system. This should be taken by anything that wants to modify a | |
524 | cgroup. It may also be taken to prevent cgroups from being | |
525 | modified, but more specific locks may be more appropriate in that | |
526 | situation. | |
527 | ||
528 | See kernel/cgroup.c for more details. | |
529 | ||
530 | Subsystems can take/release the cgroup_mutex via the functions | |
ddbcc7e8 PM |
531 | cgroup_lock()/cgroup_unlock(). |
532 | ||
533 | Accessing a task's cgroup pointer may be done in the following ways: | |
534 | - while holding cgroup_mutex | |
535 | - while holding the task's alloc_lock (via task_lock()) | |
536 | - inside an rcu_read_lock() section via rcu_dereference() | |
537 | ||
538 | 3.3 Subsystem API | |
d19e0583 | 539 | ----------------- |
ddbcc7e8 PM |
540 | |
541 | Each subsystem should: | |
542 | ||
543 | - add an entry in linux/cgroup_subsys.h | |
d51b9dae | 544 | - define a cgroup_subsys object called <name>_cgrp_subsys |
e6a1105b | 545 | |
ddbcc7e8 | 546 | Each subsystem may export the following methods. The only mandatory |
92fb9748 | 547 | methods are css_alloc/free. Any others that are null are presumed to |
ddbcc7e8 PM |
548 | be successful no-ops. |
549 | ||
99c8b231 | 550 | ``struct cgroup_subsys_state *css_alloc(struct cgroup *cgrp)`` |
8dc4f3e1 | 551 | (cgroup_mutex held by caller) |
ddbcc7e8 | 552 | |
92fb9748 | 553 | Called to allocate a subsystem state object for a cgroup. The |
ddbcc7e8 PM |
554 | subsystem should allocate its subsystem state object for the passed |
555 | cgroup, returning a pointer to the new object on success or a | |
92fb9748 | 556 | ERR_PTR() value. On success, the subsystem pointer should point to |
ddbcc7e8 PM |
557 | a structure of type cgroup_subsys_state (typically embedded in a |
558 | larger subsystem-specific object), which will be initialized by the | |
559 | cgroup system. Note that this will be called at initialization to | |
560 | create the root subsystem state for this subsystem; this case can be | |
561 | identified by the passed cgroup object having a NULL parent (since | |
562 | it's the root of the hierarchy) and may be an appropriate place for | |
563 | initialization code. | |
564 | ||
99c8b231 | 565 | ``int css_online(struct cgroup *cgrp)`` |
8dc4f3e1 | 566 | (cgroup_mutex held by caller) |
ddbcc7e8 | 567 | |
92fb9748 TH |
568 | Called after @cgrp successfully completed all allocations and made |
569 | visible to cgroup_for_each_child/descendant_*() iterators. The | |
570 | subsystem may choose to fail creation by returning -errno. This | |
571 | callback can be used to implement reliable state sharing and | |
572 | propagation along the hierarchy. See the comment on | |
a24e3b7d | 573 | cgroup_for_each_live_descendant_pre() for details. |
92fb9748 | 574 | |
99c8b231 | 575 | ``void css_offline(struct cgroup *cgrp);`` |
d7eeac19 | 576 | (cgroup_mutex held by caller) |
92fb9748 TH |
577 | |
578 | This is the counterpart of css_online() and called iff css_online() | |
579 | has succeeded on @cgrp. This signifies the beginning of the end of | |
580 | @cgrp. @cgrp is being removed and the subsystem should start dropping | |
581 | all references it's holding on @cgrp. When all references are dropped, | |
582 | cgroup removal will proceed to the next step - css_free(). After this | |
583 | callback, @cgrp should be considered dead to the subsystem. | |
584 | ||
99c8b231 | 585 | ``void css_free(struct cgroup *cgrp)`` |
92fb9748 TH |
586 | (cgroup_mutex held by caller) |
587 | ||
588 | The cgroup system is about to free @cgrp; the subsystem should free | |
589 | its subsystem state object. By the time this method is called, @cgrp | |
590 | is completely unused; @cgrp->parent is still valid. (Note - can also | |
591 | be called for a newly-created cgroup if an error occurs after this | |
592 | subsystem's create() method has been called for the new cgroup). | |
d19e0583 | 593 | |
99c8b231 | 594 | ``int can_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)`` |
8dc4f3e1 | 595 | (cgroup_mutex held by caller) |
ddbcc7e8 | 596 | |
2f7ee569 TH |
597 | Called prior to moving one or more tasks into a cgroup; if the |
598 | subsystem returns an error, this will abort the attach operation. | |
599 | @tset contains the tasks to be attached and is guaranteed to have at | |
600 | least one task in it. | |
601 | ||
602 | If there are multiple tasks in the taskset, then: | |
603 | - it's guaranteed that all are from the same thread group | |
604 | - @tset contains all tasks from the thread group whether or not | |
605 | they're switching cgroups | |
606 | - the first task is the leader | |
607 | ||
608 | Each @tset entry also contains the task's old cgroup and tasks which | |
609 | aren't switching cgroup can be skipped easily using the | |
610 | cgroup_taskset_for_each() iterator. Note that this isn't called on a | |
611 | fork. If this method returns 0 (success) then this should remain valid | |
612 | while the caller holds cgroup_mutex and it is ensured that either | |
f780bdb7 BB |
613 | attach() or cancel_attach() will be called in future. |
614 | ||
99c8b231 | 615 | ``void css_reset(struct cgroup_subsys_state *css)`` |
b4536f0c TH |
616 | (cgroup_mutex held by caller) |
617 | ||
618 | An optional operation which should restore @css's configuration to the | |
619 | initial state. This is currently only used on the unified hierarchy | |
620 | when a subsystem is disabled on a cgroup through | |
621 | "cgroup.subtree_control" but should remain enabled because other | |
622 | subsystems depend on it. cgroup core makes such a css invisible by | |
623 | removing the associated interface files and invokes this callback so | |
624 | that the hidden subsystem can return to the initial neutral state. | |
625 | This prevents unexpected resource control from a hidden css and | |
626 | ensures that the configuration is in the initial state when it is made | |
627 | visible again later. | |
628 | ||
99c8b231 | 629 | ``void cancel_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)`` |
2468c723 DN |
630 | (cgroup_mutex held by caller) |
631 | ||
632 | Called when a task attach operation has failed after can_attach() has succeeded. | |
633 | A subsystem whose can_attach() has some side-effects should provide this | |
88393161 | 634 | function, so that the subsystem can implement a rollback. If not, not necessary. |
2468c723 | 635 | This will be called only about subsystems whose can_attach() operation have |
2f7ee569 | 636 | succeeded. The parameters are identical to can_attach(). |
2468c723 | 637 | |
99c8b231 | 638 | ``void attach(struct cgroup *cgrp, struct cgroup_taskset *tset)`` |
18e7f1f0 | 639 | (cgroup_mutex held by caller) |
ddbcc7e8 PM |
640 | |
641 | Called after the task has been attached to the cgroup, to allow any | |
642 | post-attachment activity that requires memory allocations or blocking. | |
2f7ee569 | 643 | The parameters are identical to can_attach(). |
f780bdb7 | 644 | |
99c8b231 | 645 | ``void fork(struct task_struct *task)`` |
ddbcc7e8 | 646 | |
e8d55fde | 647 | Called when a task is forked into a cgroup. |
ddbcc7e8 | 648 | |
99c8b231 | 649 | ``void exit(struct task_struct *task)`` |
ddbcc7e8 | 650 | |
d19e0583 | 651 | Called during task exit. |
ddbcc7e8 | 652 | |
99c8b231 | 653 | ``void free(struct task_struct *task)`` |
afcf6c8b TH |
654 | |
655 | Called when the task_struct is freed. | |
656 | ||
99c8b231 | 657 | ``void bind(struct cgroup *root)`` |
26d5bbe5 TH |
658 | (cgroup_mutex held by caller) |
659 | ||
660 | Called when a cgroup subsystem is rebound to a different hierarchy | |
661 | and root cgroup. Currently this will only involve movement between | |
662 | the default hierarchy (which never has sub-cgroups) and a hierarchy | |
663 | that is being created/destroyed (and hence has no sub-cgroups). | |
ddbcc7e8 | 664 | |
19ec2567 AR |
665 | 4. Extended attribute usage |
666 | =========================== | |
667 | ||
668 | cgroup filesystem supports certain types of extended attributes in its | |
669 | directories and files. The current supported types are: | |
99c8b231 | 670 | |
19ec2567 AR |
671 | - Trusted (XATTR_TRUSTED) |
672 | - Security (XATTR_SECURITY) | |
673 | ||
674 | Both require CAP_SYS_ADMIN capability to set. | |
675 | ||
676 | Like in tmpfs, the extended attributes in cgroup filesystem are stored | |
677 | using kernel memory and it's advised to keep the usage at minimum. This | |
678 | is the reason why user defined extended attributes are not supported, since | |
679 | any user can do it and there's no limit in the value size. | |
680 | ||
681 | The current known users for this feature are SELinux to limit cgroup usage | |
682 | in containers and systemd for assorted meta data like main PID in a cgroup | |
683 | (systemd creates a cgroup per service). | |
684 | ||
685 | 5. Questions | |
ddbcc7e8 PM |
686 | ============ |
687 | ||
99c8b231 | 688 | :: |
ddbcc7e8 | 689 | |
99c8b231 MCC |
690 | Q: what's up with this '/bin/echo' ? |
691 | A: bash's builtin 'echo' command does not check calls to write() against | |
692 | errors. If you use it in the cgroup file system, you won't be | |
693 | able to tell whether a command succeeded or failed. | |
ddbcc7e8 | 694 | |
99c8b231 MCC |
695 | Q: When I attach processes, only the first of the line gets really attached ! |
696 | A: We can only return one error code per call to write(). So you should also | |
697 | put only ONE PID. |