Commit | Line | Data |
---|---|---|
151f4e2b | 1 | ================= |
83144186 | 2 | Freezing of tasks |
151f4e2b MCC |
3 | ================= |
4 | ||
5 | (C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL | |
83144186 RW |
6 | |
7 | I. What is the freezing of tasks? | |
151f4e2b | 8 | ================================= |
83144186 RW |
9 | |
10 | The freezing of tasks is a mechanism by which user space processes and some | |
11 | kernel threads are controlled during hibernation or system-wide suspend (on some | |
12 | architectures). | |
13 | ||
14 | II. How does it work? | |
151f4e2b | 15 | ===================== |
83144186 | 16 | |
4bbf0b6a KH |
17 | There is one per-task flag (PF_NOFREEZE) and three per-task states |
18 | (TASK_FROZEN, TASK_FREEZABLE and __TASK_FREEZABLE_UNSAFE) used for that. | |
19 | The tasks that have PF_NOFREEZE unset (all user space tasks and some kernel | |
20 | threads) are regarded as 'freezable' and treated in a special way before the | |
21 | system enters a sleep state as well as before a hibernation image is created | |
22 | (hibernation is directly covered by what follows, but the description applies | |
23 | to system-wide suspend too). | |
83144186 RW |
24 | |
25 | Namely, as the first step of the hibernation procedure the function | |
26e0f90f | 26 | freeze_processes() (defined in kernel/power/process.c) is called. A system-wide |
4bbf0b6a KH |
27 | static key freezer_active (as opposed to a per-task flag or state) is used to |
28 | indicate whether the system is to undergo a freezing operation. And | |
29 | freeze_processes() sets this static key. After this, it executes | |
30 | try_to_freeze_tasks() that sends a fake signal to all user space processes, and | |
31 | wakes up all the kernel threads. All freezable tasks must react to that by | |
32 | calling try_to_freeze(), which results in a call to __refrigerator() (defined | |
33 | in kernel/freezer.c), which changes the task's state to TASK_FROZEN, and makes | |
34 | it loop until it is woken by an explicit TASK_FROZEN wakeup. Then, that task | |
35 | is regarded as 'frozen' and so the set of functions handling this mechanism is | |
36 | referred to as 'the freezer' (these functions are defined in | |
37 | kernel/power/process.c, kernel/freezer.c & include/linux/freezer.h). User space | |
38 | tasks are generally frozen before kernel threads. | |
83144186 | 39 | |
a0acae0e TH |
40 | __refrigerator() must not be called directly. Instead, use the |
41 | try_to_freeze() function (defined in include/linux/freezer.h), that checks | |
26e0f90f | 42 | if the task is to be frozen and makes the task enter __refrigerator(). |
83144186 RW |
43 | |
44 | For user space processes try_to_freeze() is called automatically from the | |
45 | signal-handling code, but the freezable kernel threads need to call it | |
d5d8c597 | 46 | explicitly in suitable places or use the wait_event_freezable() or |
4bbf0b6a KH |
47 | wait_event_freezable_timeout() macros (defined in include/linux/wait.h) |
48 | that put the task to sleep (TASK_INTERRUPTIBLE) or freeze it (TASK_FROZEN) if | |
49 | freezer_active is set. The main loop of a freezable kernel thread may look | |
151f4e2b | 50 | like the following one:: |
83144186 | 51 | |
d5d8c597 | 52 | set_freezable(); |
4bbf0b6a KH |
53 | |
54 | while (true) { | |
55 | struct task_struct *tsk = NULL; | |
56 | ||
57 | wait_event_freezable(oom_reaper_wait, oom_reaper_list != NULL); | |
58 | spin_lock_irq(&oom_reaper_lock); | |
59 | if (oom_reaper_list != NULL) { | |
60 | tsk = oom_reaper_list; | |
61 | oom_reaper_list = tsk->oom_reaper_list; | |
62 | } | |
63 | spin_unlock_irq(&oom_reaper_lock); | |
64 | ||
65 | if (tsk) | |
66 | oom_reap_task(tsk); | |
67 | } | |
68 | ||
69 | (from mm/oom_kill.c::oom_reaper()). | |
70 | ||
71 | If a freezable kernel thread is not put to the frozen state after the freezer | |
72 | has initiated a freezing operation, the freezing of tasks will fail and the | |
73 | entire system-wide transition will be cancelled. For this reason, freezable | |
74 | kernel threads must call try_to_freeze() somewhere or use one of the | |
d5d8c597 | 75 | wait_event_freezable() and wait_event_freezable_timeout() macros. |
83144186 RW |
76 | |
77 | After the system memory state has been restored from a hibernation image and | |
78 | devices have been reinitialized, the function thaw_processes() is called in | |
4bbf0b6a KH |
79 | order to wake up each frozen task. Then, the tasks that have been frozen leave |
80 | __refrigerator() and continue running. | |
83144186 | 81 | |
9045a050 | 82 | |
151f4e2b | 83 | Rationale behind the functions dealing with freezing and thawing of tasks |
9045a050 SB |
84 | ------------------------------------------------------------------------- |
85 | ||
86 | freeze_processes(): | |
87 | - freezes only userspace tasks | |
88 | ||
89 | freeze_kernel_threads(): | |
90 | - freezes all tasks (including kernel threads) because we can't freeze | |
91 | kernel threads without freezing userspace tasks | |
92 | ||
93 | thaw_kernel_threads(): | |
94 | - thaws only kernel threads; this is particularly useful if we need to do | |
95 | anything special in between thawing of kernel threads and thawing of | |
96 | userspace tasks, or if we want to postpone the thawing of userspace tasks | |
97 | ||
98 | thaw_processes(): | |
99 | - thaws all tasks (including kernel threads) because we can't thaw userspace | |
100 | tasks without thawing kernel threads | |
101 | ||
102 | ||
83144186 | 103 | III. Which kernel threads are freezable? |
151f4e2b | 104 | ======================================== |
83144186 RW |
105 | |
106 | Kernel threads are not freezable by default. However, a kernel thread may clear | |
107 | PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE | |
3a7cbd50 | 108 | directly is not allowed). From this point it is regarded as freezable |
4bbf0b6a KH |
109 | and must call try_to_freeze() or variants of wait_event_freezable() in a |
110 | suitable place. | |
83144186 RW |
111 | |
112 | IV. Why do we do that? | |
151f4e2b | 113 | ====================== |
83144186 RW |
114 | |
115 | Generally speaking, there is a couple of reasons to use the freezing of tasks: | |
116 | ||
117 | 1. The principal reason is to prevent filesystems from being damaged after | |
151f4e2b MCC |
118 | hibernation. At the moment we have no simple means of checkpointing |
119 | filesystems, so if there are any modifications made to filesystem data and/or | |
120 | metadata on disks, we cannot bring them back to the state from before the | |
121 | modifications. At the same time each hibernation image contains some | |
122 | filesystem-related information that must be consistent with the state of the | |
123 | on-disk data and metadata after the system memory state has been restored | |
124 | from the image (otherwise the filesystems will be damaged in a nasty way, | |
125 | usually making them almost impossible to repair). We therefore freeze | |
126 | tasks that might cause the on-disk filesystems' data and metadata to be | |
127 | modified after the hibernation image has been created and before the | |
128 | system is finally powered off. The majority of these are user space | |
129 | processes, but if any of the kernel threads may cause something like this | |
130 | to happen, they have to be freezable. | |
83144186 | 131 | |
27763653 | 132 | 2. Next, to create the hibernation image we need to free a sufficient amount of |
151f4e2b MCC |
133 | memory (approximately 50% of available RAM) and we need to do that before |
134 | devices are deactivated, because we generally need them for swapping out. | |
135 | Then, after the memory for the image has been freed, we don't want tasks | |
136 | to allocate additional memory and we prevent them from doing that by | |
137 | freezing them earlier. [Of course, this also means that device drivers | |
138 | should not allocate substantial amounts of memory from their .suspend() | |
139 | callbacks before hibernation, but this is a separate issue.] | |
27763653 RW |
140 | |
141 | 3. The third reason is to prevent user space processes and some kernel threads | |
151f4e2b MCC |
142 | from interfering with the suspending and resuming of devices. A user space |
143 | process running on a second CPU while we are suspending devices may, for | |
144 | example, be troublesome and without the freezing of tasks we would need some | |
145 | safeguards against race conditions that might occur in such a case. | |
83144186 RW |
146 | |
147 | Although Linus Torvalds doesn't like the freezing of tasks, he said this in one | |
05a5f51c | 148 | of the discussions on LKML (https://lore.kernel.org/r/alpine.LFD.0.98.0704271801020.9964@woody.linux-foundation.org): |
83144186 RW |
149 | |
150 | "RJW:> Why we freeze tasks at all or why we freeze kernel threads? | |
151 | ||
152 | Linus: In many ways, 'at all'. | |
153 | ||
151f4e2b | 154 | I **do** realize the IO request queue issues, and that we cannot actually do |
83144186 RW |
155 | s2ram with some devices in the middle of a DMA. So we want to be able to |
156 | avoid *that*, there's no question about that. And I suspect that stopping | |
157 | user threads and then waiting for a sync is practically one of the easier | |
158 | ways to do so. | |
159 | ||
160 | So in practice, the 'at all' may become a 'why freeze kernel threads?' and | |
161 | freezing user threads I don't find really objectionable." | |
162 | ||
163 | Still, there are kernel threads that may want to be freezable. For example, if | |
5eb6f9ad VK |
164 | a kernel thread that belongs to a device driver accesses the device directly, it |
165 | in principle needs to know when the device is suspended, so that it doesn't try | |
166 | to access it at that time. However, if the kernel thread is freezable, it will | |
167 | be frozen before the driver's .suspend() callback is executed and it will be | |
83144186 RW |
168 | thawed after the driver's .resume() callback has run, so it won't be accessing |
169 | the device while it's suspended. | |
170 | ||
27763653 | 171 | 4. Another reason for freezing tasks is to prevent user space processes from |
151f4e2b MCC |
172 | realizing that hibernation (or suspend) operation takes place. Ideally, user |
173 | space processes should not notice that such a system-wide operation has | |
174 | occurred and should continue running without any problems after the restore | |
175 | (or resume from suspend). Unfortunately, in the most general case this | |
176 | is quite difficult to achieve without the freezing of tasks. Consider, | |
177 | for example, a process that depends on all CPUs being online while it's | |
178 | running. Since we need to disable nonboot CPUs during the hibernation, | |
179 | if this process is not frozen, it may notice that the number of CPUs has | |
180 | changed and may start to work incorrectly because of that. | |
83144186 RW |
181 | |
182 | V. Are there any problems related to the freezing of tasks? | |
151f4e2b | 183 | =========================================================== |
83144186 RW |
184 | |
185 | Yes, there are. | |
186 | ||
187 | First of all, the freezing of kernel threads may be tricky if they depend one | |
188 | on another. For example, if kernel thread A waits for a completion (in the | |
189 | TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B | |
190 | and B is frozen in the meantime, then A will be blocked until B is thawed, which | |
191 | may be undesirable. That's why kernel threads are not freezable by default. | |
192 | ||
193 | Second, there are the following two problems related to the freezing of user | |
194 | space processes: | |
151f4e2b | 195 | |
83144186 RW |
196 | 1. Putting processes into an uninterruptible sleep distorts the load average. |
197 | 2. Now that we have FUSE, plus the framework for doing device drivers in | |
151f4e2b MCC |
198 | userspace, it gets even more complicated because some userspace processes are |
199 | now doing the sorts of things that kernel threads do | |
200 | (https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html). | |
83144186 RW |
201 | |
202 | The problem 1. seems to be fixable, although it hasn't been fixed so far. The | |
203 | other one is more serious, but it seems that we can work around it by using | |
204 | hibernation (and suspend) notifiers (in that case, though, we won't be able to | |
205 | avoid the realization by the user space processes that the hibernation is taking | |
206 | place). | |
207 | ||
208 | There are also problems that the freezing of tasks tends to expose, although | |
209 | they are not directly related to it. For example, if request_firmware() is | |
210 | called from a device driver's .resume() routine, it will timeout and eventually | |
211 | fail, because the user land process that should respond to the request is frozen | |
212 | at this point. So, seemingly, the failure is due to the freezing of tasks. | |
213 | Suppose, however, that the firmware file is located on a filesystem accessible | |
214 | only through another device that hasn't been resumed yet. In that case, | |
215 | request_firmware() will fail regardless of whether or not the freezing of tasks | |
216 | is used. Consequently, the problem is not really related to the freezing of | |
fccdb5ae ON |
217 | tasks, since it generally exists anyway. |
218 | ||
219 | A driver must have all firmwares it may need in RAM before suspend() is called. | |
220 | If keeping them is not practical, for example due to their size, they must be | |
730c4c05 RW |
221 | requested early enough using the suspend notifier API described in |
222 | Documentation/driver-api/pm/notifiers.rst. | |
cba3176e SB |
223 | |
224 | VI. Are there any precautions to be taken to prevent freezing failures? | |
151f4e2b | 225 | ======================================================================= |
cba3176e SB |
226 | |
227 | Yes, there are. | |
228 | ||
1992b66d BH |
229 | First of all, grabbing the 'system_transition_mutex' lock to mutually exclude a |
230 | piece of code from system-wide sleep such as suspend/hibernation is not | |
231 | encouraged. If possible, that piece of code must instead hook onto the | |
232 | suspend/hibernation notifiers to achieve mutual exclusion. Look at the | |
233 | CPU-Hotplug code (kernel/cpu.c) for an example. | |
234 | ||
235 | However, if that is not feasible, and grabbing 'system_transition_mutex' is | |
236 | deemed necessary, it is strongly discouraged to directly call | |
237 | mutex_[un]lock(&system_transition_mutex) since that could lead to freezing | |
238 | failures, because if the suspend/hibernate code successfully acquired the | |
239 | 'system_transition_mutex' lock, and hence that other entity failed to acquire | |
240 | the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE state. As a | |
241 | consequence, the freezer would not be able to freeze that task, leading to | |
242 | freezing failure. | |
cba3176e SB |
243 | |
244 | However, the [un]lock_system_sleep() APIs are safe to use in this scenario, | |
245 | since they ask the freezer to skip freezing this task, since it is anyway | |
1992b66d BH |
246 | "frozen enough" as it is blocked on 'system_transition_mutex', which will be |
247 | released only after the entire suspend/hibernation sequence is complete. So, to | |
248 | summarize, use [un]lock_system_sleep() instead of directly using | |
55f2503c | 249 | mutex_[un]lock(&system_transition_mutex). That would prevent freezing failures. |
957d1282 LF |
250 | |
251 | V. Miscellaneous | |
151f4e2b MCC |
252 | ================ |
253 | ||
957d1282 | 254 | /sys/power/pm_freeze_timeout controls how long it will cost at most to freeze |
1992b66d BH |
255 | all user space processes or all freezable kernel threads, in unit of |
256 | millisecond. The default value is 20000, with range of unsigned integer. |