Commit | Line | Data |
---|---|---|
151f4e2b | 1 | ================= |
83144186 | 2 | Freezing of tasks |
151f4e2b MCC |
3 | ================= |
4 | ||
5 | (C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL | |
83144186 RW |
6 | |
7 | I. What is the freezing of tasks? | |
151f4e2b | 8 | ================================= |
83144186 RW |
9 | |
10 | The freezing of tasks is a mechanism by which user space processes and some | |
11 | kernel threads are controlled during hibernation or system-wide suspend (on some | |
12 | architectures). | |
13 | ||
14 | II. How does it work? | |
151f4e2b | 15 | ===================== |
83144186 | 16 | |
26e0f90f | 17 | There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN |
83144186 RW |
18 | and PF_FREEZER_SKIP (the last one is auxiliary). The tasks that have |
19 | PF_NOFREEZE unset (all user space processes and some kernel threads) are | |
20 | regarded as 'freezable' and treated in a special way before the system enters a | |
21 | suspend state as well as before a hibernation image is created (in what follows | |
22 | we only consider hibernation, but the description also applies to suspend). | |
23 | ||
24 | Namely, as the first step of the hibernation procedure the function | |
26e0f90f MPS |
25 | freeze_processes() (defined in kernel/power/process.c) is called. A system-wide |
26 | variable system_freezing_cnt (as opposed to a per-task flag) is used to indicate | |
27 | whether the system is to undergo a freezing operation. And freeze_processes() | |
28 | sets this variable. After this, it executes try_to_freeze_tasks() that sends a | |
29 | fake signal to all user space processes, and wakes up all the kernel threads. | |
30 | All freezable tasks must react to that by calling try_to_freeze(), which | |
31 | results in a call to __refrigerator() (defined in kernel/freezer.c), which sets | |
32 | the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes | |
33 | it loop until PF_FROZEN is cleared for it. Then, we say that the task is | |
34 | 'frozen' and therefore the set of functions handling this mechanism is referred | |
35 | to as 'the freezer' (these functions are defined in kernel/power/process.c, | |
36 | kernel/freezer.c & include/linux/freezer.h). User space processes are generally | |
37 | frozen before kernel threads. | |
83144186 | 38 | |
a0acae0e TH |
39 | __refrigerator() must not be called directly. Instead, use the |
40 | try_to_freeze() function (defined in include/linux/freezer.h), that checks | |
26e0f90f | 41 | if the task is to be frozen and makes the task enter __refrigerator(). |
83144186 RW |
42 | |
43 | For user space processes try_to_freeze() is called automatically from the | |
44 | signal-handling code, but the freezable kernel threads need to call it | |
d5d8c597 RW |
45 | explicitly in suitable places or use the wait_event_freezable() or |
46 | wait_event_freezable_timeout() macros (defined in include/linux/freezer.h) | |
26e0f90f MPS |
47 | that combine interruptible sleep with checking if the task is to be frozen and |
48 | calling try_to_freeze(). The main loop of a freezable kernel thread may look | |
151f4e2b | 49 | like the following one:: |
83144186 | 50 | |
d5d8c597 | 51 | set_freezable(); |
83144186 RW |
52 | do { |
53 | hub_events(); | |
d5d8c597 RW |
54 | wait_event_freezable(khubd_wait, |
55 | !list_empty(&hub_event_list) || | |
56 | kthread_should_stop()); | |
57 | } while (!kthread_should_stop() || !list_empty(&hub_event_list)); | |
83144186 RW |
58 | |
59 | (from drivers/usb/core/hub.c::hub_thread()). | |
60 | ||
61 | If a freezable kernel thread fails to call try_to_freeze() after the freezer has | |
26e0f90f | 62 | initiated a freezing operation, the freezing of tasks will fail and the entire |
83144186 | 63 | hibernation operation will be cancelled. For this reason, freezable kernel |
d5d8c597 RW |
64 | threads must call try_to_freeze() somewhere or use one of the |
65 | wait_event_freezable() and wait_event_freezable_timeout() macros. | |
83144186 RW |
66 | |
67 | After the system memory state has been restored from a hibernation image and | |
68 | devices have been reinitialized, the function thaw_processes() is called in | |
69 | order to clear the PF_FROZEN flag for each frozen task. Then, the tasks that | |
a0acae0e | 70 | have been frozen leave __refrigerator() and continue running. |
83144186 | 71 | |
9045a050 | 72 | |
151f4e2b | 73 | Rationale behind the functions dealing with freezing and thawing of tasks |
9045a050 SB |
74 | ------------------------------------------------------------------------- |
75 | ||
76 | freeze_processes(): | |
77 | - freezes only userspace tasks | |
78 | ||
79 | freeze_kernel_threads(): | |
80 | - freezes all tasks (including kernel threads) because we can't freeze | |
81 | kernel threads without freezing userspace tasks | |
82 | ||
83 | thaw_kernel_threads(): | |
84 | - thaws only kernel threads; this is particularly useful if we need to do | |
85 | anything special in between thawing of kernel threads and thawing of | |
86 | userspace tasks, or if we want to postpone the thawing of userspace tasks | |
87 | ||
88 | thaw_processes(): | |
89 | - thaws all tasks (including kernel threads) because we can't thaw userspace | |
90 | tasks without thawing kernel threads | |
91 | ||
92 | ||
83144186 | 93 | III. Which kernel threads are freezable? |
151f4e2b | 94 | ======================================== |
83144186 RW |
95 | |
96 | Kernel threads are not freezable by default. However, a kernel thread may clear | |
97 | PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE | |
3a7cbd50 | 98 | directly is not allowed). From this point it is regarded as freezable |
83144186 RW |
99 | and must call try_to_freeze() in a suitable place. |
100 | ||
101 | IV. Why do we do that? | |
151f4e2b | 102 | ====================== |
83144186 RW |
103 | |
104 | Generally speaking, there is a couple of reasons to use the freezing of tasks: | |
105 | ||
106 | 1. The principal reason is to prevent filesystems from being damaged after | |
151f4e2b MCC |
107 | hibernation. At the moment we have no simple means of checkpointing |
108 | filesystems, so if there are any modifications made to filesystem data and/or | |
109 | metadata on disks, we cannot bring them back to the state from before the | |
110 | modifications. At the same time each hibernation image contains some | |
111 | filesystem-related information that must be consistent with the state of the | |
112 | on-disk data and metadata after the system memory state has been restored | |
113 | from the image (otherwise the filesystems will be damaged in a nasty way, | |
114 | usually making them almost impossible to repair). We therefore freeze | |
115 | tasks that might cause the on-disk filesystems' data and metadata to be | |
116 | modified after the hibernation image has been created and before the | |
117 | system is finally powered off. The majority of these are user space | |
118 | processes, but if any of the kernel threads may cause something like this | |
119 | to happen, they have to be freezable. | |
83144186 | 120 | |
27763653 | 121 | 2. Next, to create the hibernation image we need to free a sufficient amount of |
151f4e2b MCC |
122 | memory (approximately 50% of available RAM) and we need to do that before |
123 | devices are deactivated, because we generally need them for swapping out. | |
124 | Then, after the memory for the image has been freed, we don't want tasks | |
125 | to allocate additional memory and we prevent them from doing that by | |
126 | freezing them earlier. [Of course, this also means that device drivers | |
127 | should not allocate substantial amounts of memory from their .suspend() | |
128 | callbacks before hibernation, but this is a separate issue.] | |
27763653 RW |
129 | |
130 | 3. The third reason is to prevent user space processes and some kernel threads | |
151f4e2b MCC |
131 | from interfering with the suspending and resuming of devices. A user space |
132 | process running on a second CPU while we are suspending devices may, for | |
133 | example, be troublesome and without the freezing of tasks we would need some | |
134 | safeguards against race conditions that might occur in such a case. | |
83144186 RW |
135 | |
136 | Although Linus Torvalds doesn't like the freezing of tasks, he said this in one | |
05a5f51c | 137 | of the discussions on LKML (https://lore.kernel.org/r/alpine.LFD.0.98.0704271801020.9964@woody.linux-foundation.org): |
83144186 RW |
138 | |
139 | "RJW:> Why we freeze tasks at all or why we freeze kernel threads? | |
140 | ||
141 | Linus: In many ways, 'at all'. | |
142 | ||
151f4e2b | 143 | I **do** realize the IO request queue issues, and that we cannot actually do |
83144186 RW |
144 | s2ram with some devices in the middle of a DMA. So we want to be able to |
145 | avoid *that*, there's no question about that. And I suspect that stopping | |
146 | user threads and then waiting for a sync is practically one of the easier | |
147 | ways to do so. | |
148 | ||
149 | So in practice, the 'at all' may become a 'why freeze kernel threads?' and | |
150 | freezing user threads I don't find really objectionable." | |
151 | ||
152 | Still, there are kernel threads that may want to be freezable. For example, if | |
5eb6f9ad VK |
153 | a kernel thread that belongs to a device driver accesses the device directly, it |
154 | in principle needs to know when the device is suspended, so that it doesn't try | |
155 | to access it at that time. However, if the kernel thread is freezable, it will | |
156 | be frozen before the driver's .suspend() callback is executed and it will be | |
83144186 RW |
157 | thawed after the driver's .resume() callback has run, so it won't be accessing |
158 | the device while it's suspended. | |
159 | ||
27763653 | 160 | 4. Another reason for freezing tasks is to prevent user space processes from |
151f4e2b MCC |
161 | realizing that hibernation (or suspend) operation takes place. Ideally, user |
162 | space processes should not notice that such a system-wide operation has | |
163 | occurred and should continue running without any problems after the restore | |
164 | (or resume from suspend). Unfortunately, in the most general case this | |
165 | is quite difficult to achieve without the freezing of tasks. Consider, | |
166 | for example, a process that depends on all CPUs being online while it's | |
167 | running. Since we need to disable nonboot CPUs during the hibernation, | |
168 | if this process is not frozen, it may notice that the number of CPUs has | |
169 | changed and may start to work incorrectly because of that. | |
83144186 RW |
170 | |
171 | V. Are there any problems related to the freezing of tasks? | |
151f4e2b | 172 | =========================================================== |
83144186 RW |
173 | |
174 | Yes, there are. | |
175 | ||
176 | First of all, the freezing of kernel threads may be tricky if they depend one | |
177 | on another. For example, if kernel thread A waits for a completion (in the | |
178 | TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B | |
179 | and B is frozen in the meantime, then A will be blocked until B is thawed, which | |
180 | may be undesirable. That's why kernel threads are not freezable by default. | |
181 | ||
182 | Second, there are the following two problems related to the freezing of user | |
183 | space processes: | |
151f4e2b | 184 | |
83144186 RW |
185 | 1. Putting processes into an uninterruptible sleep distorts the load average. |
186 | 2. Now that we have FUSE, plus the framework for doing device drivers in | |
151f4e2b MCC |
187 | userspace, it gets even more complicated because some userspace processes are |
188 | now doing the sorts of things that kernel threads do | |
189 | (https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html). | |
83144186 RW |
190 | |
191 | The problem 1. seems to be fixable, although it hasn't been fixed so far. The | |
192 | other one is more serious, but it seems that we can work around it by using | |
193 | hibernation (and suspend) notifiers (in that case, though, we won't be able to | |
194 | avoid the realization by the user space processes that the hibernation is taking | |
195 | place). | |
196 | ||
197 | There are also problems that the freezing of tasks tends to expose, although | |
198 | they are not directly related to it. For example, if request_firmware() is | |
199 | called from a device driver's .resume() routine, it will timeout and eventually | |
200 | fail, because the user land process that should respond to the request is frozen | |
201 | at this point. So, seemingly, the failure is due to the freezing of tasks. | |
202 | Suppose, however, that the firmware file is located on a filesystem accessible | |
203 | only through another device that hasn't been resumed yet. In that case, | |
204 | request_firmware() will fail regardless of whether or not the freezing of tasks | |
205 | is used. Consequently, the problem is not really related to the freezing of | |
fccdb5ae ON |
206 | tasks, since it generally exists anyway. |
207 | ||
208 | A driver must have all firmwares it may need in RAM before suspend() is called. | |
209 | If keeping them is not practical, for example due to their size, they must be | |
730c4c05 RW |
210 | requested early enough using the suspend notifier API described in |
211 | Documentation/driver-api/pm/notifiers.rst. | |
cba3176e SB |
212 | |
213 | VI. Are there any precautions to be taken to prevent freezing failures? | |
151f4e2b | 214 | ======================================================================= |
cba3176e SB |
215 | |
216 | Yes, there are. | |
217 | ||
1992b66d BH |
218 | First of all, grabbing the 'system_transition_mutex' lock to mutually exclude a |
219 | piece of code from system-wide sleep such as suspend/hibernation is not | |
220 | encouraged. If possible, that piece of code must instead hook onto the | |
221 | suspend/hibernation notifiers to achieve mutual exclusion. Look at the | |
222 | CPU-Hotplug code (kernel/cpu.c) for an example. | |
223 | ||
224 | However, if that is not feasible, and grabbing 'system_transition_mutex' is | |
225 | deemed necessary, it is strongly discouraged to directly call | |
226 | mutex_[un]lock(&system_transition_mutex) since that could lead to freezing | |
227 | failures, because if the suspend/hibernate code successfully acquired the | |
228 | 'system_transition_mutex' lock, and hence that other entity failed to acquire | |
229 | the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE state. As a | |
230 | consequence, the freezer would not be able to freeze that task, leading to | |
231 | freezing failure. | |
cba3176e SB |
232 | |
233 | However, the [un]lock_system_sleep() APIs are safe to use in this scenario, | |
234 | since they ask the freezer to skip freezing this task, since it is anyway | |
1992b66d BH |
235 | "frozen enough" as it is blocked on 'system_transition_mutex', which will be |
236 | released only after the entire suspend/hibernation sequence is complete. So, to | |
237 | summarize, use [un]lock_system_sleep() instead of directly using | |
55f2503c | 238 | mutex_[un]lock(&system_transition_mutex). That would prevent freezing failures. |
957d1282 LF |
239 | |
240 | V. Miscellaneous | |
151f4e2b MCC |
241 | ================ |
242 | ||
957d1282 | 243 | /sys/power/pm_freeze_timeout controls how long it will cost at most to freeze |
1992b66d BH |
244 | all user space processes or all freezable kernel threads, in unit of |
245 | millisecond. The default value is 20000, with range of unsigned integer. |