Commit | Line | Data |
---|---|---|
a5e4da91 | 1 | .. _unevictable_lru: |
c24b7201 | 2 | |
a5e4da91 MR |
3 | ============================== |
4 | Unevictable LRU Infrastructure | |
5 | ============================== | |
c24b7201 | 6 | |
a5e4da91 MR |
7 | .. contents:: :local: |
8 | ||
9 | ||
10 | Introduction | |
c24b7201 DH |
11 | ============ |
12 | ||
13 | This document describes the Linux memory manager's "Unevictable LRU" | |
14 | infrastructure and the use of this to manage several types of "unevictable" | |
15 | pages. | |
16 | ||
17 | The document attempts to provide the overall rationale behind this mechanism | |
18 | and the rationale for some of the design decisions that drove the | |
19 | implementation. The latter design rationale is discussed in the context of an | |
20 | implementation description. Admittedly, one can obtain the implementation | |
21 | details - the "what does it do?" - by reading the code. One hopes that the | |
22 | descriptions below add value by provide the answer to "why does it do that?". | |
23 | ||
24 | ||
a5e4da91 MR |
25 | |
26 | The Unevictable LRU | |
c24b7201 DH |
27 | =================== |
28 | ||
29 | The Unevictable LRU facility adds an additional LRU list to track unevictable | |
30 | pages and to hide these pages from vmscan. This mechanism is based on a patch | |
31 | by Larry Woodman of Red Hat to address several scalability problems with page | |
fa07e787 | 32 | reclaim in Linux. The problems have been observed at customer sites on large |
c24b7201 DH |
33 | memory x86_64 systems. |
34 | ||
35 | To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of | |
36 | main memory will have over 32 million 4k pages in a single zone. When a large | |
37 | fraction of these pages are not evictable for any reason [see below], vmscan | |
38 | will spend a lot of time scanning the LRU lists looking for the small fraction | |
39 | of pages that are evictable. This can result in a situation where all CPUs are | |
40 | spending 100% of their time in vmscan for hours or days on end, with the system | |
41 | completely unresponsive. | |
42 | ||
43 | The unevictable list addresses the following classes of unevictable pages: | |
44 | ||
a5e4da91 | 45 | * Those owned by ramfs. |
c24b7201 | 46 | |
a5e4da91 | 47 | * Those mapped into SHM_LOCK'd shared memory regions. |
c24b7201 | 48 | |
a5e4da91 | 49 | * Those mapped into VM_LOCKED [mlock()ed] VMAs. |
c24b7201 DH |
50 | |
51 | The infrastructure may also be able to handle other conditions that make pages | |
fa07e787 LS |
52 | unevictable, either by definition or by circumstance, in the future. |
53 | ||
54 | ||
a5e4da91 | 55 | The Unevictable Page List |
c24b7201 | 56 | ------------------------- |
fa07e787 LS |
57 | |
58 | The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list | |
59 | called the "unevictable" list and an associated page flag, PG_unevictable, to | |
c24b7201 DH |
60 | indicate that the page is being managed on the unevictable list. |
61 | ||
62 | The PG_unevictable flag is analogous to, and mutually exclusive with, the | |
63 | PG_active flag in that it indicates on which LRU list a page resides when | |
e6e8dd50 | 64 | PG_lru is set. |
fa07e787 LS |
65 | |
66 | The Unevictable LRU infrastructure maintains unevictable pages on an additional | |
67 | LRU list for a few reasons: | |
68 | ||
c24b7201 DH |
69 | (1) We get to "treat unevictable pages just like we treat other pages in the |
70 | system - which means we get to use the same code to manipulate them, the | |
71 | same code to isolate them (for migrate, etc.), the same code to keep track | |
72 | of the statistics, etc..." [Rik van Riel] | |
73 | ||
74 | (2) We want to be able to migrate unevictable pages between nodes for memory | |
75 | defragmentation, workload management and memory hotplug. The linux kernel | |
76 | can only migrate pages that it can successfully isolate from the LRU | |
77 | lists. If we were to maintain pages elsewhere than on an LRU-like list, | |
78 | where they can be found by isolate_lru_page(), we would prevent their | |
79 | migration, unless we reworked migration code to find the unevictable pages | |
80 | itself. | |
fa07e787 | 81 | |
fa07e787 | 82 | |
c24b7201 DH |
83 | The unevictable list does not differentiate between file-backed and anonymous, |
84 | swap-backed pages. This differentiation is only important while the pages are, | |
85 | in fact, evictable. | |
fa07e787 | 86 | |
c24b7201 DH |
87 | The unevictable list benefits from the "arrayification" of the per-zone LRU |
88 | lists and statistics originally proposed and posted by Christoph Lameter. | |
fa07e787 | 89 | |
c24b7201 DH |
90 | The unevictable list does not use the LRU pagevec mechanism. Rather, |
91 | unevictable pages are placed directly on the page's zone's unevictable list | |
92 | under the zone lru_lock. This allows us to prevent the stranding of pages on | |
93 | the unevictable list when one task has the page isolated from the LRU and other | |
94 | tasks are changing the "evictability" state of the page. | |
fa07e787 | 95 | |
fa07e787 | 96 | |
a5e4da91 | 97 | Memory Control Group Interaction |
c24b7201 | 98 | -------------------------------- |
fa07e787 | 99 | |
c24b7201 | 100 | The unevictable LRU facility interacts with the memory control group [aka |
99c8b231 | 101 | memory controller; see Documentation/cgroup-v1/memory.rst] by extending the |
c24b7201 DH |
102 | lru_list enum. |
103 | ||
104 | The memory controller data structure automatically gets a per-zone unevictable | |
105 | list as a result of the "arrayification" of the per-zone LRU lists (one per | |
106 | lru_list enum element). The memory controller tracks the movement of pages to | |
107 | and from the unevictable list. | |
fa07e787 | 108 | |
fa07e787 LS |
109 | When a memory control group comes under memory pressure, the controller will |
110 | not attempt to reclaim pages on the unevictable list. This has a couple of | |
c24b7201 DH |
111 | effects: |
112 | ||
113 | (1) Because the pages are "hidden" from reclaim on the unevictable list, the | |
114 | reclaim process can be more efficient, dealing only with pages that have a | |
115 | chance of being reclaimed. | |
116 | ||
117 | (2) On the other hand, if too many of the pages charged to the control group | |
118 | are unevictable, the evictable portion of the working set of the tasks in | |
119 | the control group may not fit into the available memory. This can cause | |
120 | the control group to thrash or to OOM-kill tasks. | |
121 | ||
122 | ||
a5e4da91 MR |
123 | .. _mark_addr_space_unevict: |
124 | ||
125 | Marking Address Spaces Unevictable | |
c24b7201 DH |
126 | ---------------------------------- |
127 | ||
128 | For facilities such as ramfs none of the pages attached to the address space | |
129 | may be evicted. To prevent eviction of any such pages, the AS_UNEVICTABLE | |
130 | address space flag is provided, and this can be manipulated by a filesystem | |
131 | using a number of wrapper functions: | |
132 | ||
a5e4da91 | 133 | * ``void mapping_set_unevictable(struct address_space *mapping);`` |
c24b7201 DH |
134 | |
135 | Mark the address space as being completely unevictable. | |
136 | ||
a5e4da91 | 137 | * ``void mapping_clear_unevictable(struct address_space *mapping);`` |
c24b7201 DH |
138 | |
139 | Mark the address space as being evictable. | |
140 | ||
a5e4da91 | 141 | * ``int mapping_unevictable(struct address_space *mapping);`` |
c24b7201 DH |
142 | |
143 | Query the address space, and return true if it is completely | |
144 | unevictable. | |
145 | ||
64e3d12f | 146 | These are currently used in three places in the kernel: |
c24b7201 DH |
147 | |
148 | (1) By ramfs to mark the address spaces of its inodes when they are created, | |
149 | and this mark remains for the life of the inode. | |
150 | ||
151 | (2) By SYSV SHM to mark SHM_LOCK'd address spaces until SHM_UNLOCK is called. | |
152 | ||
153 | Note that SHM_LOCK is not required to page in the locked pages if they're | |
154 | swapped out; the application must touch the pages manually if it wants to | |
155 | ensure they're in memory. | |
156 | ||
64e3d12f KHY |
157 | (3) By the i915 driver to mark pinned address space until it's unpinned. The |
158 | amount of unevictable memory marked by i915 driver is roughly the bounded | |
159 | object size in debugfs/dri/0/i915_gem_objects. | |
160 | ||
c24b7201 | 161 | |
a5e4da91 | 162 | Detecting Unevictable Pages |
c24b7201 DH |
163 | --------------------------- |
164 | ||
165 | The function page_evictable() in vmscan.c determines whether a page is | |
a5e4da91 MR |
166 | evictable or not using the query function outlined above [see section |
167 | :ref:`Marking address spaces unevictable <mark_addr_space_unevict>`] | |
168 | to check the AS_UNEVICTABLE flag. | |
c24b7201 DH |
169 | |
170 | For address spaces that are so marked after being populated (as SHM regions | |
171 | might be), the lock action (eg: SHM_LOCK) can be lazy, and need not populate | |
172 | the page tables for the region as does, for example, mlock(), nor need it make | |
173 | any special effort to push any pages in the SHM_LOCK'd area to the unevictable | |
174 | list. Instead, vmscan will do this if and when it encounters the pages during | |
175 | a reclamation scan. | |
176 | ||
177 | On an unlock action (such as SHM_UNLOCK), the unlocker (eg: shmctl()) must scan | |
178 | the pages in the region and "rescue" them from the unevictable list if no other | |
179 | condition is keeping them unevictable. If an unevictable region is destroyed, | |
180 | the pages are also "rescued" from the unevictable list in the process of | |
181 | freeing them. | |
182 | ||
183 | page_evictable() also checks for mlocked pages by testing an additional page | |
39b5f29a HD |
184 | flag, PG_mlocked (as wrapped by PageMlocked()), which is set when a page is |
185 | faulted into a VM_LOCKED vma, or found in a vma being VM_LOCKED. | |
fa07e787 LS |
186 | |
187 | ||
a5e4da91 | 188 | Vmscan's Handling of Unevictable Pages |
c24b7201 | 189 | -------------------------------------- |
fa07e787 LS |
190 | |
191 | If unevictable pages are culled in the fault path, or moved to the unevictable | |
c24b7201 DH |
192 | list at mlock() or mmap() time, vmscan will not encounter the pages until they |
193 | have become evictable again (via munlock() for example) and have been "rescued" | |
194 | from the unevictable list. However, there may be situations where we decide, | |
195 | for the sake of expediency, to leave a unevictable page on one of the regular | |
196 | active/inactive LRU lists for vmscan to deal with. vmscan checks for such | |
197 | pages in all of the shrink_{active|inactive|page}_list() functions and will | |
198 | "cull" such pages that it encounters: that is, it diverts those pages to the | |
199 | unevictable list for the zone being scanned. | |
200 | ||
201 | There may be situations where a page is mapped into a VM_LOCKED VMA, but the | |
202 | page is not marked as PG_mlocked. Such pages will make it all the way to | |
fa07e787 | 203 | shrink_page_list() where they will be detected when vmscan walks the reverse |
c24b7201 DH |
204 | map in try_to_unmap(). If try_to_unmap() returns SWAP_MLOCK, |
205 | shrink_page_list() will cull the page at that point. | |
fa07e787 | 206 | |
c24b7201 DH |
207 | To "cull" an unevictable page, vmscan simply puts the page back on the LRU list |
208 | using putback_lru_page() - the inverse operation to isolate_lru_page() - after | |
209 | dropping the page lock. Because the condition which makes the page unevictable | |
210 | may change once the page is unlocked, putback_lru_page() will recheck the | |
211 | unevictable state of a page that it places on the unevictable list. If the | |
212 | page has become unevictable, putback_lru_page() removes it from the list and | |
213 | retries, including the page_unevictable() test. Because such a race is a rare | |
214 | event and movement of pages onto the unevictable list should be rare, these | |
215 | extra evictabilty checks should not occur in the majority of calls to | |
216 | putback_lru_page(). | |
fa07e787 LS |
217 | |
218 | ||
a5e4da91 | 219 | MLOCKED Pages |
c24b7201 | 220 | ============= |
fa07e787 | 221 | |
c24b7201 DH |
222 | The unevictable page list is also useful for mlock(), in addition to ramfs and |
223 | SYSV SHM. Note that mlock() is only available in CONFIG_MMU=y situations; in | |
224 | NOMMU situations, all mappings are effectively mlocked. | |
225 | ||
226 | ||
a5e4da91 | 227 | History |
c24b7201 DH |
228 | ------- |
229 | ||
230 | The "Unevictable mlocked Pages" infrastructure is based on work originally | |
fa07e787 | 231 | posted by Nick Piggin in an RFC patch entitled "mm: mlocked pages off LRU". |
c24b7201 DH |
232 | Nick posted his patch as an alternative to a patch posted by Christoph Lameter |
233 | to achieve the same objective: hiding mlocked pages from vmscan. | |
234 | ||
235 | In Nick's patch, he used one of the struct page LRU list link fields as a count | |
236 | of VM_LOCKED VMAs that map the page. This use of the link field for a count | |
237 | prevented the management of the pages on an LRU list, and thus mlocked pages | |
238 | were not migratable as isolate_lru_page() could not find them, and the LRU list | |
239 | link field was not available to the migration subsystem. | |
240 | ||
241 | Nick resolved this by putting mlocked pages back on the lru list before | |
242 | attempting to isolate them, thus abandoning the count of VM_LOCKED VMAs. When | |
243 | Nick's patch was integrated with the Unevictable LRU work, the count was | |
244 | replaced by walking the reverse map to determine whether any VM_LOCKED VMAs | |
245 | mapped the page. More on this below. | |
246 | ||
247 | ||
a5e4da91 | 248 | Basic Management |
c24b7201 DH |
249 | ---------------- |
250 | ||
251 | mlocked pages - pages mapped into a VM_LOCKED VMA - are a class of unevictable | |
252 | pages. When such a page has been "noticed" by the memory management subsystem, | |
253 | the page is marked with the PG_mlocked flag. This can be manipulated using the | |
254 | PageMlocked() functions. | |
255 | ||
256 | A PG_mlocked page will be placed on the unevictable list when it is added to | |
257 | the LRU. Such pages can be "noticed" by memory management in several places: | |
258 | ||
259 | (1) in the mlock()/mlockall() system call handlers; | |
260 | ||
261 | (2) in the mmap() system call handler when mmapping a region with the | |
262 | MAP_LOCKED flag; | |
263 | ||
264 | (3) mmapping a region in a task that has called mlockall() with the MCL_FUTURE | |
265 | flag | |
266 | ||
267 | (4) in the fault path, if mlocked pages are "culled" in the fault path, | |
268 | and when a VM_LOCKED stack segment is expanded; or | |
269 | ||
270 | (5) as mentioned above, in vmscan:shrink_page_list() when attempting to | |
271 | reclaim a page in a VM_LOCKED VMA via try_to_unmap() | |
272 | ||
273 | all of which result in the VM_LOCKED flag being set for the VMA if it doesn't | |
274 | already have it set. | |
275 | ||
276 | mlocked pages become unlocked and rescued from the unevictable list when: | |
277 | ||
278 | (1) mapped in a range unlocked via the munlock()/munlockall() system calls; | |
279 | ||
280 | (2) munmap()'d out of the last VM_LOCKED VMA that maps the page, including | |
281 | unmapping at task exit; | |
282 | ||
283 | (3) when the page is truncated from the last VM_LOCKED VMA of an mmapped file; | |
284 | or | |
285 | ||
286 | (4) before a page is COW'd in a VM_LOCKED VMA. | |
287 | ||
288 | ||
a5e4da91 | 289 | mlock()/mlockall() System Call Handling |
c24b7201 | 290 | --------------------------------------- |
fa07e787 | 291 | |
a5e4da91 | 292 | Both [do\_]mlock() and [do\_]mlockall() system call handlers call mlock_fixup() |
c24b7201 | 293 | for each VMA in the range specified by the call. In the case of mlockall(), |
fa07e787 | 294 | this is the entire active address space of the task. Note that mlock_fixup() |
c24b7201 DH |
295 | is used for both mlocking and munlocking a range of memory. A call to mlock() |
296 | an already VM_LOCKED VMA, or to munlock() a VMA that is not VM_LOCKED is | |
297 | treated as a no-op, and mlock_fixup() simply returns. | |
298 | ||
299 | If the VMA passes some filtering as described in "Filtering Special Vmas" | |
300 | below, mlock_fixup() will attempt to merge the VMA with its neighbors or split | |
301 | off a subset of the VMA if the range does not cover the entire VMA. Once the | |
302 | VMA has been merged or split or neither, mlock_fixup() will call | |
fc05f566 | 303 | populate_vma_page_range() to fault in the pages via get_user_pages() and to |
c24b7201 DH |
304 | mark the pages as mlocked via mlock_vma_page(). |
305 | ||
306 | Note that the VMA being mlocked might be mapped with PROT_NONE. In this case, | |
307 | get_user_pages() will be unable to fault in the pages. That's okay. If pages | |
308 | do end up getting faulted into this VM_LOCKED VMA, we'll handle them in the | |
fa07e787 LS |
309 | fault path or in vmscan. |
310 | ||
311 | Also note that a page returned by get_user_pages() could be truncated or | |
c24b7201 | 312 | migrated out from under us, while we're trying to mlock it. To detect this, |
fc05f566 | 313 | populate_vma_page_range() checks page_mapping() after acquiring the page lock. |
c24b7201 DH |
314 | If the page is still associated with its mapping, we'll go ahead and call |
315 | mlock_vma_page(). If the mapping is gone, we just unlock the page and move on. | |
316 | In the worst case, this will result in a page mapped in a VM_LOCKED VMA | |
317 | remaining on a normal LRU list without being PageMlocked(). Again, vmscan will | |
318 | detect and cull such pages. | |
319 | ||
320 | mlock_vma_page() will call TestSetPageMlocked() for each page returned by | |
321 | get_user_pages(). We use TestSetPageMlocked() because the page might already | |
322 | be mlocked by another task/VMA and we don't want to do extra work. We | |
323 | especially do not want to count an mlocked page more than once in the | |
324 | statistics. If the page was already mlocked, mlock_vma_page() need do nothing | |
325 | more. | |
fa07e787 LS |
326 | |
327 | If the page was NOT already mlocked, mlock_vma_page() attempts to isolate the | |
328 | page from the LRU, as it is likely on the appropriate active or inactive list | |
c24b7201 DH |
329 | at that time. If the isolate_lru_page() succeeds, mlock_vma_page() will put |
330 | back the page - by calling putback_lru_page() - which will notice that the page | |
331 | is now mlocked and divert the page to the zone's unevictable list. If | |
fa07e787 | 332 | mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle |
c24b7201 | 333 | it later if and when it attempts to reclaim the page. |
fa07e787 LS |
334 | |
335 | ||
a5e4da91 | 336 | Filtering Special VMAs |
c24b7201 | 337 | ---------------------- |
fa07e787 | 338 | |
c24b7201 | 339 | mlock_fixup() filters several classes of "special" VMAs: |
fa07e787 | 340 | |
c24b7201 | 341 | 1) VMAs with VM_IO or VM_PFNMAP set are skipped entirely. The pages behind |
fa07e787 | 342 | these mappings are inherently pinned, so we don't need to mark them as |
c24b7201 DH |
343 | mlocked. In any case, most of the pages have no struct page in which to so |
344 | mark the page. Because of this, get_user_pages() will fail for these VMAs, | |
345 | so there is no sense in attempting to visit them. | |
346 | ||
347 | 2) VMAs mapping hugetlbfs page are already effectively pinned into memory. We | |
348 | neither need nor want to mlock() these pages. However, to preserve the | |
349 | prior behavior of mlock() - before the unevictable/mlock changes - | |
350 | mlock_fixup() will call make_pages_present() in the hugetlbfs VMA range to | |
351 | allocate the huge pages and populate the ptes. | |
352 | ||
314e51b9 KK |
353 | 3) VMAs with VM_DONTEXPAND are generally userspace mappings of kernel pages, |
354 | such as the VDSO page, relay channel pages, etc. These pages | |
fa07e787 | 355 | are inherently unevictable and are not managed on the LRU lists. |
c24b7201 | 356 | mlock_fixup() treats these VMAs the same as hugetlbfs VMAs. It calls |
fa07e787 LS |
357 | make_pages_present() to populate the ptes. |
358 | ||
c24b7201 | 359 | Note that for all of these special VMAs, mlock_fixup() does not set the |
fa07e787 | 360 | VM_LOCKED flag. Therefore, we won't have to deal with them later during |
c24b7201 DH |
361 | munlock(), munmap() or task exit. Neither does mlock_fixup() account these |
362 | VMAs against the task's "locked_vm". | |
363 | ||
a5e4da91 | 364 | .. _munlock_munlockall_handling: |
c24b7201 | 365 | |
a5e4da91 | 366 | munlock()/munlockall() System Call Handling |
c24b7201 DH |
367 | ------------------------------------------- |
368 | ||
369 | The munlock() and munlockall() system calls are handled by the same functions - | |
370 | do_mlock[all]() - as the mlock() and mlockall() system calls with the unlock vs | |
371 | lock operation indicated by an argument. So, these system calls are also | |
372 | handled by mlock_fixup(). Again, if called for an already munlocked VMA, | |
373 | mlock_fixup() simply returns. Because of the VMA filtering discussed above, | |
374 | VM_LOCKED will not be set in any "special" VMAs. So, these VMAs will be | |
fa07e787 LS |
375 | ignored for munlock. |
376 | ||
c24b7201 DH |
377 | If the VMA is VM_LOCKED, mlock_fixup() again attempts to merge or split off the |
378 | specified range. The range is then munlocked via the function | |
fc05f566 | 379 | populate_vma_page_range() - the same function used to mlock a VMA range - |
fa07e787 LS |
380 | passing a flag to indicate that munlock() is being performed. |
381 | ||
c24b7201 | 382 | Because the VMA access protections could have been changed to PROT_NONE after |
63d6c5ad | 383 | faulting in and mlocking pages, get_user_pages() was unreliable for visiting |
c24b7201 | 384 | these pages for munlocking. Because we don't want to leave pages mlocked, |
fa07e787 | 385 | get_user_pages() was enhanced to accept a flag to ignore the permissions when |
c24b7201 DH |
386 | fetching the pages - all of which should be resident as a result of previous |
387 | mlocking. | |
fa07e787 | 388 | |
fc05f566 | 389 | For munlock(), populate_vma_page_range() unlocks individual pages by calling |
fa07e787 | 390 | munlock_vma_page(). munlock_vma_page() unconditionally clears the PG_mlocked |
c24b7201 DH |
391 | flag using TestClearPageMlocked(). As with mlock_vma_page(), |
392 | munlock_vma_page() use the Test*PageMlocked() function to handle the case where | |
393 | the page might have already been unlocked by another task. If the page was | |
394 | mlocked, munlock_vma_page() updates that zone statistics for the number of | |
395 | mlocked pages. Note, however, that at this point we haven't checked whether | |
396 | the page is mapped by other VM_LOCKED VMAs. | |
397 | ||
398 | We can't call try_to_munlock(), the function that walks the reverse map to | |
399 | check for other VM_LOCKED VMAs, without first isolating the page from the LRU. | |
fa07e787 | 400 | try_to_munlock() is a variant of try_to_unmap() and thus requires that the page |
c24b7201 DH |
401 | not be on an LRU list [more on these below]. However, the call to |
402 | isolate_lru_page() could fail, in which case we couldn't try_to_munlock(). So, | |
403 | we go ahead and clear PG_mlocked up front, as this might be the only chance we | |
404 | have. If we can successfully isolate the page, we go ahead and | |
fa07e787 | 405 | try_to_munlock(), which will restore the PG_mlocked flag and update the zone |
c24b7201 | 406 | page statistics if it finds another VMA holding the page mlocked. If we fail |
fa07e787 | 407 | to isolate the page, we'll have left a potentially mlocked page on the LRU. |
c24b7201 DH |
408 | This is fine, because we'll catch it later if and if vmscan tries to reclaim |
409 | the page. This should be relatively rare. | |
410 | ||
411 | ||
a5e4da91 | 412 | Migrating MLOCKED Pages |
c24b7201 DH |
413 | ----------------------- |
414 | ||
415 | A page that is being migrated has been isolated from the LRU lists and is held | |
416 | locked across unmapping of the page, updating the page's address space entry | |
417 | and copying the contents and state, until the page table entry has been | |
418 | replaced with an entry that refers to the new page. Linux supports migration | |
419 | of mlocked pages and other unevictable pages. This involves simply moving the | |
420 | PG_mlocked and PG_unevictable states from the old page to the new page. | |
421 | ||
422 | Note that page migration can race with mlocking or munlocking of the same page. | |
423 | This has been discussed from the mlock/munlock perspective in the respective | |
424 | sections above. Both processes (migration and m[un]locking) hold the page | |
425 | locked. This provides the first level of synchronization. Page migration | |
426 | zeros out the page_mapping of the old page before unlocking it, so m[un]lock | |
427 | can skip these pages by testing the page mapping under page lock. | |
428 | ||
429 | To complete page migration, we place the new and old pages back onto the LRU | |
430 | after dropping the page lock. The "unneeded" page - old page on success, new | |
431 | page on failure - will be freed when the reference count held by the migration | |
432 | process is released. To ensure that we don't strand pages on the unevictable | |
433 | list because of a race between munlock and migration, page migration uses the | |
434 | putback_lru_page() function to add migrated pages back to the LRU. | |
435 | ||
436 | ||
a5e4da91 | 437 | Compacting MLOCKED Pages |
922c0551 EM |
438 | ------------------------ |
439 | ||
440 | The unevictable LRU can be scanned for compactable regions and the default | |
441 | behavior is to do so. /proc/sys/vm/compact_unevictable_allowed controls | |
57043247 | 442 | this behavior (see Documentation/admin-guide/sysctl/vm.rst). Once scanning of the |
922c0551 EM |
443 | unevictable LRU is enabled, the work of compaction is mostly handled by |
444 | the page migration code and the same work flow as described in MIGRATING | |
445 | MLOCKED PAGES will apply. | |
446 | ||
a5e4da91 | 447 | MLOCKING Transparent Huge Pages |
6fb8ddfc KS |
448 | ------------------------------- |
449 | ||
450 | A transparent huge page is represented by a single entry on an LRU list. | |
451 | Therefore, we can only make unevictable an entire compound page, not | |
452 | individual subpages. | |
453 | ||
454 | If a user tries to mlock() part of a huge page, we want the rest of the | |
455 | page to be reclaimable. | |
456 | ||
457 | We cannot just split the page on partial mlock() as split_huge_page() can | |
458 | fail and new intermittent failure mode for the syscall is undesirable. | |
459 | ||
460 | We handle this by keeping PTE-mapped huge pages on normal LRU lists: the | |
461 | PMD on border of VM_LOCKED VMA will be split into PTE table. | |
462 | ||
463 | This way the huge page is accessible for vmscan. Under memory pressure the | |
464 | page will be split, subpages which belong to VM_LOCKED VMAs will be moved | |
465 | to unevictable LRU and the rest can be reclaimed. | |
466 | ||
467 | See also comment in follow_trans_huge_pmd(). | |
922c0551 | 468 | |
a5e4da91 | 469 | mmap(MAP_LOCKED) System Call Handling |
c24b7201 | 470 | ------------------------------------- |
fa07e787 | 471 | |
df5cbb27 | 472 | In addition the mlock()/mlockall() system calls, an application can request |
c24b7201 | 473 | that a region of memory be mlocked supplying the MAP_LOCKED flag to the mmap() |
9b012a29 MH |
474 | call. There is one important and subtle difference here, though. mmap() + mlock() |
475 | will fail if the range cannot be faulted in (e.g. because mm_populate fails) | |
476 | and returns with ENOMEM while mmap(MAP_LOCKED) will not fail. The mmaped | |
477 | area will still have properties of the locked area - aka. pages will not get | |
478 | swapped out - but major page faults to fault memory in might still happen. | |
479 | ||
480 | Furthermore, any mmap() call or brk() call that expands the heap by a | |
fa07e787 | 481 | task that has previously called mlockall() with the MCL_FUTURE flag will result |
c24b7201 DH |
482 | in the newly mapped memory being mlocked. Before the unevictable/mlock |
483 | changes, the kernel simply called make_pages_present() to allocate pages and | |
484 | populate the page table. | |
fa07e787 LS |
485 | |
486 | To mlock a range of memory under the unevictable/mlock infrastructure, the | |
487 | mmap() handler and task address space expansion functions call | |
fc05f566 KS |
488 | populate_vma_page_range() specifying the vma and the address range to mlock. |
489 | ||
490 | The callers of populate_vma_page_range() will have already added the memory range | |
c24b7201 | 491 | to be mlocked to the task's "locked_vm". To account for filtered VMAs, |
fc05f566 | 492 | populate_vma_page_range() returns the number of pages NOT mlocked. All of the |
c24b7201 DH |
493 | callers then subtract a non-negative return value from the task's locked_vm. A |
494 | negative return value represent an error - for example, from get_user_pages() | |
495 | attempting to fault in a VMA with PROT_NONE access. In this case, we leave the | |
496 | memory range accounted as locked_vm, as the protections could be changed later | |
497 | and pages allocated into that region. | |
fa07e787 LS |
498 | |
499 | ||
a5e4da91 | 500 | munmap()/exit()/exec() System Call Handling |
c24b7201 | 501 | ------------------------------------------- |
fa07e787 LS |
502 | |
503 | When unmapping an mlocked region of memory, whether by an explicit call to | |
504 | munmap() or via an internal unmap from exit() or exec() processing, we must | |
c24b7201 | 505 | munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. |
63d6c5ad HD |
506 | Before the unevictable/mlock changes, mlocking did not mark the pages in any |
507 | way, so unmapping them required no processing. | |
fa07e787 LS |
508 | |
509 | To munlock a range of memory under the unevictable/mlock infrastructure, the | |
c24b7201 | 510 | munmap() handler and task address space call tear down function |
fa07e787 | 511 | munlock_vma_pages_all(). The name reflects the observation that one always |
c24b7201 DH |
512 | specifies the entire VMA range when munlock()ing during unmap of a region. |
513 | Because of the VMA filtering when mlocking() regions, only "normal" VMAs that | |
fa07e787 LS |
514 | actually contain mlocked pages will be passed to munlock_vma_pages_all(). |
515 | ||
c24b7201 | 516 | munlock_vma_pages_all() clears the VM_LOCKED VMA flag and, like mlock_fixup() |
fa07e787 | 517 | for the munlock case, calls __munlock_vma_pages_range() to walk the page table |
c24b7201 DH |
518 | for the VMA's memory range and munlock_vma_page() each resident page mapped by |
519 | the VMA. This effectively munlocks the page, only if this is the last | |
520 | VM_LOCKED VMA that maps the page. | |
fa07e787 | 521 | |
fa07e787 | 522 | |
c24b7201 DH |
523 | try_to_unmap() |
524 | -------------- | |
fa07e787 | 525 | |
c24b7201 | 526 | Pages can, of course, be mapped into multiple VMAs. Some of these VMAs may |
fa07e787 | 527 | have VM_LOCKED flag set. It is possible for a page mapped into one or more |
c24b7201 DH |
528 | VM_LOCKED VMAs not to have the PG_mlocked flag set and therefore reside on one |
529 | of the active or inactive LRU lists. This could happen if, for example, a task | |
530 | in the process of munlocking the page could not isolate the page from the LRU. | |
531 | As a result, vmscan/shrink_page_list() might encounter such a page as described | |
532 | in section "vmscan's handling of unevictable pages". To handle this situation, | |
533 | try_to_unmap() checks for VM_LOCKED VMAs while it is walking a page's reverse | |
534 | map. | |
fa07e787 LS |
535 | |
536 | try_to_unmap() is always called, by either vmscan for reclaim or for page | |
c24b7201 | 537 | migration, with the argument page locked and isolated from the LRU. Separate |
b87537d9 HD |
538 | functions handle anonymous and mapped file and KSM pages, as these types of |
539 | pages have different reverse map lookup mechanisms, with different locking. | |
540 | In each case, whether rmap_walk_anon() or rmap_walk_file() or rmap_walk_ksm(), | |
541 | it will call try_to_unmap_one() for every VMA which might contain the page. | |
c24b7201 | 542 | |
b87537d9 HD |
543 | When trying to reclaim, if try_to_unmap_one() finds the page in a VM_LOCKED |
544 | VMA, it will then mlock the page via mlock_vma_page() instead of unmapping it, | |
545 | and return SWAP_MLOCK to indicate that the page is unevictable: and the scan | |
546 | stops there. | |
c24b7201 | 547 | |
b87537d9 HD |
548 | mlock_vma_page() is called while holding the page table's lock (in addition |
549 | to the page lock, and the rmap lock): to serialize against concurrent mlock or | |
550 | munlock or munmap system calls, mm teardown (munlock_vma_pages_all), reclaim, | |
551 | holepunching, and truncation of file pages and their anonymous COWed pages. | |
c24b7201 | 552 | |
c24b7201 | 553 | |
a5e4da91 | 554 | try_to_munlock() Reverse Map Scan |
c24b7201 DH |
555 | --------------------------------- |
556 | ||
a5e4da91 MR |
557 | .. warning:: |
558 | [!] TODO/FIXME: a better name might be page_mlocked() - analogous to the | |
559 | page_referenced() reverse map walker. | |
c24b7201 | 560 | |
a5e4da91 MR |
561 | When munlock_vma_page() [see section :ref:`munlock()/munlockall() System Call |
562 | Handling <munlock_munlockall_handling>` above] tries to munlock a | |
563 | page, it needs to determine whether or not the page is mapped by any | |
564 | VM_LOCKED VMA without actually attempting to unmap all PTEs from the | |
565 | page. For this purpose, the unevictable/mlock infrastructure | |
c24b7201 | 566 | introduced a variant of try_to_unmap() called try_to_munlock(). |
fa07e787 LS |
567 | |
568 | try_to_munlock() calls the same functions as try_to_unmap() for anonymous and | |
b87537d9 | 569 | mapped file and KSM pages with a flag argument specifying unlock versus unmap |
fa07e787 | 570 | processing. Again, these functions walk the respective reverse maps looking |
7a14239a | 571 | for VM_LOCKED VMAs. When such a VMA is found, as in the try_to_unmap() case, |
b87537d9 HD |
572 | the functions mlock the page via mlock_vma_page() and return SWAP_MLOCK. This |
573 | undoes the pre-clearing of the page's PG_mlocked done by munlock_vma_page. | |
c24b7201 | 574 | |
c24b7201 DH |
575 | Note that try_to_munlock()'s reverse map walk must visit every VMA in a page's |
576 | reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA. | |
b87537d9 | 577 | However, the scan can terminate when it encounters a VM_LOCKED VMA. |
c24b7201 DH |
578 | Although try_to_munlock() might be called a great many times when munlocking a |
579 | large region or tearing down a large address space that has been mlocked via | |
580 | mlockall(), overall this is a fairly rare event. | |
581 | ||
582 | ||
a5e4da91 | 583 | Page Reclaim in shrink_*_list() |
c24b7201 DH |
584 | ------------------------------- |
585 | ||
586 | shrink_active_list() culls any obviously unevictable pages - i.e. | |
39b5f29a | 587 | !page_evictable(page) - diverting these to the unevictable list. |
c24b7201 DH |
588 | However, shrink_active_list() only sees unevictable pages that made it onto the |
589 | active/inactive lru lists. Note that these pages do not have PageUnevictable | |
590 | set - otherwise they would be on the unevictable list and shrink_active_list | |
591 | would never see them. | |
fa07e787 LS |
592 | |
593 | Some examples of these unevictable pages on the LRU lists are: | |
594 | ||
c24b7201 DH |
595 | (1) ramfs pages that have been placed on the LRU lists when first allocated. |
596 | ||
597 | (2) SHM_LOCK'd shared memory pages. shmctl(SHM_LOCK) does not attempt to | |
598 | allocate or fault in the pages in the shared memory region. This happens | |
599 | when an application accesses the page the first time after SHM_LOCK'ing | |
600 | the segment. | |
fa07e787 | 601 | |
c24b7201 DH |
602 | (3) mlocked pages that could not be isolated from the LRU and moved to the |
603 | unevictable list in mlock_vma_page(). | |
fa07e787 | 604 | |
c24b7201 DH |
605 | shrink_inactive_list() also diverts any unevictable pages that it finds on the |
606 | inactive lists to the appropriate zone's unevictable list. | |
fa07e787 | 607 | |
c24b7201 DH |
608 | shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd |
609 | after shrink_active_list() had moved them to the inactive list, or pages mapped | |
610 | into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to | |
611 | recheck via try_to_munlock(). shrink_inactive_list() won't notice the latter, | |
612 | but will pass on to shrink_page_list(). | |
fa07e787 LS |
613 | |
614 | shrink_page_list() again culls obviously unevictable pages that it could | |
63d6c5ad | 615 | encounter for similar reason to shrink_inactive_list(). Pages mapped into |
c24b7201 | 616 | VM_LOCKED VMAs but without PG_mlocked set will make it all the way to |
63d6c5ad HD |
617 | try_to_unmap(). shrink_page_list() will divert them to the unevictable list |
618 | when try_to_unmap() returns SWAP_MLOCK, as discussed above. |