Commit | Line | Data |
---|---|---|
a5e4da91 | 1 | .. _unevictable_lru: |
c24b7201 | 2 | |
a5e4da91 MR |
3 | ============================== |
4 | Unevictable LRU Infrastructure | |
5 | ============================== | |
c24b7201 | 6 | |
a5e4da91 MR |
7 | .. contents:: :local: |
8 | ||
9 | ||
10 | Introduction | |
c24b7201 DH |
11 | ============ |
12 | ||
13 | This document describes the Linux memory manager's "Unevictable LRU" | |
14 | infrastructure and the use of this to manage several types of "unevictable" | |
15 | pages. | |
16 | ||
17 | The document attempts to provide the overall rationale behind this mechanism | |
18 | and the rationale for some of the design decisions that drove the | |
19 | implementation. The latter design rationale is discussed in the context of an | |
20 | implementation description. Admittedly, one can obtain the implementation | |
21 | details - the "what does it do?" - by reading the code. One hopes that the | |
22 | descriptions below add value by provide the answer to "why does it do that?". | |
23 | ||
24 | ||
a5e4da91 MR |
25 | |
26 | The Unevictable LRU | |
c24b7201 DH |
27 | =================== |
28 | ||
29 | The Unevictable LRU facility adds an additional LRU list to track unevictable | |
30 | pages and to hide these pages from vmscan. This mechanism is based on a patch | |
31 | by Larry Woodman of Red Hat to address several scalability problems with page | |
fa07e787 | 32 | reclaim in Linux. The problems have been observed at customer sites on large |
c24b7201 DH |
33 | memory x86_64 systems. |
34 | ||
35 | To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of | |
15b44736 | 36 | main memory will have over 32 million 4k pages in a single node. When a large |
c24b7201 DH |
37 | fraction of these pages are not evictable for any reason [see below], vmscan |
38 | will spend a lot of time scanning the LRU lists looking for the small fraction | |
39 | of pages that are evictable. This can result in a situation where all CPUs are | |
40 | spending 100% of their time in vmscan for hours or days on end, with the system | |
41 | completely unresponsive. | |
42 | ||
43 | The unevictable list addresses the following classes of unevictable pages: | |
44 | ||
a5e4da91 | 45 | * Those owned by ramfs. |
c24b7201 | 46 | |
a5e4da91 | 47 | * Those mapped into SHM_LOCK'd shared memory regions. |
c24b7201 | 48 | |
a5e4da91 | 49 | * Those mapped into VM_LOCKED [mlock()ed] VMAs. |
c24b7201 DH |
50 | |
51 | The infrastructure may also be able to handle other conditions that make pages | |
fa07e787 LS |
52 | unevictable, either by definition or by circumstance, in the future. |
53 | ||
54 | ||
577e9846 HD |
55 | The Unevictable LRU Page List |
56 | ----------------------------- | |
57 | ||
58 | The Unevictable LRU page list is a lie. It was never an LRU-ordered list, but a | |
59 | companion to the LRU-ordered anonymous and file, active and inactive page lists; | |
60 | and now it is not even a page list. But following familiar convention, here in | |
61 | this document and in the source, we often imagine it as a fifth LRU page list. | |
fa07e787 | 62 | |
15b44736 | 63 | The Unevictable LRU infrastructure consists of an additional, per-node, LRU list |
fa07e787 | 64 | called the "unevictable" list and an associated page flag, PG_unevictable, to |
c24b7201 DH |
65 | indicate that the page is being managed on the unevictable list. |
66 | ||
67 | The PG_unevictable flag is analogous to, and mutually exclusive with, the | |
68 | PG_active flag in that it indicates on which LRU list a page resides when | |
e6e8dd50 | 69 | PG_lru is set. |
fa07e787 | 70 | |
577e9846 HD |
71 | The Unevictable LRU infrastructure maintains unevictable pages as if they were |
72 | on an additional LRU list for a few reasons: | |
fa07e787 | 73 | |
c24b7201 DH |
74 | (1) We get to "treat unevictable pages just like we treat other pages in the |
75 | system - which means we get to use the same code to manipulate them, the | |
76 | same code to isolate them (for migrate, etc.), the same code to keep track | |
77 | of the statistics, etc..." [Rik van Riel] | |
78 | ||
79 | (2) We want to be able to migrate unevictable pages between nodes for memory | |
577e9846 | 80 | defragmentation, workload management and memory hotplug. The Linux kernel |
c24b7201 | 81 | can only migrate pages that it can successfully isolate from the LRU |
577e9846 HD |
82 | lists (or "Movable" pages: outside of consideration here). If we were to |
83 | maintain pages elsewhere than on an LRU-like list, where they can be | |
84 | detected by isolate_lru_page(), we would prevent their migration. | |
fa07e787 | 85 | |
c24b7201 DH |
86 | The unevictable list does not differentiate between file-backed and anonymous, |
87 | swap-backed pages. This differentiation is only important while the pages are, | |
88 | in fact, evictable. | |
fa07e787 | 89 | |
15b44736 | 90 | The unevictable list benefits from the "arrayification" of the per-node LRU |
c24b7201 | 91 | lists and statistics originally proposed and posted by Christoph Lameter. |
fa07e787 | 92 | |
fa07e787 | 93 | |
a5e4da91 | 94 | Memory Control Group Interaction |
c24b7201 | 95 | -------------------------------- |
fa07e787 | 96 | |
c24b7201 | 97 | The unevictable LRU facility interacts with the memory control group [aka |
577e9846 HD |
98 | memory controller; see Documentation/admin-guide/cgroup-v1/memory.rst] by |
99 | extending the lru_list enum. | |
c24b7201 | 100 | |
15b44736 HD |
101 | The memory controller data structure automatically gets a per-node unevictable |
102 | list as a result of the "arrayification" of the per-node LRU lists (one per | |
c24b7201 DH |
103 | lru_list enum element). The memory controller tracks the movement of pages to |
104 | and from the unevictable list. | |
fa07e787 | 105 | |
fa07e787 LS |
106 | When a memory control group comes under memory pressure, the controller will |
107 | not attempt to reclaim pages on the unevictable list. This has a couple of | |
c24b7201 DH |
108 | effects: |
109 | ||
110 | (1) Because the pages are "hidden" from reclaim on the unevictable list, the | |
111 | reclaim process can be more efficient, dealing only with pages that have a | |
112 | chance of being reclaimed. | |
113 | ||
114 | (2) On the other hand, if too many of the pages charged to the control group | |
115 | are unevictable, the evictable portion of the working set of the tasks in | |
116 | the control group may not fit into the available memory. This can cause | |
117 | the control group to thrash or to OOM-kill tasks. | |
118 | ||
119 | ||
a5e4da91 MR |
120 | .. _mark_addr_space_unevict: |
121 | ||
122 | Marking Address Spaces Unevictable | |
c24b7201 DH |
123 | ---------------------------------- |
124 | ||
125 | For facilities such as ramfs none of the pages attached to the address space | |
126 | may be evicted. To prevent eviction of any such pages, the AS_UNEVICTABLE | |
127 | address space flag is provided, and this can be manipulated by a filesystem | |
128 | using a number of wrapper functions: | |
129 | ||
a5e4da91 | 130 | * ``void mapping_set_unevictable(struct address_space *mapping);`` |
c24b7201 DH |
131 | |
132 | Mark the address space as being completely unevictable. | |
133 | ||
a5e4da91 | 134 | * ``void mapping_clear_unevictable(struct address_space *mapping);`` |
c24b7201 DH |
135 | |
136 | Mark the address space as being evictable. | |
137 | ||
a5e4da91 | 138 | * ``int mapping_unevictable(struct address_space *mapping);`` |
c24b7201 DH |
139 | |
140 | Query the address space, and return true if it is completely | |
141 | unevictable. | |
142 | ||
64e3d12f | 143 | These are currently used in three places in the kernel: |
c24b7201 DH |
144 | |
145 | (1) By ramfs to mark the address spaces of its inodes when they are created, | |
146 | and this mark remains for the life of the inode. | |
147 | ||
148 | (2) By SYSV SHM to mark SHM_LOCK'd address spaces until SHM_UNLOCK is called. | |
c24b7201 DH |
149 | Note that SHM_LOCK is not required to page in the locked pages if they're |
150 | swapped out; the application must touch the pages manually if it wants to | |
151 | ensure they're in memory. | |
152 | ||
64e3d12f KHY |
153 | (3) By the i915 driver to mark pinned address space until it's unpinned. The |
154 | amount of unevictable memory marked by i915 driver is roughly the bounded | |
155 | object size in debugfs/dri/0/i915_gem_objects. | |
156 | ||
c24b7201 | 157 | |
a5e4da91 | 158 | Detecting Unevictable Pages |
c24b7201 DH |
159 | --------------------------- |
160 | ||
577e9846 | 161 | The function page_evictable() in mm/internal.h determines whether a page is |
a5e4da91 MR |
162 | evictable or not using the query function outlined above [see section |
163 | :ref:`Marking address spaces unevictable <mark_addr_space_unevict>`] | |
164 | to check the AS_UNEVICTABLE flag. | |
c24b7201 DH |
165 | |
166 | For address spaces that are so marked after being populated (as SHM regions | |
577e9846 | 167 | might be), the lock action (e.g. SHM_LOCK) can be lazy, and need not populate |
c24b7201 DH |
168 | the page tables for the region as does, for example, mlock(), nor need it make |
169 | any special effort to push any pages in the SHM_LOCK'd area to the unevictable | |
170 | list. Instead, vmscan will do this if and when it encounters the pages during | |
171 | a reclamation scan. | |
172 | ||
577e9846 | 173 | On an unlock action (such as SHM_UNLOCK), the unlocker (e.g. shmctl()) must scan |
c24b7201 DH |
174 | the pages in the region and "rescue" them from the unevictable list if no other |
175 | condition is keeping them unevictable. If an unevictable region is destroyed, | |
176 | the pages are also "rescued" from the unevictable list in the process of | |
177 | freeing them. | |
178 | ||
179 | page_evictable() also checks for mlocked pages by testing an additional page | |
39b5f29a | 180 | flag, PG_mlocked (as wrapped by PageMlocked()), which is set when a page is |
577e9846 | 181 | faulted into a VM_LOCKED VMA, or found in a VMA being VM_LOCKED. |
fa07e787 LS |
182 | |
183 | ||
a5e4da91 | 184 | Vmscan's Handling of Unevictable Pages |
c24b7201 | 185 | -------------------------------------- |
fa07e787 LS |
186 | |
187 | If unevictable pages are culled in the fault path, or moved to the unevictable | |
c24b7201 DH |
188 | list at mlock() or mmap() time, vmscan will not encounter the pages until they |
189 | have become evictable again (via munlock() for example) and have been "rescued" | |
190 | from the unevictable list. However, there may be situations where we decide, | |
577e9846 | 191 | for the sake of expediency, to leave an unevictable page on one of the regular |
c24b7201 DH |
192 | active/inactive LRU lists for vmscan to deal with. vmscan checks for such |
193 | pages in all of the shrink_{active|inactive|page}_list() functions and will | |
194 | "cull" such pages that it encounters: that is, it diverts those pages to the | |
577e9846 | 195 | unevictable list for the memory cgroup and node being scanned. |
c24b7201 DH |
196 | |
197 | There may be situations where a page is mapped into a VM_LOCKED VMA, but the | |
198 | page is not marked as PG_mlocked. Such pages will make it all the way to | |
577e9846 | 199 | shrink_active_list() or shrink_page_list() where they will be detected when |
9a7d7a80 | 200 | vmscan walks the reverse map in folio_referenced() or try_to_unmap(). The page |
577e9846 | 201 | is culled to the unevictable list when it is released by the shrinker. |
fa07e787 | 202 | |
c24b7201 DH |
203 | To "cull" an unevictable page, vmscan simply puts the page back on the LRU list |
204 | using putback_lru_page() - the inverse operation to isolate_lru_page() - after | |
205 | dropping the page lock. Because the condition which makes the page unevictable | |
577e9846 HD |
206 | may change once the page is unlocked, __pagevec_lru_add_fn() will recheck the |
207 | unevictable state of a page before placing it on the unevictable list. | |
fa07e787 LS |
208 | |
209 | ||
a5e4da91 | 210 | MLOCKED Pages |
c24b7201 | 211 | ============= |
fa07e787 | 212 | |
c24b7201 DH |
213 | The unevictable page list is also useful for mlock(), in addition to ramfs and |
214 | SYSV SHM. Note that mlock() is only available in CONFIG_MMU=y situations; in | |
215 | NOMMU situations, all mappings are effectively mlocked. | |
216 | ||
217 | ||
a5e4da91 | 218 | History |
c24b7201 DH |
219 | ------- |
220 | ||
221 | The "Unevictable mlocked Pages" infrastructure is based on work originally | |
fa07e787 | 222 | posted by Nick Piggin in an RFC patch entitled "mm: mlocked pages off LRU". |
c24b7201 DH |
223 | Nick posted his patch as an alternative to a patch posted by Christoph Lameter |
224 | to achieve the same objective: hiding mlocked pages from vmscan. | |
225 | ||
226 | In Nick's patch, he used one of the struct page LRU list link fields as a count | |
577e9846 HD |
227 | of VM_LOCKED VMAs that map the page (Rik van Riel had the same idea three years |
228 | earlier). But this use of the link field for a count prevented the management | |
229 | of the pages on an LRU list, and thus mlocked pages were not migratable as | |
230 | isolate_lru_page() could not detect them, and the LRU list link field was not | |
231 | available to the migration subsystem. | |
c24b7201 | 232 | |
577e9846 | 233 | Nick resolved this by putting mlocked pages back on the LRU list before |
c24b7201 DH |
234 | attempting to isolate them, thus abandoning the count of VM_LOCKED VMAs. When |
235 | Nick's patch was integrated with the Unevictable LRU work, the count was | |
577e9846 HD |
236 | replaced by walking the reverse map when munlocking, to determine whether any |
237 | other VM_LOCKED VMAs still mapped the page. | |
238 | ||
239 | However, walking the reverse map for each page when munlocking was ugly and | |
240 | inefficient, and could lead to catastrophic contention on a file's rmap lock, | |
241 | when many processes which had it mlocked were trying to exit. In 5.18, the | |
242 | idea of keeping mlock_count in Unevictable LRU list link field was revived and | |
243 | put to work, without preventing the migration of mlocked pages. This is why | |
244 | the "Unevictable LRU list" cannot be a linked list of pages now; but there was | |
245 | no use for that linked list anyway - though its size is maintained for meminfo. | |
c24b7201 DH |
246 | |
247 | ||
a5e4da91 | 248 | Basic Management |
c24b7201 DH |
249 | ---------------- |
250 | ||
251 | mlocked pages - pages mapped into a VM_LOCKED VMA - are a class of unevictable | |
252 | pages. When such a page has been "noticed" by the memory management subsystem, | |
253 | the page is marked with the PG_mlocked flag. This can be manipulated using the | |
254 | PageMlocked() functions. | |
255 | ||
256 | A PG_mlocked page will be placed on the unevictable list when it is added to | |
257 | the LRU. Such pages can be "noticed" by memory management in several places: | |
258 | ||
577e9846 | 259 | (1) in the mlock()/mlock2()/mlockall() system call handlers; |
c24b7201 DH |
260 | |
261 | (2) in the mmap() system call handler when mmapping a region with the | |
262 | MAP_LOCKED flag; | |
263 | ||
264 | (3) mmapping a region in a task that has called mlockall() with the MCL_FUTURE | |
577e9846 | 265 | flag; |
c24b7201 | 266 | |
577e9846 | 267 | (4) in the fault path and when a VM_LOCKED stack segment is expanded; or |
c24b7201 DH |
268 | |
269 | (5) as mentioned above, in vmscan:shrink_page_list() when attempting to | |
9a7d7a80 | 270 | reclaim a page in a VM_LOCKED VMA by folio_referenced() or try_to_unmap(). |
c24b7201 DH |
271 | |
272 | mlocked pages become unlocked and rescued from the unevictable list when: | |
273 | ||
274 | (1) mapped in a range unlocked via the munlock()/munlockall() system calls; | |
275 | ||
276 | (2) munmap()'d out of the last VM_LOCKED VMA that maps the page, including | |
277 | unmapping at task exit; | |
278 | ||
279 | (3) when the page is truncated from the last VM_LOCKED VMA of an mmapped file; | |
280 | or | |
281 | ||
282 | (4) before a page is COW'd in a VM_LOCKED VMA. | |
283 | ||
284 | ||
577e9846 HD |
285 | mlock()/mlock2()/mlockall() System Call Handling |
286 | ------------------------------------------------ | |
fa07e787 | 287 | |
577e9846 | 288 | mlock(), mlock2() and mlockall() system call handlers proceed to mlock_fixup() |
c24b7201 | 289 | for each VMA in the range specified by the call. In the case of mlockall(), |
fa07e787 | 290 | this is the entire active address space of the task. Note that mlock_fixup() |
c24b7201 | 291 | is used for both mlocking and munlocking a range of memory. A call to mlock() |
577e9846 HD |
292 | an already VM_LOCKED VMA, or to munlock() a VMA that is not VM_LOCKED, is |
293 | treated as a no-op and mlock_fixup() simply returns. | |
c24b7201 | 294 | |
577e9846 | 295 | If the VMA passes some filtering as described in "Filtering Special VMAs" |
c24b7201 | 296 | below, mlock_fixup() will attempt to merge the VMA with its neighbors or split |
577e9846 HD |
297 | off a subset of the VMA if the range does not cover the entire VMA. Any pages |
298 | already present in the VMA are then marked as mlocked by mlock_page() via | |
299 | mlock_pte_range() via walk_page_range() via mlock_vma_pages_range(). | |
300 | ||
301 | Before returning from the system call, do_mlock() or mlockall() will call | |
302 | __mm_populate() to fault in the remaining pages via get_user_pages() and to | |
303 | mark those pages as mlocked as they are faulted. | |
c24b7201 DH |
304 | |
305 | Note that the VMA being mlocked might be mapped with PROT_NONE. In this case, | |
306 | get_user_pages() will be unable to fault in the pages. That's okay. If pages | |
577e9846 HD |
307 | do end up getting faulted into this VM_LOCKED VMA, they will be handled in the |
308 | fault path - which is also how mlock2()'s MLOCK_ONFAULT areas are handled. | |
309 | ||
310 | For each PTE (or PMD) being faulted into a VMA, the page add rmap function | |
311 | calls mlock_vma_page(), which calls mlock_page() when the VMA is VM_LOCKED | |
312 | (unless it is a PTE mapping of a part of a transparent huge page). Or when | |
313 | it is a newly allocated anonymous page, lru_cache_add_inactive_or_unevictable() | |
314 | calls mlock_new_page() instead: similar to mlock_page(), but can make better | |
315 | judgments, since this page is held exclusively and known not to be on LRU yet. | |
316 | ||
317 | mlock_page() sets PageMlocked immediately, then places the page on the CPU's | |
318 | mlock pagevec, to batch up the rest of the work to be done under lru_lock by | |
319 | __mlock_page(). __mlock_page() sets PageUnevictable, initializes mlock_count | |
320 | and moves the page to unevictable state ("the unevictable LRU", but with | |
321 | mlock_count in place of LRU threading). Or if the page was already PageLRU | |
322 | and PageUnevictable and PageMlocked, it simply increments the mlock_count. | |
323 | ||
324 | But in practice that may not work ideally: the page may not yet be on an LRU, or | |
325 | it may have been temporarily isolated from LRU. In such cases the mlock_count | |
326 | field cannot be touched, but will be set to 0 later when __pagevec_lru_add_fn() | |
327 | returns the page to "LRU". Races prohibit mlock_count from being set to 1 then: | |
328 | rather than risk stranding a page indefinitely as unevictable, always err with | |
329 | mlock_count on the low side, so that when munlocked the page will be rescued to | |
330 | an evictable LRU, then perhaps be mlocked again later if vmscan finds it in a | |
331 | VM_LOCKED VMA. | |
fa07e787 LS |
332 | |
333 | ||
a5e4da91 | 334 | Filtering Special VMAs |
c24b7201 | 335 | ---------------------- |
fa07e787 | 336 | |
c24b7201 | 337 | mlock_fixup() filters several classes of "special" VMAs: |
fa07e787 | 338 | |
c24b7201 | 339 | 1) VMAs with VM_IO or VM_PFNMAP set are skipped entirely. The pages behind |
fa07e787 | 340 | these mappings are inherently pinned, so we don't need to mark them as |
c24b7201 DH |
341 | mlocked. In any case, most of the pages have no struct page in which to so |
342 | mark the page. Because of this, get_user_pages() will fail for these VMAs, | |
343 | so there is no sense in attempting to visit them. | |
344 | ||
345 | 2) VMAs mapping hugetlbfs page are already effectively pinned into memory. We | |
577e9846 HD |
346 | neither need nor want to mlock() these pages. But __mm_populate() includes |
347 | hugetlbfs ranges, allocating the huge pages and populating the PTEs. | |
c24b7201 | 348 | |
314e51b9 | 349 | 3) VMAs with VM_DONTEXPAND are generally userspace mappings of kernel pages, |
577e9846 HD |
350 | such as the VDSO page, relay channel pages, etc. These pages are inherently |
351 | unevictable and are not managed on the LRU lists. __mm_populate() includes | |
352 | these ranges, populating the PTEs if not already populated. | |
353 | ||
354 | 4) VMAs with VM_MIXEDMAP set are not marked VM_LOCKED, but __mm_populate() | |
355 | includes these ranges, populating the PTEs if not already populated. | |
fa07e787 | 356 | |
c24b7201 | 357 | Note that for all of these special VMAs, mlock_fixup() does not set the |
fa07e787 | 358 | VM_LOCKED flag. Therefore, we won't have to deal with them later during |
c24b7201 DH |
359 | munlock(), munmap() or task exit. Neither does mlock_fixup() account these |
360 | VMAs against the task's "locked_vm". | |
361 | ||
362 | ||
a5e4da91 | 363 | munlock()/munlockall() System Call Handling |
c24b7201 DH |
364 | ------------------------------------------- |
365 | ||
577e9846 HD |
366 | The munlock() and munlockall() system calls are handled by the same |
367 | mlock_fixup() function as mlock(), mlock2() and mlockall() system calls are. | |
368 | If called to munlock an already munlocked VMA, mlock_fixup() simply returns. | |
369 | Because of the VMA filtering discussed above, VM_LOCKED will not be set in | |
370 | any "special" VMAs. So, those VMAs will be ignored for munlock. | |
fa07e787 | 371 | |
c24b7201 | 372 | If the VMA is VM_LOCKED, mlock_fixup() again attempts to merge or split off the |
577e9846 HD |
373 | specified range. All pages in the VMA are then munlocked by munlock_page() via |
374 | mlock_pte_range() via walk_page_range() via mlock_vma_pages_range() - the same | |
375 | function used when mlocking a VMA range, with new flags for the VMA indicating | |
376 | that it is munlock() being performed. | |
377 | ||
378 | munlock_page() uses the mlock pagevec to batch up work to be done under | |
379 | lru_lock by __munlock_page(). __munlock_page() decrements the page's | |
380 | mlock_count, and when that reaches 0 it clears PageMlocked and clears | |
381 | PageUnevictable, moving the page from unevictable state to inactive LRU. | |
382 | ||
383 | But in practice that may not work ideally: the page may not yet have reached | |
384 | "the unevictable LRU", or it may have been temporarily isolated from it. In | |
385 | those cases its mlock_count field is unusable and must be assumed to be 0: so | |
386 | that the page will be rescued to an evictable LRU, then perhaps be mlocked | |
387 | again later if vmscan finds it in a VM_LOCKED VMA. | |
c24b7201 DH |
388 | |
389 | ||
a5e4da91 | 390 | Migrating MLOCKED Pages |
c24b7201 DH |
391 | ----------------------- |
392 | ||
393 | A page that is being migrated has been isolated from the LRU lists and is held | |
394 | locked across unmapping of the page, updating the page's address space entry | |
395 | and copying the contents and state, until the page table entry has been | |
396 | replaced with an entry that refers to the new page. Linux supports migration | |
577e9846 HD |
397 | of mlocked pages and other unevictable pages. PG_mlocked is cleared from the |
398 | the old page when it is unmapped from the last VM_LOCKED VMA, and set when the | |
399 | new page is mapped in place of migration entry in a VM_LOCKED VMA. If the page | |
400 | was unevictable because mlocked, PG_unevictable follows PG_mlocked; but if the | |
401 | page was unevictable for other reasons, PG_unevictable is copied explicitly. | |
c24b7201 DH |
402 | |
403 | Note that page migration can race with mlocking or munlocking of the same page. | |
577e9846 HD |
404 | There is mostly no problem since page migration requires unmapping all PTEs of |
405 | the old page (including munlock where VM_LOCKED), then mapping in the new page | |
406 | (including mlock where VM_LOCKED). The page table locks provide sufficient | |
407 | synchronization. | |
c24b7201 | 408 | |
577e9846 HD |
409 | However, since mlock_vma_pages_range() starts by setting VM_LOCKED on a VMA, |
410 | before mlocking any pages already present, if one of those pages were migrated | |
411 | before mlock_pte_range() reached it, it would get counted twice in mlock_count. | |
412 | To prevent that, mlock_vma_pages_range() temporarily marks the VMA as VM_IO, | |
413 | so that mlock_vma_page() will skip it. | |
414 | ||
415 | To complete page migration, we place the old and new pages back onto the LRU | |
416 | afterwards. The "unneeded" page - old page on success, new page on failure - | |
417 | is freed when the reference count held by the migration process is released. | |
c24b7201 DH |
418 | |
419 | ||
a5e4da91 | 420 | Compacting MLOCKED Pages |
922c0551 EM |
421 | ------------------------ |
422 | ||
577e9846 HD |
423 | The memory map can be scanned for compactable regions and the default behavior |
424 | is to let unevictable pages be moved. /proc/sys/vm/compact_unevictable_allowed | |
425 | controls this behavior (see Documentation/admin-guide/sysctl/vm.rst). The work | |
426 | of compaction is mostly handled by the page migration code and the same work | |
427 | flow as described in Migrating MLOCKED Pages will apply. | |
428 | ||
922c0551 | 429 | |
a5e4da91 | 430 | MLOCKING Transparent Huge Pages |
6fb8ddfc KS |
431 | ------------------------------- |
432 | ||
433 | A transparent huge page is represented by a single entry on an LRU list. | |
434 | Therefore, we can only make unevictable an entire compound page, not | |
435 | individual subpages. | |
436 | ||
577e9846 HD |
437 | If a user tries to mlock() part of a huge page, and no user mlock()s the |
438 | whole of the huge page, we want the rest of the page to be reclaimable. | |
6fb8ddfc KS |
439 | |
440 | We cannot just split the page on partial mlock() as split_huge_page() can | |
577e9846 | 441 | fail and a new intermittent failure mode for the syscall is undesirable. |
6fb8ddfc | 442 | |
577e9846 HD |
443 | We handle this by keeping PTE-mlocked huge pages on evictable LRU lists: |
444 | the PMD on the border of a VM_LOCKED VMA will be split into a PTE table. | |
6fb8ddfc | 445 | |
577e9846 | 446 | This way the huge page is accessible for vmscan. Under memory pressure the |
6fb8ddfc | 447 | page will be split, subpages which belong to VM_LOCKED VMAs will be moved |
577e9846 HD |
448 | to the unevictable LRU and the rest can be reclaimed. |
449 | ||
450 | /proc/meminfo's Unevictable and Mlocked amounts do not include those parts | |
451 | of a transparent huge page which are mapped only by PTEs in VM_LOCKED VMAs. | |
6fb8ddfc | 452 | |
922c0551 | 453 | |
a5e4da91 | 454 | mmap(MAP_LOCKED) System Call Handling |
c24b7201 | 455 | ------------------------------------- |
fa07e787 | 456 | |
577e9846 HD |
457 | In addition to the mlock(), mlock2() and mlockall() system calls, an application |
458 | can request that a region of memory be mlocked by supplying the MAP_LOCKED flag | |
459 | to the mmap() call. There is one important and subtle difference here, though. | |
460 | mmap() + mlock() will fail if the range cannot be faulted in (e.g. because | |
461 | mm_populate fails) and returns with ENOMEM while mmap(MAP_LOCKED) will not fail. | |
462 | The mmaped area will still have properties of the locked area - pages will not | |
463 | get swapped out - but major page faults to fault memory in might still happen. | |
9b012a29 | 464 | |
577e9846 HD |
465 | Furthermore, any mmap() call or brk() call that expands the heap by a task |
466 | that has previously called mlockall() with the MCL_FUTURE flag will result | |
c24b7201 | 467 | in the newly mapped memory being mlocked. Before the unevictable/mlock |
577e9846 HD |
468 | changes, the kernel simply called make_pages_present() to allocate pages |
469 | and populate the page table. | |
fa07e787 | 470 | |
577e9846 HD |
471 | To mlock a range of memory under the unevictable/mlock infrastructure, |
472 | the mmap() handler and task address space expansion functions call | |
fc05f566 KS |
473 | populate_vma_page_range() specifying the vma and the address range to mlock. |
474 | ||
fa07e787 | 475 | |
a5e4da91 | 476 | munmap()/exit()/exec() System Call Handling |
c24b7201 | 477 | ------------------------------------------- |
fa07e787 LS |
478 | |
479 | When unmapping an mlocked region of memory, whether by an explicit call to | |
480 | munmap() or via an internal unmap from exit() or exec() processing, we must | |
c24b7201 | 481 | munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. |
63d6c5ad HD |
482 | Before the unevictable/mlock changes, mlocking did not mark the pages in any |
483 | way, so unmapping them required no processing. | |
fa07e787 | 484 | |
577e9846 HD |
485 | For each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls |
486 | munlock_vma_page(), which calls munlock_page() when the VMA is VM_LOCKED | |
487 | (unless it was a PTE mapping of a part of a transparent huge page). | |
488 | ||
489 | munlock_page() uses the mlock pagevec to batch up work to be done under | |
490 | lru_lock by __munlock_page(). __munlock_page() decrements the page's | |
491 | mlock_count, and when that reaches 0 it clears PageMlocked and clears | |
492 | PageUnevictable, moving the page from unevictable state to inactive LRU. | |
493 | ||
494 | But in practice that may not work ideally: the page may not yet have reached | |
495 | "the unevictable LRU", or it may have been temporarily isolated from it. In | |
496 | those cases its mlock_count field is unusable and must be assumed to be 0: so | |
497 | that the page will be rescued to an evictable LRU, then perhaps be mlocked | |
498 | again later if vmscan finds it in a VM_LOCKED VMA. | |
499 | ||
500 | ||
501 | Truncating MLOCKED Pages | |
502 | ------------------------ | |
503 | ||
504 | File truncation or hole punching forcibly unmaps the deleted pages from | |
505 | userspace; truncation even unmaps and deletes any private anonymous pages | |
506 | which had been Copied-On-Write from the file pages now being truncated. | |
507 | ||
508 | Mlocked pages can be munlocked and deleted in this way: like with munmap(), | |
509 | for each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls | |
510 | munlock_vma_page(), which calls munlock_page() when the VMA is VM_LOCKED | |
511 | (unless it was a PTE mapping of a part of a transparent huge page). | |
512 | ||
513 | However, if there is a racing munlock(), since mlock_vma_pages_range() starts | |
514 | munlocking by clearing VM_LOCKED from a VMA, before munlocking all the pages | |
515 | present, if one of those pages were unmapped by truncation or hole punch before | |
516 | mlock_pte_range() reached it, it would not be recognized as mlocked by this VMA, | |
517 | and would not be counted out of mlock_count. In this rare case, a page may | |
518 | still appear as PageMlocked after it has been fully unmapped: and it is left to | |
519 | release_pages() (or __page_cache_release()) to clear it and update statistics | |
520 | before freeing (this event is counted in /proc/vmstat unevictable_pgs_cleared, | |
521 | which is usually 0). | |
c24b7201 DH |
522 | |
523 | ||
a5e4da91 | 524 | Page Reclaim in shrink_*_list() |
c24b7201 DH |
525 | ------------------------------- |
526 | ||
577e9846 HD |
527 | vmscan's shrink_active_list() culls any obviously unevictable pages - |
528 | i.e. !page_evictable(page) pages - diverting those to the unevictable list. | |
c24b7201 | 529 | However, shrink_active_list() only sees unevictable pages that made it onto the |
577e9846 HD |
530 | active/inactive LRU lists. Note that these pages do not have PageUnevictable |
531 | set - otherwise they would be on the unevictable list and shrink_active_list() | |
c24b7201 | 532 | would never see them. |
fa07e787 LS |
533 | |
534 | Some examples of these unevictable pages on the LRU lists are: | |
535 | ||
c24b7201 DH |
536 | (1) ramfs pages that have been placed on the LRU lists when first allocated. |
537 | ||
538 | (2) SHM_LOCK'd shared memory pages. shmctl(SHM_LOCK) does not attempt to | |
539 | allocate or fault in the pages in the shared memory region. This happens | |
540 | when an application accesses the page the first time after SHM_LOCK'ing | |
541 | the segment. | |
fa07e787 | 542 | |
577e9846 HD |
543 | (3) pages still mapped into VM_LOCKED VMAs, which should be marked mlocked, |
544 | but events left mlock_count too low, so they were munlocked too early. | |
fa07e787 | 545 | |
577e9846 HD |
546 | vmscan's shrink_inactive_list() and shrink_page_list() also divert obviously |
547 | unevictable pages found on the inactive lists to the appropriate memory cgroup | |
548 | and node unevictable list. | |
fa07e787 | 549 | |
9a7d7a80 | 550 | rmap's folio_referenced_one(), called via vmscan's shrink_active_list() or |
577e9846 HD |
551 | shrink_page_list(), and rmap's try_to_unmap_one() called via shrink_page_list(), |
552 | check for (3) pages still mapped into VM_LOCKED VMAs, and call mlock_vma_page() | |
553 | to correct them. Such pages are culled to the unevictable list when released | |
554 | by the shrinker. |