Commit | Line | Data |
---|---|---|
88ececc2 MR |
1 | .. _hugetlbfs_reserve: |
2 | ||
3 | ===================== | |
4 | Hugetlbfs Reservation | |
5 | ===================== | |
6 | ||
7 | Overview | |
8 | ======== | |
9 | ||
10 | Huge pages as described at :ref:`hugetlbpage` are typically | |
70bc0dc5 MK |
11 | preallocated for application use. These huge pages are instantiated in a |
12 | task's address space at page fault time if the VMA indicates huge pages are | |
13 | to be used. If no huge page exists at page fault time, the task is sent | |
14 | a SIGBUS and often dies an unhappy death. Shortly after huge page support | |
15 | was added, it was determined that it would be better to detect a shortage | |
16 | of huge pages at mmap() time. The idea is that if there were not enough | |
17 | huge pages to cover the mapping, the mmap() would fail. This was first | |
18 | done with a simple check in the code at mmap() time to determine if there | |
19 | were enough free huge pages to cover the mapping. Like most things in the | |
20 | kernel, the code has evolved over time. However, the basic idea was to | |
21 | 'reserve' huge pages at mmap() time to ensure that huge pages would be | |
22 | available for page faults in that mapping. The description below attempts to | |
23 | describe how huge page reserve processing is done in the v4.10 kernel. | |
24 | ||
25 | ||
26 | Audience | |
88ececc2 | 27 | ======== |
70bc0dc5 MK |
28 | This description is primarily targeted at kernel developers who are modifying |
29 | hugetlbfs code. | |
30 | ||
31 | ||
32 | The Data Structures | |
88ececc2 MR |
33 | =================== |
34 | ||
70bc0dc5 MK |
35 | resv_huge_pages |
36 | This is a global (per-hstate) count of reserved huge pages. Reserved | |
37 | huge pages are only available to the task which reserved them. | |
38 | Therefore, the number of huge pages generally available is computed | |
88ececc2 | 39 | as (``free_huge_pages - resv_huge_pages``). |
70bc0dc5 | 40 | Reserve Map |
88ececc2 MR |
41 | A reserve map is described by the structure:: |
42 | ||
43 | struct resv_map { | |
44 | struct kref refs; | |
45 | spinlock_t lock; | |
46 | struct list_head regions; | |
47 | long adds_in_progress; | |
48 | struct list_head region_cache; | |
49 | long region_cache_count; | |
50 | }; | |
51 | ||
70bc0dc5 MK |
52 | There is one reserve map for each huge page mapping in the system. |
53 | The regions list within the resv_map describes the regions within | |
88ececc2 MR |
54 | the mapping. A region is described as:: |
55 | ||
56 | struct file_region { | |
57 | struct list_head link; | |
58 | long from; | |
59 | long to; | |
60 | }; | |
61 | ||
70bc0dc5 MK |
62 | The 'from' and 'to' fields of the file region structure are huge page |
63 | indices into the mapping. Depending on the type of mapping, a | |
64 | region in the reserv_map may indicate reservations exist for the | |
65 | range, or reservations do not exist. | |
66 | Flags for MAP_PRIVATE Reservations | |
67 | These are stored in the bottom bits of the reservation map pointer. | |
88ececc2 MR |
68 | |
69 | ``#define HPAGE_RESV_OWNER (1UL << 0)`` | |
70 | Indicates this task is the owner of the reservations | |
71 | associated with the mapping. | |
72 | ``#define HPAGE_RESV_UNMAPPED (1UL << 1)`` | |
73 | Indicates task originally mapping this range (and creating | |
74 | reserves) has unmapped a page from this task (the child) | |
75 | due to a failed COW. | |
70bc0dc5 MK |
76 | Page Flags |
77 | The PagePrivate page flag is used to indicate that a huge page | |
78 | reservation must be restored when the huge page is freed. More | |
79 | details will be discussed in the "Freeing huge pages" section. | |
80 | ||
81 | ||
82 | Reservation Map Location (Private or Shared) | |
88ececc2 MR |
83 | ============================================ |
84 | ||
70bc0dc5 MK |
85 | A huge page mapping or segment is either private or shared. If private, |
86 | it is typically only available to a single address space (task). If shared, | |
87 | it can be mapped into multiple address spaces (tasks). The location and | |
41f0a954 | 88 | semantics of the reservation map is significantly different for the two types |
70bc0dc5 | 89 | of mappings. Location differences are: |
88ececc2 | 90 | |
41f0a954 | 91 | - For private mappings, the reservation map hangs off the VMA structure. |
70bc0dc5 MK |
92 | Specifically, vma->vm_private_data. This reserve map is created at the |
93 | time the mapping (mmap(MAP_PRIVATE)) is created. | |
94 | - For shared mappings, the reservation map hangs off the inode. Specifically, | |
95 | inode->i_mapping->private_data. Since shared mappings are always backed | |
96 | by files in the hugetlbfs filesystem, the hugetlbfs code ensures each inode | |
97 | contains a reservation map. As a result, the reservation map is allocated | |
98 | when the inode is created. | |
99 | ||
100 | ||
101 | Creating Reservations | |
88ececc2 | 102 | ===================== |
70bc0dc5 MK |
103 | Reservations are created when a huge page backed shared memory segment is |
104 | created (shmget(SHM_HUGETLB)) or a mapping is created via mmap(MAP_HUGETLB). | |
88ececc2 | 105 | These operations result in a call to the routine hugetlb_reserve_pages():: |
70bc0dc5 | 106 | |
88ececc2 MR |
107 | int hugetlb_reserve_pages(struct inode *inode, |
108 | long from, long to, | |
109 | struct vm_area_struct *vma, | |
110 | vm_flags_t vm_flags) | |
70bc0dc5 | 111 | |
41f0a954 | 112 | The first thing hugetlb_reserve_pages() does is check if the NORESERVE |
70bc0dc5 | 113 | flag was specified in either the shmget() or mmap() call. If NORESERVE |
41f0a954 | 114 | was specified, then this routine returns immediately as no reservations |
70bc0dc5 MK |
115 | are desired. |
116 | ||
117 | The arguments 'from' and 'to' are huge page indices into the mapping or | |
118 | underlying file. For shmget(), 'from' is always 0 and 'to' corresponds to | |
119 | the length of the segment/mapping. For mmap(), the offset argument could | |
41f0a954 | 120 | be used to specify the offset into the underlying file. In such a case, |
70bc0dc5 MK |
121 | the 'from' and 'to' arguments have been adjusted by this offset. |
122 | ||
123 | One of the big differences between PRIVATE and SHARED mappings is the way | |
124 | in which reservations are represented in the reservation map. | |
88ececc2 | 125 | |
70bc0dc5 MK |
126 | - For shared mappings, an entry in the reservation map indicates a reservation |
127 | exists or did exist for the corresponding page. As reservations are | |
128 | consumed, the reservation map is not modified. | |
129 | - For private mappings, the lack of an entry in the reservation map indicates | |
130 | a reservation exists for the corresponding page. As reservations are | |
131 | consumed, entries are added to the reservation map. Therefore, the | |
132 | reservation map can also be used to determine which reservations have | |
133 | been consumed. | |
134 | ||
135 | For private mappings, hugetlb_reserve_pages() creates the reservation map and | |
136 | hangs it off the VMA structure. In addition, the HPAGE_RESV_OWNER flag is set | |
137 | to indicate this VMA owns the reservations. | |
138 | ||
139 | The reservation map is consulted to determine how many huge page reservations | |
140 | are needed for the current mapping/segment. For private mappings, this is | |
41f0a954 RC |
141 | always the value (to - from). However, for shared mappings it is possible that |
142 | some reservations may already exist within the range (to - from). See the | |
88ececc2 MR |
143 | section :ref:`Reservation Map Modifications <resv_map_modifications>` |
144 | for details on how this is accomplished. | |
70bc0dc5 MK |
145 | |
146 | The mapping may be associated with a subpool. If so, the subpool is consulted | |
147 | to ensure there is sufficient space for the mapping. It is possible that the | |
148 | subpool has set aside reservations that can be used for the mapping. See the | |
88ececc2 | 149 | section :ref:`Subpool Reservations <sub_pool_resv>` for more details. |
70bc0dc5 MK |
150 | |
151 | After consulting the reservation map and subpool, the number of needed new | |
152 | reservations is known. The routine hugetlb_acct_memory() is called to check | |
153 | for and take the requested number of reservations. hugetlb_acct_memory() | |
154 | calls into routines that potentially allocate and adjust surplus page counts. | |
155 | However, within those routines the code is simply checking to ensure there | |
156 | are enough free huge pages to accommodate the reservation. If there are, | |
157 | the global reservation count resv_huge_pages is adjusted something like the | |
88ececc2 MR |
158 | following:: |
159 | ||
70bc0dc5 MK |
160 | if (resv_needed <= (resv_huge_pages - free_huge_pages)) |
161 | resv_huge_pages += resv_needed; | |
88ececc2 | 162 | |
70bc0dc5 MK |
163 | Note that the global lock hugetlb_lock is held when checking and adjusting |
164 | these counters. | |
165 | ||
166 | If there were enough free huge pages and the global count resv_huge_pages | |
167 | was adjusted, then the reservation map associated with the mapping is | |
168 | modified to reflect the reservations. In the case of a shared mapping, a | |
41f0a954 | 169 | file_region will exist that includes the range 'from' - 'to'. For private |
70bc0dc5 MK |
170 | mappings, no modifications are made to the reservation map as lack of an |
171 | entry indicates a reservation exists. | |
172 | ||
173 | If hugetlb_reserve_pages() was successful, the global reservation count and | |
174 | reservation map associated with the mapping will be modified as required to | |
175 | ensure reservations exist for the range 'from' - 'to'. | |
176 | ||
88ececc2 | 177 | .. _consume_resv: |
70bc0dc5 MK |
178 | |
179 | Consuming Reservations/Allocating a Huge Page | |
88ececc2 MR |
180 | ============================================= |
181 | ||
70bc0dc5 MK |
182 | Reservations are consumed when huge pages associated with the reservations |
183 | are allocated and instantiated in the corresponding mapping. The allocation | |
88ececc2 MR |
184 | is performed within the routine alloc_huge_page():: |
185 | ||
186 | struct page *alloc_huge_page(struct vm_area_struct *vma, | |
187 | unsigned long addr, int avoid_reserve) | |
188 | ||
70bc0dc5 MK |
189 | alloc_huge_page is passed a VMA pointer and a virtual address, so it can |
190 | consult the reservation map to determine if a reservation exists. In addition, | |
191 | alloc_huge_page takes the argument avoid_reserve which indicates reserves | |
192 | should not be used even if it appears they have been set aside for the | |
193 | specified address. The avoid_reserve argument is most often used in the case | |
194 | of Copy on Write and Page Migration where additional copies of an existing | |
195 | page are being allocated. | |
196 | ||
197 | The helper routine vma_needs_reservation() is called to determine if a | |
198 | reservation exists for the address within the mapping(vma). See the section | |
88ececc2 MR |
199 | :ref:`Reservation Map Helper Routines <resv_map_helpers>` for detailed |
200 | information on what this routine does. | |
201 | The value returned from vma_needs_reservation() is generally | |
70bc0dc5 MK |
202 | 0 or 1. 0 if a reservation exists for the address, 1 if no reservation exists. |
203 | If a reservation does not exist, and there is a subpool associated with the | |
204 | mapping the subpool is consulted to determine if it contains reservations. | |
205 | If the subpool contains reservations, one can be used for this allocation. | |
206 | However, in every case the avoid_reserve argument overrides the use of | |
207 | a reservation for the allocation. After determining whether a reservation | |
208 | exists and can be used for the allocation, the routine dequeue_huge_page_vma() | |
209 | is called. This routine takes two arguments related to reservations: | |
88ececc2 | 210 | |
70bc0dc5 MK |
211 | - avoid_reserve, this is the same value/argument passed to alloc_huge_page() |
212 | - chg, even though this argument is of type long only the values 0 or 1 are | |
213 | passed to dequeue_huge_page_vma. If the value is 0, it indicates a | |
214 | reservation exists (see the section "Memory Policy and Reservations" for | |
215 | possible issues). If the value is 1, it indicates a reservation does not | |
216 | exist and the page must be taken from the global free pool if possible. | |
88ececc2 | 217 | |
70bc0dc5 MK |
218 | The free lists associated with the memory policy of the VMA are searched for |
219 | a free page. If a page is found, the value free_huge_pages is decremented | |
220 | when the page is removed from the free list. If there was a reservation | |
88ececc2 MR |
221 | associated with the page, the following adjustments are made:: |
222 | ||
70bc0dc5 MK |
223 | SetPagePrivate(page); /* Indicates allocating this page consumed |
224 | * a reservation, and if an error is | |
225 | * encountered such that the page must be | |
226 | * freed, the reservation will be restored. */ | |
227 | resv_huge_pages--; /* Decrement the global reservation count */ | |
88ececc2 | 228 | |
70bc0dc5 MK |
229 | Note, if no huge page can be found that satisfies the VMA's memory policy |
230 | an attempt will be made to allocate one using the buddy allocator. This | |
231 | brings up the issue of surplus huge pages and overcommit which is beyond | |
232 | the scope reservations. Even if a surplus page is allocated, the same | |
233 | reservation based adjustments as above will be made: SetPagePrivate(page) and | |
234 | resv_huge_pages--. | |
235 | ||
236 | After obtaining a new huge page, (page)->private is set to the value of | |
237 | the subpool associated with the page if it exists. This will be used for | |
238 | subpool accounting when the page is freed. | |
239 | ||
240 | The routine vma_commit_reservation() is then called to adjust the reserve | |
241 | map based on the consumption of the reservation. In general, this involves | |
242 | ensuring the page is represented within a file_region structure of the region | |
41f0a954 | 243 | map. For shared mappings where the reservation was present, an entry |
70bc0dc5 MK |
244 | in the reserve map already existed so no change is made. However, if there |
245 | was no reservation in a shared mapping or this was a private mapping a new | |
246 | entry must be created. | |
247 | ||
248 | It is possible that the reserve map could have been changed between the call | |
249 | to vma_needs_reservation() at the beginning of alloc_huge_page() and the | |
250 | call to vma_commit_reservation() after the page was allocated. This would | |
251 | be possible if hugetlb_reserve_pages was called for the same page in a shared | |
252 | mapping. In such cases, the reservation count and subpool free page count | |
253 | will be off by one. This rare condition can be identified by comparing the | |
254 | return value from vma_needs_reservation and vma_commit_reservation. If such | |
255 | a race is detected, the subpool and global reserve counts are adjusted to | |
88ececc2 MR |
256 | compensate. See the section |
257 | :ref:`Reservation Map Helper Routines <resv_map_helpers>` for more | |
70bc0dc5 MK |
258 | information on these routines. |
259 | ||
260 | ||
261 | Instantiate Huge Pages | |
88ececc2 MR |
262 | ====================== |
263 | ||
70bc0dc5 MK |
264 | After huge page allocation, the page is typically added to the page tables |
265 | of the allocating task. Before this, pages in a shared mapping are added | |
266 | to the page cache and pages in private mappings are added to an anonymous | |
267 | reverse mapping. In both cases, the PagePrivate flag is cleared. Therefore, | |
268 | when a huge page that has been instantiated is freed no adjustment is made | |
269 | to the global reservation count (resv_huge_pages). | |
270 | ||
271 | ||
272 | Freeing Huge Pages | |
88ececc2 MR |
273 | ================== |
274 | ||
70bc0dc5 MK |
275 | Huge page freeing is performed by the routine free_huge_page(). This routine |
276 | is the destructor for hugetlbfs compound pages. As a result, it is only | |
277 | passed a pointer to the page struct. When a huge page is freed, reservation | |
278 | accounting may need to be performed. This would be the case if the page was | |
279 | associated with a subpool that contained reserves, or the page is being freed | |
280 | on an error path where a global reserve count must be restored. | |
281 | ||
282 | The page->private field points to any subpool associated with the page. | |
283 | If the PagePrivate flag is set, it indicates the global reserve count should | |
88ececc2 MR |
284 | be adjusted (see the section |
285 | :ref:`Consuming Reservations/Allocating a Huge Page <consume_resv>` | |
70bc0dc5 MK |
286 | for information on how these are set). |
287 | ||
288 | The routine first calls hugepage_subpool_put_pages() for the page. If this | |
289 | routine returns a value of 0 (which does not equal the value passed 1) it | |
290 | indicates reserves are associated with the subpool, and this newly free page | |
291 | must be used to keep the number of subpool reserves above the minimum size. | |
292 | Therefore, the global resv_huge_pages counter is incremented in this case. | |
293 | ||
294 | If the PagePrivate flag was set in the page, the global resv_huge_pages counter | |
295 | will always be incremented. | |
296 | ||
88ececc2 | 297 | .. _sub_pool_resv: |
70bc0dc5 MK |
298 | |
299 | Subpool Reservations | |
88ececc2 MR |
300 | ==================== |
301 | ||
70bc0dc5 MK |
302 | There is a struct hstate associated with each huge page size. The hstate |
303 | tracks all huge pages of the specified size. A subpool represents a subset | |
304 | of pages within a hstate that is associated with a mounted hugetlbfs | |
305 | filesystem. | |
306 | ||
307 | When a hugetlbfs filesystem is mounted a min_size option can be specified | |
308 | which indicates the minimum number of huge pages required by the filesystem. | |
309 | If this option is specified, the number of huge pages corresponding to | |
310 | min_size are reserved for use by the filesystem. This number is tracked in | |
311 | the min_hpages field of a struct hugepage_subpool. At mount time, | |
312 | hugetlb_acct_memory(min_hpages) is called to reserve the specified number of | |
313 | huge pages. If they can not be reserved, the mount fails. | |
314 | ||
315 | The routines hugepage_subpool_get/put_pages() are called when pages are | |
316 | obtained from or released back to a subpool. They perform all subpool | |
317 | accounting, and track any reservations associated with the subpool. | |
318 | hugepage_subpool_get/put_pages are passed the number of huge pages by which | |
319 | to adjust the subpool 'used page' count (down for get, up for put). Normally, | |
320 | they return the same value that was passed or an error if not enough pages | |
321 | exist in the subpool. | |
322 | ||
323 | However, if reserves are associated with the subpool a return value less | |
324 | than the passed value may be returned. This return value indicates the | |
325 | number of additional global pool adjustments which must be made. For example, | |
326 | suppose a subpool contains 3 reserved huge pages and someone asks for 5. | |
327 | The 3 reserved pages associated with the subpool can be used to satisfy part | |
328 | of the request. But, 2 pages must be obtained from the global pools. To | |
329 | relay this information to the caller, the value 2 is returned. The caller | |
330 | is then responsible for attempting to obtain the additional two pages from | |
331 | the global pools. | |
332 | ||
333 | ||
334 | COW and Reservations | |
88ececc2 MR |
335 | ==================== |
336 | ||
70bc0dc5 MK |
337 | Since shared mappings all point to and use the same underlying pages, the |
338 | biggest reservation concern for COW is private mappings. In this case, | |
339 | two tasks can be pointing at the same previously allocated page. One task | |
340 | attempts to write to the page, so a new page must be allocated so that each | |
341 | task points to its own page. | |
342 | ||
343 | When the page was originally allocated, the reservation for that page was | |
344 | consumed. When an attempt to allocate a new page is made as a result of | |
345 | COW, it is possible that no free huge pages are free and the allocation | |
346 | will fail. | |
347 | ||
348 | When the private mapping was originally created, the owner of the mapping | |
349 | was noted by setting the HPAGE_RESV_OWNER bit in the pointer to the reservation | |
350 | map of the owner. Since the owner created the mapping, the owner owns all | |
351 | the reservations associated with the mapping. Therefore, when a write fault | |
352 | occurs and there is no page available, different action is taken for the owner | |
353 | and non-owner of the reservation. | |
354 | ||
355 | In the case where the faulting task is not the owner, the fault will fail and | |
356 | the task will typically receive a SIGBUS. | |
357 | ||
358 | If the owner is the faulting task, we want it to succeed since it owned the | |
359 | original reservation. To accomplish this, the page is unmapped from the | |
360 | non-owning task. In this way, the only reference is from the owning task. | |
361 | In addition, the HPAGE_RESV_UNMAPPED bit is set in the reservation map pointer | |
362 | of the non-owning task. The non-owning task may receive a SIGBUS if it later | |
363 | faults on a non-present page. But, the original owner of the | |
364 | mapping/reservation will behave as expected. | |
365 | ||
366 | ||
88ececc2 MR |
367 | .. _resv_map_modifications: |
368 | ||
70bc0dc5 | 369 | Reservation Map Modifications |
88ececc2 MR |
370 | ============================= |
371 | ||
70bc0dc5 MK |
372 | The following low level routines are used to make modifications to a |
373 | reservation map. Typically, these routines are not called directly. Rather, | |
374 | a reservation map helper routine is called which calls one of these low level | |
375 | routines. These low level routines are fairly well documented in the source | |
88ececc2 MR |
376 | code (mm/hugetlb.c). These routines are:: |
377 | ||
378 | long region_chg(struct resv_map *resv, long f, long t); | |
379 | long region_add(struct resv_map *resv, long f, long t); | |
380 | void region_abort(struct resv_map *resv, long f, long t); | |
381 | long region_count(struct resv_map *resv, long f, long t); | |
70bc0dc5 MK |
382 | |
383 | Operations on the reservation map typically involve two operations: | |
88ececc2 | 384 | |
70bc0dc5 MK |
385 | 1) region_chg() is called to examine the reserve map and determine how |
386 | many pages in the specified range [f, t) are NOT currently represented. | |
387 | ||
388 | The calling code performs global checks and allocations to determine if | |
389 | there are enough huge pages for the operation to succeed. | |
390 | ||
88ececc2 MR |
391 | 2) |
392 | a) If the operation can succeed, region_add() is called to actually modify | |
393 | the reservation map for the same range [f, t) previously passed to | |
394 | region_chg(). | |
395 | b) If the operation can not succeed, region_abort is called for the same | |
396 | range [f, t) to abort the operation. | |
70bc0dc5 MK |
397 | |
398 | Note that this is a two step process where region_add() and region_abort() | |
399 | are guaranteed to succeed after a prior call to region_chg() for the same | |
400 | range. region_chg() is responsible for pre-allocating any data structures | |
401 | necessary to ensure the subsequent operations (specifically region_add())) | |
402 | will succeed. | |
403 | ||
404 | As mentioned above, region_chg() determines the number of pages in the range | |
405 | which are NOT currently represented in the map. This number is returned to | |
406 | the caller. region_add() returns the number of pages in the range added to | |
407 | the map. In most cases, the return value of region_add() is the same as the | |
408 | return value of region_chg(). However, in the case of shared mappings it is | |
409 | possible for changes to the reservation map to be made between the calls to | |
410 | region_chg() and region_add(). In this case, the return value of region_add() | |
411 | will not match the return value of region_chg(). It is likely that in such | |
412 | cases global counts and subpool accounting will be incorrect and in need of | |
413 | adjustment. It is the responsibility of the caller to check for this condition | |
414 | and make the appropriate adjustments. | |
415 | ||
416 | The routine region_del() is called to remove regions from a reservation map. | |
417 | It is typically called in the following situations: | |
88ececc2 | 418 | |
70bc0dc5 MK |
419 | - When a file in the hugetlbfs filesystem is being removed, the inode will |
420 | be released and the reservation map freed. Before freeing the reservation | |
421 | map, all the individual file_region structures must be freed. In this case | |
422 | region_del is passed the range [0, LONG_MAX). | |
423 | - When a hugetlbfs file is being truncated. In this case, all allocated pages | |
424 | after the new file size must be freed. In addition, any file_region entries | |
425 | in the reservation map past the new end of file must be deleted. In this | |
426 | case, region_del is passed the range [new_end_of_file, LONG_MAX). | |
427 | - When a hole is being punched in a hugetlbfs file. In this case, huge pages | |
428 | are removed from the middle of the file one at a time. As the pages are | |
429 | removed, region_del() is called to remove the corresponding entry from the | |
430 | reservation map. In this case, region_del is passed the range | |
431 | [page_idx, page_idx + 1). | |
88ececc2 | 432 | |
70bc0dc5 MK |
433 | In every case, region_del() will return the number of pages removed from the |
434 | reservation map. In VERY rare cases, region_del() can fail. This can only | |
435 | happen in the hole punch case where it has to split an existing file_region | |
436 | entry and can not allocate a new structure. In this error case, region_del() | |
437 | will return -ENOMEM. The problem here is that the reservation map will | |
438 | indicate that there is a reservation for the page. However, the subpool and | |
439 | global reservation counts will not reflect the reservation. To handle this | |
440 | situation, the routine hugetlb_fix_reserve_counts() is called to adjust the | |
441 | counters so that they correspond with the reservation map entry that could | |
442 | not be deleted. | |
443 | ||
444 | region_count() is called when unmapping a private huge page mapping. In | |
445 | private mappings, the lack of a entry in the reservation map indicates that | |
446 | a reservation exists. Therefore, by counting the number of entries in the | |
447 | reservation map we know how many reservations were consumed and how many are | |
448 | outstanding (outstanding = (end - start) - region_count(resv, start, end)). | |
449 | Since the mapping is going away, the subpool and global reservation counts | |
450 | are decremented by the number of outstanding reservations. | |
451 | ||
88ececc2 | 452 | .. _resv_map_helpers: |
70bc0dc5 MK |
453 | |
454 | Reservation Map Helper Routines | |
88ececc2 MR |
455 | =============================== |
456 | ||
70bc0dc5 MK |
457 | Several helper routines exist to query and modify the reservation maps. |
458 | These routines are only interested with reservations for a specific huge | |
459 | page, so they just pass in an address instead of a range. In addition, | |
460 | they pass in the associated VMA. From the VMA, the type of mapping (private | |
461 | or shared) and the location of the reservation map (inode or VMA) can be | |
462 | determined. These routines simply call the underlying routines described | |
463 | in the section "Reservation Map Modifications". However, they do take into | |
464 | account the 'opposite' meaning of reservation map entries for private and | |
88ececc2 MR |
465 | shared mappings and hide this detail from the caller:: |
466 | ||
467 | long vma_needs_reservation(struct hstate *h, | |
468 | struct vm_area_struct *vma, | |
469 | unsigned long addr) | |
70bc0dc5 | 470 | |
70bc0dc5 | 471 | This routine calls region_chg() for the specified page. If no reservation |
88ececc2 MR |
472 | exists, 1 is returned. If a reservation exists, 0 is returned:: |
473 | ||
474 | long vma_commit_reservation(struct hstate *h, | |
475 | struct vm_area_struct *vma, | |
476 | unsigned long addr) | |
70bc0dc5 | 477 | |
70bc0dc5 MK |
478 | This calls region_add() for the specified page. As in the case of region_chg |
479 | and region_add, this routine is to be called after a previous call to | |
480 | vma_needs_reservation. It will add a reservation entry for the page. It | |
481 | returns 1 if the reservation was added and 0 if not. The return value should | |
482 | be compared with the return value of the previous call to | |
483 | vma_needs_reservation. An unexpected difference indicates the reservation | |
88ececc2 MR |
484 | map was modified between calls:: |
485 | ||
486 | void vma_end_reservation(struct hstate *h, | |
487 | struct vm_area_struct *vma, | |
488 | unsigned long addr) | |
70bc0dc5 | 489 | |
70bc0dc5 MK |
490 | This calls region_abort() for the specified page. As in the case of region_chg |
491 | and region_abort, this routine is to be called after a previous call to | |
492 | vma_needs_reservation. It will abort/end the in progress reservation add | |
88ececc2 MR |
493 | operation:: |
494 | ||
495 | long vma_add_reservation(struct hstate *h, | |
496 | struct vm_area_struct *vma, | |
497 | unsigned long addr) | |
70bc0dc5 | 498 | |
70bc0dc5 MK |
499 | This is a special wrapper routine to help facilitate reservation cleanup |
500 | on error paths. It is only called from the routine restore_reserve_on_error(). | |
501 | This routine is used in conjunction with vma_needs_reservation in an attempt | |
502 | to add a reservation to the reservation map. It takes into account the | |
503 | different reservation map semantics for private and shared mappings. Hence, | |
504 | region_add is called for shared mappings (as an entry present in the map | |
505 | indicates a reservation), and region_del is called for private mappings (as | |
506 | the absence of an entry in the map indicates a reservation). See the section | |
507 | "Reservation cleanup in error paths" for more information on what needs to | |
508 | be done on error paths. | |
509 | ||
510 | ||
511 | Reservation Cleanup in Error Paths | |
88ececc2 MR |
512 | ================================== |
513 | ||
514 | As mentioned in the section | |
515 | :ref:`Reservation Map Helper Routines <resv_map_helpers>`, reservation | |
70bc0dc5 MK |
516 | map modifications are performed in two steps. First vma_needs_reservation |
517 | is called before a page is allocated. If the allocation is successful, | |
518 | then vma_commit_reservation is called. If not, vma_end_reservation is called. | |
519 | Global and subpool reservation counts are adjusted based on success or failure | |
520 | of the operation and all is well. | |
521 | ||
522 | Additionally, after a huge page is instantiated the PagePrivate flag is | |
523 | cleared so that accounting when the page is ultimately freed is correct. | |
524 | ||
525 | However, there are several instances where errors are encountered after a huge | |
526 | page is allocated but before it is instantiated. In this case, the page | |
527 | allocation has consumed the reservation and made the appropriate subpool, | |
528 | reservation map and global count adjustments. If the page is freed at this | |
529 | time (before instantiation and clearing of PagePrivate), then free_huge_page | |
530 | will increment the global reservation count. However, the reservation map | |
531 | indicates the reservation was consumed. This resulting inconsistent state | |
532 | will cause the 'leak' of a reserved huge page. The global reserve count will | |
533 | be higher than it should and prevent allocation of a pre-allocated page. | |
534 | ||
535 | The routine restore_reserve_on_error() attempts to handle this situation. It | |
536 | is fairly well documented. The intention of this routine is to restore | |
537 | the reservation map to the way it was before the page allocation. In this | |
538 | way, the state of the reservation map will correspond to the global reservation | |
539 | count after the page is freed. | |
540 | ||
541 | The routine restore_reserve_on_error itself may encounter errors while | |
542 | attempting to restore the reservation map entry. In this case, it will | |
543 | simply clear the PagePrivate flag of the page. In this way, the global | |
544 | reserve count will not be incremented when the page is freed. However, the | |
545 | reservation map will continue to look as though the reservation was consumed. | |
546 | A page can still be allocated for the address, but it will not use a reserved | |
547 | page as originally intended. | |
548 | ||
549 | There is some code (most notably userfaultfd) which can not call | |
550 | restore_reserve_on_error. In this case, it simply modifies the PagePrivate | |
551 | so that a reservation will not be leaked when the huge page is freed. | |
552 | ||
553 | ||
554 | Reservations and Memory Policy | |
88ececc2 | 555 | ============================== |
70bc0dc5 MK |
556 | Per-node huge page lists existed in struct hstate when git was first used |
557 | to manage Linux code. The concept of reservations was added some time later. | |
558 | When reservations were added, no attempt was made to take memory policy | |
559 | into account. While cpusets are not exactly the same as memory policy, this | |
560 | comment in hugetlb_acct_memory sums up the interaction between reservations | |
88ececc2 MR |
561 | and cpusets/memory policy:: |
562 | ||
70bc0dc5 MK |
563 | /* |
564 | * When cpuset is configured, it breaks the strict hugetlb page | |
565 | * reservation as the accounting is done on a global variable. Such | |
566 | * reservation is completely rubbish in the presence of cpuset because | |
567 | * the reservation is not checked against page availability for the | |
568 | * current cpuset. Application can still potentially OOM'ed by kernel | |
569 | * with lack of free htlb page in cpuset that the task is in. | |
570 | * Attempt to enforce strict accounting with cpuset is almost | |
571 | * impossible (or too ugly) because cpuset is too fluid that | |
572 | * task or memory node can be dynamically moved between cpusets. | |
573 | * | |
574 | * The change of semantics for shared hugetlb mapping with cpuset is | |
575 | * undesirable. However, in order to preserve some of the semantics, | |
576 | * we fall back to check against current free page availability as | |
577 | * a best attempt and hopefully to minimize the impact of changing | |
578 | * semantics that cpuset has. | |
579 | */ | |
580 | ||
581 | Huge page reservations were added to prevent unexpected page allocation | |
582 | failures (OOM) at page fault time. However, if an application makes use | |
583 | of cpusets or memory policy there is no guarantee that huge pages will be | |
584 | available on the required nodes. This is true even if there are a sufficient | |
585 | number of global reservations. | |
586 | ||
946280cd MR |
587 | Hugetlbfs regression testing |
588 | ============================ | |
70bc0dc5 | 589 | |
946280cd MR |
590 | The most complete set of hugetlb tests are in the libhugetlbfs repository. |
591 | If you modify any hugetlb related code, use the libhugetlbfs test suite | |
592 | to check for regressions. In addition, if you add any new hugetlb | |
593 | functionality, please add appropriate tests to libhugetlbfs. | |
594 | ||
595 | -- | |
70bc0dc5 | 596 | Mike Kravetz, 7 April 2017 |