Commit | Line | Data |
---|---|---|
1c9bf22c AA |
1 | = Transparent Hugepage Support = |
2 | ||
3 | == Objective == | |
4 | ||
5 | Performance critical computing applications dealing with large memory | |
6 | working sets are already running on top of libhugetlbfs and in turn | |
7 | hugetlbfs. Transparent Hugepage Support is an alternative means of | |
8 | using huge pages for the backing of virtual memory with huge pages | |
9 | that supports the automatic promotion and demotion of page sizes and | |
10 | without the shortcomings of hugetlbfs. | |
11 | ||
12 | Currently it only works for anonymous memory mappings but in the | |
13 | future it can expand over the pagecache layer starting with tmpfs. | |
14 | ||
15 | The reason applications are running faster is because of two | |
16 | factors. The first factor is almost completely irrelevant and it's not | |
17 | of significant interest because it'll also have the downside of | |
18 | requiring larger clear-page copy-page in page faults which is a | |
19 | potentially negative effect. The first factor consists in taking a | |
20 | single page fault for each 2M virtual region touched by userland (so | |
21 | reducing the enter/exit kernel frequency by a 512 times factor). This | |
22 | only matters the first time the memory is accessed for the lifetime of | |
23 | a memory mapping. The second long lasting and much more important | |
24 | factor will affect all subsequent accesses to the memory for the whole | |
25 | runtime of the application. The second factor consist of two | |
26 | components: 1) the TLB miss will run faster (especially with | |
27 | virtualization using nested pagetables but almost always also on bare | |
28 | metal without virtualization) and 2) a single TLB entry will be | |
29 | mapping a much larger amount of virtual memory in turn reducing the | |
30 | number of TLB misses. With virtualization and nested pagetables the | |
31 | TLB can be mapped of larger size only if both KVM and the Linux guest | |
32 | are using hugepages but a significant speedup already happens if only | |
33 | one of the two is using hugepages just because of the fact the TLB | |
34 | miss is going to run faster. | |
35 | ||
36 | == Design == | |
37 | ||
38 | - "graceful fallback": mm components which don't have transparent | |
39 | hugepage knowledge fall back to breaking a transparent hugepage and | |
40 | working on the regular pages and their respective regular pmd/pte | |
41 | mappings | |
42 | ||
43 | - if a hugepage allocation fails because of memory fragmentation, | |
44 | regular pages should be gracefully allocated instead and mixed in | |
45 | the same vma without any failure or significant delay and without | |
46 | userland noticing | |
47 | ||
48 | - if some task quits and more hugepages become available (either | |
49 | immediately in the buddy or through the VM), guest physical memory | |
50 | backed by regular pages should be relocated on hugepages | |
51 | automatically (with khugepaged) | |
52 | ||
53 | - it doesn't require memory reservation and in turn it uses hugepages | |
54 | whenever possible (the only possible reservation here is kernelcore= | |
55 | to avoid unmovable pages to fragment all the memory but such a tweak | |
56 | is not specific to transparent hugepage support and it's a generic | |
57 | feature that applies to all dynamic high order allocations in the | |
58 | kernel) | |
59 | ||
60 | - this initial support only offers the feature in the anonymous memory | |
61 | regions but it'd be ideal to move it to tmpfs and the pagecache | |
62 | later | |
63 | ||
64 | Transparent Hugepage Support maximizes the usefulness of free memory | |
65 | if compared to the reservation approach of hugetlbfs by allowing all | |
66 | unused memory to be used as cache or other movable (or even unmovable | |
67 | entities). It doesn't require reservation to prevent hugepage | |
68 | allocation failures to be noticeable from userland. It allows paging | |
69 | and all other advanced VM features to be available on the | |
70 | hugepages. It requires no modifications for applications to take | |
71 | advantage of it. | |
72 | ||
73 | Applications however can be further optimized to take advantage of | |
74 | this feature, like for example they've been optimized before to avoid | |
75 | a flood of mmap system calls for every malloc(4k). Optimizing userland | |
76 | is by far not mandatory and khugepaged already can take care of long | |
77 | lived page allocations even for hugepage unaware applications that | |
78 | deals with large amounts of memory. | |
79 | ||
80 | In certain cases when hugepages are enabled system wide, application | |
81 | may end up allocating more memory resources. An application may mmap a | |
82 | large region but only touch 1 byte of it, in that case a 2M page might | |
83 | be allocated instead of a 4k page for no good. This is why it's | |
84 | possible to disable hugepages system-wide and to only have them inside | |
85 | MADV_HUGEPAGE madvise regions. | |
86 | ||
87 | Embedded systems should enable hugepages only inside madvise regions | |
88 | to eliminate any risk of wasting any precious byte of memory and to | |
89 | only run faster. | |
90 | ||
91 | Applications that gets a lot of benefit from hugepages and that don't | |
92 | risk to lose memory by using hugepages, should use | |
93 | madvise(MADV_HUGEPAGE) on their critical mmapped regions. | |
94 | ||
95 | == sysfs == | |
96 | ||
97 | Transparent Hugepage Support can be entirely disabled (mostly for | |
98 | debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to | |
99 | avoid the risk of consuming more memory resources) or enabled system | |
100 | wide. This can be achieved with one of: | |
101 | ||
102 | echo always >/sys/kernel/mm/transparent_hugepage/enabled | |
103 | echo madvise >/sys/kernel/mm/transparent_hugepage/enabled | |
104 | echo never >/sys/kernel/mm/transparent_hugepage/enabled | |
105 | ||
106 | It's also possible to limit defrag efforts in the VM to generate | |
107 | hugepages in case they're not immediately free to madvise regions or | |
108 | to never try to defrag memory and simply fallback to regular pages | |
109 | unless hugepages are immediately available. Clearly if we spend CPU | |
110 | time to defrag memory, we would expect to gain even more by the fact | |
111 | we use hugepages later instead of regular pages. This isn't always | |
112 | guaranteed, but it may be more likely in case the allocation is for a | |
113 | MADV_HUGEPAGE region. | |
114 | ||
115 | echo always >/sys/kernel/mm/transparent_hugepage/defrag | |
116 | echo madvise >/sys/kernel/mm/transparent_hugepage/defrag | |
117 | echo never >/sys/kernel/mm/transparent_hugepage/defrag | |
118 | ||
119 | khugepaged will be automatically started when | |
120 | transparent_hugepage/enabled is set to "always" or "madvise, and it'll | |
121 | be automatically shutdown if it's set to "never". | |
122 | ||
123 | khugepaged runs usually at low frequency so while one may not want to | |
124 | invoke defrag algorithms synchronously during the page faults, it | |
125 | should be worth invoking defrag at least in khugepaged. However it's | |
126 | also possible to disable defrag in khugepaged: | |
127 | ||
128 | echo yes >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag | |
129 | echo no >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag | |
130 | ||
131 | You can also control how many pages khugepaged should scan at each | |
132 | pass: | |
133 | ||
134 | /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan | |
135 | ||
136 | and how many milliseconds to wait in khugepaged between each pass (you | |
137 | can set this to 0 to run khugepaged at 100% utilization of one core): | |
138 | ||
139 | /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs | |
140 | ||
141 | and how many milliseconds to wait in khugepaged if there's an hugepage | |
142 | allocation failure to throttle the next allocation attempt. | |
143 | ||
144 | /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs | |
145 | ||
146 | The khugepaged progress can be seen in the number of pages collapsed: | |
147 | ||
148 | /sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed | |
149 | ||
150 | for each pass: | |
151 | ||
152 | /sys/kernel/mm/transparent_hugepage/khugepaged/full_scans | |
153 | ||
154 | == Boot parameter == | |
155 | ||
156 | You can change the sysfs boot time defaults of Transparent Hugepage | |
157 | Support by passing the parameter "transparent_hugepage=always" or | |
158 | "transparent_hugepage=madvise" or "transparent_hugepage=never" | |
159 | (without "") to the kernel command line. | |
160 | ||
161 | == Need of application restart == | |
162 | ||
163 | The transparent_hugepage/enabled values only affect future | |
164 | behavior. So to make them effective you need to restart any | |
165 | application that could have been using hugepages. This also applies to | |
166 | the regions registered in khugepaged. | |
167 | ||
168 | == get_user_pages and follow_page == | |
169 | ||
170 | get_user_pages and follow_page if run on a hugepage, will return the | |
171 | head or tail pages as usual (exactly as they would do on | |
172 | hugetlbfs). Most gup users will only care about the actual physical | |
173 | address of the page and its temporary pinning to release after the I/O | |
174 | is complete, so they won't ever notice the fact the page is huge. But | |
175 | if any driver is going to mangle over the page structure of the tail | |
176 | page (like for checking page->mapping or other bits that are relevant | |
177 | for the head page and not the tail page), it should be updated to jump | |
178 | to check head page instead (while serializing properly against | |
179 | split_huge_page() to avoid the head and tail pages to disappear from | |
180 | under it, see the futex code to see an example of that, hugetlbfs also | |
181 | needed special handling in futex code for similar reasons). | |
182 | ||
183 | NOTE: these aren't new constraints to the GUP API, and they match the | |
184 | same constrains that applies to hugetlbfs too, so any driver capable | |
185 | of handling GUP on hugetlbfs will also work fine on transparent | |
186 | hugepage backed mappings. | |
187 | ||
188 | In case you can't handle compound pages if they're returned by | |
189 | follow_page, the FOLL_SPLIT bit can be specified as parameter to | |
190 | follow_page, so that it will split the hugepages before returning | |
191 | them. Migration for example passes FOLL_SPLIT as parameter to | |
192 | follow_page because it's not hugepage aware and in fact it can't work | |
193 | at all on hugetlbfs (but it instead works fine on transparent | |
194 | hugepages thanks to FOLL_SPLIT). migration simply can't deal with | |
195 | hugepages being returned (as it's not only checking the pfn of the | |
196 | page and pinning it during the copy but it pretends to migrate the | |
197 | memory in regular page sizes and with regular pte/pmd mappings). | |
198 | ||
199 | == Optimizing the applications == | |
200 | ||
201 | To be guaranteed that the kernel will map a 2M page immediately in any | |
202 | memory region, the mmap region has to be hugepage naturally | |
203 | aligned. posix_memalign() can provide that guarantee. | |
204 | ||
205 | == Hugetlbfs == | |
206 | ||
207 | You can use hugetlbfs on a kernel that has transparent hugepage | |
208 | support enabled just fine as always. No difference can be noted in | |
209 | hugetlbfs other than there will be less overall fragmentation. All | |
210 | usual features belonging to hugetlbfs are preserved and | |
211 | unaffected. libhugetlbfs will also work fine as usual. | |
212 | ||
213 | == Graceful fallback == | |
214 | ||
215 | Code walking pagetables but unware about huge pmds can simply call | |
216 | split_huge_page_pmd(mm, pmd) where the pmd is the one returned by | |
217 | pmd_offset. It's trivial to make the code transparent hugepage aware | |
218 | by just grepping for "pmd_offset" and adding split_huge_page_pmd where | |
219 | missing after pmd_offset returns the pmd. Thanks to the graceful | |
220 | fallback design, with a one liner change, you can avoid to write | |
221 | hundred if not thousand of lines of complex code to make your code | |
222 | hugepage aware. | |
223 | ||
224 | If you're not walking pagetables but you run into a physical hugepage | |
225 | but you can't handle it natively in your code, you can split it by | |
226 | calling split_huge_page(page). This is what the Linux VM does before | |
227 | it tries to swapout the hugepage for example. | |
228 | ||
229 | Example to make mremap.c transparent hugepage aware with a one liner | |
230 | change: | |
231 | ||
232 | diff --git a/mm/mremap.c b/mm/mremap.c | |
233 | --- a/mm/mremap.c | |
234 | +++ b/mm/mremap.c | |
235 | @@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru | |
236 | return NULL; | |
237 | ||
238 | pmd = pmd_offset(pud, addr); | |
239 | + split_huge_page_pmd(mm, pmd); | |
240 | if (pmd_none_or_clear_bad(pmd)) | |
241 | return NULL; | |
242 | ||
243 | == Locking in hugepage aware code == | |
244 | ||
245 | We want as much code as possible hugepage aware, as calling | |
246 | split_huge_page() or split_huge_page_pmd() has a cost. | |
247 | ||
248 | To make pagetable walks huge pmd aware, all you need to do is to call | |
249 | pmd_trans_huge() on the pmd returned by pmd_offset. You must hold the | |
250 | mmap_sem in read (or write) mode to be sure an huge pmd cannot be | |
251 | created from under you by khugepaged (khugepaged collapse_huge_page | |
252 | takes the mmap_sem in write mode in addition to the anon_vma lock). If | |
253 | pmd_trans_huge returns false, you just fallback in the old code | |
254 | paths. If instead pmd_trans_huge returns true, you have to take the | |
255 | mm->page_table_lock and re-run pmd_trans_huge. Taking the | |
256 | page_table_lock will prevent the huge pmd to be converted into a | |
257 | regular pmd from under you (split_huge_page can run in parallel to the | |
258 | pagetable walk). If the second pmd_trans_huge returns false, you | |
259 | should just drop the page_table_lock and fallback to the old code as | |
260 | before. Otherwise you should run pmd_trans_splitting on the pmd. In | |
261 | case pmd_trans_splitting returns true, it means split_huge_page is | |
262 | already in the middle of splitting the page. So if pmd_trans_splitting | |
263 | returns true it's enough to drop the page_table_lock and call | |
264 | wait_split_huge_page and then fallback the old code paths. You are | |
265 | guaranteed by the time wait_split_huge_page returns, the pmd isn't | |
266 | huge anymore. If pmd_trans_splitting returns false, you can proceed to | |
267 | process the huge pmd and the hugepage natively. Once finished you can | |
268 | drop the page_table_lock. | |
269 | ||
270 | == compound_lock, get_user_pages and put_page == | |
271 | ||
272 | split_huge_page internally has to distribute the refcounts in the head | |
273 | page to the tail pages before clearing all PG_head/tail bits from the | |
274 | page structures. It can do that easily for refcounts taken by huge pmd | |
275 | mappings. But the GUI API as created by hugetlbfs (that returns head | |
276 | and tail pages if running get_user_pages on an address backed by any | |
277 | hugepage), requires the refcount to be accounted on the tail pages and | |
278 | not only in the head pages, if we want to be able to run | |
279 | split_huge_page while there are gup pins established on any tail | |
280 | page. Failure to be able to run split_huge_page if there's any gup pin | |
281 | on any tail page, would mean having to split all hugepages upfront in | |
282 | get_user_pages which is unacceptable as too many gup users are | |
283 | performance critical and they must work natively on hugepages like | |
284 | they work natively on hugetlbfs already (hugetlbfs is simpler because | |
285 | hugetlbfs pages cannot be splitted so there wouldn't be requirement of | |
286 | accounting the pins on the tail pages for hugetlbfs). If we wouldn't | |
287 | account the gup refcounts on the tail pages during gup, we won't know | |
288 | anymore which tail page is pinned by gup and which is not while we run | |
289 | split_huge_page. But we still have to add the gup pin to the head page | |
290 | too, to know when we can free the compound page in case it's never | |
291 | splitted during its lifetime. That requires changing not just | |
292 | get_page, but put_page as well so that when put_page runs on a tail | |
293 | page (and only on a tail page) it will find its respective head page, | |
294 | and then it will decrease the head page refcount in addition to the | |
295 | tail page refcount. To obtain a head page reliably and to decrease its | |
296 | refcount without race conditions, put_page has to serialize against | |
297 | __split_huge_page_refcount using a special per-page lock called | |
298 | compound_lock. |