Commit | Line | Data |
---|---|---|
1da177e4 LT |
1 | |
2 | The intent of this file is to give a brief summary of hugetlbpage support in | |
3 | the Linux kernel. This support is built on top of multiple page size support | |
4 | that is provided by most modern architectures. For example, i386 | |
5 | architecture supports 4K and 4M (2M in PAE mode) page sizes, ia64 | |
6 | architecture supports multiple page sizes 4K, 8K, 64K, 256K, 1M, 4M, 16M, | |
7 | 256M and ppc64 supports 4K and 16M. A TLB is a cache of virtual-to-physical | |
8 | translations. Typically this is a very scarce resource on processor. | |
9 | Operating systems try to make best use of limited number of TLB resources. | |
10 | This optimization is more critical now as bigger and bigger physical memories | |
11 | (several GBs) are more readily available. | |
12 | ||
13 | Users can use the huge page support in Linux kernel by either using the mmap | |
267b4c28 | 14 | system call or standard SYSV shared memory system calls (shmget, shmat). |
1da177e4 | 15 | |
5c7ad510 MBY |
16 | First the Linux kernel needs to be built with the CONFIG_HUGETLBFS |
17 | (present under "File systems") and CONFIG_HUGETLB_PAGE (selected | |
18 | automatically when CONFIG_HUGETLBFS is selected) configuration | |
19 | options. | |
1da177e4 | 20 | |
267b4c28 LS |
21 | The /proc/meminfo file provides information about the total number of |
22 | persistent hugetlb pages in the kernel's huge page pool. It also displays | |
23 | information about the number of free, reserved and surplus huge pages and the | |
24 | default huge page size. The huge page size is needed for generating the | |
25 | proper alignment and size of the arguments to system calls that map huge page | |
26 | regions. | |
1da177e4 | 27 | |
267b4c28 | 28 | The output of "cat /proc/meminfo" will include lines like: |
1da177e4 LT |
29 | |
30 | ..... | |
d5dbac87 NA |
31 | HugePages_Total: vvv |
32 | HugePages_Free: www | |
33 | HugePages_Rsvd: xxx | |
34 | HugePages_Surp: yyy | |
5e122271 RD |
35 | Hugepagesize: zzz kB |
36 | ||
37 | where: | |
41a25e7e LS |
38 | HugePages_Total is the size of the pool of huge pages. |
39 | HugePages_Free is the number of huge pages in the pool that are not yet | |
40 | allocated. | |
41 | HugePages_Rsvd is short for "reserved," and is the number of huge pages for | |
42 | which a commitment to allocate from the pool has been made, | |
43 | but no allocation has yet been made. Reserved huge pages | |
44 | guarantee that an application will be able to allocate a | |
45 | huge page from the pool of huge pages at fault time. | |
46 | HugePages_Surp is short for "surplus," and is the number of huge pages in | |
47 | the pool above the value in /proc/sys/vm/nr_hugepages. The | |
48 | maximum number of surplus huge pages is controlled by | |
49 | /proc/sys/vm/nr_overcommit_hugepages. | |
1da177e4 LT |
50 | |
51 | /proc/filesystems should also show a filesystem of type "hugetlbfs" configured | |
52 | in the kernel. | |
53 | ||
267b4c28 LS |
54 | /proc/sys/vm/nr_hugepages indicates the current number of "persistent" huge |
55 | pages in the kernel's huge page pool. "Persistent" huge pages will be | |
56 | returned to the huge page pool when freed by a task. A user with root | |
57 | privileges can dynamically allocate more or free some persistent huge pages | |
58 | by increasing or decreasing the value of 'nr_hugepages'. | |
1da177e4 | 59 | |
267b4c28 LS |
60 | Pages that are used as huge pages are reserved inside the kernel and cannot |
61 | be used for other purposes. Huge pages cannot be swapped out under | |
62 | memory pressure. | |
1da177e4 | 63 | |
267b4c28 LS |
64 | Once a number of huge pages have been pre-allocated to the kernel huge page |
65 | pool, a user with appropriate privilege can use either the mmap system call | |
66 | or shared memory system calls to use the huge pages. See the discussion of | |
67 | Using Huge Pages, below. | |
1da177e4 | 68 | |
267b4c28 LS |
69 | The administrator can allocate persistent huge pages on the kernel boot |
70 | command line by specifying the "hugepages=N" parameter, where 'N' = the | |
71 | number of huge pages requested. This is the most reliable method of | |
72 | allocating huge pages as memory has not yet become fragmented. | |
41a25e7e | 73 | |
267b4c28 | 74 | Some platforms support multiple huge page sizes. To allocate huge pages |
41a25e7e LS |
75 | of a specific size, one must preceed the huge pages boot command parameters |
76 | with a huge page size selection parameter "hugepagesz=<size>". <size> must | |
77 | be specified in bytes with optional scale suffix [kKmMgG]. The default huge | |
78 | page size may be selected with the "default_hugepagesz=<size>" boot parameter. | |
79 | ||
267b4c28 LS |
80 | When multiple huge page sizes are supported, /proc/sys/vm/nr_hugepages |
81 | indicates the current number of pre-allocated huge pages of the default size. | |
82 | Thus, one can use the following command to dynamically allocate/deallocate | |
83 | default sized persistent huge pages: | |
1da177e4 LT |
84 | |
85 | echo 20 > /proc/sys/vm/nr_hugepages | |
86 | ||
267b4c28 LS |
87 | This command will try to adjust the number of default sized huge pages in the |
88 | huge page pool to 20, allocating or freeing huge pages, as required. | |
89 | ||
41a25e7e | 90 | On a NUMA platform, the kernel will attempt to distribute the huge page pool |
267b4c28 LS |
91 | over all the set of allowed nodes specified by the NUMA memory policy of the |
92 | task that modifies nr_hugepages. The default for the allowed nodes--when the | |
9b5e5d0f LS |
93 | task has default memory policy--is all on-line nodes with memory. Allowed |
94 | nodes with insufficient available, contiguous memory for a huge page will be | |
95 | silently skipped when allocating persistent huge pages. See the discussion | |
96 | below of the interaction of task memory policy, cpusets and per node attributes | |
97 | with the allocation and freeing of persistent huge pages. | |
41a25e7e LS |
98 | |
99 | The success or failure of huge page allocation depends on the amount of | |
267b4c28 | 100 | physically contiguous memory that is present in system at the time of the |
41a25e7e LS |
101 | allocation attempt. If the kernel is unable to allocate huge pages from |
102 | some nodes in a NUMA system, it will attempt to make up the difference by | |
103 | allocating extra pages on other nodes with sufficient available contiguous | |
104 | memory, if any. | |
105 | ||
267b4c28 LS |
106 | System administrators may want to put this command in one of the local rc |
107 | init files. This will enable the kernel to allocate huge pages early in | |
108 | the boot process when the possibility of getting physical contiguous pages | |
109 | is still very high. Administrators can verify the number of huge pages | |
110 | actually allocated by checking the sysctl or meminfo. To check the per node | |
41a25e7e LS |
111 | distribution of huge pages in a NUMA system, use: |
112 | ||
113 | cat /sys/devices/system/node/node*/meminfo | fgrep Huge | |
114 | ||
115 | /proc/sys/vm/nr_overcommit_hugepages specifies how large the pool of | |
116 | huge pages can grow, if more huge pages than /proc/sys/vm/nr_hugepages are | |
117 | requested by applications. Writing any non-zero value into this file | |
267b4c28 LS |
118 | indicates that the hugetlb subsystem is allowed to try to obtain that |
119 | number of "surplus" huge pages from the kernel's normal page pool, when the | |
120 | persistent huge page pool is exhausted. As these surplus huge pages become | |
121 | unused, they are freed back to the kernel's normal page pool. | |
d5dbac87 | 122 | |
267b4c28 | 123 | When increasing the huge page pool size via nr_hugepages, any existing surplus |
41a25e7e LS |
124 | pages will first be promoted to persistent huge pages. Then, additional |
125 | huge pages will be allocated, if necessary and if possible, to fulfill | |
267b4c28 | 126 | the new persistent huge page pool size. |
41a25e7e | 127 | |
267b4c28 | 128 | The administrator may shrink the pool of persistent huge pages for |
41a25e7e LS |
129 | the default huge page size by setting the nr_hugepages sysctl to a |
130 | smaller value. The kernel will attempt to balance the freeing of huge pages | |
267b4c28 LS |
131 | across all nodes in the memory policy of the task modifying nr_hugepages. |
132 | Any free huge pages on the selected nodes will be freed back to the kernel's | |
133 | normal page pool. | |
134 | ||
135 | Caveat: Shrinking the persistent huge page pool via nr_hugepages such that | |
136 | it becomes less than the number of huge pages in use will convert the balance | |
137 | of the in-use huge pages to surplus huge pages. This will occur even if | |
138 | the number of surplus pages it would exceed the overcommit value. As long as | |
139 | this condition holds--that is, until nr_hugepages+nr_overcommit_hugepages is | |
140 | increased sufficiently, or the surplus huge pages go out of use and are freed-- | |
141 | no more surplus huge pages will be allowed to be allocated. | |
1da177e4 | 142 | |
41a25e7e | 143 | With support for multiple huge page pools at run-time available, much of |
267b4c28 LS |
144 | the huge page userspace interface in /proc/sys/vm has been duplicated in sysfs. |
145 | The /proc interfaces discussed above have been retained for backwards | |
146 | compatibility. The root huge page control directory in sysfs is: | |
a3437870 NA |
147 | |
148 | /sys/kernel/mm/hugepages | |
149 | ||
41a25e7e | 150 | For each huge page size supported by the running kernel, a subdirectory |
267b4c28 | 151 | will exist, of the form: |
a3437870 NA |
152 | |
153 | hugepages-${size}kB | |
154 | ||
155 | Inside each of these directories, the same set of files will exist: | |
156 | ||
157 | nr_hugepages | |
267b4c28 | 158 | nr_hugepages_mempolicy |
a3437870 NA |
159 | nr_overcommit_hugepages |
160 | free_hugepages | |
161 | resv_hugepages | |
162 | surplus_hugepages | |
163 | ||
41a25e7e | 164 | which function as described above for the default huge page-sized case. |
a3437870 | 165 | |
267b4c28 LS |
166 | |
167 | Interaction of Task Memory Policy with Huge Page Allocation/Freeing | |
168 | ||
169 | Whether huge pages are allocated and freed via the /proc interface or | |
170 | the /sysfs interface using the nr_hugepages_mempolicy attribute, the NUMA | |
171 | nodes from which huge pages are allocated or freed are controlled by the | |
172 | NUMA memory policy of the task that modifies the nr_hugepages_mempolicy | |
173 | sysctl or attribute. When the nr_hugepages attribute is used, mempolicy | |
174 | is ignored. | |
175 | ||
176 | The recommended method to allocate or free huge pages to/from the kernel | |
177 | huge page pool, using the nr_hugepages example above, is: | |
178 | ||
179 | numactl --interleave <node-list> echo 20 \ | |
180 | >/proc/sys/vm/nr_hugepages_mempolicy | |
181 | ||
182 | or, more succinctly: | |
183 | ||
184 | numactl -m <node-list> echo 20 >/proc/sys/vm/nr_hugepages_mempolicy | |
185 | ||
186 | This will allocate or free abs(20 - nr_hugepages) to or from the nodes | |
187 | specified in <node-list>, depending on whether number of persistent huge pages | |
188 | is initially less than or greater than 20, respectively. No huge pages will be | |
189 | allocated nor freed on any node not included in the specified <node-list>. | |
190 | ||
191 | When adjusting the persistent hugepage count via nr_hugepages_mempolicy, any | |
192 | memory policy mode--bind, preferred, local or interleave--may be used. The | |
193 | resulting effect on persistent huge page allocation is as follows: | |
194 | ||
195 | 1) Regardless of mempolicy mode [see Documentation/vm/numa_memory_policy.txt], | |
196 | persistent huge pages will be distributed across the node or nodes | |
197 | specified in the mempolicy as if "interleave" had been specified. | |
198 | However, if a node in the policy does not contain sufficient contiguous | |
199 | memory for a huge page, the allocation will not "fallback" to the nearest | |
200 | neighbor node with sufficient contiguous memory. To do this would cause | |
201 | undesirable imbalance in the distribution of the huge page pool, or | |
202 | possibly, allocation of persistent huge pages on nodes not allowed by | |
203 | the task's memory policy. | |
204 | ||
205 | 2) One or more nodes may be specified with the bind or interleave policy. | |
206 | If more than one node is specified with the preferred policy, only the | |
207 | lowest numeric id will be used. Local policy will select the node where | |
208 | the task is running at the time the nodes_allowed mask is constructed. | |
209 | For local policy to be deterministic, the task must be bound to a cpu or | |
210 | cpus in a single node. Otherwise, the task could be migrated to some | |
211 | other node at any time after launch and the resulting node will be | |
212 | indeterminate. Thus, local policy is not very useful for this purpose. | |
213 | Any of the other mempolicy modes may be used to specify a single node. | |
214 | ||
215 | 3) The nodes allowed mask will be derived from any non-default task mempolicy, | |
216 | whether this policy was set explicitly by the task itself or one of its | |
217 | ancestors, such as numactl. This means that if the task is invoked from a | |
218 | shell with non-default policy, that policy will be used. One can specify a | |
219 | node list of "all" with numactl --interleave or --membind [-m] to achieve | |
220 | interleaving over all nodes in the system or cpuset. | |
221 | ||
222 | 4) Any task mempolicy specifed--e.g., using numactl--will be constrained by | |
223 | the resource limits of any cpuset in which the task runs. Thus, there will | |
224 | be no way for a task with non-default policy running in a cpuset with a | |
225 | subset of the system nodes to allocate huge pages outside the cpuset | |
226 | without first moving to a cpuset that contains all of the desired nodes. | |
227 | ||
228 | 5) Boot-time huge page allocation attempts to distribute the requested number | |
9b5e5d0f | 229 | of huge pages over all on-lines nodes with memory. |
267b4c28 LS |
230 | |
231 | Per Node Hugepages Attributes | |
232 | ||
233 | A subset of the contents of the root huge page control directory in sysfs, | |
234 | described above, has been replicated under each "node" system device in: | |
235 | ||
236 | /sys/devices/system/node/node[0-9]*/hugepages/ | |
237 | ||
238 | Under this directory, the subdirectory for each supported huge page size | |
239 | contains the following attribute files: | |
240 | ||
241 | nr_hugepages | |
242 | free_hugepages | |
243 | surplus_hugepages | |
244 | ||
245 | The free_' and surplus_' attribute files are read-only. They return the number | |
246 | of free and surplus [overcommitted] huge pages, respectively, on the parent | |
247 | node. | |
248 | ||
249 | The nr_hugepages attribute returns the total number of huge pages on the | |
250 | specified node. When this attribute is written, the number of persistent huge | |
251 | pages on the parent node will be adjusted to the specified value, if sufficient | |
252 | resources exist, regardless of the task's mempolicy or cpuset constraints. | |
253 | ||
254 | Note that the number of overcommit and reserve pages remain global quantities, | |
255 | as we don't know until fault time, when the faulting task's mempolicy is | |
256 | applied, from which node the huge page allocation will be attempted. | |
257 | ||
258 | ||
259 | Using Huge Pages | |
260 | ||
41a25e7e | 261 | If the user applications are going to request huge pages using mmap system |
1da177e4 LT |
262 | call, then it is required that system administrator mount a file system of |
263 | type hugetlbfs: | |
264 | ||
e73a75fa RD |
265 | mount -t hugetlbfs \ |
266 | -o uid=<value>,gid=<value>,mode=<value>,size=<value>,nr_inodes=<value> \ | |
267 | none /mnt/huge | |
1da177e4 LT |
268 | |
269 | This command mounts a (pseudo) filesystem of type hugetlbfs on the directory | |
41a25e7e | 270 | /mnt/huge. Any files created on /mnt/huge uses huge pages. The uid and gid |
1da177e4 LT |
271 | options sets the owner and group of the root of the file system. By default |
272 | the uid and gid of the current process are taken. The mode option sets the | |
273 | mode of root of file system to value & 0777. This value is given in octal. | |
274 | By default the value 0755 is picked. The size option sets the maximum value of | |
275 | memory (huge pages) allowed for that filesystem (/mnt/huge). The size is | |
21a26d49 | 276 | rounded down to HPAGE_SIZE. The option nr_inodes sets the maximum number of |
e73a75fa | 277 | inodes that /mnt/huge can use. If the size or nr_inodes option is not |
1da177e4 | 278 | provided on command line then no limits are set. For size and nr_inodes |
5c7ad510 | 279 | options, you can use [G|g]/[M|m]/[K|k] to represent giga/mega/kilo. For |
e73a75fa | 280 | example, size=2K has the same meaning as size=2048. |
1da177e4 | 281 | |
d5dbac87 NA |
282 | While read system calls are supported on files that reside on hugetlb |
283 | file systems, write system calls are not. | |
1da177e4 | 284 | |
21a26d49 | 285 | Regular chown, chgrp, and chmod commands (with right permissions) could be |
1da177e4 LT |
286 | used to change the file attributes on hugetlbfs. |
287 | ||
288 | Also, it is important to note that no such mount command is required if the | |
94bf5cea EM |
289 | applications are going to use only shmat/shmget system calls or mmap with |
290 | MAP_HUGETLB. Users who wish to use hugetlb page via shared memory segment | |
291 | should be a member of a supplementary group and system admin needs to | |
292 | configure that gid into /proc/sys/vm/hugetlb_shm_group. It is possible for | |
293 | same or different applications to use any combination of mmaps and shm* | |
294 | calls, though the mount of filesystem will be required for using mmap calls | |
295 | without MAP_HUGETLB. For an example of how to use mmap with MAP_HUGETLB see | |
296 | map_hugetlb.c. | |
1da177e4 LT |
297 | |
298 | ******************************************************************* | |
299 | ||
300 | /* | |
41a25e7e | 301 | * Example of using huge page memory in a user application using Sys V shared |
1da177e4 LT |
302 | * memory system calls. In this example the app is requesting 256MB of |
303 | * memory that is backed by huge pages. The application uses the flag | |
304 | * SHM_HUGETLB in the shmget system call to inform the kernel that it is | |
41a25e7e | 305 | * requesting huge pages. |
1da177e4 LT |
306 | * |
307 | * For the ia64 architecture, the Linux kernel reserves Region number 4 for | |
267b4c28 LS |
308 | * huge pages. That means that if one requires a fixed address, a huge page |
309 | * aligned address starting with 0x800000... will be required. If a fixed | |
310 | * address is not required, the kernel will select an address in the proper | |
311 | * range. | |
312 | * Other architectures, such as ppc64, i386 or x86_64 are not so constrained. | |
1da177e4 LT |
313 | * |
314 | * Note: The default shared memory limit is quite low on many kernels, | |
315 | * you may need to increase it via: | |
316 | * | |
317 | * echo 268435456 > /proc/sys/kernel/shmmax | |
318 | * | |
319 | * This will increase the maximum size per shared memory segment to 256MB. | |
320 | * The other limit that you will hit eventually is shmall which is the | |
321 | * total amount of shared memory in pages. To set it to 16GB on a system | |
322 | * with a 4kB pagesize do: | |
323 | * | |
324 | * echo 4194304 > /proc/sys/kernel/shmall | |
325 | */ | |
326 | #include <stdlib.h> | |
327 | #include <stdio.h> | |
328 | #include <sys/types.h> | |
329 | #include <sys/ipc.h> | |
330 | #include <sys/shm.h> | |
331 | #include <sys/mman.h> | |
332 | ||
333 | #ifndef SHM_HUGETLB | |
334 | #define SHM_HUGETLB 04000 | |
335 | #endif | |
336 | ||
337 | #define LENGTH (256UL*1024*1024) | |
338 | ||
339 | #define dprintf(x) printf(x) | |
340 | ||
267b4c28 | 341 | #define ADDR (void *)(0x0UL) /* let kernel choose address */ |
1da177e4 | 342 | #define SHMAT_FLAGS (0) |
1da177e4 LT |
343 | |
344 | int main(void) | |
345 | { | |
346 | int shmid; | |
347 | unsigned long i; | |
348 | char *shmaddr; | |
349 | ||
350 | if ((shmid = shmget(2, LENGTH, | |
351 | SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W)) < 0) { | |
352 | perror("shmget"); | |
353 | exit(1); | |
354 | } | |
355 | printf("shmid: 0x%x\n", shmid); | |
356 | ||
357 | shmaddr = shmat(shmid, ADDR, SHMAT_FLAGS); | |
358 | if (shmaddr == (char *)-1) { | |
359 | perror("Shared memory attach failure"); | |
360 | shmctl(shmid, IPC_RMID, NULL); | |
361 | exit(2); | |
362 | } | |
363 | printf("shmaddr: %p\n", shmaddr); | |
364 | ||
365 | dprintf("Starting the writes:\n"); | |
366 | for (i = 0; i < LENGTH; i++) { | |
367 | shmaddr[i] = (char)(i); | |
368 | if (!(i % (1024 * 1024))) | |
369 | dprintf("."); | |
370 | } | |
371 | dprintf("\n"); | |
372 | ||
373 | dprintf("Starting the Check..."); | |
374 | for (i = 0; i < LENGTH; i++) | |
375 | if (shmaddr[i] != (char)i) | |
376 | printf("\nIndex %lu mismatched\n", i); | |
377 | dprintf("Done.\n"); | |
378 | ||
379 | if (shmdt((const void *)shmaddr) != 0) { | |
380 | perror("Detach failure"); | |
381 | shmctl(shmid, IPC_RMID, NULL); | |
382 | exit(3); | |
383 | } | |
384 | ||
385 | shmctl(shmid, IPC_RMID, NULL); | |
386 | ||
387 | return 0; | |
388 | } | |
389 | ||
390 | ******************************************************************* | |
391 | ||
392 | /* | |
41a25e7e | 393 | * Example of using huge page memory in a user application using the mmap |
1da177e4 LT |
394 | * system call. Before running this application, make sure that the |
395 | * administrator has mounted the hugetlbfs filesystem (on some directory | |
396 | * like /mnt) using the command mount -t hugetlbfs nodev /mnt. In this | |
397 | * example, the app is requesting memory of size 256MB that is backed by | |
398 | * huge pages. | |
399 | * | |
267b4c28 LS |
400 | * For the ia64 architecture, the Linux kernel reserves Region number 4 for |
401 | * huge pages. That means that if one requires a fixed address, a huge page | |
402 | * aligned address starting with 0x800000... will be required. If a fixed | |
403 | * address is not required, the kernel will select an address in the proper | |
404 | * range. | |
405 | * Other architectures, such as ppc64, i386 or x86_64 are not so constrained. | |
1da177e4 LT |
406 | */ |
407 | #include <stdlib.h> | |
408 | #include <stdio.h> | |
409 | #include <unistd.h> | |
410 | #include <sys/mman.h> | |
411 | #include <fcntl.h> | |
412 | ||
413 | #define FILE_NAME "/mnt/hugepagefile" | |
414 | #define LENGTH (256UL*1024*1024) | |
415 | #define PROTECTION (PROT_READ | PROT_WRITE) | |
416 | ||
267b4c28 | 417 | #define ADDR (void *)(0x0UL) /* let kernel choose address */ |
1da177e4 | 418 | #define FLAGS (MAP_SHARED) |
1da177e4 LT |
419 | |
420 | void check_bytes(char *addr) | |
421 | { | |
422 | printf("First hex is %x\n", *((unsigned int *)addr)); | |
423 | } | |
424 | ||
425 | void write_bytes(char *addr) | |
426 | { | |
427 | unsigned long i; | |
428 | ||
429 | for (i = 0; i < LENGTH; i++) | |
430 | *(addr + i) = (char)i; | |
431 | } | |
432 | ||
433 | void read_bytes(char *addr) | |
434 | { | |
435 | unsigned long i; | |
436 | ||
437 | check_bytes(addr); | |
438 | for (i = 0; i < LENGTH; i++) | |
439 | if (*(addr + i) != (char)i) { | |
440 | printf("Mismatch at %lu\n", i); | |
441 | break; | |
442 | } | |
443 | } | |
444 | ||
445 | int main(void) | |
446 | { | |
447 | void *addr; | |
448 | int fd; | |
449 | ||
450 | fd = open(FILE_NAME, O_CREAT | O_RDWR, 0755); | |
451 | if (fd < 0) { | |
452 | perror("Open failed"); | |
453 | exit(1); | |
454 | } | |
455 | ||
456 | addr = mmap(ADDR, LENGTH, PROTECTION, FLAGS, fd, 0); | |
457 | if (addr == MAP_FAILED) { | |
458 | perror("mmap"); | |
459 | unlink(FILE_NAME); | |
460 | exit(1); | |
461 | } | |
462 | ||
463 | printf("Returned address is %p\n", addr); | |
464 | check_bytes(addr); | |
465 | write_bytes(addr); | |
466 | read_bytes(addr); | |
467 | ||
468 | munmap(addr, LENGTH); | |
469 | close(fd); | |
470 | unlink(FILE_NAME); | |
471 | ||
472 | return 0; | |
473 | } |