| 1 | ============== |
| 2 | Memory Hotplug |
| 3 | ============== |
| 4 | |
| 5 | :Created: Jul 28 2007 |
| 6 | :Updated: Add description of notifier of memory hotplug: Oct 11 2007 |
| 7 | |
| 8 | This document is about memory hotplug including how-to-use and current status. |
| 9 | Because Memory Hotplug is still under development, contents of this text will |
| 10 | be changed often. |
| 11 | |
| 12 | .. CONTENTS |
| 13 | |
| 14 | 1. Introduction |
| 15 | 1.1 purpose of memory hotplug |
| 16 | 1.2. Phases of memory hotplug |
| 17 | 1.3. Unit of Memory online/offline operation |
| 18 | 2. Kernel Configuration |
| 19 | 3. sysfs files for memory hotplug |
| 20 | 4. Physical memory hot-add phase |
| 21 | 4.1 Hardware(Firmware) Support |
| 22 | 4.2 Notify memory hot-add event by hand |
| 23 | 5. Logical Memory hot-add phase |
| 24 | 5.1. State of memory |
| 25 | 5.2. How to online memory |
| 26 | 6. Logical memory remove |
| 27 | 6.1 Memory offline and ZONE_MOVABLE |
| 28 | 6.2. How to offline memory |
| 29 | 7. Physical memory remove |
| 30 | 8. Memory hotplug event notifier |
| 31 | 9. Future Work List |
| 32 | |
| 33 | |
| 34 | .. note:: |
| 35 | |
| 36 | (1) x86_64's has special implementation for memory hotplug. |
| 37 | This text does not describe it. |
| 38 | (2) This text assumes that sysfs is mounted at /sys. |
| 39 | |
| 40 | |
| 41 | Introduction |
| 42 | ============ |
| 43 | |
| 44 | purpose of memory hotplug |
| 45 | ------------------------- |
| 46 | |
| 47 | Memory Hotplug allows users to increase/decrease the amount of memory. |
| 48 | Generally, there are two purposes. |
| 49 | |
| 50 | (A) For changing the amount of memory. |
| 51 | This is to allow a feature like capacity on demand. |
| 52 | (B) For installing/removing DIMMs or NUMA-nodes physically. |
| 53 | This is to exchange DIMMs/NUMA-nodes, reduce power consumption, etc. |
| 54 | |
| 55 | (A) is required by highly virtualized environments and (B) is required by |
| 56 | hardware which supports memory power management. |
| 57 | |
| 58 | Linux memory hotplug is designed for both purpose. |
| 59 | |
| 60 | |
| 61 | Phases of memory hotplug |
| 62 | ------------------------ |
| 63 | |
| 64 | There are 2 phases in Memory Hotplug: |
| 65 | |
| 66 | 1) Physical Memory Hotplug phase |
| 67 | 2) Logical Memory Hotplug phase. |
| 68 | |
| 69 | The First phase is to communicate hardware/firmware and make/erase |
| 70 | environment for hotplugged memory. Basically, this phase is necessary |
| 71 | for the purpose (B), but this is good phase for communication between |
| 72 | highly virtualized environments too. |
| 73 | |
| 74 | When memory is hotplugged, the kernel recognizes new memory, makes new memory |
| 75 | management tables, and makes sysfs files for new memory's operation. |
| 76 | |
| 77 | If firmware supports notification of connection of new memory to OS, |
| 78 | this phase is triggered automatically. ACPI can notify this event. If not, |
| 79 | "probe" operation by system administration is used instead. |
| 80 | (see :ref:`memory_hotplug_physical_mem`). |
| 81 | |
| 82 | Logical Memory Hotplug phase is to change memory state into |
| 83 | available/unavailable for users. Amount of memory from user's view is |
| 84 | changed by this phase. The kernel makes all memory in it as free pages |
| 85 | when a memory range is available. |
| 86 | |
| 87 | In this document, this phase is described as online/offline. |
| 88 | |
| 89 | Logical Memory Hotplug phase is triggered by write of sysfs file by system |
| 90 | administrator. For the hot-add case, it must be executed after Physical Hotplug |
| 91 | phase by hand. |
| 92 | (However, if you writes udev's hotplug scripts for memory hotplug, these |
| 93 | phases can be execute in seamless way.) |
| 94 | |
| 95 | |
| 96 | Unit of Memory online/offline operation |
| 97 | --------------------------------------- |
| 98 | |
| 99 | Memory hotplug uses SPARSEMEM memory model which allows memory to be divided |
| 100 | into chunks of the same size. These chunks are called "sections". The size of |
| 101 | a memory section is architecture dependent. For example, power uses 16MiB, ia64 |
| 102 | uses 1GiB. |
| 103 | |
| 104 | Memory sections are combined into chunks referred to as "memory blocks". The |
| 105 | size of a memory block is architecture dependent and represents the logical |
| 106 | unit upon which memory online/offline operations are to be performed. The |
| 107 | default size of a memory block is the same as memory section size unless an |
| 108 | architecture specifies otherwise. (see :ref:`memory_hotplug_sysfs_files`.) |
| 109 | |
| 110 | To determine the size (in bytes) of a memory block please read this file: |
| 111 | |
| 112 | /sys/devices/system/memory/block_size_bytes |
| 113 | |
| 114 | |
| 115 | Kernel Configuration |
| 116 | ==================== |
| 117 | |
| 118 | To use memory hotplug feature, kernel must be compiled with following |
| 119 | config options. |
| 120 | |
| 121 | - For all memory hotplug: |
| 122 | - Memory model -> Sparse Memory (CONFIG_SPARSEMEM) |
| 123 | - Allow for memory hot-add (CONFIG_MEMORY_HOTPLUG) |
| 124 | |
| 125 | - To enable memory removal, the following are also necessary: |
| 126 | - Allow for memory hot remove (CONFIG_MEMORY_HOTREMOVE) |
| 127 | - Page Migration (CONFIG_MIGRATION) |
| 128 | |
| 129 | - For ACPI memory hotplug, the following are also necessary: |
| 130 | - Memory hotplug (under ACPI Support menu) (CONFIG_ACPI_HOTPLUG_MEMORY) |
| 131 | - This option can be kernel module. |
| 132 | |
| 133 | - As a related configuration, if your box has a feature of NUMA-node hotplug |
| 134 | via ACPI, then this option is necessary too. |
| 135 | |
| 136 | - ACPI0004,PNP0A05 and PNP0A06 Container Driver (under ACPI Support menu) |
| 137 | (CONFIG_ACPI_CONTAINER). |
| 138 | |
| 139 | This option can be kernel module too. |
| 140 | |
| 141 | |
| 142 | .. _memory_hotplug_sysfs_files: |
| 143 | |
| 144 | sysfs files for memory hotplug |
| 145 | ============================== |
| 146 | |
| 147 | All memory blocks have their device information in sysfs. Each memory block |
| 148 | is described under /sys/devices/system/memory as: |
| 149 | |
| 150 | /sys/devices/system/memory/memoryXXX |
| 151 | (XXX is the memory block id.) |
| 152 | |
| 153 | For the memory block covered by the sysfs directory. It is expected that all |
| 154 | memory sections in this range are present and no memory holes exist in the |
| 155 | range. Currently there is no way to determine if there is a memory hole, but |
| 156 | the existence of one should not affect the hotplug capabilities of the memory |
| 157 | block. |
| 158 | |
| 159 | For example, assume 1GiB memory block size. A device for a memory starting at |
| 160 | 0x100000000 is /sys/device/system/memory/memory4:: |
| 161 | |
| 162 | (0x100000000 / 1Gib = 4) |
| 163 | |
| 164 | This device covers address range [0x100000000 ... 0x140000000) |
| 165 | |
| 166 | Under each memory block, you can see 5 files: |
| 167 | |
| 168 | - /sys/devices/system/memory/memoryXXX/phys_index |
| 169 | - /sys/devices/system/memory/memoryXXX/phys_device |
| 170 | - /sys/devices/system/memory/memoryXXX/state |
| 171 | - /sys/devices/system/memory/memoryXXX/removable |
| 172 | - /sys/devices/system/memory/memoryXXX/valid_zones |
| 173 | |
| 174 | =================== ============================================================ |
| 175 | ``phys_index`` read-only and contains memory block id, same as XXX. |
| 176 | ``state`` read-write |
| 177 | |
| 178 | - at read: contains online/offline state of memory. |
| 179 | - at write: user can specify "online_kernel", |
| 180 | |
| 181 | "online_movable", "online", "offline" command |
| 182 | which will be performed on all sections in the block. |
| 183 | ``phys_device`` read-only: designed to show the name of physical memory |
| 184 | device. This is not well implemented now. |
| 185 | ``removable`` read-only: contains an integer value indicating |
| 186 | whether the memory block is removable or not |
| 187 | removable. A value of 1 indicates that the memory |
| 188 | block is removable and a value of 0 indicates that |
| 189 | it is not removable. A memory block is removable only if |
| 190 | every section in the block is removable. |
| 191 | ``valid_zones`` read-only: designed to show which zones this memory block |
| 192 | can be onlined to. |
| 193 | |
| 194 | The first column shows it`s default zone. |
| 195 | |
| 196 | "memory6/valid_zones: Normal Movable" shows this memoryblock |
| 197 | can be onlined to ZONE_NORMAL by default and to ZONE_MOVABLE |
| 198 | by online_movable. |
| 199 | |
| 200 | "memory7/valid_zones: Movable Normal" shows this memoryblock |
| 201 | can be onlined to ZONE_MOVABLE by default and to ZONE_NORMAL |
| 202 | by online_kernel. |
| 203 | =================== ============================================================ |
| 204 | |
| 205 | .. note:: |
| 206 | |
| 207 | These directories/files appear after physical memory hotplug phase. |
| 208 | |
| 209 | If CONFIG_NUMA is enabled the memoryXXX/ directories can also be accessed |
| 210 | via symbolic links located in the /sys/devices/system/node/node* directories. |
| 211 | |
| 212 | For example: |
| 213 | /sys/devices/system/node/node0/memory9 -> ../../memory/memory9 |
| 214 | |
| 215 | A backlink will also be created: |
| 216 | /sys/devices/system/memory/memory9/node0 -> ../../node/node0 |
| 217 | |
| 218 | .. _memory_hotplug_physical_mem: |
| 219 | |
| 220 | Physical memory hot-add phase |
| 221 | ============================= |
| 222 | |
| 223 | Hardware(Firmware) Support |
| 224 | -------------------------- |
| 225 | |
| 226 | On x86_64/ia64 platform, memory hotplug by ACPI is supported. |
| 227 | |
| 228 | In general, the firmware (ACPI) which supports memory hotplug defines |
| 229 | memory class object of _HID "PNP0C80". When a notify is asserted to PNP0C80, |
| 230 | Linux's ACPI handler does hot-add memory to the system and calls a hotplug udev |
| 231 | script. This will be done automatically. |
| 232 | |
| 233 | But scripts for memory hotplug are not contained in generic udev package(now). |
| 234 | You may have to write it by yourself or online/offline memory by hand. |
| 235 | Please see :ref:`memory_hotplug_how_to_online_memory` and |
| 236 | :ref:`memory_hotplug_how_to_offline_memory`. |
| 237 | |
| 238 | If firmware supports NUMA-node hotplug, and defines an object _HID "ACPI0004", |
| 239 | "PNP0A05", or "PNP0A06", notification is asserted to it, and ACPI handler |
| 240 | calls hotplug code for all of objects which are defined in it. |
| 241 | If memory device is found, memory hotplug code will be called. |
| 242 | |
| 243 | |
| 244 | Notify memory hot-add event by hand |
| 245 | ----------------------------------- |
| 246 | |
| 247 | On some architectures, the firmware may not notify the kernel of a memory |
| 248 | hotplug event. Therefore, the memory "probe" interface is supported to |
| 249 | explicitly notify the kernel. This interface depends on |
| 250 | CONFIG_ARCH_MEMORY_PROBE and can be configured on powerpc, sh, and x86 |
| 251 | if hotplug is supported, although for x86 this should be handled by ACPI |
| 252 | notification. |
| 253 | |
| 254 | Probe interface is located at |
| 255 | /sys/devices/system/memory/probe |
| 256 | |
| 257 | You can tell the physical address of new memory to the kernel by:: |
| 258 | |
| 259 | % echo start_address_of_new_memory > /sys/devices/system/memory/probe |
| 260 | |
| 261 | Then, [start_address_of_new_memory, start_address_of_new_memory + |
| 262 | memory_block_size] memory range is hot-added. In this case, hotplug script is |
| 263 | not called (in current implementation). You'll have to online memory by |
| 264 | yourself. Please see :ref:`memory_hotplug_how_to_online_memory`. |
| 265 | |
| 266 | |
| 267 | Logical Memory hot-add phase |
| 268 | ============================ |
| 269 | |
| 270 | State of memory |
| 271 | --------------- |
| 272 | |
| 273 | To see (online/offline) state of a memory block, read 'state' file:: |
| 274 | |
| 275 | % cat /sys/device/system/memory/memoryXXX/state |
| 276 | |
| 277 | |
| 278 | - If the memory block is online, you'll read "online". |
| 279 | - If the memory block is offline, you'll read "offline". |
| 280 | |
| 281 | |
| 282 | .. _memory_hotplug_how_to_online_memory: |
| 283 | |
| 284 | How to online memory |
| 285 | -------------------- |
| 286 | |
| 287 | When the memory is hot-added, the kernel decides whether or not to "online" |
| 288 | it according to the policy which can be read from "auto_online_blocks" file:: |
| 289 | |
| 290 | % cat /sys/devices/system/memory/auto_online_blocks |
| 291 | |
| 292 | The default depends on the CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel config |
| 293 | option. If it is disabled the default is "offline" which means the newly added |
| 294 | memory is not in a ready-to-use state and you have to "online" the newly added |
| 295 | memory blocks manually. Automatic onlining can be requested by writing "online" |
| 296 | to "auto_online_blocks" file:: |
| 297 | |
| 298 | % echo online > /sys/devices/system/memory/auto_online_blocks |
| 299 | |
| 300 | This sets a global policy and impacts all memory blocks that will subsequently |
| 301 | be hotplugged. Currently offline blocks keep their state. It is possible, under |
| 302 | certain circumstances, that some memory blocks will be added but will fail to |
| 303 | online. User space tools can check their "state" files |
| 304 | (/sys/devices/system/memory/memoryXXX/state) and try to online them manually. |
| 305 | |
| 306 | If the automatic onlining wasn't requested, failed, or some memory block was |
| 307 | offlined it is possible to change the individual block's state by writing to the |
| 308 | "state" file:: |
| 309 | |
| 310 | % echo online > /sys/devices/system/memory/memoryXXX/state |
| 311 | |
| 312 | This onlining will not change the ZONE type of the target memory block, |
| 313 | If the memory block doesn't belong to any zone an appropriate kernel zone |
| 314 | (usually ZONE_NORMAL) will be used unless movable_node kernel command line |
| 315 | option is specified when ZONE_MOVABLE will be used. |
| 316 | |
| 317 | You can explicitly request to associate it with ZONE_MOVABLE by:: |
| 318 | |
| 319 | % echo online_movable > /sys/devices/system/memory/memoryXXX/state |
| 320 | |
| 321 | .. note:: current limit: this memory block must be adjacent to ZONE_MOVABLE |
| 322 | |
| 323 | Or you can explicitly request a kernel zone (usually ZONE_NORMAL) by:: |
| 324 | |
| 325 | % echo online_kernel > /sys/devices/system/memory/memoryXXX/state |
| 326 | |
| 327 | .. note:: current limit: this memory block must be adjacent to ZONE_NORMAL |
| 328 | |
| 329 | An explicit zone onlining can fail (e.g. when the range is already within |
| 330 | and existing and incompatible zone already). |
| 331 | |
| 332 | After this, memory block XXX's state will be 'online' and the amount of |
| 333 | available memory will be increased. |
| 334 | |
| 335 | This may be changed in future. |
| 336 | |
| 337 | |
| 338 | |
| 339 | Logical memory remove |
| 340 | ===================== |
| 341 | |
| 342 | Memory offline and ZONE_MOVABLE |
| 343 | ------------------------------- |
| 344 | |
| 345 | Memory offlining is more complicated than memory online. Because memory offline |
| 346 | has to make the whole memory block be unused, memory offline can fail if |
| 347 | the memory block includes memory which cannot be freed. |
| 348 | |
| 349 | In general, memory offline can use 2 techniques. |
| 350 | |
| 351 | (1) reclaim and free all memory in the memory block. |
| 352 | (2) migrate all pages in the memory block. |
| 353 | |
| 354 | In the current implementation, Linux's memory offline uses method (2), freeing |
| 355 | all pages in the memory block by page migration. But not all pages are |
| 356 | migratable. Under current Linux, migratable pages are anonymous pages and |
| 357 | page caches. For offlining a memory block by migration, the kernel has to |
| 358 | guarantee that the memory block contains only migratable pages. |
| 359 | |
| 360 | Now, a boot option for making a memory block which consists of migratable pages |
| 361 | is supported. By specifying "kernelcore=" or "movablecore=" boot option, you can |
| 362 | create ZONE_MOVABLE...a zone which is just used for movable pages. |
| 363 | (See also Documentation/admin-guide/kernel-parameters.rst) |
| 364 | |
| 365 | Assume the system has "TOTAL" amount of memory at boot time, this boot option |
| 366 | creates ZONE_MOVABLE as following. |
| 367 | |
| 368 | 1) When kernelcore=YYYY boot option is used, |
| 369 | Size of memory not for movable pages (not for offline) is YYYY. |
| 370 | Size of memory for movable pages (for offline) is TOTAL-YYYY. |
| 371 | |
| 372 | 2) When movablecore=ZZZZ boot option is used, |
| 373 | Size of memory not for movable pages (not for offline) is TOTAL - ZZZZ. |
| 374 | Size of memory for movable pages (for offline) is ZZZZ. |
| 375 | |
| 376 | .. note:: |
| 377 | |
| 378 | Unfortunately, there is no information to show which memory block belongs |
| 379 | to ZONE_MOVABLE. This is TBD. |
| 380 | |
| 381 | .. _memory_hotplug_how_to_offline_memory: |
| 382 | |
| 383 | How to offline memory |
| 384 | --------------------- |
| 385 | |
| 386 | You can offline a memory block by using the same sysfs interface that was used |
| 387 | in memory onlining:: |
| 388 | |
| 389 | % echo offline > /sys/devices/system/memory/memoryXXX/state |
| 390 | |
| 391 | If offline succeeds, the state of the memory block is changed to be "offline". |
| 392 | If it fails, some error core (like -EBUSY) will be returned by the kernel. |
| 393 | Even if a memory block does not belong to ZONE_MOVABLE, you can try to offline |
| 394 | it. If it doesn't contain 'unmovable' memory, you'll get success. |
| 395 | |
| 396 | A memory block under ZONE_MOVABLE is considered to be able to be offlined |
| 397 | easily. But under some busy state, it may return -EBUSY. Even if a memory |
| 398 | block cannot be offlined due to -EBUSY, you can retry offlining it and may be |
| 399 | able to offline it (or not). (For example, a page is referred to by some kernel |
| 400 | internal call and released soon.) |
| 401 | |
| 402 | Consideration: |
| 403 | Memory hotplug's design direction is to make the possibility of memory |
| 404 | offlining higher and to guarantee unplugging memory under any situation. But |
| 405 | it needs more work. Returning -EBUSY under some situation may be good because |
| 406 | the user can decide to retry more or not by himself. Currently, memory |
| 407 | offlining code does some amount of retry with 120 seconds timeout. |
| 408 | |
| 409 | Physical memory remove |
| 410 | ====================== |
| 411 | |
| 412 | Need more implementation yet.... |
| 413 | - Notification completion of remove works by OS to firmware. |
| 414 | - Guard from remove if not yet. |
| 415 | |
| 416 | Memory hotplug event notifier |
| 417 | ============================= |
| 418 | |
| 419 | Hotplugging events are sent to a notification queue. |
| 420 | |
| 421 | There are six types of notification defined in include/linux/memory.h: |
| 422 | |
| 423 | MEM_GOING_ONLINE |
| 424 | Generated before new memory becomes available in order to be able to |
| 425 | prepare subsystems to handle memory. The page allocator is still unable |
| 426 | to allocate from the new memory. |
| 427 | |
| 428 | MEM_CANCEL_ONLINE |
| 429 | Generated if MEMORY_GOING_ONLINE fails. |
| 430 | |
| 431 | MEM_ONLINE |
| 432 | Generated when memory has successfully brought online. The callback may |
| 433 | allocate pages from the new memory. |
| 434 | |
| 435 | MEM_GOING_OFFLINE |
| 436 | Generated to begin the process of offlining memory. Allocations are no |
| 437 | longer possible from the memory but some of the memory to be offlined |
| 438 | is still in use. The callback can be used to free memory known to a |
| 439 | subsystem from the indicated memory block. |
| 440 | |
| 441 | MEM_CANCEL_OFFLINE |
| 442 | Generated if MEMORY_GOING_OFFLINE fails. Memory is available again from |
| 443 | the memory block that we attempted to offline. |
| 444 | |
| 445 | MEM_OFFLINE |
| 446 | Generated after offlining memory is complete. |
| 447 | |
| 448 | A callback routine can be registered by calling:: |
| 449 | |
| 450 | hotplug_memory_notifier(callback_func, priority) |
| 451 | |
| 452 | Callback functions with higher values of priority are called before callback |
| 453 | functions with lower values. |
| 454 | |
| 455 | A callback function must have the following prototype:: |
| 456 | |
| 457 | int callback_func( |
| 458 | struct notifier_block *self, unsigned long action, void *arg); |
| 459 | |
| 460 | The first argument of the callback function (self) is a pointer to the block |
| 461 | of the notifier chain that points to the callback function itself. |
| 462 | The second argument (action) is one of the event types described above. |
| 463 | The third argument (arg) passes a pointer of struct memory_notify:: |
| 464 | |
| 465 | struct memory_notify { |
| 466 | unsigned long start_pfn; |
| 467 | unsigned long nr_pages; |
| 468 | int status_change_nid_normal; |
| 469 | int status_change_nid_high; |
| 470 | int status_change_nid; |
| 471 | } |
| 472 | |
| 473 | - start_pfn is start_pfn of online/offline memory. |
| 474 | - nr_pages is # of pages of online/offline memory. |
| 475 | - status_change_nid_normal is set node id when N_NORMAL_MEMORY of nodemask |
| 476 | is (will be) set/clear, if this is -1, then nodemask status is not changed. |
| 477 | - status_change_nid_high is set node id when N_HIGH_MEMORY of nodemask |
| 478 | is (will be) set/clear, if this is -1, then nodemask status is not changed. |
| 479 | - status_change_nid is set node id when N_MEMORY of nodemask is (will be) |
| 480 | set/clear. It means a new(memoryless) node gets new memory by online and a |
| 481 | node loses all memory. If this is -1, then nodemask status is not changed. |
| 482 | |
| 483 | If status_changed_nid* >= 0, callback should create/discard structures for the |
| 484 | node if necessary. |
| 485 | |
| 486 | The callback routine shall return one of the values |
| 487 | NOTIFY_DONE, NOTIFY_OK, NOTIFY_BAD, NOTIFY_STOP |
| 488 | defined in include/linux/notifier.h |
| 489 | |
| 490 | NOTIFY_DONE and NOTIFY_OK have no effect on the further processing. |
| 491 | |
| 492 | NOTIFY_BAD is used as response to the MEM_GOING_ONLINE, MEM_GOING_OFFLINE, |
| 493 | MEM_ONLINE, or MEM_OFFLINE action to cancel hotplugging. It stops |
| 494 | further processing of the notification queue. |
| 495 | |
| 496 | NOTIFY_STOP stops further processing of the notification queue. |
| 497 | |
| 498 | Future Work |
| 499 | =========== |
| 500 | |
| 501 | - allowing memory hot-add to ZONE_MOVABLE. maybe we need some switch like |
| 502 | sysctl or new control file. |
| 503 | - showing memory block and physical device relationship. |
| 504 | - test and make it better memory offlining. |
| 505 | - support HugeTLB page migration and offlining. |
| 506 | - memmap removing at memory offline. |
| 507 | - physical remove memory. |