1 .. SPDX-License-Identifier: GPL-2.0
2 .. include:: <isonum.txt>
4 ===========================================
5 User Interface for Resource Control feature
6 ===========================================
8 :Copyright: |copy| 2016 Intel Corporation
9 :Authors: - Fenghua Yu <fenghua.yu@intel.com>
10 - Tony Luck <tony.luck@intel.com>
11 - Vikas Shivappa <vikas.shivappa@intel.com>
14 Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT).
15 AMD refers to this feature as AMD Platform Quality of Service(AMD QoS).
17 This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo
20 =============================================== ================================
21 RDT (Resource Director Technology) Allocation "rdt_a"
22 CAT (Cache Allocation Technology) "cat_l3", "cat_l2"
23 CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2"
24 CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc"
25 MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local"
26 MBA (Memory Bandwidth Allocation) "mba"
27 SMBA (Slow Memory Bandwidth Allocation) ""
28 BMEC (Bandwidth Monitoring Event Configuration) ""
29 =============================================== ================================
31 Historically, new features were made visible by default in /proc/cpuinfo. This
32 resulted in the feature flags becoming hard to parse by humans. Adding a new
33 flag to /proc/cpuinfo should be avoided if user space can obtain information
34 about the feature from resctrl's info directory.
36 To use the feature mount the file system::
38 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps]] /sys/fs/resctrl
43 Enable code/data prioritization in L3 cache allocations.
45 Enable code/data prioritization in L2 cache allocations.
47 Enable the MBA Software Controller(mba_sc) to specify MBA
50 L2 and L3 CDP are controlled separately.
52 RDT features are orthogonal. A particular system may support only
53 monitoring, only control, or both monitoring and control. Cache
54 pseudo-locking is a unique way of using cache control to "pin" or
55 "lock" data in the cache. Details can be found in
56 "Cache Pseudo-Locking".
59 The mount succeeds if either of allocation or monitoring is present, but
60 only those files and directories supported by the system will be created.
61 For more details on the behavior of the interface during monitoring
62 and allocation, see the "Resource alloc and monitor groups" section.
67 The 'info' directory contains information about the enabled
68 resources. Each resource has its own subdirectory. The subdirectory
69 names reflect the resource names.
71 Each subdirectory contains the following files with respect to
74 Cache resource(L3/L2) subdirectory contains the following files
75 related to allocation:
78 The number of CLOSIDs which are valid for this
79 resource. The kernel uses the smallest number of
80 CLOSIDs of all enabled resources as limit.
82 The bitmask which is valid for this resource.
83 This mask is equivalent to 100%.
85 The minimum number of consecutive bits which
86 must be set when writing a mask.
89 Bitmask of shareable resource with other executing
90 entities (e.g. I/O). User can use this when
91 setting up exclusive cache partitions. Note that
92 some platforms support devices that have their
93 own settings for cache use which can over-ride
96 Annotated capacity bitmasks showing how all
97 instances of the resource are used. The legend is:
100 Corresponding region is unused. When the system's
101 resources have been allocated and a "0" is found
102 in "bit_usage" it is a sign that resources are
106 Corresponding region is used by hardware only
107 but available for software use. If a resource
108 has bits set in "shareable_bits" but not all
109 of these bits appear in the resource groups'
110 schematas then the bits appearing in
111 "shareable_bits" but no resource group will
114 Corresponding region is available for sharing and
115 used by hardware and software. These are the
116 bits that appear in "shareable_bits" as
117 well as a resource group's allocation.
119 Corresponding region is used by software
120 and available for sharing.
122 Corresponding region is used exclusively by
123 one resource group. No sharing allowed.
125 Corresponding region is pseudo-locked. No
128 Memory bandwidth(MB) subdirectory contains the following files
129 with respect to allocation:
132 The minimum memory bandwidth percentage which
136 The granularity in which the memory bandwidth
137 percentage is allocated. The allocated
138 b/w percentage is rounded off to the next
139 control step available on the hardware. The
140 available bandwidth control steps are:
141 min_bandwidth + N * bandwidth_gran.
144 Indicates if the delay scale is linear or
145 non-linear. This field is purely informational
148 "thread_throttle_mode":
149 Indicator on Intel systems of how tasks running on threads
150 of a physical core are throttled in cases where they
151 request different memory bandwidth percentages:
154 the smallest percentage is applied
157 bandwidth percentages are directly applied to
158 the threads running on the core
160 If RDT monitoring is available there will be an "L3_MON" directory
161 with the following files:
164 The number of RMIDs available. This is the
165 upper bound for how many "CTRL_MON" + "MON"
166 groups can be created.
169 Lists the monitoring events if
170 monitoring is enabled for the resource.
173 # cat /sys/fs/resctrl/info/L3_MON/mon_features
178 If the system supports Bandwidth Monitoring Event
179 Configuration (BMEC), then the bandwidth events will
180 be configurable. The output will be::
182 # cat /sys/fs/resctrl/info/L3_MON/mon_features
185 mbm_total_bytes_config
187 mbm_local_bytes_config
189 "mbm_total_bytes_config", "mbm_local_bytes_config":
190 Read/write files containing the configuration for the mbm_total_bytes
191 and mbm_local_bytes events, respectively, when the Bandwidth
192 Monitoring Event Configuration (BMEC) feature is supported.
193 The event configuration settings are domain specific and affect
194 all the CPUs in the domain. When either event configuration is
195 changed, the bandwidth counters for all RMIDs of both events
196 (mbm_total_bytes as well as mbm_local_bytes) are cleared for that
197 domain. The next read for every RMID will report "Unavailable"
198 and subsequent reads will report the valid value.
200 Following are the types of events supported:
202 ==== ========================================================
204 ==== ========================================================
205 6 Dirty Victims from the QOS domain to all types of memory
206 5 Reads to slow memory in the non-local NUMA domain
207 4 Reads to slow memory in the local NUMA domain
208 3 Non-temporal writes to non-local NUMA domain
209 2 Non-temporal writes to local NUMA domain
210 1 Reads to memory in the non-local NUMA domain
211 0 Reads to memory in the local NUMA domain
212 ==== ========================================================
214 By default, the mbm_total_bytes configuration is set to 0x7f to count
215 all the event types and the mbm_local_bytes configuration is set to
216 0x15 to count all the local memory events.
220 * To view the current configuration::
223 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
224 0=0x7f;1=0x7f;2=0x7f;3=0x7f
226 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
227 0=0x15;1=0x15;3=0x15;4=0x15
229 * To change the mbm_total_bytes to count only reads on domain 0,
230 the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary
231 (in hexadecimal 0x33):
234 # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
236 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
237 0=0x33;1=0x7f;2=0x7f;3=0x7f
239 * To change the mbm_local_bytes to count all the slow memory reads on
240 domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b
241 in binary (in hexadecimal 0x30):
244 # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
246 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
247 0=0x30;1=0x30;3=0x15;4=0x15
249 "max_threshold_occupancy":
250 Read/write file provides the largest value (in
251 bytes) at which a previously used LLC_occupancy
252 counter can be considered for re-use.
254 Finally, in the top level of the "info" directory there is a file
255 named "last_cmd_status". This is reset with every "command" issued
256 via the file system (making new directories or writing to any of the
257 control files). If the command was successful, it will read as "ok".
258 If the command failed, it will provide more information that can be
259 conveyed in the error returns from file operations. E.g.
262 # echo L3:0=f7 > schemata
263 bash: echo: write error: Invalid argument
264 # cat info/last_cmd_status
265 mask f7 has non-consecutive 1-bits
267 Resource alloc and monitor groups
268 =================================
270 Resource groups are represented as directories in the resctrl file
271 system. The default group is the root directory which, immediately
272 after mounting, owns all the tasks and cpus in the system and can make
273 full use of all resources.
275 On a system with RDT control features additional directories can be
276 created in the root directory that specify different amounts of each
277 resource (see "schemata" below). The root and these additional top level
278 directories are referred to as "CTRL_MON" groups below.
280 On a system with RDT monitoring the root directory and other top level
281 directories contain a directory named "mon_groups" in which additional
282 directories can be created to monitor subsets of tasks in the CTRL_MON
283 group that is their ancestor. These are called "MON" groups in the rest
286 Removing a directory will move all tasks and cpus owned by the group it
287 represents to the parent. Removing one of the created CTRL_MON groups
288 will automatically remove all MON groups below it.
290 All groups contain the following files:
293 Reading this file shows the list of all tasks that belong to
294 this group. Writing a task id to the file will add a task to the
295 group. If the group is a CTRL_MON group the task is removed from
296 whichever previous CTRL_MON group owned the task and also from
297 any MON group that owned the task. If the group is a MON group,
298 then the task must already belong to the CTRL_MON parent of this
299 group. The task is removed from any previous MON group.
303 Reading this file shows a bitmask of the logical CPUs owned by
304 this group. Writing a mask to this file will add and remove
305 CPUs to/from this group. As with the tasks file a hierarchy is
306 maintained where MON groups may only include CPUs owned by the
307 parent CTRL_MON group.
308 When the resource group is in pseudo-locked mode this file will
309 only be readable, reflecting the CPUs associated with the
310 pseudo-locked region.
314 Just like "cpus", only using ranges of CPUs instead of bitmasks.
317 When control is enabled all CTRL_MON groups will also contain:
320 A list of all the resources available to this group.
321 Each resource has its own line and format - see below for details.
324 Mirrors the display of the "schemata" file to display the size in
325 bytes of each allocation instead of the bits representing the
329 The "mode" of the resource group dictates the sharing of its
330 allocations. A "shareable" resource group allows sharing of its
331 allocations while an "exclusive" resource group does not. A
332 cache pseudo-locked region is created by first writing
333 "pseudo-locksetup" to the "mode" file before writing the cache
334 pseudo-locked region's schemata to the resource group's "schemata"
335 file. On successful pseudo-locked region creation the mode will
336 automatically change to "pseudo-locked".
338 When monitoring is enabled all MON groups will also contain:
341 This contains a set of files organized by L3 domain and by
342 RDT event. E.g. on a system with two L3 domains there will
343 be subdirectories "mon_L3_00" and "mon_L3_01". Each of these
344 directories have one file per event (e.g. "llc_occupancy",
345 "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
346 files provide a read out of the current value of the event for
347 all tasks in the group. In CTRL_MON groups these files provide
348 the sum for all tasks in the CTRL_MON group and all tasks in
349 MON groups. Please see example section for more details on usage.
351 Resource allocation rules
352 -------------------------
354 When a task is running the following rules define which resources are
357 1) If the task is a member of a non-default group, then the schemata
358 for that group is used.
360 2) Else if the task belongs to the default group, but is running on a
361 CPU that is assigned to some specific group, then the schemata for the
364 3) Otherwise the schemata for the default group is used.
366 Resource monitoring rules
367 -------------------------
368 1) If a task is a member of a MON group, or non-default CTRL_MON group
369 then RDT events for the task will be reported in that group.
371 2) If a task is a member of the default CTRL_MON group, but is running
372 on a CPU that is assigned to some specific group, then the RDT events
373 for the task will be reported in that group.
375 3) Otherwise RDT events for the task will be reported in the root level
379 Notes on cache occupancy monitoring and control
380 ===============================================
381 When moving a task from one group to another you should remember that
382 this only affects *new* cache allocations by the task. E.g. you may have
383 a task in a monitor group showing 3 MB of cache occupancy. If you move
384 to a new group and immediately check the occupancy of the old and new
385 groups you will likely see that the old group is still showing 3 MB and
386 the new group zero. When the task accesses locations still in cache from
387 before the move, the h/w does not update any counters. On a busy system
388 you will likely see the occupancy in the old group go down as cache lines
389 are evicted and re-used while the occupancy in the new group rises as
390 the task accesses memory and loads into the cache are counted based on
391 membership in the new group.
393 The same applies to cache allocation control. Moving a task to a group
394 with a smaller cache partition will not evict any cache lines. The
395 process may continue to use them from the old partition.
397 Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID)
398 to identify a control group and a monitoring group respectively. Each of
399 the resource groups are mapped to these IDs based on the kind of group. The
400 number of CLOSid and RMID are limited by the hardware and hence the creation of
401 a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID
402 and creation of "MON" group may fail if we run out of RMIDs.
404 max_threshold_occupancy - generic concepts
405 ------------------------------------------
407 Note that an RMID once freed may not be immediately available for use as
408 the RMID is still tagged the cache lines of the previous user of RMID.
409 Hence such RMIDs are placed on limbo list and checked back if the cache
410 occupancy has gone down. If there is a time when system has a lot of
411 limbo RMIDs but which are not ready to be used, user may see an -EBUSY
414 max_threshold_occupancy is a user configurable value to determine the
415 occupancy at which an RMID can be freed.
417 Schemata files - general concepts
418 ---------------------------------
419 Each line in the file describes one resource. The line starts with
420 the name of the resource, followed by specific values to be applied
421 in each of the instances of that resource on the system.
425 On current generation systems there is one L3 cache per socket and L2
426 caches are generally just shared by the hyperthreads on a core, but this
427 isn't an architectural requirement. We could have multiple separate L3
428 caches on a socket, multiple cores could share an L2 cache. So instead
429 of using "socket" or "core" to define the set of logical cpus sharing
430 a resource we use a "Cache ID". At a given cache level this will be a
431 unique number across the whole system (but it isn't guaranteed to be a
432 contiguous sequence, there may be gaps). To find the ID for each logical
433 CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id
435 Cache Bit Masks (CBM)
436 ---------------------
437 For cache resources we describe the portion of the cache that is available
438 for allocation using a bitmask. The maximum value of the mask is defined
439 by each cpu model (and may be different for different cache levels). It
440 is found using CPUID, but is also provided in the "info" directory of
441 the resctrl file system in "info/{resource}/cbm_mask". Intel hardware
442 requires that these masks have all the '1' bits in a contiguous block. So
443 0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9
444 and 0xA are not. On a system with a 20-bit mask each bit represents 5%
445 of the capacity of the cache. You could partition the cache into four
446 equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
448 Memory bandwidth Allocation and monitoring
449 ==========================================
451 For Memory bandwidth resource, by default the user controls the resource
452 by indicating the percentage of total memory bandwidth.
454 The minimum bandwidth percentage value for each cpu model is predefined
455 and can be looked up through "info/MB/min_bandwidth". The bandwidth
456 granularity that is allocated is also dependent on the cpu model and can
457 be looked up at "info/MB/bandwidth_gran". The available bandwidth
458 control steps are: min_bw + N * bw_gran. Intermediate values are rounded
459 to the next control step available on the hardware.
461 The bandwidth throttling is a core specific mechanism on some of Intel
462 SKUs. Using a high bandwidth and a low bandwidth setting on two threads
463 sharing a core may result in both threads being throttled to use the
464 low bandwidth (see "thread_throttle_mode").
466 The fact that Memory bandwidth allocation(MBA) may be a core
467 specific mechanism where as memory bandwidth monitoring(MBM) is done at
468 the package level may lead to confusion when users try to apply control
469 via the MBA and then monitor the bandwidth to see if the controls are
470 effective. Below are such scenarios:
472 1. User may *not* see increase in actual bandwidth when percentage
473 values are increased:
475 This can occur when aggregate L2 external bandwidth is more than L3
476 external bandwidth. Consider an SKL SKU with 24 cores on a package and
477 where L2 external is 10GBps (hence aggregate L2 external bandwidth is
478 240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20
479 threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3
480 bandwidth of 100GBps although the percentage value specified is only 50%
481 << 100%. Hence increasing the bandwidth percentage will not yield any
482 more bandwidth. This is because although the L2 external bandwidth still
483 has capacity, the L3 external bandwidth is fully used. Also note that
484 this would be dependent on number of cores the benchmark is run on.
486 2. Same bandwidth percentage may mean different actual bandwidth
487 depending on # of threads:
489 For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4
490 thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although
491 they have same percentage bandwidth of 10%. This is simply because as
492 threads start using more cores in an rdtgroup, the actual bandwidth may
493 increase or vary although user specified bandwidth percentage is same.
495 In order to mitigate this and make the interface more user friendly,
496 resctrl added support for specifying the bandwidth in MBps as well. The
497 kernel underneath would use a software feedback mechanism or a "Software
498 Controller(mba_sc)" which reads the actual bandwidth using MBM counters
499 and adjust the memory bandwidth percentages to ensure::
501 "actual bandwidth < user specified bandwidth".
503 By default, the schemata would take the bandwidth percentage values
504 where as user can switch to the "MBA software controller" mode using
505 a mount option 'mba_MBps'. The schemata format is specified in the below
508 L3 schemata file details (code and data prioritization disabled)
509 ----------------------------------------------------------------
510 With CDP disabled the L3 schemata format is::
512 L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
514 L3 schemata file details (CDP enabled via mount option to resctrl)
515 ------------------------------------------------------------------
516 When CDP is enabled L3 control is split into two separate resources
517 so you can specify independent masks for code and data like this::
519 L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
520 L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
522 L2 schemata file details
523 ------------------------
524 CDP is supported at L2 using the 'cdpl2' mount option. The schemata
527 L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
531 L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
532 L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
535 Memory bandwidth Allocation (default mode)
536 ------------------------------------------
538 Memory b/w domain is L3 cache.
541 MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
543 Memory bandwidth Allocation specified in MBps
544 ---------------------------------------------
546 Memory bandwidth domain is L3 cache.
549 MB:<cache_id0>=bw_MBps0;<cache_id1>=bw_MBps1;...
551 Slow Memory Bandwidth Allocation (SMBA)
552 ---------------------------------------
553 AMD hardware supports Slow Memory Bandwidth Allocation (SMBA).
554 CXL.memory is the only supported "slow" memory device. With the
555 support of SMBA, the hardware enables bandwidth allocation on
556 the slow memory devices. If there are multiple such devices in
557 the system, the throttling logic groups all the slow sources
558 together and applies the limit on them as a whole.
560 The presence of SMBA (with CXL.memory) is independent of slow memory
561 devices presence. If there are no such devices on the system, then
562 configuring SMBA will have no impact on the performance of the system.
564 The bandwidth domain for slow memory is L3 cache. Its schemata file
568 SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
570 Reading/writing the schemata file
571 ---------------------------------
572 Reading the schemata file will show the state of all resources
573 on all domains. When writing you only need to specify those values
574 which you wish to change. E.g.
578 L3DATA:0=fffff;1=fffff;2=fffff;3=fffff
579 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
580 # echo "L3DATA:2=3c0;" > schemata
582 L3DATA:0=fffff;1=fffff;2=3c0;3=fffff
583 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
585 Reading/writing the schemata file (on AMD systems)
586 --------------------------------------------------
587 Reading the schemata file will show the current bandwidth limit on all
588 domains. The allocated resources are in multiples of one eighth GB/s.
589 When writing to the file, you need to specify what cache id you wish to
590 configure the bandwidth limit.
592 For example, to allocate 2GB/s limit on the first cache id:
597 MB:0=2048;1=2048;2=2048;3=2048
598 L3:0=ffff;1=ffff;2=ffff;3=ffff
600 # echo "MB:1=16" > schemata
602 MB:0=2048;1= 16;2=2048;3=2048
603 L3:0=ffff;1=ffff;2=ffff;3=ffff
605 Reading/writing the schemata file (on AMD systems) with SMBA feature
606 --------------------------------------------------------------------
607 Reading and writing the schemata file is the same as without SMBA in
610 For example, to allocate 8GB/s limit on the first cache id:
615 SMBA:0=2048;1=2048;2=2048;3=2048
616 MB:0=2048;1=2048;2=2048;3=2048
617 L3:0=ffff;1=ffff;2=ffff;3=ffff
619 # echo "SMBA:1=64" > schemata
621 SMBA:0=2048;1= 64;2=2048;3=2048
622 MB:0=2048;1=2048;2=2048;3=2048
623 L3:0=ffff;1=ffff;2=ffff;3=ffff
627 CAT enables a user to specify the amount of cache space that an
628 application can fill. Cache pseudo-locking builds on the fact that a
629 CPU can still read and write data pre-allocated outside its current
630 allocated area on a cache hit. With cache pseudo-locking, data can be
631 preloaded into a reserved portion of cache that no application can
632 fill, and from that point on will only serve cache hits. The cache
633 pseudo-locked memory is made accessible to user space where an
634 application can map it into its virtual address space and thus have
635 a region of memory with reduced average read latency.
637 The creation of a cache pseudo-locked region is triggered by a request
638 from the user to do so that is accompanied by a schemata of the region
639 to be pseudo-locked. The cache pseudo-locked region is created as follows:
641 - Create a CAT allocation CLOSNEW with a CBM matching the schemata
642 from the user of the cache region that will contain the pseudo-locked
643 memory. This region must not overlap with any current CAT allocation/CLOS
644 on the system and no future overlap with this cache region is allowed
645 while the pseudo-locked region exists.
646 - Create a contiguous region of memory of the same size as the cache
648 - Flush the cache, disable hardware prefetchers, disable preemption.
649 - Make CLOSNEW the active CLOS and touch the allocated memory to load
651 - Set the previous CLOS as active.
652 - At this point the closid CLOSNEW can be released - the cache
653 pseudo-locked region is protected as long as its CBM does not appear in
654 any CAT allocation. Even though the cache pseudo-locked region will from
655 this point on not appear in any CBM of any CLOS an application running with
656 any CLOS will be able to access the memory in the pseudo-locked region since
657 the region continues to serve cache hits.
658 - The contiguous region of memory loaded into the cache is exposed to
659 user-space as a character device.
661 Cache pseudo-locking increases the probability that data will remain
662 in the cache via carefully configuring the CAT feature and controlling
663 application behavior. There is no guarantee that data is placed in
664 cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
665 “locked” data from cache. Power management C-states may shrink or
666 power off cache. Deeper C-states will automatically be restricted on
667 pseudo-locked region creation.
669 It is required that an application using a pseudo-locked region runs
670 with affinity to the cores (or a subset of the cores) associated
671 with the cache on which the pseudo-locked region resides. A sanity check
672 within the code will not allow an application to map pseudo-locked memory
673 unless it runs with affinity to cores associated with the cache on which the
674 pseudo-locked region resides. The sanity check is only done during the
675 initial mmap() handling, there is no enforcement afterwards and the
676 application self needs to ensure it remains affine to the correct cores.
678 Pseudo-locking is accomplished in two stages:
680 1) During the first stage the system administrator allocates a portion
681 of cache that should be dedicated to pseudo-locking. At this time an
682 equivalent portion of memory is allocated, loaded into allocated
683 cache portion, and exposed as a character device.
684 2) During the second stage a user-space application maps (mmap()) the
685 pseudo-locked memory into its address space.
687 Cache Pseudo-Locking Interface
688 ------------------------------
689 A pseudo-locked region is created using the resctrl interface as follows:
691 1) Create a new resource group by creating a new directory in /sys/fs/resctrl.
692 2) Change the new resource group's mode to "pseudo-locksetup" by writing
693 "pseudo-locksetup" to the "mode" file.
694 3) Write the schemata of the pseudo-locked region to the "schemata" file. All
695 bits within the schemata should be "unused" according to the "bit_usage"
698 On successful pseudo-locked region creation the "mode" file will contain
699 "pseudo-locked" and a new character device with the same name as the resource
700 group will exist in /dev/pseudo_lock. This character device can be mmap()'ed
701 by user space in order to obtain access to the pseudo-locked memory region.
703 An example of cache pseudo-locked region creation and usage can be found below.
705 Cache Pseudo-Locking Debugging Interface
706 ----------------------------------------
707 The pseudo-locking debugging interface is enabled by default (if
708 CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl.
710 There is no explicit way for the kernel to test if a provided memory
711 location is present in the cache. The pseudo-locking debugging interface uses
712 the tracing infrastructure to provide two ways to measure cache residency of
713 the pseudo-locked region:
715 1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data
716 from these measurements are best visualized using a hist trigger (see
717 example below). In this test the pseudo-locked region is traversed at
718 a stride of 32 bytes while hardware prefetchers and preemption
719 are disabled. This also provides a substitute visualization of cache
721 2) Cache hit and miss measurements using model specific precision counters if
722 available. Depending on the levels of cache on the system the pseudo_lock_l2
723 and pseudo_lock_l3 tracepoints are available.
725 When a pseudo-locked region is created a new debugfs directory is created for
726 it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single
727 write-only file, pseudo_lock_measure, is present in this directory. The
728 measurement of the pseudo-locked region depends on the number written to this
732 writing "1" to the pseudo_lock_measure file will trigger the latency
733 measurement captured in the pseudo_lock_mem_latency tracepoint. See
736 writing "2" to the pseudo_lock_measure file will trigger the L2 cache
737 residency (cache hits and misses) measurement captured in the
738 pseudo_lock_l2 tracepoint. See example below.
740 writing "3" to the pseudo_lock_measure file will trigger the L3 cache
741 residency (cache hits and misses) measurement captured in the
742 pseudo_lock_l3 tracepoint.
744 All measurements are recorded with the tracing infrastructure. This requires
745 the relevant tracepoints to be enabled before the measurement is triggered.
747 Example of latency debugging interface
748 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
749 In this example a pseudo-locked region named "newlock" was created. Here is
750 how we can measure the latency in cycles of reading from this region and
751 visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS
754 # :> /sys/kernel/tracing/trace
755 # echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger
756 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
757 # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
758 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
759 # cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist
763 # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active]
766 { latency: 456 } hitcount: 1
767 { latency: 50 } hitcount: 83
768 { latency: 36 } hitcount: 96
769 { latency: 44 } hitcount: 174
770 { latency: 48 } hitcount: 195
771 { latency: 46 } hitcount: 262
772 { latency: 42 } hitcount: 693
773 { latency: 40 } hitcount: 3204
774 { latency: 38 } hitcount: 3484
781 Example of cache hits/misses debugging
782 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
783 In this example a pseudo-locked region named "newlock" was created on the L2
784 cache of a platform. Here is how we can obtain details of the cache hits
785 and misses using the platform's precision counters.
788 # :> /sys/kernel/tracing/trace
789 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
790 # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
791 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
792 # cat /sys/kernel/tracing/trace
797 # / _----=> need-resched
798 # | / _---=> hardirq/softirq
799 # || / _--=> preempt-depth
801 # TASK-PID CPU# |||| TIMESTAMP FUNCTION
803 pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0
806 Examples for RDT allocation usage
807 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
811 On a two socket machine (one L3 cache per socket) with just four bits
812 for cache bit masks, minimum b/w of 10% with a memory bandwidth
816 # mount -t resctrl resctrl /sys/fs/resctrl
819 # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata
820 # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata
822 The default resource group is unmodified, so we have access to all parts
823 of all caches (its schemata file reads "L3:0=f;1=f").
825 Tasks that are under the control of group "p0" may only allocate from the
826 "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
827 Tasks in group "p1" use the "lower" 50% of cache on both sockets.
829 Similarly, tasks that are under the control of group "p0" may use a
830 maximum memory b/w of 50% on socket0 and 50% on socket 1.
831 Tasks in group "p1" may also use 50% memory b/w on both sockets.
832 Note that unlike cache masks, memory b/w cannot specify whether these
833 allocations can overlap or not. The allocations specifies the maximum
834 b/w that the group may be able to use and the system admin can configure
837 If resctrl is using the software controller (mba_sc) then user can enter the
838 max b/w in MB rather than the percentage values.
841 # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata
842 # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata
844 In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w
845 of 1024MB where as on socket 1 they would use 500MB.
849 Again two sockets, but this time with a more realistic 20-bit mask.
851 Two real time tasks pid=1234 running on processor 0 and pid=5678 running on
852 processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy
853 neighbors, each of the two real-time tasks exclusively occupies one quarter
854 of L3 cache on socket 0.
857 # mount -t resctrl resctrl /sys/fs/resctrl
860 First we reset the schemata for the default group so that the "upper"
861 50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by
864 # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata
866 Next we make a resource group for our first real time task and give
867 it access to the "top" 25% of the cache on socket 0.
871 # echo "L3:0=f8000;1=fffff" > p0/schemata
873 Finally we move our first real time task into this resource group. We
874 also use taskset(1) to ensure the task always runs on a dedicated CPU
875 on socket 0. Most uses of resource groups will also constrain which
876 processors tasks run on.
879 # echo 1234 > p0/tasks
882 Ditto for the second real time task (with the remaining 25% of cache)::
885 # echo "L3:0=7c00;1=fffff" > p1/schemata
886 # echo 5678 > p1/tasks
889 For the same 2 socket system with memory b/w resource and CAT L3 the
890 schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is
893 For our first real time task this would request 20% memory b/w on socket 0.
896 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
898 For our second real time task this would request an other 20% memory b/w
902 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
906 A single socket system which has real-time tasks running on core 4-7 and
907 non real-time workload assigned to core 0-3. The real-time tasks share text
908 and data, so a per task association is not required and due to interaction
909 with the kernel it's desired that the kernel on these cores shares L3 with
913 # mount -t resctrl resctrl /sys/fs/resctrl
916 First we reset the schemata for the default group so that the "upper"
917 50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0
918 cannot be used by ordinary tasks::
920 # echo "L3:0=3ff\nMB:0=50" > schemata
922 Next we make a resource group for our real time cores and give it access
923 to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on
928 # echo "L3:0=ffc00\nMB:0=50" > p0/schemata
930 Finally we move core 4-7 over to the new group and make sure that the
931 kernel and the tasks running there get 50% of the cache. They should
932 also get 50% of memory bandwidth assuming that the cores 4-7 are SMT
933 siblings and only the real time threads are scheduled on the cores 4-7.
940 The resource groups in previous examples were all in the default "shareable"
941 mode allowing sharing of their cache allocations. If one resource group
942 configures a cache allocation then nothing prevents another resource group
943 to overlap with that allocation.
945 In this example a new exclusive resource group will be created on a L2 CAT
946 system with two L2 cache instances that can be configured with an 8-bit
947 capacity bitmask. The new exclusive resource group will be configured to use
948 25% of each cache instance.
951 # mount -t resctrl resctrl /sys/fs/resctrl/
954 First, we observe that the default group is configured to allocate to all L2
960 We could attempt to create the new resource group at this point, but it will
961 fail because of the overlap with the schemata of the default group::
964 # echo 'L2:0=0x3;1=0x3' > p0/schemata
967 # echo exclusive > p0/mode
968 -sh: echo: write error: Invalid argument
969 # cat info/last_cmd_status
972 To ensure that there is no overlap with another resource group the default
973 resource group's schemata has to change, making it possible for the new
974 resource group to become exclusive.
977 # echo 'L2:0=0xfc;1=0xfc' > schemata
978 # echo exclusive > p0/mode
982 p0/schemata:L2:0=03;1=03
983 p0/size:L2:0=262144;1=262144
985 A new resource group will on creation not overlap with an exclusive resource
992 p1/schemata:L2:0=fc;1=fc
993 p1/size:L2:0=786432;1=786432
995 The bit_usage will reflect how the cache is used::
997 # cat info/L2/bit_usage
998 0=SSSSSSEE;1=SSSSSSEE
1000 A resource group cannot be forced to overlap with an exclusive resource group::
1002 # echo 'L2:0=0x1;1=0x1' > p1/schemata
1003 -sh: echo: write error: Invalid argument
1004 # cat info/last_cmd_status
1005 overlaps with exclusive group
1007 Example of Cache Pseudo-Locking
1008 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1009 Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked
1010 region is exposed at /dev/pseudo_lock/newlock that can be provided to
1011 application for argument to mmap().
1014 # mount -t resctrl resctrl /sys/fs/resctrl/
1015 # cd /sys/fs/resctrl
1017 Ensure that there are bits available that can be pseudo-locked, since only
1018 unused bits can be pseudo-locked the bits to be pseudo-locked needs to be
1019 removed from the default resource group's schemata::
1021 # cat info/L2/bit_usage
1022 0=SSSSSSSS;1=SSSSSSSS
1023 # echo 'L2:1=0xfc' > schemata
1024 # cat info/L2/bit_usage
1025 0=SSSSSSSS;1=SSSSSS00
1027 Create a new resource group that will be associated with the pseudo-locked
1028 region, indicate that it will be used for a pseudo-locked region, and
1029 configure the requested pseudo-locked region capacity bitmask::
1032 # echo pseudo-locksetup > newlock/mode
1033 # echo 'L2:1=0x3' > newlock/schemata
1035 On success the resource group's mode will change to pseudo-locked, the
1036 bit_usage will reflect the pseudo-locked region, and the character device
1037 exposing the pseudo-locked region will exist::
1041 # cat info/L2/bit_usage
1042 0=SSSSSSSS;1=SSSSSSPP
1043 # ls -l /dev/pseudo_lock/newlock
1044 crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock
1049 * Example code to access one page of pseudo-locked cache region
1058 #include <sys/mman.h>
1061 * It is required that the application runs with affinity to only
1062 * cores associated with the pseudo-locked region. Here the cpu
1063 * is hardcoded for convenience of example.
1065 static int cpuid = 2;
1067 int main(int argc, char *argv[])
1075 page_size = sysconf(_SC_PAGESIZE);
1078 CPU_SET(cpuid, &cpuset);
1079 ret = sched_setaffinity(0, sizeof(cpuset), &cpuset);
1081 perror("sched_setaffinity");
1085 dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR);
1091 mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
1093 if (mapping == MAP_FAILED) {
1099 /* Application interacts with pseudo-locked memory @mapping */
1101 ret = munmap(mapping, page_size);
1112 Locking between applications
1113 ----------------------------
1115 Certain operations on the resctrl filesystem, composed of read/writes
1116 to/from multiple files, must be atomic.
1118 As an example, the allocation of an exclusive reservation of L3 cache
1121 1. Read the cbmmasks from each directory or the per-resource "bit_usage"
1122 2. Find a contiguous set of bits in the global CBM bitmask that is clear
1123 in any of the directory cbmmasks
1124 3. Create a new directory
1125 4. Set the bits found in step 2 to the new directory "schemata" file
1127 If two applications attempt to allocate space concurrently then they can
1128 end up allocating the same bits so the reservations are shared instead of
1131 To coordinate atomic operations on the resctrlfs and to avoid the problem
1132 above, the following locking procedure is recommended:
1134 Locking is based on flock, which is available in libc and also as a shell
1139 A) Take flock(LOCK_EX) on /sys/fs/resctrl
1140 B) Read/write the directory structure.
1145 A) Take flock(LOCK_SH) on /sys/fs/resctrl
1146 B) If success read the directory structure.
1151 # Atomically read directory structure
1152 $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl
1154 # Read directory contents and create new subdirectory
1157 find /sys/fs/resctrl/ > output.txt
1158 mask = function-of(output.txt)
1159 mkdir /sys/fs/resctrl/newres/
1160 echo mask > /sys/fs/resctrl/newres/schemata
1162 $ flock /sys/fs/resctrl/ ./create-dir.sh
1167 * Example code do take advisory locks
1168 * before accessing resctrl filesystem
1170 #include <sys/file.h>
1173 void resctrl_take_shared_lock(int fd)
1177 /* take shared lock on resctrl filesystem */
1178 ret = flock(fd, LOCK_SH);
1185 void resctrl_take_exclusive_lock(int fd)
1189 /* release lock on resctrl filesystem */
1190 ret = flock(fd, LOCK_EX);
1197 void resctrl_release_lock(int fd)
1201 /* take shared lock on resctrl filesystem */
1202 ret = flock(fd, LOCK_UN);
1213 fd = open("/sys/fs/resctrl", O_DIRECTORY);
1218 resctrl_take_shared_lock(fd);
1219 /* code to read directory contents */
1220 resctrl_release_lock(fd);
1222 resctrl_take_exclusive_lock(fd);
1223 /* code to read and write directory contents */
1224 resctrl_release_lock(fd);
1227 Examples for RDT Monitoring along with allocation usage
1228 =======================================================
1229 Reading monitored data
1230 ----------------------
1231 Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would
1232 show the current snapshot of LLC occupancy of the corresponding MON
1233 group or CTRL_MON group.
1236 Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group)
1237 ------------------------------------------------------------------------
1238 On a two socket machine (one L3 cache per socket) with just four bits
1239 for cache bit masks::
1241 # mount -t resctrl resctrl /sys/fs/resctrl
1242 # cd /sys/fs/resctrl
1244 # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata
1245 # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata
1246 # echo 5678 > p1/tasks
1247 # echo 5679 > p1/tasks
1249 The default resource group is unmodified, so we have access to all parts
1250 of all caches (its schemata file reads "L3:0=f;1=f").
1252 Tasks that are under the control of group "p0" may only allocate from the
1253 "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
1254 Tasks in group "p1" use the "lower" 50% of cache on both sockets.
1256 Create monitor groups and assign a subset of tasks to each monitor group.
1259 # cd /sys/fs/resctrl/p1/mon_groups
1261 # echo 5678 > m11/tasks
1262 # echo 5679 > m12/tasks
1264 fetch data (data shown in bytes)
1267 # cat m11/mon_data/mon_L3_00/llc_occupancy
1269 # cat m11/mon_data/mon_L3_01/llc_occupancy
1271 # cat m12/mon_data/mon_L3_00/llc_occupancy
1274 The parent ctrl_mon group shows the aggregated data.
1277 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
1280 Example 2 (Monitor a task from its creation)
1281 --------------------------------------------
1282 On a two socket machine (one L3 cache per socket)::
1284 # mount -t resctrl resctrl /sys/fs/resctrl
1285 # cd /sys/fs/resctrl
1288 An RMID is allocated to the group once its created and hence the <cmd>
1289 below is monitored from its creation.
1292 # echo $$ > /sys/fs/resctrl/p1/tasks
1297 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
1300 Example 3 (Monitor without CAT support or before creating CAT groups)
1301 ---------------------------------------------------------------------
1303 Assume a system like HSW has only CQM and no CAT support. In this case
1304 the resctrl will still mount but cannot create CTRL_MON directories.
1305 But user can create different MON groups within the root group thereby
1306 able to monitor all tasks including kernel threads.
1308 This can also be used to profile jobs cache size footprint before being
1309 able to allocate them to different allocation groups.
1312 # mount -t resctrl resctrl /sys/fs/resctrl
1313 # cd /sys/fs/resctrl
1314 # mkdir mon_groups/m01
1315 # mkdir mon_groups/m02
1317 # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks
1318 # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks
1320 Monitor the groups separately and also get per domain data. From the
1321 below its apparent that the tasks are mostly doing work on
1325 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy
1327 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy
1329 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy
1331 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy
1335 Example 4 (Monitor real time tasks)
1336 -----------------------------------
1338 A single socket system which has real time tasks running on cores 4-7
1339 and non real time tasks on other cpus. We want to monitor the cache
1340 occupancy of the real time threads on these cores.
1343 # mount -t resctrl resctrl /sys/fs/resctrl
1344 # cd /sys/fs/resctrl
1347 Move the cpus 4-7 over to p1::
1351 View the llc occupancy snapshot::
1353 # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy
1359 Intel MBM Counters May Report System Memory Bandwidth Incorrectly
1360 -----------------------------------------------------------------
1362 Errata SKX99 for Skylake server and BDF102 for Broadwell server.
1364 Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics
1365 according to the assigned Resource Monitor ID (RMID) for that logical
1366 core. The IA32_QM_CTR register (MSR 0xC8E), used to report these
1367 metrics, may report incorrect system bandwidth for certain RMID values.
1369 Implication: Due to the errata, system memory bandwidth may not match
1372 Workaround: MBM total and local readings are corrected according to the
1373 following correction factor table:
1375 +---------------+---------------+---------------+-----------------+
1376 |core count |rmid count |rmid threshold |correction factor|
1377 +---------------+---------------+---------------+-----------------+
1378 |1 |8 |0 |1.000000 |
1379 +---------------+---------------+---------------+-----------------+
1380 |2 |16 |0 |1.000000 |
1381 +---------------+---------------+---------------+-----------------+
1382 |3 |24 |15 |0.969650 |
1383 +---------------+---------------+---------------+-----------------+
1384 |4 |32 |0 |1.000000 |
1385 +---------------+---------------+---------------+-----------------+
1386 |6 |48 |31 |0.969650 |
1387 +---------------+---------------+---------------+-----------------+
1388 |7 |56 |47 |1.142857 |
1389 +---------------+---------------+---------------+-----------------+
1390 |8 |64 |0 |1.000000 |
1391 +---------------+---------------+---------------+-----------------+
1392 |9 |72 |63 |1.185115 |
1393 +---------------+---------------+---------------+-----------------+
1394 |10 |80 |63 |1.066553 |
1395 +---------------+---------------+---------------+-----------------+
1396 |11 |88 |79 |1.454545 |
1397 +---------------+---------------+---------------+-----------------+
1398 |12 |96 |0 |1.000000 |
1399 +---------------+---------------+---------------+-----------------+
1400 |13 |104 |95 |1.230769 |
1401 +---------------+---------------+---------------+-----------------+
1402 |14 |112 |95 |1.142857 |
1403 +---------------+---------------+---------------+-----------------+
1404 |15 |120 |95 |1.066667 |
1405 +---------------+---------------+---------------+-----------------+
1406 |16 |128 |0 |1.000000 |
1407 +---------------+---------------+---------------+-----------------+
1408 |17 |136 |127 |1.254863 |
1409 +---------------+---------------+---------------+-----------------+
1410 |18 |144 |127 |1.185255 |
1411 +---------------+---------------+---------------+-----------------+
1412 |19 |152 |0 |1.000000 |
1413 +---------------+---------------+---------------+-----------------+
1414 |20 |160 |127 |1.066667 |
1415 +---------------+---------------+---------------+-----------------+
1416 |21 |168 |0 |1.000000 |
1417 +---------------+---------------+---------------+-----------------+
1418 |22 |176 |159 |1.454334 |
1419 +---------------+---------------+---------------+-----------------+
1420 |23 |184 |0 |1.000000 |
1421 +---------------+---------------+---------------+-----------------+
1422 |24 |192 |127 |0.969744 |
1423 +---------------+---------------+---------------+-----------------+
1424 |25 |200 |191 |1.280246 |
1425 +---------------+---------------+---------------+-----------------+
1426 |26 |208 |191 |1.230921 |
1427 +---------------+---------------+---------------+-----------------+
1428 |27 |216 |0 |1.000000 |
1429 +---------------+---------------+---------------+-----------------+
1430 |28 |224 |191 |1.143118 |
1431 +---------------+---------------+---------------+-----------------+
1433 If rmid > rmid threshold, MBM total and local values should be multiplied
1434 by the correction factor.
1438 1. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update:
1439 http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html
1441 2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update:
1442 http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf
1444 3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual:
1445 https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html
1447 for further information.