Commit | Line | Data |
---|---|---|
1cd7af50 CD |
1 | .. SPDX-License-Identifier: GPL-2.0 |
2 | .. include:: <isonum.txt> | |
3 | ||
4 | =========================================== | |
a6f771c9 | 5 | User Interface for Resource Control feature |
1cd7af50 | 6 | =========================================== |
a6f771c9 | 7 | |
1cd7af50 CD |
8 | :Copyright: |copy| 2016 Intel Corporation |
9 | :Authors: - Fenghua Yu <fenghua.yu@intel.com> | |
10 | - Tony Luck <tony.luck@intel.com> | |
11 | - Vikas Shivappa <vikas.shivappa@intel.com> | |
f20e5789 | 12 | |
f20e5789 | 13 | |
1cd7af50 CD |
14 | Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). |
15 | AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). | |
f20e5789 | 16 | |
e6d42931 | 17 | This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo |
a6f771c9 | 18 | flag bits: |
f20e5789 | 19 | |
0a363fb2 | 20 | =============================================== ================================ |
1cd7af50 CD |
21 | RDT (Resource Director Technology) Allocation "rdt_a" |
22 | CAT (Cache Allocation Technology) "cat_l3", "cat_l2" | |
23 | CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2" | |
24 | CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc" | |
25 | MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" | |
26 | MBA (Memory Bandwidth Allocation) "mba" | |
0a363fb2 BM |
27 | SMBA (Slow Memory Bandwidth Allocation) "" |
28 | BMEC (Bandwidth Monitoring Event Configuration) "" | |
29 | =============================================== ================================ | |
30 | ||
31 | Historically, new features were made visible by default in /proc/cpuinfo. This | |
32 | resulted in the feature flags becoming hard to parse by humans. Adding a new | |
33 | flag to /proc/cpuinfo should be avoided if user space can obtain information | |
34 | about the feature from resctrl's info directory. | |
1cd7af50 CD |
35 | |
36 | To use the feature mount the file system:: | |
f20e5789 | 37 | |
cb07d71f | 38 | # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl |
f20e5789 FY |
39 | |
40 | mount options are: | |
41 | ||
1cd7af50 CD |
42 | "cdp": |
43 | Enable code/data prioritization in L3 cache allocations. | |
44 | "cdpl2": | |
45 | Enable code/data prioritization in L2 cache allocations. | |
46 | "mba_MBps": | |
47 | Enable the MBA Software Controller(mba_sc) to specify MBA | |
a8ed59a3 | 48 | bandwidth in MiBps |
cb07d71f BM |
49 | "debug": |
50 | Make debug files accessible. Available debug files are annotated with | |
51 | "Available only with debug option". | |
aa55d5a4 | 52 | |
57794aab | 53 | L2 and L3 CDP are controlled separately. |
f20e5789 | 54 | |
1640ae94 | 55 | RDT features are orthogonal. A particular system may support only |
e17e7330 RC |
56 | monitoring, only control, or both monitoring and control. Cache |
57 | pseudo-locking is a unique way of using cache control to "pin" or | |
58 | "lock" data in the cache. Details can be found in | |
59 | "Cache Pseudo-Locking". | |
60 | ||
1640ae94 VS |
61 | |
62 | The mount succeeds if either of allocation or monitoring is present, but | |
63 | only those files and directories supported by the system will be created. | |
64 | For more details on the behavior of the interface during monitoring | |
65 | and allocation, see the "Resource alloc and monitor groups" section. | |
f20e5789 | 66 | |
458b0d6e | 67 | Info directory |
1cd7af50 | 68 | ============== |
458b0d6e TG |
69 | |
70 | The 'info' directory contains information about the enabled | |
71 | resources. Each resource has its own subdirectory. The subdirectory | |
a9cad3d4 | 72 | names reflect the resource names. |
1640ae94 VS |
73 | |
74 | Each subdirectory contains the following files with respect to | |
75 | allocation: | |
76 | ||
77 | Cache resource(L3/L2) subdirectory contains the following files | |
78 | related to allocation: | |
458b0d6e | 79 | |
1cd7af50 CD |
80 | "num_closids": |
81 | The number of CLOSIDs which are valid for this | |
82 | resource. The kernel uses the smallest number of | |
83 | CLOSIDs of all enabled resources as limit. | |
84 | "cbm_mask": | |
85 | The bitmask which is valid for this resource. | |
86 | This mask is equivalent to 100%. | |
87 | "min_cbm_bits": | |
88 | The minimum number of consecutive bits which | |
89 | must be set when writing a mask. | |
90 | ||
91 | "shareable_bits": | |
92 | Bitmask of shareable resource with other executing | |
93 | entities (e.g. I/O). User can use this when | |
94 | setting up exclusive cache partitions. Note that | |
95 | some platforms support devices that have their | |
96 | own settings for cache use which can over-ride | |
97 | these bits. | |
98 | "bit_usage": | |
99 | Annotated capacity bitmasks showing how all | |
100 | instances of the resource are used. The legend is: | |
101 | ||
102 | "0": | |
103 | Corresponding region is unused. When the system's | |
cba1aab8 RC |
104 | resources have been allocated and a "0" is found |
105 | in "bit_usage" it is a sign that resources are | |
106 | wasted. | |
1cd7af50 CD |
107 | |
108 | "H": | |
109 | Corresponding region is used by hardware only | |
cba1aab8 RC |
110 | but available for software use. If a resource |
111 | has bits set in "shareable_bits" but not all | |
112 | of these bits appear in the resource groups' | |
113 | schematas then the bits appearing in | |
114 | "shareable_bits" but no resource group will | |
115 | be marked as "H". | |
1cd7af50 CD |
116 | "X": |
117 | Corresponding region is available for sharing and | |
cba1aab8 RC |
118 | used by hardware and software. These are the |
119 | bits that appear in "shareable_bits" as | |
120 | well as a resource group's allocation. | |
1cd7af50 CD |
121 | "S": |
122 | Corresponding region is used by software | |
cba1aab8 | 123 | and available for sharing. |
1cd7af50 CD |
124 | "E": |
125 | Corresponding region is used exclusively by | |
cba1aab8 | 126 | one resource group. No sharing allowed. |
1cd7af50 CD |
127 | "P": |
128 | Corresponding region is pseudo-locked. No | |
e17e7330 | 129 | sharing allowed. |
aaa5fa35 FY |
130 | "sparse_masks": |
131 | Indicates if non-contiguous 1s value in CBM is supported. | |
132 | ||
133 | "0": | |
134 | Only contiguous 1s value in CBM is supported. | |
135 | "1": | |
136 | Non-contiguous 1s value in CBM is supported. | |
0dd2d749 | 137 | |
57794aab | 138 | Memory bandwidth(MB) subdirectory contains the following files |
1640ae94 | 139 | with respect to allocation: |
a9cad3d4 | 140 | |
1cd7af50 CD |
141 | "min_bandwidth": |
142 | The minimum memory bandwidth percentage which | |
143 | user can request. | |
a9cad3d4 | 144 | |
1cd7af50 CD |
145 | "bandwidth_gran": |
146 | The granularity in which the memory bandwidth | |
147 | percentage is allocated. The allocated | |
148 | b/w percentage is rounded off to the next | |
149 | control step available on the hardware. The | |
150 | available bandwidth control steps are: | |
151 | min_bandwidth + N * bandwidth_gran. | |
a9cad3d4 | 152 | |
1cd7af50 CD |
153 | "delay_linear": |
154 | Indicates if the delay scale is linear or | |
155 | non-linear. This field is purely informational | |
156 | only. | |
458b0d6e | 157 | |
29b6bd41 FY |
158 | "thread_throttle_mode": |
159 | Indicator on Intel systems of how tasks running on threads | |
160 | of a physical core are throttled in cases where they | |
161 | request different memory bandwidth percentages: | |
162 | ||
163 | "max": | |
164 | the smallest percentage is applied | |
165 | to all threads | |
166 | "per-thread": | |
167 | bandwidth percentages are directly applied to | |
168 | the threads running on the core | |
169 | ||
1640ae94 VS |
170 | If RDT monitoring is available there will be an "L3_MON" directory |
171 | with the following files: | |
172 | ||
1cd7af50 CD |
173 | "num_rmids": |
174 | The number of RMIDs available. This is the | |
175 | upper bound for how many "CTRL_MON" + "MON" | |
176 | groups can be created. | |
1640ae94 | 177 | |
1cd7af50 CD |
178 | "mon_features": |
179 | Lists the monitoring events if | |
180 | monitoring is enabled for the resource. | |
0a363fb2 BM |
181 | Example:: |
182 | ||
183 | # cat /sys/fs/resctrl/info/L3_MON/mon_features | |
184 | llc_occupancy | |
185 | mbm_total_bytes | |
186 | mbm_local_bytes | |
187 | ||
188 | If the system supports Bandwidth Monitoring Event | |
189 | Configuration (BMEC), then the bandwidth events will | |
190 | be configurable. The output will be:: | |
191 | ||
192 | # cat /sys/fs/resctrl/info/L3_MON/mon_features | |
193 | llc_occupancy | |
194 | mbm_total_bytes | |
195 | mbm_total_bytes_config | |
196 | mbm_local_bytes | |
197 | mbm_local_bytes_config | |
198 | ||
199 | "mbm_total_bytes_config", "mbm_local_bytes_config": | |
200 | Read/write files containing the configuration for the mbm_total_bytes | |
201 | and mbm_local_bytes events, respectively, when the Bandwidth | |
202 | Monitoring Event Configuration (BMEC) feature is supported. | |
203 | The event configuration settings are domain specific and affect | |
204 | all the CPUs in the domain. When either event configuration is | |
205 | changed, the bandwidth counters for all RMIDs of both events | |
206 | (mbm_total_bytes as well as mbm_local_bytes) are cleared for that | |
207 | domain. The next read for every RMID will report "Unavailable" | |
208 | and subsequent reads will report the valid value. | |
209 | ||
210 | Following are the types of events supported: | |
211 | ||
212 | ==== ======================================================== | |
213 | Bits Description | |
214 | ==== ======================================================== | |
215 | 6 Dirty Victims from the QOS domain to all types of memory | |
216 | 5 Reads to slow memory in the non-local NUMA domain | |
217 | 4 Reads to slow memory in the local NUMA domain | |
218 | 3 Non-temporal writes to non-local NUMA domain | |
219 | 2 Non-temporal writes to local NUMA domain | |
220 | 1 Reads to memory in the non-local NUMA domain | |
221 | 0 Reads to memory in the local NUMA domain | |
222 | ==== ======================================================== | |
223 | ||
224 | By default, the mbm_total_bytes configuration is set to 0x7f to count | |
225 | all the event types and the mbm_local_bytes configuration is set to | |
226 | 0x15 to count all the local memory events. | |
227 | ||
228 | Examples: | |
229 | ||
230 | * To view the current configuration:: | |
231 | :: | |
232 | ||
233 | # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config | |
234 | 0=0x7f;1=0x7f;2=0x7f;3=0x7f | |
235 | ||
236 | # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config | |
237 | 0=0x15;1=0x15;3=0x15;4=0x15 | |
238 | ||
239 | * To change the mbm_total_bytes to count only reads on domain 0, | |
240 | the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary | |
241 | (in hexadecimal 0x33): | |
242 | :: | |
243 | ||
244 | # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config | |
245 | ||
246 | # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config | |
247 | 0=0x33;1=0x7f;2=0x7f;3=0x7f | |
248 | ||
249 | * To change the mbm_local_bytes to count all the slow memory reads on | |
250 | domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b | |
251 | in binary (in hexadecimal 0x30): | |
252 | :: | |
253 | ||
254 | # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config | |
255 | ||
256 | # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config | |
257 | 0=0x30;1=0x30;3=0x15;4=0x15 | |
1640ae94 VS |
258 | |
259 | "max_threshold_occupancy": | |
1cd7af50 CD |
260 | Read/write file provides the largest value (in |
261 | bytes) at which a previously used LLC_occupancy | |
262 | counter can be considered for re-use. | |
1640ae94 | 263 | |
165d3ad8 TL |
264 | Finally, in the top level of the "info" directory there is a file |
265 | named "last_cmd_status". This is reset with every "command" issued | |
266 | via the file system (making new directories or writing to any of the | |
267 | control files). If the command was successful, it will read as "ok". | |
268 | If the command failed, it will provide more information that can be | |
269 | conveyed in the error returns from file operations. E.g. | |
1cd7af50 | 270 | :: |
165d3ad8 TL |
271 | |
272 | # echo L3:0=f7 > schemata | |
273 | bash: echo: write error: Invalid argument | |
274 | # cat info/last_cmd_status | |
275 | mask f7 has non-consecutive 1-bits | |
1640ae94 VS |
276 | |
277 | Resource alloc and monitor groups | |
1cd7af50 | 278 | ================================= |
1640ae94 | 279 | |
f20e5789 | 280 | Resource groups are represented as directories in the resctrl file |
1640ae94 VS |
281 | system. The default group is the root directory which, immediately |
282 | after mounting, owns all the tasks and cpus in the system and can make | |
283 | full use of all resources. | |
284 | ||
285 | On a system with RDT control features additional directories can be | |
286 | created in the root directory that specify different amounts of each | |
287 | resource (see "schemata" below). The root and these additional top level | |
288 | directories are referred to as "CTRL_MON" groups below. | |
289 | ||
290 | On a system with RDT monitoring the root directory and other top level | |
291 | directories contain a directory named "mon_groups" in which additional | |
292 | directories can be created to monitor subsets of tasks in the CTRL_MON | |
293 | group that is their ancestor. These are called "MON" groups in the rest | |
294 | of this document. | |
295 | ||
296 | Removing a directory will move all tasks and cpus owned by the group it | |
297 | represents to the parent. Removing one of the created CTRL_MON groups | |
298 | will automatically remove all MON groups below it. | |
299 | ||
e0a6ede2 PN |
300 | Moving MON group directories to a new parent CTRL_MON group is supported |
301 | for the purpose of changing the resource allocations of a MON group | |
302 | without impacting its monitoring data or assigned tasks. This operation | |
303 | is not allowed for MON groups which monitor CPUs. No other move | |
304 | operation is currently allowed other than simply renaming a CTRL_MON or | |
305 | MON group. | |
306 | ||
1640ae94 VS |
307 | All groups contain the following files: |
308 | ||
309 | "tasks": | |
310 | Reading this file shows the list of all tasks that belong to | |
311 | this group. Writing a task id to the file will add a task to the | |
fe2a20ea BM |
312 | group. Multiple tasks can be added by separating the task ids |
313 | with commas. Tasks will be assigned sequentially. Multiple | |
314 | failures are not supported. A single failure encountered while | |
315 | attempting to assign a task will cause the operation to abort and | |
316 | already added tasks before the failure will remain in the group. | |
317 | Failures will be logged to /sys/fs/resctrl/info/last_cmd_status. | |
318 | ||
319 | If the group is a CTRL_MON group the task is removed from | |
1640ae94 VS |
320 | whichever previous CTRL_MON group owned the task and also from |
321 | any MON group that owned the task. If the group is a MON group, | |
322 | then the task must already belong to the CTRL_MON parent of this | |
323 | group. The task is removed from any previous MON group. | |
324 | ||
325 | ||
326 | "cpus": | |
327 | Reading this file shows a bitmask of the logical CPUs owned by | |
328 | this group. Writing a mask to this file will add and remove | |
329 | CPUs to/from this group. As with the tasks file a hierarchy is | |
330 | maintained where MON groups may only include CPUs owned by the | |
331 | parent CTRL_MON group. | |
57794aab | 332 | When the resource group is in pseudo-locked mode this file will |
33dc3e41 RC |
333 | only be readable, reflecting the CPUs associated with the |
334 | pseudo-locked region. | |
1640ae94 VS |
335 | |
336 | ||
337 | "cpus_list": | |
338 | Just like "cpus", only using ranges of CPUs instead of bitmasks. | |
f20e5789 | 339 | |
f20e5789 | 340 | |
1640ae94 | 341 | When control is enabled all CTRL_MON groups will also contain: |
f20e5789 | 342 | |
1640ae94 VS |
343 | "schemata": |
344 | A list of all the resources available to this group. | |
345 | Each resource has its own line and format - see below for details. | |
f20e5789 | 346 | |
cba1aab8 RC |
347 | "size": |
348 | Mirrors the display of the "schemata" file to display the size in | |
349 | bytes of each allocation instead of the bits representing the | |
350 | allocation. | |
351 | ||
352 | "mode": | |
353 | The "mode" of the resource group dictates the sharing of its | |
354 | allocations. A "shareable" resource group allows sharing of its | |
e17e7330 RC |
355 | allocations while an "exclusive" resource group does not. A |
356 | cache pseudo-locked region is created by first writing | |
357 | "pseudo-locksetup" to the "mode" file before writing the cache | |
358 | pseudo-locked region's schemata to the resource group's "schemata" | |
359 | file. On successful pseudo-locked region creation the mode will | |
360 | automatically change to "pseudo-locked". | |
cba1aab8 | 361 | |
ca8dad22 BM |
362 | "ctrl_hw_id": |
363 | Available only with debug option. The identifier used by hardware | |
364 | for the control group. On x86 this is the CLOSID. | |
365 | ||
1640ae94 | 366 | When monitoring is enabled all MON groups will also contain: |
4ffa3c97 | 367 | |
1640ae94 VS |
368 | "mon_data": |
369 | This contains a set of files organized by L3 domain and by | |
370 | RDT event. E.g. on a system with two L3 domains there will | |
371 | be subdirectories "mon_L3_00" and "mon_L3_01". Each of these | |
372 | directories have one file per event (e.g. "llc_occupancy", | |
373 | "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these | |
374 | files provide a read out of the current value of the event for | |
375 | all tasks in the group. In CTRL_MON groups these files provide | |
376 | the sum for all tasks in the CTRL_MON group and all tasks in | |
377 | MON groups. Please see example section for more details on usage. | |
f20e5789 | 378 | |
4cee14bc BM |
379 | "mon_hw_id": |
380 | Available only with debug option. The identifier used by hardware | |
381 | for the monitor group. On x86 this is the RMID. | |
382 | ||
1640ae94 VS |
383 | Resource allocation rules |
384 | ------------------------- | |
1cd7af50 | 385 | |
1640ae94 VS |
386 | When a task is running the following rules define which resources are |
387 | available to it: | |
f20e5789 FY |
388 | |
389 | 1) If the task is a member of a non-default group, then the schemata | |
1640ae94 | 390 | for that group is used. |
f20e5789 FY |
391 | |
392 | 2) Else if the task belongs to the default group, but is running on a | |
1640ae94 VS |
393 | CPU that is assigned to some specific group, then the schemata for the |
394 | CPU's group is used. | |
f20e5789 FY |
395 | |
396 | 3) Otherwise the schemata for the default group is used. | |
397 | ||
1640ae94 VS |
398 | Resource monitoring rules |
399 | ------------------------- | |
400 | 1) If a task is a member of a MON group, or non-default CTRL_MON group | |
401 | then RDT events for the task will be reported in that group. | |
402 | ||
403 | 2) If a task is a member of the default CTRL_MON group, but is running | |
404 | on a CPU that is assigned to some specific group, then the RDT events | |
405 | for the task will be reported in that group. | |
406 | ||
407 | 3) Otherwise RDT events for the task will be reported in the root level | |
408 | "mon_data" group. | |
409 | ||
410 | ||
411 | Notes on cache occupancy monitoring and control | |
1cd7af50 | 412 | =============================================== |
1640ae94 VS |
413 | When moving a task from one group to another you should remember that |
414 | this only affects *new* cache allocations by the task. E.g. you may have | |
415 | a task in a monitor group showing 3 MB of cache occupancy. If you move | |
416 | to a new group and immediately check the occupancy of the old and new | |
417 | groups you will likely see that the old group is still showing 3 MB and | |
418 | the new group zero. When the task accesses locations still in cache from | |
419 | before the move, the h/w does not update any counters. On a busy system | |
420 | you will likely see the occupancy in the old group go down as cache lines | |
421 | are evicted and re-used while the occupancy in the new group rises as | |
422 | the task accesses memory and loads into the cache are counted based on | |
423 | membership in the new group. | |
424 | ||
425 | The same applies to cache allocation control. Moving a task to a group | |
426 | with a smaller cache partition will not evict any cache lines. The | |
427 | process may continue to use them from the old partition. | |
428 | ||
429 | Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID) | |
430 | to identify a control group and a monitoring group respectively. Each of | |
431 | the resource groups are mapped to these IDs based on the kind of group. The | |
432 | number of CLOSid and RMID are limited by the hardware and hence the creation of | |
433 | a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID | |
434 | and creation of "MON" group may fail if we run out of RMIDs. | |
435 | ||
436 | max_threshold_occupancy - generic concepts | |
437 | ------------------------------------------ | |
438 | ||
439 | Note that an RMID once freed may not be immediately available for use as | |
440 | the RMID is still tagged the cache lines of the previous user of RMID. | |
441 | Hence such RMIDs are placed on limbo list and checked back if the cache | |
442 | occupancy has gone down. If there is a time when system has a lot of | |
443 | limbo RMIDs but which are not ready to be used, user may see an -EBUSY | |
444 | during mkdir. | |
445 | ||
446 | max_threshold_occupancy is a user configurable value to determine the | |
447 | occupancy at which an RMID can be freed. | |
f20e5789 FY |
448 | |
449 | Schemata files - general concepts | |
450 | --------------------------------- | |
451 | Each line in the file describes one resource. The line starts with | |
452 | the name of the resource, followed by specific values to be applied | |
453 | in each of the instances of that resource on the system. | |
454 | ||
455 | Cache IDs | |
456 | --------- | |
457 | On current generation systems there is one L3 cache per socket and L2 | |
458 | caches are generally just shared by the hyperthreads on a core, but this | |
459 | isn't an architectural requirement. We could have multiple separate L3 | |
460 | caches on a socket, multiple cores could share an L2 cache. So instead | |
461 | of using "socket" or "core" to define the set of logical cpus sharing | |
462 | a resource we use a "Cache ID". At a given cache level this will be a | |
463 | unique number across the whole system (but it isn't guaranteed to be a | |
464 | contiguous sequence, there may be gaps). To find the ID for each logical | |
465 | CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id | |
466 | ||
467 | Cache Bit Masks (CBM) | |
468 | --------------------- | |
469 | For cache resources we describe the portion of the cache that is available | |
470 | for allocation using a bitmask. The maximum value of the mask is defined | |
471 | by each cpu model (and may be different for different cache levels). It | |
472 | is found using CPUID, but is also provided in the "info" directory of | |
aaa5fa35 | 473 | the resctrl file system in "info/{resource}/cbm_mask". Some Intel hardware |
f20e5789 FY |
474 | requires that these masks have all the '1' bits in a contiguous block. So |
475 | 0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9 | |
aaa5fa35 FY |
476 | and 0xA are not. Check /sys/fs/resctrl/info/{resource}/sparse_masks |
477 | if non-contiguous 1s value is supported. On a system with a 20-bit mask | |
478 | each bit represents 5% of the capacity of the cache. You could partition | |
479 | the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. | |
f20e5789 | 480 | |
d6c64a4f | 481 | Memory bandwidth Allocation and monitoring |
1cd7af50 | 482 | ========================================== |
d6c64a4f VS |
483 | |
484 | For Memory bandwidth resource, by default the user controls the resource | |
485 | by indicating the percentage of total memory bandwidth. | |
a9cad3d4 VS |
486 | |
487 | The minimum bandwidth percentage value for each cpu model is predefined | |
488 | and can be looked up through "info/MB/min_bandwidth". The bandwidth | |
489 | granularity that is allocated is also dependent on the cpu model and can | |
490 | be looked up at "info/MB/bandwidth_gran". The available bandwidth | |
491 | control steps are: min_bw + N * bw_gran. Intermediate values are rounded | |
492 | to the next control step available on the hardware. | |
493 | ||
494 | The bandwidth throttling is a core specific mechanism on some of Intel | |
495 | SKUs. Using a high bandwidth and a low bandwidth setting on two threads | |
29b6bd41 FY |
496 | sharing a core may result in both threads being throttled to use the |
497 | low bandwidth (see "thread_throttle_mode"). | |
498 | ||
499 | The fact that Memory bandwidth allocation(MBA) may be a core | |
d6c64a4f VS |
500 | specific mechanism where as memory bandwidth monitoring(MBM) is done at |
501 | the package level may lead to confusion when users try to apply control | |
502 | via the MBA and then monitor the bandwidth to see if the controls are | |
503 | effective. Below are such scenarios: | |
504 | ||
505 | 1. User may *not* see increase in actual bandwidth when percentage | |
506 | values are increased: | |
507 | ||
508 | This can occur when aggregate L2 external bandwidth is more than L3 | |
509 | external bandwidth. Consider an SKL SKU with 24 cores on a package and | |
510 | where L2 external is 10GBps (hence aggregate L2 external bandwidth is | |
511 | 240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20 | |
512 | threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3 | |
513 | bandwidth of 100GBps although the percentage value specified is only 50% | |
57794aab | 514 | << 100%. Hence increasing the bandwidth percentage will not yield any |
d6c64a4f VS |
515 | more bandwidth. This is because although the L2 external bandwidth still |
516 | has capacity, the L3 external bandwidth is fully used. Also note that | |
517 | this would be dependent on number of cores the benchmark is run on. | |
518 | ||
519 | 2. Same bandwidth percentage may mean different actual bandwidth | |
520 | depending on # of threads: | |
521 | ||
522 | For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4 | |
523 | thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although | |
524 | they have same percentage bandwidth of 10%. This is simply because as | |
525 | threads start using more cores in an rdtgroup, the actual bandwidth may | |
526 | increase or vary although user specified bandwidth percentage is same. | |
527 | ||
528 | In order to mitigate this and make the interface more user friendly, | |
a8ed59a3 | 529 | resctrl added support for specifying the bandwidth in MiBps as well. The |
d6c64a4f VS |
530 | kernel underneath would use a software feedback mechanism or a "Software |
531 | Controller(mba_sc)" which reads the actual bandwidth using MBM counters | |
57794aab | 532 | and adjust the memory bandwidth percentages to ensure:: |
d6c64a4f VS |
533 | |
534 | "actual bandwidth < user specified bandwidth". | |
535 | ||
536 | By default, the schemata would take the bandwidth percentage values | |
537 | where as user can switch to the "MBA software controller" mode using | |
538 | a mount option 'mba_MBps'. The schemata format is specified in the below | |
539 | sections. | |
f20e5789 | 540 | |
1640ae94 VS |
541 | L3 schemata file details (code and data prioritization disabled) |
542 | ---------------------------------------------------------------- | |
1cd7af50 | 543 | With CDP disabled the L3 schemata format is:: |
f20e5789 FY |
544 | |
545 | L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... | |
546 | ||
1640ae94 VS |
547 | L3 schemata file details (CDP enabled via mount option to resctrl) |
548 | ------------------------------------------------------------------ | |
f20e5789 | 549 | When CDP is enabled L3 control is split into two separate resources |
1cd7af50 | 550 | so you can specify independent masks for code and data like this:: |
f20e5789 | 551 | |
7c7a4995 JM |
552 | L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... |
553 | L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... | |
f20e5789 | 554 | |
1640ae94 VS |
555 | L2 schemata file details |
556 | ------------------------ | |
7c7a4995 JM |
557 | CDP is supported at L2 using the 'cdpl2' mount option. The schemata |
558 | format is either:: | |
f20e5789 FY |
559 | |
560 | L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... | |
561 | ||
7c7a4995 JM |
562 | or |
563 | ||
564 | L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... | |
565 | L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... | |
566 | ||
567 | ||
d6c64a4f VS |
568 | Memory bandwidth Allocation (default mode) |
569 | ------------------------------------------ | |
a9cad3d4 VS |
570 | |
571 | Memory b/w domain is L3 cache. | |
1cd7af50 | 572 | :: |
a9cad3d4 VS |
573 | |
574 | MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... | |
575 | ||
a8ed59a3 | 576 | Memory bandwidth Allocation specified in MiBps |
91491e5f | 577 | ---------------------------------------------- |
d6c64a4f VS |
578 | |
579 | Memory bandwidth domain is L3 cache. | |
1cd7af50 | 580 | :: |
d6c64a4f | 581 | |
a8ed59a3 | 582 | MB:<cache_id0>=bw_MiBps0;<cache_id1>=bw_MiBps1;... |
d6c64a4f | 583 | |
0a363fb2 BM |
584 | Slow Memory Bandwidth Allocation (SMBA) |
585 | --------------------------------------- | |
586 | AMD hardware supports Slow Memory Bandwidth Allocation (SMBA). | |
587 | CXL.memory is the only supported "slow" memory device. With the | |
588 | support of SMBA, the hardware enables bandwidth allocation on | |
589 | the slow memory devices. If there are multiple such devices in | |
590 | the system, the throttling logic groups all the slow sources | |
591 | together and applies the limit on them as a whole. | |
592 | ||
593 | The presence of SMBA (with CXL.memory) is independent of slow memory | |
594 | devices presence. If there are no such devices on the system, then | |
595 | configuring SMBA will have no impact on the performance of the system. | |
596 | ||
597 | The bandwidth domain for slow memory is L3 cache. Its schemata file | |
598 | is formatted as: | |
599 | :: | |
600 | ||
601 | SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... | |
602 | ||
c4026b7b TL |
603 | Reading/writing the schemata file |
604 | --------------------------------- | |
605 | Reading the schemata file will show the state of all resources | |
606 | on all domains. When writing you only need to specify those values | |
607 | which you wish to change. E.g. | |
1cd7af50 | 608 | :: |
c4026b7b | 609 | |
1cd7af50 CD |
610 | # cat schemata |
611 | L3DATA:0=fffff;1=fffff;2=fffff;3=fffff | |
612 | L3CODE:0=fffff;1=fffff;2=fffff;3=fffff | |
613 | # echo "L3DATA:2=3c0;" > schemata | |
614 | # cat schemata | |
615 | L3DATA:0=fffff;1=fffff;2=3c0;3=fffff | |
616 | L3CODE:0=fffff;1=fffff;2=fffff;3=fffff | |
c4026b7b | 617 | |
0a363fb2 BM |
618 | Reading/writing the schemata file (on AMD systems) |
619 | -------------------------------------------------- | |
620 | Reading the schemata file will show the current bandwidth limit on all | |
621 | domains. The allocated resources are in multiples of one eighth GB/s. | |
622 | When writing to the file, you need to specify what cache id you wish to | |
623 | configure the bandwidth limit. | |
624 | ||
625 | For example, to allocate 2GB/s limit on the first cache id: | |
626 | ||
627 | :: | |
628 | ||
629 | # cat schemata | |
630 | MB:0=2048;1=2048;2=2048;3=2048 | |
631 | L3:0=ffff;1=ffff;2=ffff;3=ffff | |
632 | ||
633 | # echo "MB:1=16" > schemata | |
634 | # cat schemata | |
635 | MB:0=2048;1= 16;2=2048;3=2048 | |
636 | L3:0=ffff;1=ffff;2=ffff;3=ffff | |
637 | ||
638 | Reading/writing the schemata file (on AMD systems) with SMBA feature | |
639 | -------------------------------------------------------------------- | |
640 | Reading and writing the schemata file is the same as without SMBA in | |
641 | above section. | |
642 | ||
643 | For example, to allocate 8GB/s limit on the first cache id: | |
644 | ||
645 | :: | |
646 | ||
647 | # cat schemata | |
648 | SMBA:0=2048;1=2048;2=2048;3=2048 | |
649 | MB:0=2048;1=2048;2=2048;3=2048 | |
650 | L3:0=ffff;1=ffff;2=ffff;3=ffff | |
651 | ||
652 | # echo "SMBA:1=64" > schemata | |
653 | # cat schemata | |
654 | SMBA:0=2048;1= 64;2=2048;3=2048 | |
655 | MB:0=2048;1=2048;2=2048;3=2048 | |
656 | L3:0=ffff;1=ffff;2=ffff;3=ffff | |
657 | ||
e17e7330 | 658 | Cache Pseudo-Locking |
1cd7af50 | 659 | ==================== |
e17e7330 RC |
660 | CAT enables a user to specify the amount of cache space that an |
661 | application can fill. Cache pseudo-locking builds on the fact that a | |
662 | CPU can still read and write data pre-allocated outside its current | |
663 | allocated area on a cache hit. With cache pseudo-locking, data can be | |
664 | preloaded into a reserved portion of cache that no application can | |
665 | fill, and from that point on will only serve cache hits. The cache | |
666 | pseudo-locked memory is made accessible to user space where an | |
667 | application can map it into its virtual address space and thus have | |
668 | a region of memory with reduced average read latency. | |
669 | ||
670 | The creation of a cache pseudo-locked region is triggered by a request | |
671 | from the user to do so that is accompanied by a schemata of the region | |
672 | to be pseudo-locked. The cache pseudo-locked region is created as follows: | |
1cd7af50 | 673 | |
e17e7330 RC |
674 | - Create a CAT allocation CLOSNEW with a CBM matching the schemata |
675 | from the user of the cache region that will contain the pseudo-locked | |
676 | memory. This region must not overlap with any current CAT allocation/CLOS | |
677 | on the system and no future overlap with this cache region is allowed | |
678 | while the pseudo-locked region exists. | |
679 | - Create a contiguous region of memory of the same size as the cache | |
680 | region. | |
681 | - Flush the cache, disable hardware prefetchers, disable preemption. | |
682 | - Make CLOSNEW the active CLOS and touch the allocated memory to load | |
683 | it into the cache. | |
684 | - Set the previous CLOS as active. | |
685 | - At this point the closid CLOSNEW can be released - the cache | |
686 | pseudo-locked region is protected as long as its CBM does not appear in | |
687 | any CAT allocation. Even though the cache pseudo-locked region will from | |
688 | this point on not appear in any CBM of any CLOS an application running with | |
689 | any CLOS will be able to access the memory in the pseudo-locked region since | |
690 | the region continues to serve cache hits. | |
691 | - The contiguous region of memory loaded into the cache is exposed to | |
692 | user-space as a character device. | |
693 | ||
694 | Cache pseudo-locking increases the probability that data will remain | |
695 | in the cache via carefully configuring the CAT feature and controlling | |
696 | application behavior. There is no guarantee that data is placed in | |
697 | cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict | |
698 | “locked” data from cache. Power management C-states may shrink or | |
6fc0de37 RC |
699 | power off cache. Deeper C-states will automatically be restricted on |
700 | pseudo-locked region creation. | |
e17e7330 RC |
701 | |
702 | It is required that an application using a pseudo-locked region runs | |
703 | with affinity to the cores (or a subset of the cores) associated | |
704 | with the cache on which the pseudo-locked region resides. A sanity check | |
705 | within the code will not allow an application to map pseudo-locked memory | |
706 | unless it runs with affinity to cores associated with the cache on which the | |
707 | pseudo-locked region resides. The sanity check is only done during the | |
708 | initial mmap() handling, there is no enforcement afterwards and the | |
709 | application self needs to ensure it remains affine to the correct cores. | |
710 | ||
711 | Pseudo-locking is accomplished in two stages: | |
1cd7af50 | 712 | |
e17e7330 RC |
713 | 1) During the first stage the system administrator allocates a portion |
714 | of cache that should be dedicated to pseudo-locking. At this time an | |
715 | equivalent portion of memory is allocated, loaded into allocated | |
716 | cache portion, and exposed as a character device. | |
717 | 2) During the second stage a user-space application maps (mmap()) the | |
718 | pseudo-locked memory into its address space. | |
719 | ||
720 | Cache Pseudo-Locking Interface | |
721 | ------------------------------ | |
722 | A pseudo-locked region is created using the resctrl interface as follows: | |
723 | ||
724 | 1) Create a new resource group by creating a new directory in /sys/fs/resctrl. | |
725 | 2) Change the new resource group's mode to "pseudo-locksetup" by writing | |
726 | "pseudo-locksetup" to the "mode" file. | |
727 | 3) Write the schemata of the pseudo-locked region to the "schemata" file. All | |
728 | bits within the schemata should be "unused" according to the "bit_usage" | |
729 | file. | |
730 | ||
731 | On successful pseudo-locked region creation the "mode" file will contain | |
732 | "pseudo-locked" and a new character device with the same name as the resource | |
733 | group will exist in /dev/pseudo_lock. This character device can be mmap()'ed | |
734 | by user space in order to obtain access to the pseudo-locked memory region. | |
735 | ||
736 | An example of cache pseudo-locked region creation and usage can be found below. | |
737 | ||
738 | Cache Pseudo-Locking Debugging Interface | |
1cd7af50 | 739 | ---------------------------------------- |
e17e7330 RC |
740 | The pseudo-locking debugging interface is enabled by default (if |
741 | CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl. | |
742 | ||
743 | There is no explicit way for the kernel to test if a provided memory | |
744 | location is present in the cache. The pseudo-locking debugging interface uses | |
745 | the tracing infrastructure to provide two ways to measure cache residency of | |
746 | the pseudo-locked region: | |
1cd7af50 | 747 | |
e17e7330 RC |
748 | 1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data |
749 | from these measurements are best visualized using a hist trigger (see | |
750 | example below). In this test the pseudo-locked region is traversed at | |
751 | a stride of 32 bytes while hardware prefetchers and preemption | |
752 | are disabled. This also provides a substitute visualization of cache | |
753 | hits and misses. | |
754 | 2) Cache hit and miss measurements using model specific precision counters if | |
755 | available. Depending on the levels of cache on the system the pseudo_lock_l2 | |
756 | and pseudo_lock_l3 tracepoints are available. | |
e17e7330 RC |
757 | |
758 | When a pseudo-locked region is created a new debugfs directory is created for | |
759 | it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single | |
760 | write-only file, pseudo_lock_measure, is present in this directory. The | |
dd45407c RC |
761 | measurement of the pseudo-locked region depends on the number written to this |
762 | debugfs file: | |
1cd7af50 CD |
763 | |
764 | 1: | |
765 | writing "1" to the pseudo_lock_measure file will trigger the latency | |
dd45407c RC |
766 | measurement captured in the pseudo_lock_mem_latency tracepoint. See |
767 | example below. | |
1cd7af50 CD |
768 | 2: |
769 | writing "2" to the pseudo_lock_measure file will trigger the L2 cache | |
dd45407c RC |
770 | residency (cache hits and misses) measurement captured in the |
771 | pseudo_lock_l2 tracepoint. See example below. | |
1cd7af50 CD |
772 | 3: |
773 | writing "3" to the pseudo_lock_measure file will trigger the L3 cache | |
dd45407c RC |
774 | residency (cache hits and misses) measurement captured in the |
775 | pseudo_lock_l3 tracepoint. | |
776 | ||
777 | All measurements are recorded with the tracing infrastructure. This requires | |
778 | the relevant tracepoints to be enabled before the measurement is triggered. | |
e17e7330 | 779 | |
1cd7af50 CD |
780 | Example of latency debugging interface |
781 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
e17e7330 RC |
782 | In this example a pseudo-locked region named "newlock" was created. Here is |
783 | how we can measure the latency in cycles of reading from this region and | |
784 | visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS | |
1cd7af50 CD |
785 | is set:: |
786 | ||
2abfcd29 RZ |
787 | # :> /sys/kernel/tracing/trace |
788 | # echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger | |
789 | # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable | |
1cd7af50 | 790 | # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure |
2abfcd29 RZ |
791 | # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable |
792 | # cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist | |
1cd7af50 CD |
793 | |
794 | # event histogram | |
795 | # | |
796 | # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active] | |
797 | # | |
798 | ||
799 | { latency: 456 } hitcount: 1 | |
800 | { latency: 50 } hitcount: 83 | |
801 | { latency: 36 } hitcount: 96 | |
802 | { latency: 44 } hitcount: 174 | |
803 | { latency: 48 } hitcount: 195 | |
804 | { latency: 46 } hitcount: 262 | |
805 | { latency: 42 } hitcount: 693 | |
806 | { latency: 40 } hitcount: 3204 | |
807 | { latency: 38 } hitcount: 3484 | |
808 | ||
809 | Totals: | |
810 | Hits: 8192 | |
811 | Entries: 9 | |
812 | Dropped: 0 | |
813 | ||
814 | Example of cache hits/misses debugging | |
815 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
e17e7330 RC |
816 | In this example a pseudo-locked region named "newlock" was created on the L2 |
817 | cache of a platform. Here is how we can obtain details of the cache hits | |
818 | and misses using the platform's precision counters. | |
1cd7af50 | 819 | :: |
e17e7330 | 820 | |
2abfcd29 RZ |
821 | # :> /sys/kernel/tracing/trace |
822 | # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable | |
1cd7af50 | 823 | # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure |
2abfcd29 RZ |
824 | # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable |
825 | # cat /sys/kernel/tracing/trace | |
e17e7330 | 826 | |
1cd7af50 CD |
827 | # tracer: nop |
828 | # | |
829 | # _-----=> irqs-off | |
830 | # / _----=> need-resched | |
831 | # | / _---=> hardirq/softirq | |
832 | # || / _--=> preempt-depth | |
833 | # ||| / delay | |
834 | # TASK-PID CPU# |||| TIMESTAMP FUNCTION | |
835 | # | | | |||| | | | |
836 | pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0 | |
e17e7330 RC |
837 | |
838 | ||
1cd7af50 CD |
839 | Examples for RDT allocation usage |
840 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
841 | ||
842 | 1) Example 1 | |
1640ae94 | 843 | |
f20e5789 | 844 | On a two socket machine (one L3 cache per socket) with just four bits |
a9cad3d4 | 845 | for cache bit masks, minimum b/w of 10% with a memory bandwidth |
1cd7af50 CD |
846 | granularity of 10%. |
847 | :: | |
f20e5789 | 848 | |
1cd7af50 CD |
849 | # mount -t resctrl resctrl /sys/fs/resctrl |
850 | # cd /sys/fs/resctrl | |
851 | # mkdir p0 p1 | |
852 | # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata | |
853 | # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata | |
f20e5789 FY |
854 | |
855 | The default resource group is unmodified, so we have access to all parts | |
856 | of all caches (its schemata file reads "L3:0=f;1=f"). | |
857 | ||
858 | Tasks that are under the control of group "p0" may only allocate from the | |
859 | "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. | |
860 | Tasks in group "p1" use the "lower" 50% of cache on both sockets. | |
861 | ||
a9cad3d4 VS |
862 | Similarly, tasks that are under the control of group "p0" may use a |
863 | maximum memory b/w of 50% on socket0 and 50% on socket 1. | |
864 | Tasks in group "p1" may also use 50% memory b/w on both sockets. | |
865 | Note that unlike cache masks, memory b/w cannot specify whether these | |
866 | allocations can overlap or not. The allocations specifies the maximum | |
867 | b/w that the group may be able to use and the system admin can configure | |
868 | the b/w accordingly. | |
869 | ||
b5453a8e JM |
870 | If resctrl is using the software controller (mba_sc) then user can enter the |
871 | max b/w in MB rather than the percentage values. | |
1cd7af50 | 872 | :: |
d6c64a4f | 873 | |
1cd7af50 CD |
874 | # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata |
875 | # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata | |
d6c64a4f VS |
876 | |
877 | In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w | |
878 | of 1024MB where as on socket 1 they would use 500MB. | |
879 | ||
1cd7af50 CD |
880 | 2) Example 2 |
881 | ||
f20e5789 FY |
882 | Again two sockets, but this time with a more realistic 20-bit mask. |
883 | ||
884 | Two real time tasks pid=1234 running on processor 0 and pid=5678 running on | |
885 | processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy | |
886 | neighbors, each of the two real-time tasks exclusively occupies one quarter | |
887 | of L3 cache on socket 0. | |
1cd7af50 | 888 | :: |
f20e5789 | 889 | |
1cd7af50 CD |
890 | # mount -t resctrl resctrl /sys/fs/resctrl |
891 | # cd /sys/fs/resctrl | |
f20e5789 FY |
892 | |
893 | First we reset the schemata for the default group so that the "upper" | |
a9cad3d4 | 894 | 50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by |
1cd7af50 | 895 | ordinary tasks:: |
f20e5789 | 896 | |
1cd7af50 | 897 | # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata |
f20e5789 FY |
898 | |
899 | Next we make a resource group for our first real time task and give | |
900 | it access to the "top" 25% of the cache on socket 0. | |
1cd7af50 | 901 | :: |
f20e5789 | 902 | |
1cd7af50 CD |
903 | # mkdir p0 |
904 | # echo "L3:0=f8000;1=fffff" > p0/schemata | |
f20e5789 FY |
905 | |
906 | Finally we move our first real time task into this resource group. We | |
907 | also use taskset(1) to ensure the task always runs on a dedicated CPU | |
908 | on socket 0. Most uses of resource groups will also constrain which | |
909 | processors tasks run on. | |
1cd7af50 | 910 | :: |
f20e5789 | 911 | |
1cd7af50 CD |
912 | # echo 1234 > p0/tasks |
913 | # taskset -cp 1 1234 | |
f20e5789 | 914 | |
1cd7af50 | 915 | Ditto for the second real time task (with the remaining 25% of cache):: |
f20e5789 | 916 | |
1cd7af50 CD |
917 | # mkdir p1 |
918 | # echo "L3:0=7c00;1=fffff" > p1/schemata | |
919 | # echo 5678 > p1/tasks | |
920 | # taskset -cp 2 5678 | |
f20e5789 | 921 | |
a9cad3d4 VS |
922 | For the same 2 socket system with memory b/w resource and CAT L3 the |
923 | schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is | |
924 | 10): | |
925 | ||
1cd7af50 CD |
926 | For our first real time task this would request 20% memory b/w on socket 0. |
927 | :: | |
a9cad3d4 | 928 | |
1cd7af50 | 929 | # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata |
a9cad3d4 VS |
930 | |
931 | For our second real time task this would request an other 20% memory b/w | |
932 | on socket 0. | |
1cd7af50 | 933 | :: |
a9cad3d4 | 934 | |
1cd7af50 | 935 | # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata |
a9cad3d4 | 936 | |
1cd7af50 | 937 | 3) Example 3 |
f20e5789 FY |
938 | |
939 | A single socket system which has real-time tasks running on core 4-7 and | |
940 | non real-time workload assigned to core 0-3. The real-time tasks share text | |
941 | and data, so a per task association is not required and due to interaction | |
942 | with the kernel it's desired that the kernel on these cores shares L3 with | |
943 | the tasks. | |
1cd7af50 | 944 | :: |
f20e5789 | 945 | |
1cd7af50 CD |
946 | # mount -t resctrl resctrl /sys/fs/resctrl |
947 | # cd /sys/fs/resctrl | |
f20e5789 FY |
948 | |
949 | First we reset the schemata for the default group so that the "upper" | |
a9cad3d4 | 950 | 50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0 |
1cd7af50 | 951 | cannot be used by ordinary tasks:: |
f20e5789 | 952 | |
1cd7af50 | 953 | # echo "L3:0=3ff\nMB:0=50" > schemata |
f20e5789 | 954 | |
a9cad3d4 VS |
955 | Next we make a resource group for our real time cores and give it access |
956 | to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on | |
957 | socket 0. | |
1cd7af50 | 958 | :: |
f20e5789 | 959 | |
1cd7af50 CD |
960 | # mkdir p0 |
961 | # echo "L3:0=ffc00\nMB:0=50" > p0/schemata | |
f20e5789 FY |
962 | |
963 | Finally we move core 4-7 over to the new group and make sure that the | |
a9cad3d4 VS |
964 | kernel and the tasks running there get 50% of the cache. They should |
965 | also get 50% of memory bandwidth assuming that the cores 4-7 are SMT | |
966 | siblings and only the real time threads are scheduled on the cores 4-7. | |
1cd7af50 | 967 | :: |
f20e5789 | 968 | |
1cd7af50 | 969 | # echo F0 > p0/cpus |
3c2a769d | 970 | |
1cd7af50 | 971 | 4) Example 4 |
cba1aab8 RC |
972 | |
973 | The resource groups in previous examples were all in the default "shareable" | |
974 | mode allowing sharing of their cache allocations. If one resource group | |
975 | configures a cache allocation then nothing prevents another resource group | |
976 | to overlap with that allocation. | |
977 | ||
978 | In this example a new exclusive resource group will be created on a L2 CAT | |
979 | system with two L2 cache instances that can be configured with an 8-bit | |
980 | capacity bitmask. The new exclusive resource group will be configured to use | |
981 | 25% of each cache instance. | |
1cd7af50 | 982 | :: |
cba1aab8 | 983 | |
1cd7af50 CD |
984 | # mount -t resctrl resctrl /sys/fs/resctrl/ |
985 | # cd /sys/fs/resctrl | |
cba1aab8 RC |
986 | |
987 | First, we observe that the default group is configured to allocate to all L2 | |
1cd7af50 | 988 | cache:: |
cba1aab8 | 989 | |
1cd7af50 CD |
990 | # cat schemata |
991 | L2:0=ff;1=ff | |
cba1aab8 RC |
992 | |
993 | We could attempt to create the new resource group at this point, but it will | |
1cd7af50 CD |
994 | fail because of the overlap with the schemata of the default group:: |
995 | ||
996 | # mkdir p0 | |
997 | # echo 'L2:0=0x3;1=0x3' > p0/schemata | |
998 | # cat p0/mode | |
999 | shareable | |
1000 | # echo exclusive > p0/mode | |
1001 | -sh: echo: write error: Invalid argument | |
1002 | # cat info/last_cmd_status | |
1003 | schemata overlaps | |
cba1aab8 RC |
1004 | |
1005 | To ensure that there is no overlap with another resource group the default | |
1006 | resource group's schemata has to change, making it possible for the new | |
1007 | resource group to become exclusive. | |
1cd7af50 CD |
1008 | :: |
1009 | ||
1010 | # echo 'L2:0=0xfc;1=0xfc' > schemata | |
1011 | # echo exclusive > p0/mode | |
1012 | # grep . p0/* | |
1013 | p0/cpus:0 | |
1014 | p0/mode:exclusive | |
1015 | p0/schemata:L2:0=03;1=03 | |
1016 | p0/size:L2:0=262144;1=262144 | |
cba1aab8 RC |
1017 | |
1018 | A new resource group will on creation not overlap with an exclusive resource | |
1cd7af50 CD |
1019 | group:: |
1020 | ||
1021 | # mkdir p1 | |
1022 | # grep . p1/* | |
1023 | p1/cpus:0 | |
1024 | p1/mode:shareable | |
1025 | p1/schemata:L2:0=fc;1=fc | |
1026 | p1/size:L2:0=786432;1=786432 | |
1027 | ||
1028 | The bit_usage will reflect how the cache is used:: | |
1029 | ||
1030 | # cat info/L2/bit_usage | |
1031 | 0=SSSSSSEE;1=SSSSSSEE | |
1032 | ||
1033 | A resource group cannot be forced to overlap with an exclusive resource group:: | |
1034 | ||
1035 | # echo 'L2:0=0x1;1=0x1' > p1/schemata | |
1036 | -sh: echo: write error: Invalid argument | |
1037 | # cat info/last_cmd_status | |
1038 | overlaps with exclusive group | |
cba1aab8 | 1039 | |
e17e7330 | 1040 | Example of Cache Pseudo-Locking |
1cd7af50 | 1041 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
e17e7330 RC |
1042 | Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked |
1043 | region is exposed at /dev/pseudo_lock/newlock that can be provided to | |
1044 | application for argument to mmap(). | |
1cd7af50 | 1045 | :: |
e17e7330 | 1046 | |
1cd7af50 CD |
1047 | # mount -t resctrl resctrl /sys/fs/resctrl/ |
1048 | # cd /sys/fs/resctrl | |
e17e7330 RC |
1049 | |
1050 | Ensure that there are bits available that can be pseudo-locked, since only | |
1051 | unused bits can be pseudo-locked the bits to be pseudo-locked needs to be | |
1cd7af50 CD |
1052 | removed from the default resource group's schemata:: |
1053 | ||
1054 | # cat info/L2/bit_usage | |
1055 | 0=SSSSSSSS;1=SSSSSSSS | |
1056 | # echo 'L2:1=0xfc' > schemata | |
1057 | # cat info/L2/bit_usage | |
1058 | 0=SSSSSSSS;1=SSSSSS00 | |
e17e7330 RC |
1059 | |
1060 | Create a new resource group that will be associated with the pseudo-locked | |
1061 | region, indicate that it will be used for a pseudo-locked region, and | |
1cd7af50 | 1062 | configure the requested pseudo-locked region capacity bitmask:: |
e17e7330 | 1063 | |
1cd7af50 CD |
1064 | # mkdir newlock |
1065 | # echo pseudo-locksetup > newlock/mode | |
1066 | # echo 'L2:1=0x3' > newlock/schemata | |
e17e7330 RC |
1067 | |
1068 | On success the resource group's mode will change to pseudo-locked, the | |
1069 | bit_usage will reflect the pseudo-locked region, and the character device | |
1cd7af50 CD |
1070 | exposing the pseudo-locked region will exist:: |
1071 | ||
1072 | # cat newlock/mode | |
1073 | pseudo-locked | |
1074 | # cat info/L2/bit_usage | |
1075 | 0=SSSSSSSS;1=SSSSSSPP | |
1076 | # ls -l /dev/pseudo_lock/newlock | |
1077 | crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock | |
1078 | ||
1079 | :: | |
1080 | ||
1081 | /* | |
1082 | * Example code to access one page of pseudo-locked cache region | |
1083 | * from user space. | |
1084 | */ | |
1085 | #define _GNU_SOURCE | |
1086 | #include <fcntl.h> | |
1087 | #include <sched.h> | |
1088 | #include <stdio.h> | |
1089 | #include <stdlib.h> | |
1090 | #include <unistd.h> | |
1091 | #include <sys/mman.h> | |
1092 | ||
1093 | /* | |
1094 | * It is required that the application runs with affinity to only | |
1095 | * cores associated with the pseudo-locked region. Here the cpu | |
1096 | * is hardcoded for convenience of example. | |
1097 | */ | |
1098 | static int cpuid = 2; | |
1099 | ||
1100 | int main(int argc, char *argv[]) | |
1101 | { | |
1102 | cpu_set_t cpuset; | |
1103 | long page_size; | |
1104 | void *mapping; | |
1105 | int dev_fd; | |
1106 | int ret; | |
1107 | ||
1108 | page_size = sysconf(_SC_PAGESIZE); | |
1109 | ||
1110 | CPU_ZERO(&cpuset); | |
1111 | CPU_SET(cpuid, &cpuset); | |
1112 | ret = sched_setaffinity(0, sizeof(cpuset), &cpuset); | |
1113 | if (ret < 0) { | |
1114 | perror("sched_setaffinity"); | |
1115 | exit(EXIT_FAILURE); | |
1116 | } | |
1117 | ||
1118 | dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR); | |
1119 | if (dev_fd < 0) { | |
1120 | perror("open"); | |
1121 | exit(EXIT_FAILURE); | |
1122 | } | |
1123 | ||
1124 | mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, | |
1125 | dev_fd, 0); | |
1126 | if (mapping == MAP_FAILED) { | |
1127 | perror("mmap"); | |
1128 | close(dev_fd); | |
1129 | exit(EXIT_FAILURE); | |
1130 | } | |
1131 | ||
1132 | /* Application interacts with pseudo-locked memory @mapping */ | |
1133 | ||
1134 | ret = munmap(mapping, page_size); | |
1135 | if (ret < 0) { | |
1136 | perror("munmap"); | |
1137 | close(dev_fd); | |
1138 | exit(EXIT_FAILURE); | |
1139 | } | |
1140 | ||
1141 | close(dev_fd); | |
1142 | exit(EXIT_SUCCESS); | |
1143 | } | |
e17e7330 | 1144 | |
cba1aab8 RC |
1145 | Locking between applications |
1146 | ---------------------------- | |
3c2a769d MT |
1147 | |
1148 | Certain operations on the resctrl filesystem, composed of read/writes | |
1149 | to/from multiple files, must be atomic. | |
1150 | ||
1151 | As an example, the allocation of an exclusive reservation of L3 cache | |
1152 | involves: | |
1153 | ||
cba1aab8 | 1154 | 1. Read the cbmmasks from each directory or the per-resource "bit_usage" |
3c2a769d MT |
1155 | 2. Find a contiguous set of bits in the global CBM bitmask that is clear |
1156 | in any of the directory cbmmasks | |
1157 | 3. Create a new directory | |
1158 | 4. Set the bits found in step 2 to the new directory "schemata" file | |
1159 | ||
1160 | If two applications attempt to allocate space concurrently then they can | |
1161 | end up allocating the same bits so the reservations are shared instead of | |
1162 | exclusive. | |
1163 | ||
1164 | To coordinate atomic operations on the resctrlfs and to avoid the problem | |
1165 | above, the following locking procedure is recommended: | |
1166 | ||
1167 | Locking is based on flock, which is available in libc and also as a shell | |
1168 | script command | |
1169 | ||
1170 | Write lock: | |
1171 | ||
1172 | A) Take flock(LOCK_EX) on /sys/fs/resctrl | |
1173 | B) Read/write the directory structure. | |
1174 | C) funlock | |
1175 | ||
1176 | Read lock: | |
1177 | ||
1178 | A) Take flock(LOCK_SH) on /sys/fs/resctrl | |
1179 | B) If success read the directory structure. | |
1180 | C) funlock | |
1181 | ||
1cd7af50 CD |
1182 | Example with bash:: |
1183 | ||
1184 | # Atomically read directory structure | |
1185 | $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl | |
1186 | ||
1187 | # Read directory contents and create new subdirectory | |
1188 | ||
1189 | $ cat create-dir.sh | |
1190 | find /sys/fs/resctrl/ > output.txt | |
1191 | mask = function-of(output.txt) | |
1192 | mkdir /sys/fs/resctrl/newres/ | |
1193 | echo mask > /sys/fs/resctrl/newres/schemata | |
1194 | ||
1195 | $ flock /sys/fs/resctrl/ ./create-dir.sh | |
1196 | ||
1197 | Example with C:: | |
1198 | ||
1199 | /* | |
1200 | * Example code do take advisory locks | |
1201 | * before accessing resctrl filesystem | |
1202 | */ | |
1203 | #include <sys/file.h> | |
1204 | #include <stdlib.h> | |
1205 | ||
1206 | void resctrl_take_shared_lock(int fd) | |
1207 | { | |
1208 | int ret; | |
1209 | ||
1210 | /* take shared lock on resctrl filesystem */ | |
1211 | ret = flock(fd, LOCK_SH); | |
1212 | if (ret) { | |
1213 | perror("flock"); | |
1214 | exit(-1); | |
1215 | } | |
1216 | } | |
1217 | ||
1218 | void resctrl_take_exclusive_lock(int fd) | |
1219 | { | |
1220 | int ret; | |
1221 | ||
1222 | /* release lock on resctrl filesystem */ | |
1223 | ret = flock(fd, LOCK_EX); | |
1224 | if (ret) { | |
1225 | perror("flock"); | |
1226 | exit(-1); | |
1227 | } | |
1228 | } | |
1229 | ||
1230 | void resctrl_release_lock(int fd) | |
1231 | { | |
1232 | int ret; | |
1233 | ||
1234 | /* take shared lock on resctrl filesystem */ | |
1235 | ret = flock(fd, LOCK_UN); | |
1236 | if (ret) { | |
1237 | perror("flock"); | |
1238 | exit(-1); | |
1239 | } | |
1240 | } | |
1241 | ||
1242 | void main(void) | |
1243 | { | |
1244 | int fd, ret; | |
1245 | ||
1246 | fd = open("/sys/fs/resctrl", O_DIRECTORY); | |
1247 | if (fd == -1) { | |
1248 | perror("open"); | |
1249 | exit(-1); | |
1250 | } | |
1251 | resctrl_take_shared_lock(fd); | |
1252 | /* code to read directory contents */ | |
1253 | resctrl_release_lock(fd); | |
1254 | ||
1255 | resctrl_take_exclusive_lock(fd); | |
1256 | /* code to read and write directory contents */ | |
1257 | resctrl_release_lock(fd); | |
1258 | } | |
1259 | ||
1260 | Examples for RDT Monitoring along with allocation usage | |
1261 | ======================================================= | |
1640ae94 VS |
1262 | Reading monitored data |
1263 | ---------------------- | |
1264 | Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would | |
1265 | show the current snapshot of LLC occupancy of the corresponding MON | |
1266 | group or CTRL_MON group. | |
1267 | ||
1268 | ||
1269 | Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group) | |
1cd7af50 | 1270 | ------------------------------------------------------------------------ |
1640ae94 | 1271 | On a two socket machine (one L3 cache per socket) with just four bits |
1cd7af50 | 1272 | for cache bit masks:: |
1640ae94 | 1273 | |
1cd7af50 CD |
1274 | # mount -t resctrl resctrl /sys/fs/resctrl |
1275 | # cd /sys/fs/resctrl | |
1276 | # mkdir p0 p1 | |
1277 | # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata | |
1278 | # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata | |
1279 | # echo 5678 > p1/tasks | |
1280 | # echo 5679 > p1/tasks | |
1640ae94 VS |
1281 | |
1282 | The default resource group is unmodified, so we have access to all parts | |
1283 | of all caches (its schemata file reads "L3:0=f;1=f"). | |
1284 | ||
1285 | Tasks that are under the control of group "p0" may only allocate from the | |
1286 | "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. | |
1287 | Tasks in group "p1" use the "lower" 50% of cache on both sockets. | |
1288 | ||
1289 | Create monitor groups and assign a subset of tasks to each monitor group. | |
1cd7af50 | 1290 | :: |
1640ae94 | 1291 | |
1cd7af50 CD |
1292 | # cd /sys/fs/resctrl/p1/mon_groups |
1293 | # mkdir m11 m12 | |
1294 | # echo 5678 > m11/tasks | |
1295 | # echo 5679 > m12/tasks | |
1640ae94 VS |
1296 | |
1297 | fetch data (data shown in bytes) | |
1cd7af50 | 1298 | :: |
1640ae94 | 1299 | |
1cd7af50 CD |
1300 | # cat m11/mon_data/mon_L3_00/llc_occupancy |
1301 | 16234000 | |
1302 | # cat m11/mon_data/mon_L3_01/llc_occupancy | |
1303 | 14789000 | |
1304 | # cat m12/mon_data/mon_L3_00/llc_occupancy | |
1305 | 16789000 | |
1640ae94 VS |
1306 | |
1307 | The parent ctrl_mon group shows the aggregated data. | |
1cd7af50 | 1308 | :: |
1640ae94 | 1309 | |
1cd7af50 CD |
1310 | # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy |
1311 | 31234000 | |
1640ae94 VS |
1312 | |
1313 | Example 2 (Monitor a task from its creation) | |
1cd7af50 CD |
1314 | -------------------------------------------- |
1315 | On a two socket machine (one L3 cache per socket):: | |
1640ae94 | 1316 | |
1cd7af50 CD |
1317 | # mount -t resctrl resctrl /sys/fs/resctrl |
1318 | # cd /sys/fs/resctrl | |
1319 | # mkdir p0 p1 | |
1640ae94 VS |
1320 | |
1321 | An RMID is allocated to the group once its created and hence the <cmd> | |
1322 | below is monitored from its creation. | |
1cd7af50 | 1323 | :: |
1640ae94 | 1324 | |
1cd7af50 CD |
1325 | # echo $$ > /sys/fs/resctrl/p1/tasks |
1326 | # <cmd> | |
1640ae94 | 1327 | |
1cd7af50 | 1328 | Fetch the data:: |
1640ae94 | 1329 | |
1cd7af50 CD |
1330 | # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy |
1331 | 31789000 | |
1640ae94 VS |
1332 | |
1333 | Example 3 (Monitor without CAT support or before creating CAT groups) | |
1cd7af50 | 1334 | --------------------------------------------------------------------- |
1640ae94 VS |
1335 | |
1336 | Assume a system like HSW has only CQM and no CAT support. In this case | |
1337 | the resctrl will still mount but cannot create CTRL_MON directories. | |
1338 | But user can create different MON groups within the root group thereby | |
1339 | able to monitor all tasks including kernel threads. | |
1340 | ||
1341 | This can also be used to profile jobs cache size footprint before being | |
1342 | able to allocate them to different allocation groups. | |
1cd7af50 | 1343 | :: |
1640ae94 | 1344 | |
1cd7af50 CD |
1345 | # mount -t resctrl resctrl /sys/fs/resctrl |
1346 | # cd /sys/fs/resctrl | |
1347 | # mkdir mon_groups/m01 | |
1348 | # mkdir mon_groups/m02 | |
1640ae94 | 1349 | |
1cd7af50 CD |
1350 | # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks |
1351 | # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks | |
1640ae94 VS |
1352 | |
1353 | Monitor the groups separately and also get per domain data. From the | |
1354 | below its apparent that the tasks are mostly doing work on | |
1355 | domain(socket) 0. | |
1cd7af50 | 1356 | :: |
1640ae94 | 1357 | |
1cd7af50 CD |
1358 | # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy |
1359 | 31234000 | |
1360 | # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy | |
1361 | 34555 | |
1362 | # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy | |
1363 | 31234000 | |
1364 | # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy | |
1365 | 32789 | |
1640ae94 VS |
1366 | |
1367 | ||
1368 | Example 4 (Monitor real time tasks) | |
1369 | ----------------------------------- | |
1370 | ||
1371 | A single socket system which has real time tasks running on cores 4-7 | |
1372 | and non real time tasks on other cpus. We want to monitor the cache | |
1373 | occupancy of the real time threads on these cores. | |
1cd7af50 CD |
1374 | :: |
1375 | ||
1376 | # mount -t resctrl resctrl /sys/fs/resctrl | |
1377 | # cd /sys/fs/resctrl | |
1378 | # mkdir p1 | |
1640ae94 | 1379 | |
1cd7af50 | 1380 | Move the cpus 4-7 over to p1:: |
1640ae94 | 1381 | |
1cd7af50 | 1382 | # echo f0 > p1/cpus |
1640ae94 | 1383 | |
1cd7af50 | 1384 | View the llc occupancy snapshot:: |
1640ae94 | 1385 | |
1cd7af50 CD |
1386 | # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy |
1387 | 11234000 | |
d1b22e36 FY |
1388 | |
1389 | Intel RDT Errata | |
1390 | ================ | |
1391 | ||
1392 | Intel MBM Counters May Report System Memory Bandwidth Incorrectly | |
1393 | ----------------------------------------------------------------- | |
1394 | ||
1395 | Errata SKX99 for Skylake server and BDF102 for Broadwell server. | |
1396 | ||
1397 | Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics | |
1398 | according to the assigned Resource Monitor ID (RMID) for that logical | |
1399 | core. The IA32_QM_CTR register (MSR 0xC8E), used to report these | |
1400 | metrics, may report incorrect system bandwidth for certain RMID values. | |
1401 | ||
1402 | Implication: Due to the errata, system memory bandwidth may not match | |
1403 | what is reported. | |
1404 | ||
1405 | Workaround: MBM total and local readings are corrected according to the | |
1406 | following correction factor table: | |
1407 | ||
1408 | +---------------+---------------+---------------+-----------------+ | |
1409 | |core count |rmid count |rmid threshold |correction factor| | |
1410 | +---------------+---------------+---------------+-----------------+ | |
1411 | |1 |8 |0 |1.000000 | | |
1412 | +---------------+---------------+---------------+-----------------+ | |
1413 | |2 |16 |0 |1.000000 | | |
1414 | +---------------+---------------+---------------+-----------------+ | |
1415 | |3 |24 |15 |0.969650 | | |
1416 | +---------------+---------------+---------------+-----------------+ | |
1417 | |4 |32 |0 |1.000000 | | |
1418 | +---------------+---------------+---------------+-----------------+ | |
1419 | |6 |48 |31 |0.969650 | | |
1420 | +---------------+---------------+---------------+-----------------+ | |
1421 | |7 |56 |47 |1.142857 | | |
1422 | +---------------+---------------+---------------+-----------------+ | |
1423 | |8 |64 |0 |1.000000 | | |
1424 | +---------------+---------------+---------------+-----------------+ | |
1425 | |9 |72 |63 |1.185115 | | |
1426 | +---------------+---------------+---------------+-----------------+ | |
1427 | |10 |80 |63 |1.066553 | | |
1428 | +---------------+---------------+---------------+-----------------+ | |
1429 | |11 |88 |79 |1.454545 | | |
1430 | +---------------+---------------+---------------+-----------------+ | |
1431 | |12 |96 |0 |1.000000 | | |
1432 | +---------------+---------------+---------------+-----------------+ | |
1433 | |13 |104 |95 |1.230769 | | |
1434 | +---------------+---------------+---------------+-----------------+ | |
1435 | |14 |112 |95 |1.142857 | | |
1436 | +---------------+---------------+---------------+-----------------+ | |
1437 | |15 |120 |95 |1.066667 | | |
1438 | +---------------+---------------+---------------+-----------------+ | |
1439 | |16 |128 |0 |1.000000 | | |
1440 | +---------------+---------------+---------------+-----------------+ | |
1441 | |17 |136 |127 |1.254863 | | |
1442 | +---------------+---------------+---------------+-----------------+ | |
1443 | |18 |144 |127 |1.185255 | | |
1444 | +---------------+---------------+---------------+-----------------+ | |
1445 | |19 |152 |0 |1.000000 | | |
1446 | +---------------+---------------+---------------+-----------------+ | |
1447 | |20 |160 |127 |1.066667 | | |
1448 | +---------------+---------------+---------------+-----------------+ | |
1449 | |21 |168 |0 |1.000000 | | |
1450 | +---------------+---------------+---------------+-----------------+ | |
1451 | |22 |176 |159 |1.454334 | | |
1452 | +---------------+---------------+---------------+-----------------+ | |
1453 | |23 |184 |0 |1.000000 | | |
1454 | +---------------+---------------+---------------+-----------------+ | |
1455 | |24 |192 |127 |0.969744 | | |
1456 | +---------------+---------------+---------------+-----------------+ | |
1457 | |25 |200 |191 |1.280246 | | |
1458 | +---------------+---------------+---------------+-----------------+ | |
1459 | |26 |208 |191 |1.230921 | | |
1460 | +---------------+---------------+---------------+-----------------+ | |
1461 | |27 |216 |0 |1.000000 | | |
1462 | +---------------+---------------+---------------+-----------------+ | |
1463 | |28 |224 |191 |1.143118 | | |
1464 | +---------------+---------------+---------------+-----------------+ | |
1465 | ||
1466 | If rmid > rmid threshold, MBM total and local values should be multiplied | |
1467 | by the correction factor. | |
1468 | ||
1469 | See: | |
1470 | ||
1471 | 1. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update: | |
1472 | http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html | |
1473 | ||
1474 | 2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update: | |
1475 | http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf | |
1476 | ||
1477 | 3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual: | |
1478 | https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html | |
1479 | ||
1480 | for further information. |