Commit | Line | Data |
---|---|---|
a6f771c9 BM |
1 | User Interface for Resource Control feature |
2 | ||
3 | Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). | |
4 | AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). | |
f20e5789 FY |
5 | |
6 | Copyright (C) 2016 Intel Corporation | |
7 | ||
8 | Fenghua Yu <fenghua.yu@intel.com> | |
9 | Tony Luck <tony.luck@intel.com> | |
a9cad3d4 | 10 | Vikas Shivappa <vikas.shivappa@intel.com> |
f20e5789 | 11 | |
90802938 | 12 | This feature is enabled by the CONFIG_X86_RESCTRL and the x86 /proc/cpuinfo |
a6f771c9 | 13 | flag bits: |
0ff8e080 FY |
14 | RDT (Resource Director Technology) Allocation - "rdt_a" |
15 | CAT (Cache Allocation Technology) - "cat_l3", "cat_l2" | |
aa55d5a4 | 16 | CDP (Code and Data Prioritization ) - "cdp_l3", "cdp_l2" |
0ff8e080 FY |
17 | CQM (Cache QoS Monitoring) - "cqm_llc", "cqm_occup_llc" |
18 | MBM (Memory Bandwidth Monitoring) - "cqm_mbm_total", "cqm_mbm_local" | |
19 | MBA (Memory Bandwidth Allocation) - "mba" | |
f20e5789 FY |
20 | |
21 | To use the feature mount the file system: | |
22 | ||
d6c64a4f | 23 | # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps]] /sys/fs/resctrl |
f20e5789 FY |
24 | |
25 | mount options are: | |
26 | ||
27 | "cdp": Enable code/data prioritization in L3 cache allocations. | |
aa55d5a4 | 28 | "cdpl2": Enable code/data prioritization in L2 cache allocations. |
d6c64a4f VS |
29 | "mba_MBps": Enable the MBA Software Controller(mba_sc) to specify MBA |
30 | bandwidth in MBps | |
aa55d5a4 FY |
31 | |
32 | L2 and L3 CDP are controlled seperately. | |
f20e5789 | 33 | |
1640ae94 | 34 | RDT features are orthogonal. A particular system may support only |
e17e7330 RC |
35 | monitoring, only control, or both monitoring and control. Cache |
36 | pseudo-locking is a unique way of using cache control to "pin" or | |
37 | "lock" data in the cache. Details can be found in | |
38 | "Cache Pseudo-Locking". | |
39 | ||
1640ae94 VS |
40 | |
41 | The mount succeeds if either of allocation or monitoring is present, but | |
42 | only those files and directories supported by the system will be created. | |
43 | For more details on the behavior of the interface during monitoring | |
44 | and allocation, see the "Resource alloc and monitor groups" section. | |
f20e5789 | 45 | |
458b0d6e TG |
46 | Info directory |
47 | -------------- | |
48 | ||
49 | The 'info' directory contains information about the enabled | |
50 | resources. Each resource has its own subdirectory. The subdirectory | |
a9cad3d4 | 51 | names reflect the resource names. |
1640ae94 VS |
52 | |
53 | Each subdirectory contains the following files with respect to | |
54 | allocation: | |
55 | ||
56 | Cache resource(L3/L2) subdirectory contains the following files | |
57 | related to allocation: | |
458b0d6e | 58 | |
a9cad3d4 VS |
59 | "num_closids": The number of CLOSIDs which are valid for this |
60 | resource. The kernel uses the smallest number of | |
61 | CLOSIDs of all enabled resources as limit. | |
458b0d6e | 62 | |
a9cad3d4 VS |
63 | "cbm_mask": The bitmask which is valid for this resource. |
64 | This mask is equivalent to 100%. | |
458b0d6e | 65 | |
a9cad3d4 VS |
66 | "min_cbm_bits": The minimum number of consecutive bits which |
67 | must be set when writing a mask. | |
458b0d6e | 68 | |
0dd2d749 FY |
69 | "shareable_bits": Bitmask of shareable resource with other executing |
70 | entities (e.g. I/O). User can use this when | |
71 | setting up exclusive cache partitions. Note that | |
72 | some platforms support devices that have their | |
73 | own settings for cache use which can over-ride | |
74 | these bits. | |
cba1aab8 RC |
75 | "bit_usage": Annotated capacity bitmasks showing how all |
76 | instances of the resource are used. The legend is: | |
77 | "0" - Corresponding region is unused. When the system's | |
78 | resources have been allocated and a "0" is found | |
79 | in "bit_usage" it is a sign that resources are | |
80 | wasted. | |
81 | "H" - Corresponding region is used by hardware only | |
82 | but available for software use. If a resource | |
83 | has bits set in "shareable_bits" but not all | |
84 | of these bits appear in the resource groups' | |
85 | schematas then the bits appearing in | |
86 | "shareable_bits" but no resource group will | |
87 | be marked as "H". | |
88 | "X" - Corresponding region is available for sharing and | |
89 | used by hardware and software. These are the | |
90 | bits that appear in "shareable_bits" as | |
91 | well as a resource group's allocation. | |
92 | "S" - Corresponding region is used by software | |
93 | and available for sharing. | |
94 | "E" - Corresponding region is used exclusively by | |
95 | one resource group. No sharing allowed. | |
e17e7330 RC |
96 | "P" - Corresponding region is pseudo-locked. No |
97 | sharing allowed. | |
0dd2d749 | 98 | |
1640ae94 VS |
99 | Memory bandwitdh(MB) subdirectory contains the following files |
100 | with respect to allocation: | |
a9cad3d4 VS |
101 | |
102 | "min_bandwidth": The minimum memory bandwidth percentage which | |
103 | user can request. | |
104 | ||
105 | "bandwidth_gran": The granularity in which the memory bandwidth | |
106 | percentage is allocated. The allocated | |
107 | b/w percentage is rounded off to the next | |
108 | control step available on the hardware. The | |
109 | available bandwidth control steps are: | |
110 | min_bandwidth + N * bandwidth_gran. | |
111 | ||
112 | "delay_linear": Indicates if the delay scale is linear or | |
113 | non-linear. This field is purely informational | |
114 | only. | |
458b0d6e | 115 | |
1640ae94 VS |
116 | If RDT monitoring is available there will be an "L3_MON" directory |
117 | with the following files: | |
118 | ||
119 | "num_rmids": The number of RMIDs available. This is the | |
120 | upper bound for how many "CTRL_MON" + "MON" | |
121 | groups can be created. | |
122 | ||
123 | "mon_features": Lists the monitoring events if | |
124 | monitoring is enabled for the resource. | |
125 | ||
126 | "max_threshold_occupancy": | |
127 | Read/write file provides the largest value (in | |
128 | bytes) at which a previously used LLC_occupancy | |
129 | counter can be considered for re-use. | |
130 | ||
165d3ad8 TL |
131 | Finally, in the top level of the "info" directory there is a file |
132 | named "last_cmd_status". This is reset with every "command" issued | |
133 | via the file system (making new directories or writing to any of the | |
134 | control files). If the command was successful, it will read as "ok". | |
135 | If the command failed, it will provide more information that can be | |
136 | conveyed in the error returns from file operations. E.g. | |
137 | ||
138 | # echo L3:0=f7 > schemata | |
139 | bash: echo: write error: Invalid argument | |
140 | # cat info/last_cmd_status | |
141 | mask f7 has non-consecutive 1-bits | |
1640ae94 VS |
142 | |
143 | Resource alloc and monitor groups | |
144 | --------------------------------- | |
145 | ||
f20e5789 | 146 | Resource groups are represented as directories in the resctrl file |
1640ae94 VS |
147 | system. The default group is the root directory which, immediately |
148 | after mounting, owns all the tasks and cpus in the system and can make | |
149 | full use of all resources. | |
150 | ||
151 | On a system with RDT control features additional directories can be | |
152 | created in the root directory that specify different amounts of each | |
153 | resource (see "schemata" below). The root and these additional top level | |
154 | directories are referred to as "CTRL_MON" groups below. | |
155 | ||
156 | On a system with RDT monitoring the root directory and other top level | |
157 | directories contain a directory named "mon_groups" in which additional | |
158 | directories can be created to monitor subsets of tasks in the CTRL_MON | |
159 | group that is their ancestor. These are called "MON" groups in the rest | |
160 | of this document. | |
161 | ||
162 | Removing a directory will move all tasks and cpus owned by the group it | |
163 | represents to the parent. Removing one of the created CTRL_MON groups | |
164 | will automatically remove all MON groups below it. | |
165 | ||
166 | All groups contain the following files: | |
167 | ||
168 | "tasks": | |
169 | Reading this file shows the list of all tasks that belong to | |
170 | this group. Writing a task id to the file will add a task to the | |
171 | group. If the group is a CTRL_MON group the task is removed from | |
172 | whichever previous CTRL_MON group owned the task and also from | |
173 | any MON group that owned the task. If the group is a MON group, | |
174 | then the task must already belong to the CTRL_MON parent of this | |
175 | group. The task is removed from any previous MON group. | |
176 | ||
177 | ||
178 | "cpus": | |
179 | Reading this file shows a bitmask of the logical CPUs owned by | |
180 | this group. Writing a mask to this file will add and remove | |
181 | CPUs to/from this group. As with the tasks file a hierarchy is | |
182 | maintained where MON groups may only include CPUs owned by the | |
183 | parent CTRL_MON group. | |
33dc3e41 RC |
184 | When the resouce group is in pseudo-locked mode this file will |
185 | only be readable, reflecting the CPUs associated with the | |
186 | pseudo-locked region. | |
1640ae94 VS |
187 | |
188 | ||
189 | "cpus_list": | |
190 | Just like "cpus", only using ranges of CPUs instead of bitmasks. | |
f20e5789 | 191 | |
f20e5789 | 192 | |
1640ae94 | 193 | When control is enabled all CTRL_MON groups will also contain: |
f20e5789 | 194 | |
1640ae94 VS |
195 | "schemata": |
196 | A list of all the resources available to this group. | |
197 | Each resource has its own line and format - see below for details. | |
f20e5789 | 198 | |
cba1aab8 RC |
199 | "size": |
200 | Mirrors the display of the "schemata" file to display the size in | |
201 | bytes of each allocation instead of the bits representing the | |
202 | allocation. | |
203 | ||
204 | "mode": | |
205 | The "mode" of the resource group dictates the sharing of its | |
206 | allocations. A "shareable" resource group allows sharing of its | |
e17e7330 RC |
207 | allocations while an "exclusive" resource group does not. A |
208 | cache pseudo-locked region is created by first writing | |
209 | "pseudo-locksetup" to the "mode" file before writing the cache | |
210 | pseudo-locked region's schemata to the resource group's "schemata" | |
211 | file. On successful pseudo-locked region creation the mode will | |
212 | automatically change to "pseudo-locked". | |
cba1aab8 | 213 | |
1640ae94 | 214 | When monitoring is enabled all MON groups will also contain: |
4ffa3c97 | 215 | |
1640ae94 VS |
216 | "mon_data": |
217 | This contains a set of files organized by L3 domain and by | |
218 | RDT event. E.g. on a system with two L3 domains there will | |
219 | be subdirectories "mon_L3_00" and "mon_L3_01". Each of these | |
220 | directories have one file per event (e.g. "llc_occupancy", | |
221 | "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these | |
222 | files provide a read out of the current value of the event for | |
223 | all tasks in the group. In CTRL_MON groups these files provide | |
224 | the sum for all tasks in the CTRL_MON group and all tasks in | |
225 | MON groups. Please see example section for more details on usage. | |
f20e5789 | 226 | |
1640ae94 VS |
227 | Resource allocation rules |
228 | ------------------------- | |
229 | When a task is running the following rules define which resources are | |
230 | available to it: | |
f20e5789 FY |
231 | |
232 | 1) If the task is a member of a non-default group, then the schemata | |
1640ae94 | 233 | for that group is used. |
f20e5789 FY |
234 | |
235 | 2) Else if the task belongs to the default group, but is running on a | |
1640ae94 VS |
236 | CPU that is assigned to some specific group, then the schemata for the |
237 | CPU's group is used. | |
f20e5789 FY |
238 | |
239 | 3) Otherwise the schemata for the default group is used. | |
240 | ||
1640ae94 VS |
241 | Resource monitoring rules |
242 | ------------------------- | |
243 | 1) If a task is a member of a MON group, or non-default CTRL_MON group | |
244 | then RDT events for the task will be reported in that group. | |
245 | ||
246 | 2) If a task is a member of the default CTRL_MON group, but is running | |
247 | on a CPU that is assigned to some specific group, then the RDT events | |
248 | for the task will be reported in that group. | |
249 | ||
250 | 3) Otherwise RDT events for the task will be reported in the root level | |
251 | "mon_data" group. | |
252 | ||
253 | ||
254 | Notes on cache occupancy monitoring and control | |
255 | ----------------------------------------------- | |
256 | When moving a task from one group to another you should remember that | |
257 | this only affects *new* cache allocations by the task. E.g. you may have | |
258 | a task in a monitor group showing 3 MB of cache occupancy. If you move | |
259 | to a new group and immediately check the occupancy of the old and new | |
260 | groups you will likely see that the old group is still showing 3 MB and | |
261 | the new group zero. When the task accesses locations still in cache from | |
262 | before the move, the h/w does not update any counters. On a busy system | |
263 | you will likely see the occupancy in the old group go down as cache lines | |
264 | are evicted and re-used while the occupancy in the new group rises as | |
265 | the task accesses memory and loads into the cache are counted based on | |
266 | membership in the new group. | |
267 | ||
268 | The same applies to cache allocation control. Moving a task to a group | |
269 | with a smaller cache partition will not evict any cache lines. The | |
270 | process may continue to use them from the old partition. | |
271 | ||
272 | Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID) | |
273 | to identify a control group and a monitoring group respectively. Each of | |
274 | the resource groups are mapped to these IDs based on the kind of group. The | |
275 | number of CLOSid and RMID are limited by the hardware and hence the creation of | |
276 | a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID | |
277 | and creation of "MON" group may fail if we run out of RMIDs. | |
278 | ||
279 | max_threshold_occupancy - generic concepts | |
280 | ------------------------------------------ | |
281 | ||
282 | Note that an RMID once freed may not be immediately available for use as | |
283 | the RMID is still tagged the cache lines of the previous user of RMID. | |
284 | Hence such RMIDs are placed on limbo list and checked back if the cache | |
285 | occupancy has gone down. If there is a time when system has a lot of | |
286 | limbo RMIDs but which are not ready to be used, user may see an -EBUSY | |
287 | during mkdir. | |
288 | ||
289 | max_threshold_occupancy is a user configurable value to determine the | |
290 | occupancy at which an RMID can be freed. | |
f20e5789 FY |
291 | |
292 | Schemata files - general concepts | |
293 | --------------------------------- | |
294 | Each line in the file describes one resource. The line starts with | |
295 | the name of the resource, followed by specific values to be applied | |
296 | in each of the instances of that resource on the system. | |
297 | ||
298 | Cache IDs | |
299 | --------- | |
300 | On current generation systems there is one L3 cache per socket and L2 | |
301 | caches are generally just shared by the hyperthreads on a core, but this | |
302 | isn't an architectural requirement. We could have multiple separate L3 | |
303 | caches on a socket, multiple cores could share an L2 cache. So instead | |
304 | of using "socket" or "core" to define the set of logical cpus sharing | |
305 | a resource we use a "Cache ID". At a given cache level this will be a | |
306 | unique number across the whole system (but it isn't guaranteed to be a | |
307 | contiguous sequence, there may be gaps). To find the ID for each logical | |
308 | CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id | |
309 | ||
310 | Cache Bit Masks (CBM) | |
311 | --------------------- | |
312 | For cache resources we describe the portion of the cache that is available | |
313 | for allocation using a bitmask. The maximum value of the mask is defined | |
314 | by each cpu model (and may be different for different cache levels). It | |
315 | is found using CPUID, but is also provided in the "info" directory of | |
316 | the resctrl file system in "info/{resource}/cbm_mask". X86 hardware | |
317 | requires that these masks have all the '1' bits in a contiguous block. So | |
318 | 0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9 | |
319 | and 0xA are not. On a system with a 20-bit mask each bit represents 5% | |
320 | of the capacity of the cache. You could partition the cache into four | |
321 | equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. | |
322 | ||
d6c64a4f VS |
323 | Memory bandwidth Allocation and monitoring |
324 | ------------------------------------------ | |
325 | ||
326 | For Memory bandwidth resource, by default the user controls the resource | |
327 | by indicating the percentage of total memory bandwidth. | |
a9cad3d4 VS |
328 | |
329 | The minimum bandwidth percentage value for each cpu model is predefined | |
330 | and can be looked up through "info/MB/min_bandwidth". The bandwidth | |
331 | granularity that is allocated is also dependent on the cpu model and can | |
332 | be looked up at "info/MB/bandwidth_gran". The available bandwidth | |
333 | control steps are: min_bw + N * bw_gran. Intermediate values are rounded | |
334 | to the next control step available on the hardware. | |
335 | ||
336 | The bandwidth throttling is a core specific mechanism on some of Intel | |
337 | SKUs. Using a high bandwidth and a low bandwidth setting on two threads | |
338 | sharing a core will result in both threads being throttled to use the | |
d6c64a4f VS |
339 | low bandwidth. The fact that Memory bandwidth allocation(MBA) is a core |
340 | specific mechanism where as memory bandwidth monitoring(MBM) is done at | |
341 | the package level may lead to confusion when users try to apply control | |
342 | via the MBA and then monitor the bandwidth to see if the controls are | |
343 | effective. Below are such scenarios: | |
344 | ||
345 | 1. User may *not* see increase in actual bandwidth when percentage | |
346 | values are increased: | |
347 | ||
348 | This can occur when aggregate L2 external bandwidth is more than L3 | |
349 | external bandwidth. Consider an SKL SKU with 24 cores on a package and | |
350 | where L2 external is 10GBps (hence aggregate L2 external bandwidth is | |
351 | 240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20 | |
352 | threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3 | |
353 | bandwidth of 100GBps although the percentage value specified is only 50% | |
354 | << 100%. Hence increasing the bandwidth percentage will not yeild any | |
355 | more bandwidth. This is because although the L2 external bandwidth still | |
356 | has capacity, the L3 external bandwidth is fully used. Also note that | |
357 | this would be dependent on number of cores the benchmark is run on. | |
358 | ||
359 | 2. Same bandwidth percentage may mean different actual bandwidth | |
360 | depending on # of threads: | |
361 | ||
362 | For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4 | |
363 | thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although | |
364 | they have same percentage bandwidth of 10%. This is simply because as | |
365 | threads start using more cores in an rdtgroup, the actual bandwidth may | |
366 | increase or vary although user specified bandwidth percentage is same. | |
367 | ||
368 | In order to mitigate this and make the interface more user friendly, | |
369 | resctrl added support for specifying the bandwidth in MBps as well. The | |
370 | kernel underneath would use a software feedback mechanism or a "Software | |
371 | Controller(mba_sc)" which reads the actual bandwidth using MBM counters | |
372 | and adjust the memowy bandwidth percentages to ensure | |
373 | ||
374 | "actual bandwidth < user specified bandwidth". | |
375 | ||
376 | By default, the schemata would take the bandwidth percentage values | |
377 | where as user can switch to the "MBA software controller" mode using | |
378 | a mount option 'mba_MBps'. The schemata format is specified in the below | |
379 | sections. | |
f20e5789 | 380 | |
1640ae94 VS |
381 | L3 schemata file details (code and data prioritization disabled) |
382 | ---------------------------------------------------------------- | |
f20e5789 FY |
383 | With CDP disabled the L3 schemata format is: |
384 | ||
385 | L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... | |
386 | ||
1640ae94 VS |
387 | L3 schemata file details (CDP enabled via mount option to resctrl) |
388 | ------------------------------------------------------------------ | |
f20e5789 FY |
389 | When CDP is enabled L3 control is split into two separate resources |
390 | so you can specify independent masks for code and data like this: | |
391 | ||
392 | L3data:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... | |
393 | L3code:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... | |
394 | ||
1640ae94 VS |
395 | L2 schemata file details |
396 | ------------------------ | |
f20e5789 FY |
397 | L2 cache does not support code and data prioritization, so the |
398 | schemata format is always: | |
399 | ||
400 | L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... | |
401 | ||
d6c64a4f VS |
402 | Memory bandwidth Allocation (default mode) |
403 | ------------------------------------------ | |
a9cad3d4 VS |
404 | |
405 | Memory b/w domain is L3 cache. | |
406 | ||
407 | MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... | |
408 | ||
d6c64a4f VS |
409 | Memory bandwidth Allocation specified in MBps |
410 | --------------------------------------------- | |
411 | ||
412 | Memory bandwidth domain is L3 cache. | |
413 | ||
414 | MB:<cache_id0>=bw_MBps0;<cache_id1>=bw_MBps1;... | |
415 | ||
c4026b7b TL |
416 | Reading/writing the schemata file |
417 | --------------------------------- | |
418 | Reading the schemata file will show the state of all resources | |
419 | on all domains. When writing you only need to specify those values | |
420 | which you wish to change. E.g. | |
421 | ||
422 | # cat schemata | |
423 | L3DATA:0=fffff;1=fffff;2=fffff;3=fffff | |
424 | L3CODE:0=fffff;1=fffff;2=fffff;3=fffff | |
425 | # echo "L3DATA:2=3c0;" > schemata | |
426 | # cat schemata | |
427 | L3DATA:0=fffff;1=fffff;2=3c0;3=fffff | |
428 | L3CODE:0=fffff;1=fffff;2=fffff;3=fffff | |
429 | ||
e17e7330 RC |
430 | Cache Pseudo-Locking |
431 | -------------------- | |
432 | CAT enables a user to specify the amount of cache space that an | |
433 | application can fill. Cache pseudo-locking builds on the fact that a | |
434 | CPU can still read and write data pre-allocated outside its current | |
435 | allocated area on a cache hit. With cache pseudo-locking, data can be | |
436 | preloaded into a reserved portion of cache that no application can | |
437 | fill, and from that point on will only serve cache hits. The cache | |
438 | pseudo-locked memory is made accessible to user space where an | |
439 | application can map it into its virtual address space and thus have | |
440 | a region of memory with reduced average read latency. | |
441 | ||
442 | The creation of a cache pseudo-locked region is triggered by a request | |
443 | from the user to do so that is accompanied by a schemata of the region | |
444 | to be pseudo-locked. The cache pseudo-locked region is created as follows: | |
445 | - Create a CAT allocation CLOSNEW with a CBM matching the schemata | |
446 | from the user of the cache region that will contain the pseudo-locked | |
447 | memory. This region must not overlap with any current CAT allocation/CLOS | |
448 | on the system and no future overlap with this cache region is allowed | |
449 | while the pseudo-locked region exists. | |
450 | - Create a contiguous region of memory of the same size as the cache | |
451 | region. | |
452 | - Flush the cache, disable hardware prefetchers, disable preemption. | |
453 | - Make CLOSNEW the active CLOS and touch the allocated memory to load | |
454 | it into the cache. | |
455 | - Set the previous CLOS as active. | |
456 | - At this point the closid CLOSNEW can be released - the cache | |
457 | pseudo-locked region is protected as long as its CBM does not appear in | |
458 | any CAT allocation. Even though the cache pseudo-locked region will from | |
459 | this point on not appear in any CBM of any CLOS an application running with | |
460 | any CLOS will be able to access the memory in the pseudo-locked region since | |
461 | the region continues to serve cache hits. | |
462 | - The contiguous region of memory loaded into the cache is exposed to | |
463 | user-space as a character device. | |
464 | ||
465 | Cache pseudo-locking increases the probability that data will remain | |
466 | in the cache via carefully configuring the CAT feature and controlling | |
467 | application behavior. There is no guarantee that data is placed in | |
468 | cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict | |
469 | “locked” data from cache. Power management C-states may shrink or | |
6fc0de37 RC |
470 | power off cache. Deeper C-states will automatically be restricted on |
471 | pseudo-locked region creation. | |
e17e7330 RC |
472 | |
473 | It is required that an application using a pseudo-locked region runs | |
474 | with affinity to the cores (or a subset of the cores) associated | |
475 | with the cache on which the pseudo-locked region resides. A sanity check | |
476 | within the code will not allow an application to map pseudo-locked memory | |
477 | unless it runs with affinity to cores associated with the cache on which the | |
478 | pseudo-locked region resides. The sanity check is only done during the | |
479 | initial mmap() handling, there is no enforcement afterwards and the | |
480 | application self needs to ensure it remains affine to the correct cores. | |
481 | ||
482 | Pseudo-locking is accomplished in two stages: | |
483 | 1) During the first stage the system administrator allocates a portion | |
484 | of cache that should be dedicated to pseudo-locking. At this time an | |
485 | equivalent portion of memory is allocated, loaded into allocated | |
486 | cache portion, and exposed as a character device. | |
487 | 2) During the second stage a user-space application maps (mmap()) the | |
488 | pseudo-locked memory into its address space. | |
489 | ||
490 | Cache Pseudo-Locking Interface | |
491 | ------------------------------ | |
492 | A pseudo-locked region is created using the resctrl interface as follows: | |
493 | ||
494 | 1) Create a new resource group by creating a new directory in /sys/fs/resctrl. | |
495 | 2) Change the new resource group's mode to "pseudo-locksetup" by writing | |
496 | "pseudo-locksetup" to the "mode" file. | |
497 | 3) Write the schemata of the pseudo-locked region to the "schemata" file. All | |
498 | bits within the schemata should be "unused" according to the "bit_usage" | |
499 | file. | |
500 | ||
501 | On successful pseudo-locked region creation the "mode" file will contain | |
502 | "pseudo-locked" and a new character device with the same name as the resource | |
503 | group will exist in /dev/pseudo_lock. This character device can be mmap()'ed | |
504 | by user space in order to obtain access to the pseudo-locked memory region. | |
505 | ||
506 | An example of cache pseudo-locked region creation and usage can be found below. | |
507 | ||
508 | Cache Pseudo-Locking Debugging Interface | |
509 | --------------------------------------- | |
510 | The pseudo-locking debugging interface is enabled by default (if | |
511 | CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl. | |
512 | ||
513 | There is no explicit way for the kernel to test if a provided memory | |
514 | location is present in the cache. The pseudo-locking debugging interface uses | |
515 | the tracing infrastructure to provide two ways to measure cache residency of | |
516 | the pseudo-locked region: | |
517 | 1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data | |
518 | from these measurements are best visualized using a hist trigger (see | |
519 | example below). In this test the pseudo-locked region is traversed at | |
520 | a stride of 32 bytes while hardware prefetchers and preemption | |
521 | are disabled. This also provides a substitute visualization of cache | |
522 | hits and misses. | |
523 | 2) Cache hit and miss measurements using model specific precision counters if | |
524 | available. Depending on the levels of cache on the system the pseudo_lock_l2 | |
525 | and pseudo_lock_l3 tracepoints are available. | |
e17e7330 RC |
526 | |
527 | When a pseudo-locked region is created a new debugfs directory is created for | |
528 | it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single | |
529 | write-only file, pseudo_lock_measure, is present in this directory. The | |
dd45407c RC |
530 | measurement of the pseudo-locked region depends on the number written to this |
531 | debugfs file: | |
532 | 1 - writing "1" to the pseudo_lock_measure file will trigger the latency | |
533 | measurement captured in the pseudo_lock_mem_latency tracepoint. See | |
534 | example below. | |
535 | 2 - writing "2" to the pseudo_lock_measure file will trigger the L2 cache | |
536 | residency (cache hits and misses) measurement captured in the | |
537 | pseudo_lock_l2 tracepoint. See example below. | |
538 | 3 - writing "3" to the pseudo_lock_measure file will trigger the L3 cache | |
539 | residency (cache hits and misses) measurement captured in the | |
540 | pseudo_lock_l3 tracepoint. | |
541 | ||
542 | All measurements are recorded with the tracing infrastructure. This requires | |
543 | the relevant tracepoints to be enabled before the measurement is triggered. | |
e17e7330 RC |
544 | |
545 | Example of latency debugging interface: | |
546 | In this example a pseudo-locked region named "newlock" was created. Here is | |
547 | how we can measure the latency in cycles of reading from this region and | |
548 | visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS | |
549 | is set: | |
550 | # :> /sys/kernel/debug/tracing/trace | |
551 | # echo 'hist:keys=latency' > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/trigger | |
552 | # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable | |
553 | # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure | |
554 | # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable | |
555 | # cat /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/hist | |
556 | ||
557 | # event histogram | |
558 | # | |
559 | # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active] | |
560 | # | |
561 | ||
562 | { latency: 456 } hitcount: 1 | |
563 | { latency: 50 } hitcount: 83 | |
564 | { latency: 36 } hitcount: 96 | |
565 | { latency: 44 } hitcount: 174 | |
566 | { latency: 48 } hitcount: 195 | |
567 | { latency: 46 } hitcount: 262 | |
568 | { latency: 42 } hitcount: 693 | |
569 | { latency: 40 } hitcount: 3204 | |
570 | { latency: 38 } hitcount: 3484 | |
571 | ||
572 | Totals: | |
573 | Hits: 8192 | |
574 | Entries: 9 | |
575 | Dropped: 0 | |
576 | ||
577 | Example of cache hits/misses debugging: | |
578 | In this example a pseudo-locked region named "newlock" was created on the L2 | |
579 | cache of a platform. Here is how we can obtain details of the cache hits | |
580 | and misses using the platform's precision counters. | |
581 | ||
582 | # :> /sys/kernel/debug/tracing/trace | |
583 | # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable | |
584 | # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure | |
585 | # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable | |
586 | # cat /sys/kernel/debug/tracing/trace | |
587 | ||
588 | # tracer: nop | |
589 | # | |
590 | # _-----=> irqs-off | |
591 | # / _----=> need-resched | |
592 | # | / _---=> hardirq/softirq | |
593 | # || / _--=> preempt-depth | |
594 | # ||| / delay | |
595 | # TASK-PID CPU# |||| TIMESTAMP FUNCTION | |
596 | # | | | |||| | | | |
597 | pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0 | |
598 | ||
599 | ||
1640ae94 VS |
600 | Examples for RDT allocation usage: |
601 | ||
f20e5789 FY |
602 | Example 1 |
603 | --------- | |
604 | On a two socket machine (one L3 cache per socket) with just four bits | |
a9cad3d4 VS |
605 | for cache bit masks, minimum b/w of 10% with a memory bandwidth |
606 | granularity of 10% | |
f20e5789 FY |
607 | |
608 | # mount -t resctrl resctrl /sys/fs/resctrl | |
609 | # cd /sys/fs/resctrl | |
610 | # mkdir p0 p1 | |
a9cad3d4 VS |
611 | # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata |
612 | # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata | |
f20e5789 FY |
613 | |
614 | The default resource group is unmodified, so we have access to all parts | |
615 | of all caches (its schemata file reads "L3:0=f;1=f"). | |
616 | ||
617 | Tasks that are under the control of group "p0" may only allocate from the | |
618 | "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. | |
619 | Tasks in group "p1" use the "lower" 50% of cache on both sockets. | |
620 | ||
a9cad3d4 VS |
621 | Similarly, tasks that are under the control of group "p0" may use a |
622 | maximum memory b/w of 50% on socket0 and 50% on socket 1. | |
623 | Tasks in group "p1" may also use 50% memory b/w on both sockets. | |
624 | Note that unlike cache masks, memory b/w cannot specify whether these | |
625 | allocations can overlap or not. The allocations specifies the maximum | |
626 | b/w that the group may be able to use and the system admin can configure | |
627 | the b/w accordingly. | |
628 | ||
d6c64a4f VS |
629 | If the MBA is specified in MB(megabytes) then user can enter the max b/w in MB |
630 | rather than the percentage values. | |
631 | ||
632 | # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata | |
633 | # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata | |
634 | ||
635 | In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w | |
636 | of 1024MB where as on socket 1 they would use 500MB. | |
637 | ||
f20e5789 FY |
638 | Example 2 |
639 | --------- | |
640 | Again two sockets, but this time with a more realistic 20-bit mask. | |
641 | ||
642 | Two real time tasks pid=1234 running on processor 0 and pid=5678 running on | |
643 | processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy | |
644 | neighbors, each of the two real-time tasks exclusively occupies one quarter | |
645 | of L3 cache on socket 0. | |
646 | ||
647 | # mount -t resctrl resctrl /sys/fs/resctrl | |
648 | # cd /sys/fs/resctrl | |
649 | ||
650 | First we reset the schemata for the default group so that the "upper" | |
a9cad3d4 VS |
651 | 50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by |
652 | ordinary tasks: | |
f20e5789 | 653 | |
a9cad3d4 | 654 | # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata |
f20e5789 FY |
655 | |
656 | Next we make a resource group for our first real time task and give | |
657 | it access to the "top" 25% of the cache on socket 0. | |
658 | ||
659 | # mkdir p0 | |
660 | # echo "L3:0=f8000;1=fffff" > p0/schemata | |
661 | ||
662 | Finally we move our first real time task into this resource group. We | |
663 | also use taskset(1) to ensure the task always runs on a dedicated CPU | |
664 | on socket 0. Most uses of resource groups will also constrain which | |
665 | processors tasks run on. | |
666 | ||
667 | # echo 1234 > p0/tasks | |
668 | # taskset -cp 1 1234 | |
669 | ||
670 | Ditto for the second real time task (with the remaining 25% of cache): | |
671 | ||
672 | # mkdir p1 | |
673 | # echo "L3:0=7c00;1=fffff" > p1/schemata | |
674 | # echo 5678 > p1/tasks | |
675 | # taskset -cp 2 5678 | |
676 | ||
a9cad3d4 VS |
677 | For the same 2 socket system with memory b/w resource and CAT L3 the |
678 | schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is | |
679 | 10): | |
680 | ||
681 | For our first real time task this would request 20% memory b/w on socket | |
682 | 0. | |
683 | ||
684 | # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata | |
685 | ||
686 | For our second real time task this would request an other 20% memory b/w | |
687 | on socket 0. | |
688 | ||
689 | # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata | |
690 | ||
f20e5789 FY |
691 | Example 3 |
692 | --------- | |
693 | ||
694 | A single socket system which has real-time tasks running on core 4-7 and | |
695 | non real-time workload assigned to core 0-3. The real-time tasks share text | |
696 | and data, so a per task association is not required and due to interaction | |
697 | with the kernel it's desired that the kernel on these cores shares L3 with | |
698 | the tasks. | |
699 | ||
700 | # mount -t resctrl resctrl /sys/fs/resctrl | |
701 | # cd /sys/fs/resctrl | |
702 | ||
703 | First we reset the schemata for the default group so that the "upper" | |
a9cad3d4 VS |
704 | 50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0 |
705 | cannot be used by ordinary tasks: | |
f20e5789 | 706 | |
a9cad3d4 | 707 | # echo "L3:0=3ff\nMB:0=50" > schemata |
f20e5789 | 708 | |
a9cad3d4 VS |
709 | Next we make a resource group for our real time cores and give it access |
710 | to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on | |
711 | socket 0. | |
f20e5789 FY |
712 | |
713 | # mkdir p0 | |
a9cad3d4 | 714 | # echo "L3:0=ffc00\nMB:0=50" > p0/schemata |
f20e5789 FY |
715 | |
716 | Finally we move core 4-7 over to the new group and make sure that the | |
a9cad3d4 VS |
717 | kernel and the tasks running there get 50% of the cache. They should |
718 | also get 50% of memory bandwidth assuming that the cores 4-7 are SMT | |
719 | siblings and only the real time threads are scheduled on the cores 4-7. | |
f20e5789 | 720 | |
fb8fb46c | 721 | # echo F0 > p0/cpus |
3c2a769d | 722 | |
cba1aab8 RC |
723 | Example 4 |
724 | --------- | |
725 | ||
726 | The resource groups in previous examples were all in the default "shareable" | |
727 | mode allowing sharing of their cache allocations. If one resource group | |
728 | configures a cache allocation then nothing prevents another resource group | |
729 | to overlap with that allocation. | |
730 | ||
731 | In this example a new exclusive resource group will be created on a L2 CAT | |
732 | system with two L2 cache instances that can be configured with an 8-bit | |
733 | capacity bitmask. The new exclusive resource group will be configured to use | |
734 | 25% of each cache instance. | |
735 | ||
736 | # mount -t resctrl resctrl /sys/fs/resctrl/ | |
737 | # cd /sys/fs/resctrl | |
738 | ||
739 | First, we observe that the default group is configured to allocate to all L2 | |
740 | cache: | |
741 | ||
742 | # cat schemata | |
743 | L2:0=ff;1=ff | |
744 | ||
745 | We could attempt to create the new resource group at this point, but it will | |
746 | fail because of the overlap with the schemata of the default group: | |
747 | # mkdir p0 | |
748 | # echo 'L2:0=0x3;1=0x3' > p0/schemata | |
749 | # cat p0/mode | |
750 | shareable | |
751 | # echo exclusive > p0/mode | |
752 | -sh: echo: write error: Invalid argument | |
753 | # cat info/last_cmd_status | |
754 | schemata overlaps | |
755 | ||
756 | To ensure that there is no overlap with another resource group the default | |
757 | resource group's schemata has to change, making it possible for the new | |
758 | resource group to become exclusive. | |
759 | # echo 'L2:0=0xfc;1=0xfc' > schemata | |
760 | # echo exclusive > p0/mode | |
761 | # grep . p0/* | |
762 | p0/cpus:0 | |
763 | p0/mode:exclusive | |
764 | p0/schemata:L2:0=03;1=03 | |
765 | p0/size:L2:0=262144;1=262144 | |
766 | ||
767 | A new resource group will on creation not overlap with an exclusive resource | |
768 | group: | |
769 | # mkdir p1 | |
770 | # grep . p1/* | |
771 | p1/cpus:0 | |
772 | p1/mode:shareable | |
773 | p1/schemata:L2:0=fc;1=fc | |
774 | p1/size:L2:0=786432;1=786432 | |
775 | ||
776 | The bit_usage will reflect how the cache is used: | |
777 | # cat info/L2/bit_usage | |
778 | 0=SSSSSSEE;1=SSSSSSEE | |
779 | ||
780 | A resource group cannot be forced to overlap with an exclusive resource group: | |
781 | # echo 'L2:0=0x1;1=0x1' > p1/schemata | |
782 | -sh: echo: write error: Invalid argument | |
783 | # cat info/last_cmd_status | |
784 | overlaps with exclusive group | |
785 | ||
e17e7330 RC |
786 | Example of Cache Pseudo-Locking |
787 | ------------------------------- | |
788 | Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked | |
789 | region is exposed at /dev/pseudo_lock/newlock that can be provided to | |
790 | application for argument to mmap(). | |
791 | ||
792 | # mount -t resctrl resctrl /sys/fs/resctrl/ | |
793 | # cd /sys/fs/resctrl | |
794 | ||
795 | Ensure that there are bits available that can be pseudo-locked, since only | |
796 | unused bits can be pseudo-locked the bits to be pseudo-locked needs to be | |
797 | removed from the default resource group's schemata: | |
798 | # cat info/L2/bit_usage | |
799 | 0=SSSSSSSS;1=SSSSSSSS | |
800 | # echo 'L2:1=0xfc' > schemata | |
801 | # cat info/L2/bit_usage | |
802 | 0=SSSSSSSS;1=SSSSSS00 | |
803 | ||
804 | Create a new resource group that will be associated with the pseudo-locked | |
805 | region, indicate that it will be used for a pseudo-locked region, and | |
806 | configure the requested pseudo-locked region capacity bitmask: | |
807 | ||
808 | # mkdir newlock | |
809 | # echo pseudo-locksetup > newlock/mode | |
810 | # echo 'L2:1=0x3' > newlock/schemata | |
811 | ||
812 | On success the resource group's mode will change to pseudo-locked, the | |
813 | bit_usage will reflect the pseudo-locked region, and the character device | |
814 | exposing the pseudo-locked region will exist: | |
815 | ||
816 | # cat newlock/mode | |
817 | pseudo-locked | |
818 | # cat info/L2/bit_usage | |
819 | 0=SSSSSSSS;1=SSSSSSPP | |
820 | # ls -l /dev/pseudo_lock/newlock | |
821 | crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock | |
822 | ||
823 | /* | |
824 | * Example code to access one page of pseudo-locked cache region | |
825 | * from user space. | |
826 | */ | |
827 | #define _GNU_SOURCE | |
828 | #include <fcntl.h> | |
829 | #include <sched.h> | |
830 | #include <stdio.h> | |
831 | #include <stdlib.h> | |
832 | #include <unistd.h> | |
833 | #include <sys/mman.h> | |
834 | ||
835 | /* | |
836 | * It is required that the application runs with affinity to only | |
837 | * cores associated with the pseudo-locked region. Here the cpu | |
838 | * is hardcoded for convenience of example. | |
839 | */ | |
840 | static int cpuid = 2; | |
841 | ||
842 | int main(int argc, char *argv[]) | |
843 | { | |
844 | cpu_set_t cpuset; | |
845 | long page_size; | |
846 | void *mapping; | |
847 | int dev_fd; | |
848 | int ret; | |
849 | ||
850 | page_size = sysconf(_SC_PAGESIZE); | |
851 | ||
852 | CPU_ZERO(&cpuset); | |
853 | CPU_SET(cpuid, &cpuset); | |
854 | ret = sched_setaffinity(0, sizeof(cpuset), &cpuset); | |
855 | if (ret < 0) { | |
856 | perror("sched_setaffinity"); | |
857 | exit(EXIT_FAILURE); | |
858 | } | |
859 | ||
860 | dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR); | |
861 | if (dev_fd < 0) { | |
862 | perror("open"); | |
863 | exit(EXIT_FAILURE); | |
864 | } | |
865 | ||
866 | mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, | |
867 | dev_fd, 0); | |
868 | if (mapping == MAP_FAILED) { | |
869 | perror("mmap"); | |
870 | close(dev_fd); | |
871 | exit(EXIT_FAILURE); | |
872 | } | |
873 | ||
874 | /* Application interacts with pseudo-locked memory @mapping */ | |
875 | ||
876 | ret = munmap(mapping, page_size); | |
877 | if (ret < 0) { | |
878 | perror("munmap"); | |
879 | close(dev_fd); | |
880 | exit(EXIT_FAILURE); | |
881 | } | |
882 | ||
883 | close(dev_fd); | |
884 | exit(EXIT_SUCCESS); | |
885 | } | |
886 | ||
cba1aab8 RC |
887 | Locking between applications |
888 | ---------------------------- | |
3c2a769d MT |
889 | |
890 | Certain operations on the resctrl filesystem, composed of read/writes | |
891 | to/from multiple files, must be atomic. | |
892 | ||
893 | As an example, the allocation of an exclusive reservation of L3 cache | |
894 | involves: | |
895 | ||
cba1aab8 | 896 | 1. Read the cbmmasks from each directory or the per-resource "bit_usage" |
3c2a769d MT |
897 | 2. Find a contiguous set of bits in the global CBM bitmask that is clear |
898 | in any of the directory cbmmasks | |
899 | 3. Create a new directory | |
900 | 4. Set the bits found in step 2 to the new directory "schemata" file | |
901 | ||
902 | If two applications attempt to allocate space concurrently then they can | |
903 | end up allocating the same bits so the reservations are shared instead of | |
904 | exclusive. | |
905 | ||
906 | To coordinate atomic operations on the resctrlfs and to avoid the problem | |
907 | above, the following locking procedure is recommended: | |
908 | ||
909 | Locking is based on flock, which is available in libc and also as a shell | |
910 | script command | |
911 | ||
912 | Write lock: | |
913 | ||
914 | A) Take flock(LOCK_EX) on /sys/fs/resctrl | |
915 | B) Read/write the directory structure. | |
916 | C) funlock | |
917 | ||
918 | Read lock: | |
919 | ||
920 | A) Take flock(LOCK_SH) on /sys/fs/resctrl | |
921 | B) If success read the directory structure. | |
922 | C) funlock | |
923 | ||
924 | Example with bash: | |
925 | ||
926 | # Atomically read directory structure | |
927 | $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl | |
928 | ||
929 | # Read directory contents and create new subdirectory | |
930 | ||
931 | $ cat create-dir.sh | |
932 | find /sys/fs/resctrl/ > output.txt | |
933 | mask = function-of(output.txt) | |
934 | mkdir /sys/fs/resctrl/newres/ | |
935 | echo mask > /sys/fs/resctrl/newres/schemata | |
936 | ||
937 | $ flock /sys/fs/resctrl/ ./create-dir.sh | |
938 | ||
939 | Example with C: | |
940 | ||
941 | /* | |
942 | * Example code do take advisory locks | |
943 | * before accessing resctrl filesystem | |
944 | */ | |
945 | #include <sys/file.h> | |
946 | #include <stdlib.h> | |
947 | ||
948 | void resctrl_take_shared_lock(int fd) | |
949 | { | |
950 | int ret; | |
951 | ||
952 | /* take shared lock on resctrl filesystem */ | |
953 | ret = flock(fd, LOCK_SH); | |
954 | if (ret) { | |
955 | perror("flock"); | |
956 | exit(-1); | |
957 | } | |
958 | } | |
959 | ||
960 | void resctrl_take_exclusive_lock(int fd) | |
961 | { | |
962 | int ret; | |
963 | ||
964 | /* release lock on resctrl filesystem */ | |
965 | ret = flock(fd, LOCK_EX); | |
966 | if (ret) { | |
967 | perror("flock"); | |
968 | exit(-1); | |
969 | } | |
970 | } | |
971 | ||
972 | void resctrl_release_lock(int fd) | |
973 | { | |
974 | int ret; | |
975 | ||
976 | /* take shared lock on resctrl filesystem */ | |
977 | ret = flock(fd, LOCK_UN); | |
978 | if (ret) { | |
979 | perror("flock"); | |
980 | exit(-1); | |
981 | } | |
982 | } | |
983 | ||
984 | void main(void) | |
985 | { | |
986 | int fd, ret; | |
987 | ||
988 | fd = open("/sys/fs/resctrl", O_DIRECTORY); | |
989 | if (fd == -1) { | |
990 | perror("open"); | |
991 | exit(-1); | |
992 | } | |
993 | resctrl_take_shared_lock(fd); | |
994 | /* code to read directory contents */ | |
995 | resctrl_release_lock(fd); | |
996 | ||
997 | resctrl_take_exclusive_lock(fd); | |
998 | /* code to read and write directory contents */ | |
999 | resctrl_release_lock(fd); | |
1000 | } | |
1640ae94 VS |
1001 | |
1002 | Examples for RDT Monitoring along with allocation usage: | |
1003 | ||
1004 | Reading monitored data | |
1005 | ---------------------- | |
1006 | Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would | |
1007 | show the current snapshot of LLC occupancy of the corresponding MON | |
1008 | group or CTRL_MON group. | |
1009 | ||
1010 | ||
1011 | Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group) | |
1012 | --------- | |
1013 | On a two socket machine (one L3 cache per socket) with just four bits | |
1014 | for cache bit masks | |
1015 | ||
1016 | # mount -t resctrl resctrl /sys/fs/resctrl | |
1017 | # cd /sys/fs/resctrl | |
1018 | # mkdir p0 p1 | |
1019 | # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata | |
1020 | # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata | |
1021 | # echo 5678 > p1/tasks | |
1022 | # echo 5679 > p1/tasks | |
1023 | ||
1024 | The default resource group is unmodified, so we have access to all parts | |
1025 | of all caches (its schemata file reads "L3:0=f;1=f"). | |
1026 | ||
1027 | Tasks that are under the control of group "p0" may only allocate from the | |
1028 | "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. | |
1029 | Tasks in group "p1" use the "lower" 50% of cache on both sockets. | |
1030 | ||
1031 | Create monitor groups and assign a subset of tasks to each monitor group. | |
1032 | ||
1033 | # cd /sys/fs/resctrl/p1/mon_groups | |
1034 | # mkdir m11 m12 | |
1035 | # echo 5678 > m11/tasks | |
1036 | # echo 5679 > m12/tasks | |
1037 | ||
1038 | fetch data (data shown in bytes) | |
1039 | ||
1040 | # cat m11/mon_data/mon_L3_00/llc_occupancy | |
1041 | 16234000 | |
1042 | # cat m11/mon_data/mon_L3_01/llc_occupancy | |
1043 | 14789000 | |
1044 | # cat m12/mon_data/mon_L3_00/llc_occupancy | |
1045 | 16789000 | |
1046 | ||
1047 | The parent ctrl_mon group shows the aggregated data. | |
1048 | ||
1049 | # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy | |
1050 | 31234000 | |
1051 | ||
1052 | Example 2 (Monitor a task from its creation) | |
1053 | --------- | |
1054 | On a two socket machine (one L3 cache per socket) | |
1055 | ||
1056 | # mount -t resctrl resctrl /sys/fs/resctrl | |
1057 | # cd /sys/fs/resctrl | |
1058 | # mkdir p0 p1 | |
1059 | ||
1060 | An RMID is allocated to the group once its created and hence the <cmd> | |
1061 | below is monitored from its creation. | |
1062 | ||
1063 | # echo $$ > /sys/fs/resctrl/p1/tasks | |
1064 | # <cmd> | |
1065 | ||
1066 | Fetch the data | |
1067 | ||
1068 | # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy | |
1069 | 31789000 | |
1070 | ||
1071 | Example 3 (Monitor without CAT support or before creating CAT groups) | |
1072 | --------- | |
1073 | ||
1074 | Assume a system like HSW has only CQM and no CAT support. In this case | |
1075 | the resctrl will still mount but cannot create CTRL_MON directories. | |
1076 | But user can create different MON groups within the root group thereby | |
1077 | able to monitor all tasks including kernel threads. | |
1078 | ||
1079 | This can also be used to profile jobs cache size footprint before being | |
1080 | able to allocate them to different allocation groups. | |
1081 | ||
1082 | # mount -t resctrl resctrl /sys/fs/resctrl | |
1083 | # cd /sys/fs/resctrl | |
1084 | # mkdir mon_groups/m01 | |
1085 | # mkdir mon_groups/m02 | |
1086 | ||
1087 | # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks | |
1088 | # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks | |
1089 | ||
1090 | Monitor the groups separately and also get per domain data. From the | |
1091 | below its apparent that the tasks are mostly doing work on | |
1092 | domain(socket) 0. | |
1093 | ||
1094 | # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy | |
1095 | 31234000 | |
1096 | # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy | |
1097 | 34555 | |
1098 | # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy | |
1099 | 31234000 | |
1100 | # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy | |
1101 | 32789 | |
1102 | ||
1103 | ||
1104 | Example 4 (Monitor real time tasks) | |
1105 | ----------------------------------- | |
1106 | ||
1107 | A single socket system which has real time tasks running on cores 4-7 | |
1108 | and non real time tasks on other cpus. We want to monitor the cache | |
1109 | occupancy of the real time threads on these cores. | |
1110 | ||
1111 | # mount -t resctrl resctrl /sys/fs/resctrl | |
1112 | # cd /sys/fs/resctrl | |
1113 | # mkdir p1 | |
1114 | ||
1115 | Move the cpus 4-7 over to p1 | |
30009746 | 1116 | # echo f0 > p1/cpus |
1640ae94 VS |
1117 | |
1118 | View the llc occupancy snapshot | |
1119 | ||
1120 | # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy | |
1121 | 11234000 |