Commit | Line | Data |
---|---|---|
76e7fd84 AB |
1 | .. _perf_security: |
2 | ||
902a8dcc | 3 | Perf events and tool security |
76e7fd84 AB |
4 | ============================= |
5 | ||
6 | Overview | |
7 | -------- | |
8 | ||
e85a198e AB |
9 | Usage of Performance Counters for Linux (perf_events) [1]_ , [2]_ , [3]_ |
10 | can impose a considerable risk of leaking sensitive data accessed by | |
11 | monitored processes. The data leakage is possible both in scenarios of | |
12 | direct usage of perf_events system call API [2]_ and over data files | |
13 | generated by Perf tool user mode utility (Perf) [3]_ , [4]_ . The risk | |
14 | depends on the nature of data that perf_events performance monitoring | |
15 | units (PMU) [2]_ and Perf collect and expose for performance analysis. | |
16 | Collected system and performance data may be split into several | |
17 | categories: | |
18 | ||
19 | 1. System hardware and software configuration data, for example: a CPU | |
20 | model and its cache configuration, an amount of available memory and | |
21 | its topology, used kernel and Perf versions, performance monitoring | |
22 | setup including experiment time, events configuration, Perf command | |
23 | line parameters, etc. | |
24 | ||
25 | 2. User and kernel module paths and their load addresses with sizes, | |
26 | process and thread names with their PIDs and TIDs, timestamps for | |
27 | captured hardware and software events. | |
28 | ||
29 | 3. Content of kernel software counters (e.g., for context switches, page | |
30 | faults, CPU migrations), architectural hardware performance counters | |
31 | (PMC) [8]_ and machine specific registers (MSR) [9]_ that provide | |
32 | execution metrics for various monitored parts of the system (e.g., | |
33 | memory controller (IMC), interconnect (QPI/UPI) or peripheral (PCIe) | |
34 | uncore counters) without direct attribution to any execution context | |
35 | state. | |
36 | ||
37 | 4. Content of architectural execution context registers (e.g., RIP, RSP, | |
38 | RBP on x86_64), process user and kernel space memory addresses and | |
39 | data, content of various architectural MSRs that capture data from | |
40 | this category. | |
41 | ||
42 | Data that belong to the fourth category can potentially contain | |
43 | sensitive process data. If PMUs in some monitoring modes capture values | |
44 | of execution context registers or data from process memory then access | |
902a8dcc AB |
45 | to such monitoring modes requires to be ordered and secured properly. |
46 | So, perf_events performance monitoring and observability operations are | |
47 | the subject for security access control management [5]_ . | |
76e7fd84 | 48 | |
902a8dcc | 49 | perf_events access control |
76e7fd84 AB |
50 | ------------------------------- |
51 | ||
e85a198e AB |
52 | To perform security checks, the Linux implementation splits processes |
53 | into two categories [6]_ : a) privileged processes (whose effective user | |
54 | ID is 0, referred to as superuser or root), and b) unprivileged | |
55 | processes (whose effective UID is nonzero). Privileged processes bypass | |
56 | all kernel security permission checks so perf_events performance | |
57 | monitoring is fully available to privileged processes without access, | |
58 | scope and resource restrictions. | |
59 | ||
60 | Unprivileged processes are subject to a full security permission check | |
61 | based on the process's credentials [5]_ (usually: effective UID, | |
62 | effective GID, and supplementary group list). | |
63 | ||
64 | Linux divides the privileges traditionally associated with superuser | |
65 | into distinct units, known as capabilities [6]_ , which can be | |
66 | independently enabled and disabled on per-thread basis for processes and | |
67 | files of unprivileged users. | |
68 | ||
902a8dcc | 69 | Unprivileged processes with enabled CAP_PERFMON capability are treated |
e85a198e | 70 | as privileged processes with respect to perf_events performance |
902a8dcc AB |
71 | monitoring and observability operations, thus, bypass *scope* permissions |
72 | checks in the kernel. CAP_PERFMON implements the principle of least | |
73 | privilege [13]_ (POSIX 1003.1e: 2.2.2.39) for performance monitoring and | |
74 | observability operations in the kernel and provides a secure approach to | |
a15cb2c1 | 75 | performance monitoring and observability in the system. |
902a8dcc AB |
76 | |
77 | For backward compatibility reasons the access to perf_events monitoring and | |
78 | observability operations is also open for CAP_SYS_ADMIN privileged | |
79 | processes but CAP_SYS_ADMIN usage for secure monitoring and observability | |
80 | use cases is discouraged with respect to the CAP_PERFMON capability. | |
81 | If system audit records [14]_ for a process using perf_events system call | |
82 | API contain denial records of acquiring both CAP_PERFMON and CAP_SYS_ADMIN | |
83 | capabilities then providing the process with CAP_PERFMON capability singly | |
84 | is recommended as the preferred secure approach to resolve double access | |
85 | denial logging related to usage of performance monitoring and observability. | |
86 | ||
4cb3fb1c AB |
87 | Prior Linux v5.9 unprivileged processes using perf_events system call |
88 | are also subject for PTRACE_MODE_READ_REALCREDS ptrace access mode check | |
89 | [7]_ , whose outcome determines whether monitoring is permitted. | |
90 | So unprivileged processes provided with CAP_SYS_PTRACE capability are | |
91 | effectively permitted to pass the check. Starting from Linux v5.9 | |
92 | CAP_SYS_PTRACE capability is not required and CAP_PERFMON is enough to | |
93 | be provided for processes to make performance monitoring and observability | |
94 | operations. | |
e85a198e AB |
95 | |
96 | Other capabilities being granted to unprivileged processes can | |
97 | effectively enable capturing of additional data required for later | |
98 | performance analysis of monitored processes or a system. For example, | |
99 | CAP_SYSLOG capability permits reading kernel space memory addresses from | |
100 | /proc/kallsyms file. | |
76e7fd84 | 101 | |
902a8dcc | 102 | Privileged Perf users groups |
e152c7b7 AB |
103 | --------------------------------- |
104 | ||
1dd88c19 AB |
105 | Mechanisms of capabilities, privileged capability-dumb files [6]_, |
106 | file system ACLs [10]_ and sudo [15]_ utility can be used to create | |
107 | dedicated groups of privileged Perf users who are permitted to execute | |
108 | performance monitoring and observability without limits. The following | |
109 | steps can be taken to create such groups of privileged Perf users. | |
e152c7b7 | 110 | |
e85a198e AB |
111 | 1. Create perf_users group of privileged Perf users, assign perf_users |
112 | group to Perf tool executable and limit access to the executable for | |
113 | other users in the system who are not in the perf_users group: | |
e152c7b7 AB |
114 | |
115 | :: | |
116 | ||
117 | # groupadd perf_users | |
118 | # ls -alhF | |
119 | -rwxr-xr-x 2 root root 11M Oct 19 15:12 perf | |
120 | # chgrp perf_users perf | |
121 | # ls -alhF | |
122 | -rwxr-xr-x 2 root perf_users 11M Oct 19 15:12 perf | |
123 | # chmod o-rwx perf | |
124 | # ls -alhF | |
125 | -rwxr-x--- 2 root perf_users 11M Oct 19 15:12 perf | |
126 | ||
e85a198e | 127 | 2. Assign the required capabilities to the Perf tool executable file and |
902a8dcc | 128 | enable members of perf_users group with monitoring and observability |
e85a198e | 129 | privileges [6]_ : |
e152c7b7 AB |
130 | |
131 | :: | |
132 | ||
902a8dcc AB |
133 | # setcap "cap_perfmon,cap_sys_ptrace,cap_syslog=ep" perf |
134 | # setcap -v "cap_perfmon,cap_sys_ptrace,cap_syslog=ep" perf | |
e152c7b7 AB |
135 | perf: OK |
136 | # getcap perf | |
902a8dcc AB |
137 | perf = cap_sys_ptrace,cap_syslog,cap_perfmon+ep |
138 | ||
1dd88c19 | 139 | If the libcap [16]_ installed doesn't yet support "cap_perfmon", use "38" instead, |
902a8dcc AB |
140 | i.e.: |
141 | ||
142 | :: | |
143 | ||
144 | # setcap "38,cap_ipc_lock,cap_sys_ptrace,cap_syslog=ep" perf | |
145 | ||
146 | Note that you may need to have 'cap_ipc_lock' in the mix for tools such as | |
147 | 'perf top', alternatively use 'perf top -m N', to reduce the memory that | |
148 | it uses for the perf ring buffer, see the memory allocation section below. | |
149 | ||
150 | Using a libcap without support for CAP_PERFMON will make cap_get_flag(caps, 38, | |
151 | CAP_EFFECTIVE, &val) fail, which will lead the default event to be 'cycles:u', | |
152 | so as a workaround explicitly ask for the 'cycles' event, i.e.: | |
153 | ||
154 | :: | |
155 | ||
156 | # perf top -e cycles | |
157 | ||
158 | To get kernel and user samples with a perf binary with just CAP_PERFMON. | |
e152c7b7 | 159 | |
e85a198e | 160 | As a result, members of perf_users group are capable of conducting |
902a8dcc AB |
161 | performance monitoring and observability by using functionality of the |
162 | configured Perf tool executable that, when executes, passes perf_events | |
163 | subsystem scope checks. | |
e152c7b7 | 164 | |
1dd88c19 AB |
165 | In case Perf tool executable can't be assigned required capabilities (e.g. |
166 | file system is mounted with nosuid option or extended attributes are | |
167 | not supported by the file system) then creation of the capabilities | |
168 | privileged environment, naturally shell, is possible. The shell provides | |
169 | inherent processes with CAP_PERFMON and other required capabilities so that | |
170 | performance monitoring and observability operations are available in the | |
171 | environment without limits. Access to the environment can be open via sudo | |
172 | utility for members of perf_users group only. In order to create such | |
173 | environment: | |
174 | ||
175 | 1. Create shell script that uses capsh utility [16]_ to assign CAP_PERFMON | |
176 | and other required capabilities into ambient capability set of the shell | |
177 | process, lock the process security bits after enabling SECBIT_NO_SETUID_FIXUP, | |
178 | SECBIT_NOROOT and SECBIT_NO_CAP_AMBIENT_RAISE bits and then change | |
179 | the process identity to sudo caller of the script who should essentially | |
180 | be a member of perf_users group: | |
181 | ||
182 | :: | |
183 | ||
184 | # ls -alh /usr/local/bin/perf.shell | |
185 | -rwxr-xr-x. 1 root root 83 Oct 13 23:57 /usr/local/bin/perf.shell | |
186 | # cat /usr/local/bin/perf.shell | |
187 | exec /usr/sbin/capsh --iab=^cap_perfmon --secbits=239 --user=$SUDO_USER -- -l | |
188 | ||
189 | 2. Extend sudo policy at /etc/sudoers file with a rule for perf_users group: | |
190 | ||
191 | :: | |
192 | ||
193 | # grep perf_users /etc/sudoers | |
194 | %perf_users ALL=/usr/local/bin/perf.shell | |
195 | ||
196 | 3. Check that members of perf_users group have access to the privileged | |
197 | shell and have CAP_PERFMON and other required capabilities enabled | |
198 | in permitted, effective and ambient capability sets of an inherent process: | |
199 | ||
200 | :: | |
201 | ||
202 | $ id | |
203 | uid=1003(capsh_test) gid=1004(capsh_test) groups=1004(capsh_test),1000(perf_users) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 | |
204 | $ sudo perf.shell | |
205 | [sudo] password for capsh_test: | |
206 | $ grep Cap /proc/self/status | |
207 | CapInh: 0000004000000000 | |
208 | CapPrm: 0000004000000000 | |
209 | CapEff: 0000004000000000 | |
210 | CapBnd: 000000ffffffffff | |
211 | CapAmb: 0000004000000000 | |
212 | $ capsh --decode=0000004000000000 | |
213 | 0x0000004000000000=cap_perfmon | |
214 | ||
215 | As a result, members of perf_users group have access to the privileged | |
216 | environment where they can use tools employing performance monitoring APIs | |
217 | governed by CAP_PERFMON Linux capability. | |
218 | ||
e85a198e AB |
219 | This specific access control management is only available to superuser |
220 | or root running processes with CAP_SETPCAP, CAP_SETFCAP [6]_ | |
221 | capabilities. | |
e152c7b7 | 222 | |
902a8dcc | 223 | Unprivileged users |
76e7fd84 AB |
224 | ----------------------------------- |
225 | ||
902a8dcc | 226 | perf_events *scope* and *access* control for unprivileged processes |
e85a198e | 227 | is governed by perf_event_paranoid [2]_ setting: |
76e7fd84 AB |
228 | |
229 | -1: | |
e85a198e AB |
230 | Impose no *scope* and *access* restrictions on using perf_events |
231 | performance monitoring. Per-user per-cpu perf_event_mlock_kb [2]_ | |
232 | locking limit is ignored when allocating memory buffers for storing | |
233 | performance data. This is the least secure mode since allowed | |
234 | monitored *scope* is maximized and no perf_events specific limits | |
235 | are imposed on *resources* allocated for performance monitoring. | |
76e7fd84 AB |
236 | |
237 | >=0: | |
238 | *scope* includes per-process and system wide performance monitoring | |
e85a198e AB |
239 | but excludes raw tracepoints and ftrace function tracepoints |
240 | monitoring. CPU and system events happened when executing either in | |
241 | user or in kernel space can be monitored and captured for later | |
242 | analysis. Per-user per-cpu perf_event_mlock_kb locking limit is | |
243 | imposed but ignored for unprivileged processes with CAP_IPC_LOCK | |
244 | [6]_ capability. | |
76e7fd84 AB |
245 | |
246 | >=1: | |
e85a198e AB |
247 | *scope* includes per-process performance monitoring only and |
248 | excludes system wide performance monitoring. CPU and system events | |
249 | happened when executing either in user or in kernel space can be | |
250 | monitored and captured for later analysis. Per-user per-cpu | |
251 | perf_event_mlock_kb locking limit is imposed but ignored for | |
252 | unprivileged processes with CAP_IPC_LOCK capability. | |
76e7fd84 AB |
253 | |
254 | >=2: | |
e85a198e AB |
255 | *scope* includes per-process performance monitoring only. CPU and |
256 | system events happened when executing in user space only can be | |
257 | monitored and captured for later analysis. Per-user per-cpu | |
258 | perf_event_mlock_kb locking limit is imposed but ignored for | |
259 | unprivileged processes with CAP_IPC_LOCK capability. | |
76e7fd84 | 260 | |
902a8dcc | 261 | Resource control |
9d87bbae AB |
262 | --------------------------------- |
263 | ||
264 | Open file descriptors | |
265 | +++++++++++++++++++++ | |
266 | ||
e85a198e AB |
267 | The perf_events system call API [2]_ allocates file descriptors for |
268 | every configured PMU event. Open file descriptors are a per-process | |
269 | accountable resource governed by the RLIMIT_NOFILE [11]_ limit | |
270 | (ulimit -n), which is usually derived from the login shell process. When | |
271 | configuring Perf collection for a long list of events on a large server | |
272 | system, this limit can be easily hit preventing required monitoring | |
273 | configuration. RLIMIT_NOFILE limit can be increased on per-user basis | |
274 | modifying content of the limits.conf file [12]_ . Ordinarily, a Perf | |
275 | sampling session (perf record) requires an amount of open perf_event | |
276 | file descriptors that is not less than the number of monitored events | |
277 | multiplied by the number of monitored CPUs. | |
9d87bbae AB |
278 | |
279 | Memory allocation | |
280 | +++++++++++++++++ | |
281 | ||
e85a198e AB |
282 | The amount of memory available to user processes for capturing |
283 | performance monitoring data is governed by the perf_event_mlock_kb [2]_ | |
284 | setting. This perf_event specific resource setting defines overall | |
285 | per-cpu limits of memory allowed for mapping by the user processes to | |
286 | execute performance monitoring. The setting essentially extends the | |
287 | RLIMIT_MEMLOCK [11]_ limit, but only for memory regions mapped | |
288 | specifically for capturing monitored performance events and related data. | |
289 | ||
290 | For example, if a machine has eight cores and perf_event_mlock_kb limit | |
291 | is set to 516 KiB, then a user process is provided with 516 KiB * 8 = | |
292 | 4128 KiB of memory above the RLIMIT_MEMLOCK limit (ulimit -l) for | |
293 | perf_event mmap buffers. In particular, this means that, if the user | |
294 | wants to start two or more performance monitoring processes, the user is | |
295 | required to manually distribute the available 4128 KiB between the | |
296 | monitoring processes, for example, using the --mmap-pages Perf record | |
297 | mode option. Otherwise, the first started performance monitoring process | |
298 | allocates all available 4128 KiB and the other processes will fail to | |
299 | proceed due to the lack of memory. | |
300 | ||
301 | RLIMIT_MEMLOCK and perf_event_mlock_kb resource constraints are ignored | |
302 | for processes with the CAP_IPC_LOCK capability. Thus, perf_events/Perf | |
303 | privileged users can be provided with memory above the constraints for | |
304 | perf_events/Perf performance monitoring purpose by providing the Perf | |
305 | executable with CAP_IPC_LOCK capability. | |
9d87bbae | 306 | |
76e7fd84 AB |
307 | Bibliography |
308 | ------------ | |
309 | ||
310 | .. [1] `<https://lwn.net/Articles/337493/>`_ | |
311 | .. [2] `<http://man7.org/linux/man-pages/man2/perf_event_open.2.html>`_ | |
312 | .. [3] `<http://web.eece.maine.edu/~vweaver/projects/perf_events/>`_ | |
313 | .. [4] `<https://perf.wiki.kernel.org/index.php/Main_Page>`_ | |
314 | .. [5] `<https://www.kernel.org/doc/html/latest/security/credentials.html>`_ | |
315 | .. [6] `<http://man7.org/linux/man-pages/man7/capabilities.7.html>`_ | |
316 | .. [7] `<http://man7.org/linux/man-pages/man2/ptrace.2.html>`_ | |
68570ca0 AB |
317 | .. [8] `<https://en.wikipedia.org/wiki/Hardware_performance_counter>`_ |
318 | .. [9] `<https://en.wikipedia.org/wiki/Model-specific_register>`_ | |
e152c7b7 | 319 | .. [10] `<http://man7.org/linux/man-pages/man5/acl.5.html>`_ |
9d87bbae AB |
320 | .. [11] `<http://man7.org/linux/man-pages/man2/getrlimit.2.html>`_ |
321 | .. [12] `<http://man7.org/linux/man-pages/man5/limits.conf.5.html>`_ | |
902a8dcc AB |
322 | .. [13] `<https://sites.google.com/site/fullycapable>`_ |
323 | .. [14] `<http://man7.org/linux/man-pages/man8/auditd.8.html>`_ | |
1dd88c19 AB |
324 | .. [15] `<https://man7.org/linux/man-pages/man8/sudo.8.html>`_ |
325 | .. [16] `<https://git.kernel.org/pub/scm/libs/libcap/libcap.git/>`_ |