Commit | Line | Data |
---|---|---|
76e7fd84 AB |
1 | .. _perf_security: |
2 | ||
902a8dcc | 3 | Perf events and tool security |
76e7fd84 AB |
4 | ============================= |
5 | ||
6 | Overview | |
7 | -------- | |
8 | ||
e85a198e AB |
9 | Usage of Performance Counters for Linux (perf_events) [1]_ , [2]_ , [3]_ |
10 | can impose a considerable risk of leaking sensitive data accessed by | |
11 | monitored processes. The data leakage is possible both in scenarios of | |
12 | direct usage of perf_events system call API [2]_ and over data files | |
13 | generated by Perf tool user mode utility (Perf) [3]_ , [4]_ . The risk | |
14 | depends on the nature of data that perf_events performance monitoring | |
15 | units (PMU) [2]_ and Perf collect and expose for performance analysis. | |
16 | Collected system and performance data may be split into several | |
17 | categories: | |
18 | ||
19 | 1. System hardware and software configuration data, for example: a CPU | |
20 | model and its cache configuration, an amount of available memory and | |
21 | its topology, used kernel and Perf versions, performance monitoring | |
22 | setup including experiment time, events configuration, Perf command | |
23 | line parameters, etc. | |
24 | ||
25 | 2. User and kernel module paths and their load addresses with sizes, | |
26 | process and thread names with their PIDs and TIDs, timestamps for | |
27 | captured hardware and software events. | |
28 | ||
29 | 3. Content of kernel software counters (e.g., for context switches, page | |
30 | faults, CPU migrations), architectural hardware performance counters | |
31 | (PMC) [8]_ and machine specific registers (MSR) [9]_ that provide | |
32 | execution metrics for various monitored parts of the system (e.g., | |
33 | memory controller (IMC), interconnect (QPI/UPI) or peripheral (PCIe) | |
34 | uncore counters) without direct attribution to any execution context | |
35 | state. | |
36 | ||
37 | 4. Content of architectural execution context registers (e.g., RIP, RSP, | |
38 | RBP on x86_64), process user and kernel space memory addresses and | |
39 | data, content of various architectural MSRs that capture data from | |
40 | this category. | |
41 | ||
42 | Data that belong to the fourth category can potentially contain | |
43 | sensitive process data. If PMUs in some monitoring modes capture values | |
44 | of execution context registers or data from process memory then access | |
902a8dcc AB |
45 | to such monitoring modes requires to be ordered and secured properly. |
46 | So, perf_events performance monitoring and observability operations are | |
47 | the subject for security access control management [5]_ . | |
76e7fd84 | 48 | |
902a8dcc | 49 | perf_events access control |
76e7fd84 AB |
50 | ------------------------------- |
51 | ||
e85a198e AB |
52 | To perform security checks, the Linux implementation splits processes |
53 | into two categories [6]_ : a) privileged processes (whose effective user | |
54 | ID is 0, referred to as superuser or root), and b) unprivileged | |
55 | processes (whose effective UID is nonzero). Privileged processes bypass | |
56 | all kernel security permission checks so perf_events performance | |
57 | monitoring is fully available to privileged processes without access, | |
58 | scope and resource restrictions. | |
59 | ||
60 | Unprivileged processes are subject to a full security permission check | |
61 | based on the process's credentials [5]_ (usually: effective UID, | |
62 | effective GID, and supplementary group list). | |
63 | ||
64 | Linux divides the privileges traditionally associated with superuser | |
65 | into distinct units, known as capabilities [6]_ , which can be | |
66 | independently enabled and disabled on per-thread basis for processes and | |
67 | files of unprivileged users. | |
68 | ||
902a8dcc | 69 | Unprivileged processes with enabled CAP_PERFMON capability are treated |
e85a198e | 70 | as privileged processes with respect to perf_events performance |
902a8dcc AB |
71 | monitoring and observability operations, thus, bypass *scope* permissions |
72 | checks in the kernel. CAP_PERFMON implements the principle of least | |
73 | privilege [13]_ (POSIX 1003.1e: 2.2.2.39) for performance monitoring and | |
74 | observability operations in the kernel and provides a secure approach to | |
75 | perfomance monitoring and observability in the system. | |
76 | ||
77 | For backward compatibility reasons the access to perf_events monitoring and | |
78 | observability operations is also open for CAP_SYS_ADMIN privileged | |
79 | processes but CAP_SYS_ADMIN usage for secure monitoring and observability | |
80 | use cases is discouraged with respect to the CAP_PERFMON capability. | |
81 | If system audit records [14]_ for a process using perf_events system call | |
82 | API contain denial records of acquiring both CAP_PERFMON and CAP_SYS_ADMIN | |
83 | capabilities then providing the process with CAP_PERFMON capability singly | |
84 | is recommended as the preferred secure approach to resolve double access | |
85 | denial logging related to usage of performance monitoring and observability. | |
86 | ||
87 | Unprivileged processes using perf_events system call are also subject | |
e85a198e AB |
88 | for PTRACE_MODE_READ_REALCREDS ptrace access mode check [7]_ , whose |
89 | outcome determines whether monitoring is permitted. So unprivileged | |
90 | processes provided with CAP_SYS_PTRACE capability are effectively | |
91 | permitted to pass the check. | |
92 | ||
93 | Other capabilities being granted to unprivileged processes can | |
94 | effectively enable capturing of additional data required for later | |
95 | performance analysis of monitored processes or a system. For example, | |
96 | CAP_SYSLOG capability permits reading kernel space memory addresses from | |
97 | /proc/kallsyms file. | |
76e7fd84 | 98 | |
902a8dcc | 99 | Privileged Perf users groups |
e152c7b7 AB |
100 | --------------------------------- |
101 | ||
e85a198e | 102 | Mechanisms of capabilities, privileged capability-dumb files [6]_ and |
902a8dcc AB |
103 | file system ACLs [10]_ can be used to create dedicated groups of |
104 | privileged Perf users who are permitted to execute performance monitoring | |
105 | and observability without scope limits. The following steps can be | |
106 | taken to create such groups of privileged Perf users. | |
e152c7b7 | 107 | |
e85a198e AB |
108 | 1. Create perf_users group of privileged Perf users, assign perf_users |
109 | group to Perf tool executable and limit access to the executable for | |
110 | other users in the system who are not in the perf_users group: | |
e152c7b7 AB |
111 | |
112 | :: | |
113 | ||
114 | # groupadd perf_users | |
115 | # ls -alhF | |
116 | -rwxr-xr-x 2 root root 11M Oct 19 15:12 perf | |
117 | # chgrp perf_users perf | |
118 | # ls -alhF | |
119 | -rwxr-xr-x 2 root perf_users 11M Oct 19 15:12 perf | |
120 | # chmod o-rwx perf | |
121 | # ls -alhF | |
122 | -rwxr-x--- 2 root perf_users 11M Oct 19 15:12 perf | |
123 | ||
e85a198e | 124 | 2. Assign the required capabilities to the Perf tool executable file and |
902a8dcc | 125 | enable members of perf_users group with monitoring and observability |
e85a198e | 126 | privileges [6]_ : |
e152c7b7 AB |
127 | |
128 | :: | |
129 | ||
902a8dcc AB |
130 | # setcap "cap_perfmon,cap_sys_ptrace,cap_syslog=ep" perf |
131 | # setcap -v "cap_perfmon,cap_sys_ptrace,cap_syslog=ep" perf | |
e152c7b7 AB |
132 | perf: OK |
133 | # getcap perf | |
902a8dcc AB |
134 | perf = cap_sys_ptrace,cap_syslog,cap_perfmon+ep |
135 | ||
136 | If the libcap installed doesn't yet support "cap_perfmon", use "38" instead, | |
137 | i.e.: | |
138 | ||
139 | :: | |
140 | ||
141 | # setcap "38,cap_ipc_lock,cap_sys_ptrace,cap_syslog=ep" perf | |
142 | ||
143 | Note that you may need to have 'cap_ipc_lock' in the mix for tools such as | |
144 | 'perf top', alternatively use 'perf top -m N', to reduce the memory that | |
145 | it uses for the perf ring buffer, see the memory allocation section below. | |
146 | ||
147 | Using a libcap without support for CAP_PERFMON will make cap_get_flag(caps, 38, | |
148 | CAP_EFFECTIVE, &val) fail, which will lead the default event to be 'cycles:u', | |
149 | so as a workaround explicitly ask for the 'cycles' event, i.e.: | |
150 | ||
151 | :: | |
152 | ||
153 | # perf top -e cycles | |
154 | ||
155 | To get kernel and user samples with a perf binary with just CAP_PERFMON. | |
e152c7b7 | 156 | |
e85a198e | 157 | As a result, members of perf_users group are capable of conducting |
902a8dcc AB |
158 | performance monitoring and observability by using functionality of the |
159 | configured Perf tool executable that, when executes, passes perf_events | |
160 | subsystem scope checks. | |
e152c7b7 | 161 | |
e85a198e AB |
162 | This specific access control management is only available to superuser |
163 | or root running processes with CAP_SETPCAP, CAP_SETFCAP [6]_ | |
164 | capabilities. | |
e152c7b7 | 165 | |
902a8dcc | 166 | Unprivileged users |
76e7fd84 AB |
167 | ----------------------------------- |
168 | ||
902a8dcc | 169 | perf_events *scope* and *access* control for unprivileged processes |
e85a198e | 170 | is governed by perf_event_paranoid [2]_ setting: |
76e7fd84 AB |
171 | |
172 | -1: | |
e85a198e AB |
173 | Impose no *scope* and *access* restrictions on using perf_events |
174 | performance monitoring. Per-user per-cpu perf_event_mlock_kb [2]_ | |
175 | locking limit is ignored when allocating memory buffers for storing | |
176 | performance data. This is the least secure mode since allowed | |
177 | monitored *scope* is maximized and no perf_events specific limits | |
178 | are imposed on *resources* allocated for performance monitoring. | |
76e7fd84 AB |
179 | |
180 | >=0: | |
181 | *scope* includes per-process and system wide performance monitoring | |
e85a198e AB |
182 | but excludes raw tracepoints and ftrace function tracepoints |
183 | monitoring. CPU and system events happened when executing either in | |
184 | user or in kernel space can be monitored and captured for later | |
185 | analysis. Per-user per-cpu perf_event_mlock_kb locking limit is | |
186 | imposed but ignored for unprivileged processes with CAP_IPC_LOCK | |
187 | [6]_ capability. | |
76e7fd84 AB |
188 | |
189 | >=1: | |
e85a198e AB |
190 | *scope* includes per-process performance monitoring only and |
191 | excludes system wide performance monitoring. CPU and system events | |
192 | happened when executing either in user or in kernel space can be | |
193 | monitored and captured for later analysis. Per-user per-cpu | |
194 | perf_event_mlock_kb locking limit is imposed but ignored for | |
195 | unprivileged processes with CAP_IPC_LOCK capability. | |
76e7fd84 AB |
196 | |
197 | >=2: | |
e85a198e AB |
198 | *scope* includes per-process performance monitoring only. CPU and |
199 | system events happened when executing in user space only can be | |
200 | monitored and captured for later analysis. Per-user per-cpu | |
201 | perf_event_mlock_kb locking limit is imposed but ignored for | |
202 | unprivileged processes with CAP_IPC_LOCK capability. | |
76e7fd84 | 203 | |
902a8dcc | 204 | Resource control |
9d87bbae AB |
205 | --------------------------------- |
206 | ||
207 | Open file descriptors | |
208 | +++++++++++++++++++++ | |
209 | ||
e85a198e AB |
210 | The perf_events system call API [2]_ allocates file descriptors for |
211 | every configured PMU event. Open file descriptors are a per-process | |
212 | accountable resource governed by the RLIMIT_NOFILE [11]_ limit | |
213 | (ulimit -n), which is usually derived from the login shell process. When | |
214 | configuring Perf collection for a long list of events on a large server | |
215 | system, this limit can be easily hit preventing required monitoring | |
216 | configuration. RLIMIT_NOFILE limit can be increased on per-user basis | |
217 | modifying content of the limits.conf file [12]_ . Ordinarily, a Perf | |
218 | sampling session (perf record) requires an amount of open perf_event | |
219 | file descriptors that is not less than the number of monitored events | |
220 | multiplied by the number of monitored CPUs. | |
9d87bbae AB |
221 | |
222 | Memory allocation | |
223 | +++++++++++++++++ | |
224 | ||
e85a198e AB |
225 | The amount of memory available to user processes for capturing |
226 | performance monitoring data is governed by the perf_event_mlock_kb [2]_ | |
227 | setting. This perf_event specific resource setting defines overall | |
228 | per-cpu limits of memory allowed for mapping by the user processes to | |
229 | execute performance monitoring. The setting essentially extends the | |
230 | RLIMIT_MEMLOCK [11]_ limit, but only for memory regions mapped | |
231 | specifically for capturing monitored performance events and related data. | |
232 | ||
233 | For example, if a machine has eight cores and perf_event_mlock_kb limit | |
234 | is set to 516 KiB, then a user process is provided with 516 KiB * 8 = | |
235 | 4128 KiB of memory above the RLIMIT_MEMLOCK limit (ulimit -l) for | |
236 | perf_event mmap buffers. In particular, this means that, if the user | |
237 | wants to start two or more performance monitoring processes, the user is | |
238 | required to manually distribute the available 4128 KiB between the | |
239 | monitoring processes, for example, using the --mmap-pages Perf record | |
240 | mode option. Otherwise, the first started performance monitoring process | |
241 | allocates all available 4128 KiB and the other processes will fail to | |
242 | proceed due to the lack of memory. | |
243 | ||
244 | RLIMIT_MEMLOCK and perf_event_mlock_kb resource constraints are ignored | |
245 | for processes with the CAP_IPC_LOCK capability. Thus, perf_events/Perf | |
246 | privileged users can be provided with memory above the constraints for | |
247 | perf_events/Perf performance monitoring purpose by providing the Perf | |
248 | executable with CAP_IPC_LOCK capability. | |
9d87bbae | 249 | |
76e7fd84 AB |
250 | Bibliography |
251 | ------------ | |
252 | ||
253 | .. [1] `<https://lwn.net/Articles/337493/>`_ | |
254 | .. [2] `<http://man7.org/linux/man-pages/man2/perf_event_open.2.html>`_ | |
255 | .. [3] `<http://web.eece.maine.edu/~vweaver/projects/perf_events/>`_ | |
256 | .. [4] `<https://perf.wiki.kernel.org/index.php/Main_Page>`_ | |
257 | .. [5] `<https://www.kernel.org/doc/html/latest/security/credentials.html>`_ | |
258 | .. [6] `<http://man7.org/linux/man-pages/man7/capabilities.7.html>`_ | |
259 | .. [7] `<http://man7.org/linux/man-pages/man2/ptrace.2.html>`_ | |
68570ca0 AB |
260 | .. [8] `<https://en.wikipedia.org/wiki/Hardware_performance_counter>`_ |
261 | .. [9] `<https://en.wikipedia.org/wiki/Model-specific_register>`_ | |
e152c7b7 | 262 | .. [10] `<http://man7.org/linux/man-pages/man5/acl.5.html>`_ |
9d87bbae AB |
263 | .. [11] `<http://man7.org/linux/man-pages/man2/getrlimit.2.html>`_ |
264 | .. [12] `<http://man7.org/linux/man-pages/man5/limits.conf.5.html>`_ | |
902a8dcc AB |
265 | .. [13] `<https://sites.google.com/site/fullycapable>`_ |
266 | .. [14] `<http://man7.org/linux/man-pages/man8/auditd.8.html>`_ |