Commit | Line | Data |
---|---|---|
de1ba09b AM |
1 | Fault injection capabilities infrastructure |
2 | =========================================== | |
3 | ||
4 | See also drivers/md/faulty.c and "every_nth" module option for scsi_debug. | |
5 | ||
6 | ||
7 | Available fault injection capabilities | |
8 | -------------------------------------- | |
9 | ||
10 | o failslab | |
11 | ||
12 | injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) | |
13 | ||
14 | o fail_page_alloc | |
15 | ||
16 | injects page allocation failures. (alloc_pages(), get_free_pages(), ...) | |
17 | ||
18 | o fail_make_request | |
19 | ||
20 | injects disk IO errors on permitted devices by | |
21 | /sys/block/<device>/make-it-fail or | |
22 | /sys/block/<device>/<partition>/make-it-fail. (generic_make_request()) | |
23 | ||
24 | Configure fault-injection capabilities behavior | |
25 | ----------------------------------------------- | |
26 | ||
27 | o debugfs entries | |
28 | ||
29 | fault-inject-debugfs kernel module provides some debugfs entries for runtime | |
30 | configuration of fault-injection capabilities. | |
31 | ||
32 | - /debug/*/probability: | |
33 | ||
34 | likelihood of failure injection, in percent. | |
35 | Format: <percent> | |
36 | ||
37 | Note that one-failure-per-handred is a very high error rate | |
38 | for some testcases. Please set probably=100 and configure | |
39 | /debug/*/interval for such testcases. | |
40 | ||
41 | - /debug/*/interval: | |
42 | ||
43 | specifies the interval between failures, for calls to | |
44 | should_fail() that pass all the other tests. | |
45 | ||
46 | Note that if you enable this, by setting interval>1, you will | |
47 | probably want to set probability=100. | |
48 | ||
49 | - /debug/*/times: | |
50 | ||
51 | specifies how many times failures may happen at most. | |
52 | A value of -1 means "no limit". | |
53 | ||
54 | - /debug/*/space: | |
55 | ||
56 | specifies an initial resource "budget", decremented by "size" | |
57 | on each call to should_fail(,size). Failure injection is | |
58 | suppressed until "space" reaches zero. | |
59 | ||
60 | - /debug/*/verbose | |
61 | ||
62 | Format: { 0 | 1 | 2 } | |
63 | specifies the verbosity of the messages when failure is injected. | |
64 | We default to 0 (no extra messages), setting it to '1' will | |
65 | print only to tell failure happened, '2' will print call trace too - | |
66 | it is useful to debug the problems revealed by fault injection | |
67 | capabilities. | |
68 | ||
69 | - /debug/*/task-filter: | |
70 | ||
71 | Format: { 0 | 1 } | |
72 | A value of '0' disables filtering by process (default). | |
73 | Any positive value limits failures to only processes indicated by | |
74 | /proc/<pid>/make-it-fail==1. | |
75 | ||
329409ae AM |
76 | - /debug/*/require-start: |
77 | - /debug/*/require-end: | |
78 | - /debug/*/reject-start: | |
79 | - /debug/*/reject-end: | |
de1ba09b AM |
80 | |
81 | specifies the range of virtual addresses tested during | |
82 | stacktrace walking. Failure is injected only if some caller | |
329409ae AM |
83 | in the walked stacktrace lies within the required range, and |
84 | none lies within the rejected range. | |
85 | Default required range is [0,ULONG_MAX) (whole of virtual address space). | |
86 | Default rejected range is [0,0). | |
de1ba09b AM |
87 | |
88 | - /debug/*/stacktrace-depth: | |
89 | ||
90 | specifies the maximum stacktrace depth walked during search | |
91 | for a caller within [address-start,address-end). | |
92 | ||
de1ba09b AM |
93 | - /debug/fail_page_alloc/ignore-gfp-highmem: |
94 | ||
95 | Format: { 0 | 1 } | |
96 | default is 0, setting it to '1' won't inject failures into | |
97 | highmem/user allocations. | |
98 | ||
99 | - /debug/failslab/ignore-gfp-wait: | |
100 | - /debug/fail_page_alloc/ignore-gfp-wait: | |
101 | ||
102 | Format: { 0 | 1 } | |
103 | default is 0, setting it to '1' will inject failures | |
104 | only into non-sleep allocations (GFP_ATOMIC allocations). | |
105 | ||
106 | o Boot option | |
107 | ||
108 | In order to inject faults while debugfs is not available (early boot time), | |
109 | use the boot option: | |
110 | ||
111 | failslab= | |
112 | fail_page_alloc= | |
113 | fail_make_request=<interval>,<probability>,<space>,<times> | |
114 | ||
115 | How to add new fault injection capability | |
116 | ----------------------------------------- | |
117 | ||
118 | o #include <linux/fault-inject.h> | |
119 | ||
120 | o define the fault attributes | |
121 | ||
122 | DECLARE_FAULT_INJECTION(name); | |
123 | ||
124 | Please see the definition of struct fault_attr in fault-inject.h | |
125 | for details. | |
126 | ||
127 | o provide the way to configure fault attributes | |
128 | ||
129 | - boot option | |
130 | ||
131 | If you need to enable the fault injection capability from boot time, you can | |
132 | provide boot option to configure it. There is a helper function for it. | |
133 | ||
134 | setup_fault_attr(attr, str); | |
135 | ||
136 | - debugfs entries | |
137 | ||
138 | failslab, fail_page_alloc, and fail_make_request use this way. | |
139 | There is a helper function for it. | |
140 | ||
141 | init_fault_attr_entries(entries, attr, name); | |
142 | void cleanup_fault_attr_entries(entries); | |
143 | ||
144 | - module parameters | |
145 | ||
146 | If the scope of the fault injection capability is limited to a | |
147 | single kernel module, it is better to provide module parameters to | |
148 | configure the fault attributes. | |
149 | ||
150 | o add a hook to insert failures | |
151 | ||
152 | should_fail() returns 1 when failures should happen. | |
153 | ||
154 | should_fail(attr,size); | |
155 | ||
156 | Application Examples | |
157 | -------------------- | |
158 | ||
159 | o inject slab allocation failures into module init/cleanup code | |
160 | ||
161 | ------------------------------------------------------------------------------ | |
162 | #!/bin/bash | |
163 | ||
164 | FAILCMD=Documentation/fault-injection/failcmd.sh | |
165 | BLACKLIST="root_plug evbug" | |
166 | ||
167 | FAILNAME=failslab | |
168 | echo Y > /debug/$FAILNAME/task-filter | |
169 | echo 10 > /debug/$FAILNAME/probability | |
170 | echo 100 > /debug/$FAILNAME/interval | |
171 | echo -1 > /debug/$FAILNAME/times | |
172 | echo 2 > /debug/$FAILNAME/verbose | |
de1ba09b AM |
173 | echo 1 > /debug/$FAILNAME/ignore-gfp-wait |
174 | ||
175 | blacklist() | |
176 | { | |
177 | echo $BLACKLIST | grep $1 > /dev/null 2>&1 | |
178 | } | |
179 | ||
180 | oops() | |
181 | { | |
182 | dmesg | grep BUG > /dev/null 2>&1 | |
183 | } | |
184 | ||
185 | find /lib/modules/`uname -r` -name '*.ko' -exec basename {} .ko \; | | |
186 | while read i | |
187 | do | |
188 | oops && exit 1 | |
189 | ||
190 | if ! blacklist $i | |
191 | then | |
192 | echo inserting $i... | |
193 | bash $FAILCMD modprobe $i | |
194 | fi | |
195 | done | |
196 | ||
197 | lsmod | awk '{ if ($3 == 0) { print $1 } }' | | |
198 | while read i | |
199 | do | |
200 | oops && exit 1 | |
201 | ||
202 | if ! blacklist $i | |
203 | then | |
204 | echo removing $i... | |
205 | bash $FAILCMD modprobe -r $i | |
206 | fi | |
207 | done | |
208 | ||
209 | ------------------------------------------------------------------------------ | |
210 | ||
211 | o inject slab allocation failures only for a specific module | |
212 | ||
213 | ------------------------------------------------------------------------------ | |
214 | #!/bin/bash | |
215 | ||
216 | FAILMOD=Documentation/fault-injection/failmodule.sh | |
217 | ||
218 | echo injecting errors into the module $1... | |
219 | ||
220 | modprobe $1 | |
221 | bash $FAILMOD failslab $1 10 | |
222 | echo 25 > /debug/failslab/probability | |
223 | ||
224 | ------------------------------------------------------------------------------ | |
225 |