Commit | Line | Data |
---|---|---|
a241ec65 PM |
1 | RCU Torture Test Operation |
2 | ||
3 | ||
4 | CONFIG_RCU_TORTURE_TEST | |
5 | ||
6 | The CONFIG_RCU_TORTURE_TEST config option is available for all RCU | |
7 | implementations. It creates an rcutorture kernel module that can | |
8 | be loaded to run a torture test. The test periodically outputs | |
9 | status messages via printk(), which can be examined via the dmesg | |
72e9bb54 | 10 | command (perhaps grepping for "torture"). The test is started |
a241ec65 PM |
11 | when the module is loaded, and stops when the module is unloaded. |
12 | ||
6684880a JW |
13 | Module parameters are prefixed by "rcutorture." in |
14 | Documentation/admin-guide/kernel-parameters.txt. | |
a241ec65 PM |
15 | |
16 | OUTPUT | |
17 | ||
18 | The statistics output is as follows: | |
19 | ||
63cd758e | 20 | rcu-torture:--- Start of test: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 |
fae4b54f | 21 | rcu-torture: rtc: (null) ver: 155441 tfle: 0 rta: 155441 rtaf: 8884 rtf: 155440 rtmbe: 0 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 3055767 |
63cd758e PM |
22 | rcu-torture: Reader Pipe: 727860534 34213 0 0 0 0 0 0 0 0 0 |
23 | rcu-torture: Reader Batch: 727877838 17003 0 0 0 0 0 0 0 0 0 | |
24 | rcu-torture: Free-Block Circulation: 155440 155440 155440 155440 155440 155440 155440 155440 155440 155440 0 | |
25 | rcu-torture:--- End of test: SUCCESS: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 | |
a241ec65 | 26 | |
72e9bb54 | 27 | The command "dmesg | grep torture:" will extract this information on |
a241ec65 PM |
28 | most systems. On more esoteric configurations, it may be necessary to |
29 | use other commands to access the output of the printk()s used by | |
30 | the RCU torture test. The printk()s use KERN_ALERT, so they should | |
31 | be evident. ;-) | |
32 | ||
63cd758e PM |
33 | The first and last lines show the rcutorture module parameters, and the |
34 | last line shows either "SUCCESS" or "FAILURE", based on rcutorture's | |
35 | automatic determination as to whether RCU operated correctly. | |
36 | ||
a241ec65 PM |
37 | The entries are as follows: |
38 | ||
a241ec65 PM |
39 | o "rtc": The hexadecimal address of the structure currently visible |
40 | to readers. | |
41 | ||
63cd758e | 42 | o "ver": The number of times since boot that the RCU writer task |
a241ec65 PM |
43 | has changed the structure visible to readers. |
44 | ||
45 | o "tfle": If non-zero, indicates that the "torture freelist" | |
63cd758e | 46 | containing structures to be placed into the "rtc" area is empty. |
a241ec65 PM |
47 | This condition is important, since it can fool you into thinking |
48 | that RCU is working when it is not. :-/ | |
49 | ||
50 | o "rta": Number of structures allocated from the torture freelist. | |
51 | ||
52 | o "rtaf": Number of allocations from the torture freelist that have | |
63cd758e PM |
53 | failed due to the list being empty. It is not unusual for this |
54 | to be non-zero, but it is bad for it to be a large fraction of | |
55 | the value indicated by "rta". | |
a241ec65 PM |
56 | |
57 | o "rtf": Number of frees into the torture freelist. | |
58 | ||
63cd758e PM |
59 | o "rtmbe": A non-zero value indicates that rcutorture believes that |
60 | rcu_assign_pointer() and rcu_dereference() are not working | |
61 | correctly. This value should be zero. | |
62 | ||
fae4b54f PM |
63 | o "rtbe": A non-zero value indicates that one of the rcu_barrier() |
64 | family of functions is not working correctly. | |
65 | ||
63cd758e PM |
66 | o "rtbke": rcutorture was unable to create the real-time kthreads |
67 | used to force RCU priority inversion. This value should be zero. | |
68 | ||
69 | o "rtbre": Although rcutorture successfully created the kthreads | |
70 | used to force RCU priority inversion, it was unable to set them | |
71 | to the real-time priority level of 1. This value should be zero. | |
72 | ||
73 | o "rtbf": The number of times that RCU priority boosting failed | |
74 | to resolve RCU priority inversion. | |
75 | ||
76 | o "rtb": The number of times that rcutorture attempted to force | |
77 | an RCU priority inversion condition. If you are testing RCU | |
78 | priority boosting via the "test_boost" module parameter, this | |
79 | value should be non-zero. | |
80 | ||
81 | o "nt": The number of times rcutorture ran RCU read-side code from | |
82 | within a timer handler. This value should be non-zero only | |
83 | if you specified the "irqreader" module parameter. | |
84 | ||
a241ec65 PM |
85 | o "Reader Pipe": Histogram of "ages" of structures seen by readers. |
86 | If any entries past the first two are non-zero, RCU is broken. | |
87 | And rcutorture prints the error flag string "!!!" to make sure | |
88 | you notice. The age of a newly allocated structure is zero, | |
89 | it becomes one when removed from reader visibility, and is | |
90 | incremented once per grace period subsequently -- and is freed | |
91 | after passing through (RCU_TORTURE_PIPE_LEN-2) grace periods. | |
92 | ||
93 | The output displayed above was taken from a correctly working | |
94 | RCU. If you want to see what it looks like when broken, break | |
95 | it yourself. ;-) | |
96 | ||
97 | o "Reader Batch": Another histogram of "ages" of structures seen | |
98 | by readers, but in terms of counter flips (or batches) rather | |
99 | than in terms of grace periods. The legal number of non-zero | |
f85d6c71 PM |
100 | entries is again two. The reason for this separate view is that |
101 | it is sometimes easier to get the third entry to show up in the | |
a241ec65 PM |
102 | "Reader Batch" list than in the "Reader Pipe" list. |
103 | ||
104 | o "Free-Block Circulation": Shows the number of torture structures | |
105 | that have reached a given point in the pipeline. The first element | |
106 | should closely correspond to the number of structures allocated, | |
107 | the second to the number that have been removed from reader view, | |
108 | and all but the last remaining to the corresponding number of | |
109 | passes through a grace period. The last entry should be zero, | |
110 | as it is only incremented if a torture structure's counter | |
111 | somehow gets incremented farther than it should. | |
112 | ||
b2896d2e | 113 | Different implementations of RCU can provide implementation-specific |
4de5f89e | 114 | additional information. For example, Tree SRCU provides the following |
63cd758e | 115 | additional line: |
b2896d2e | 116 | |
4de5f89e | 117 | srcud-torture: Tree SRCU per-CPU(idx=0): 0(35,-21) 1(-4,24) 2(1,1) 3(-26,20) 4(28,-47) 5(-9,4) 6(-10,14) 7(-14,11) T(1,6) |
b2896d2e | 118 | |
4de5f89e PM |
119 | This line shows the per-CPU counter state, in this case for Tree SRCU |
120 | using a dynamically allocated srcu_struct (hence "srcud-" rather than | |
121 | "srcu-"). The numbers in parentheses are the values of the "old" and | |
122 | "current" counters for the corresponding CPU. The "idx" value maps the | |
123 | "old" and "current" values to the underlying array, and is useful for | |
124 | debugging. The final "T" entry contains the totals of the counters. | |
240ebbf8 | 125 | |
a241ec65 PM |
126 | |
127 | USAGE | |
128 | ||
129 | The following script may be used to torture RCU: | |
130 | ||
131 | #!/bin/sh | |
132 | ||
133 | modprobe rcutorture | |
105617da | 134 | sleep 3600 |
a241ec65 | 135 | rmmod rcutorture |
72e9bb54 | 136 | dmesg | grep torture: |
a241ec65 PM |
137 | |
138 | The output can be manually inspected for the error flag of "!!!". | |
139 | One could of course create a more elaborate script that automatically | |
9b9ec9b9 PM |
140 | checked for such errors. The "rmmod" command forces a "SUCCESS", |
141 | "FAILURE", or "RCU_HOTPLUG" indication to be printk()ed. The first | |
142 | two are self-explanatory, while the last indicates that while there | |
143 | were no RCU failures, CPU-hotplug problems were detected. | |
4de5f89e PM |
144 | |
145 | However, the tools/testing/selftests/rcutorture/bin/kvm.sh script | |
146 | provides better automation, including automatic failure analysis. | |
147 | It assumes a qemu/kvm-enabled platform, and runs guest OSes out of initrd. | |
148 | See tools/testing/selftests/rcutorture/doc/initrd.txt for instructions | |
149 | on setting up such an initrd. |