Commit | Line | Data |
---|---|---|
1da177e4 LT |
1 | Tools that manage md devices can be found at |
2 | http://www.<country>.kernel.org/pub/linux/utils/raid/.... | |
3 | ||
4 | ||
5 | Boot time assembly of RAID arrays | |
6 | --------------------------------- | |
7 | ||
8 | You can boot with your md device with the following kernel command | |
9 | lines: | |
10 | ||
11 | for old raid arrays without persistent superblocks: | |
12 | md=<md device no.>,<raid level>,<chunk size factor>,<fault level>,dev0,dev1,...,devn | |
13 | ||
14 | for raid arrays with persistent superblocks | |
15 | md=<md device no.>,dev0,dev1,...,devn | |
16 | or, to assemble a partitionable array: | |
17 | md=d<md device no.>,dev0,dev1,...,devn | |
18 | ||
19 | md device no. = the number of the md device ... | |
20 | 0 means md0, | |
21 | 1 md1, | |
22 | 2 md2, | |
23 | 3 md3, | |
24 | 4 md4 | |
25 | ||
26 | raid level = -1 linear mode | |
27 | 0 striped mode | |
28 | other modes are only supported with persistent super blocks | |
29 | ||
30 | chunk size factor = (raid-0 and raid-1 only) | |
31 | Set the chunk size as 4k << n. | |
32 | ||
33 | fault level = totally ignored | |
34 | ||
35 | dev0-devn: e.g. /dev/hda1,/dev/hdc1,/dev/sda1,/dev/sdb1 | |
36 | ||
37 | A possible loadlin line (Harald Hoyer <HarryH@Royal.Net>) looks like this: | |
38 | ||
39 | e:\loadlin\loadlin e:\zimage root=/dev/md0 md=0,0,4,0,/dev/hdb2,/dev/hdc3 ro | |
40 | ||
41 | ||
42 | Boot time autodetection of RAID arrays | |
43 | -------------------------------------- | |
44 | ||
45 | When md is compiled into the kernel (not as module), partitions of | |
46 | type 0xfd are scanned and automatically assembled into RAID arrays. | |
47 | This autodetection may be suppressed with the kernel parameter | |
48 | "raid=noautodetect". As of kernel 2.6.9, only drives with a type 0 | |
49 | superblock can be autodetected and run at boot time. | |
50 | ||
51 | The kernel parameter "raid=partitionable" (or "raid=part") means | |
52 | that all auto-detected arrays are assembled as partitionable. | |
53 | ||
6ff8d8ec N |
54 | Boot time assembly of degraded/dirty arrays |
55 | ------------------------------------------- | |
56 | ||
57 | If a raid5 or raid6 array is both dirty and degraded, it could have | |
58 | undetectable data corruption. This is because the fact that it is | |
59 | 'dirty' means that the parity cannot be trusted, and the fact that it | |
60 | is degraded means that some datablocks are missing and cannot reliably | |
61 | be reconstructed (due to no parity). | |
62 | ||
63 | For this reason, md will normally refuse to start such an array. This | |
64 | requires the sysadmin to take action to explicitly start the array | |
fff9289b | 65 | despite possible corruption. This is normally done with |
6ff8d8ec N |
66 | mdadm --assemble --force .... |
67 | ||
68 | This option is not really available if the array has the root | |
69 | filesystem on it. In order to support this booting from such an | |
70 | array, md supports a module parameter "start_dirty_degraded" which, | |
71 | when set to 1, bypassed the checks and will allows dirty degraded | |
72 | arrays to be started. | |
73 | ||
74 | So, to boot with a root filesystem of a dirty degraded raid[56], use | |
75 | ||
76 | md-mod.start_dirty_degraded=1 | |
77 | ||
1da177e4 LT |
78 | |
79 | Superblock formats | |
80 | ------------------ | |
81 | ||
82 | The md driver can support a variety of different superblock formats. | |
83 | Currently, it supports superblock formats "0.90.0" and the "md-1" format | |
84 | introduced in the 2.5 development series. | |
85 | ||
86 | The kernel will autodetect which format superblock is being used. | |
87 | ||
88 | Superblock format '0' is treated differently to others for legacy | |
89 | reasons - it is the original superblock format. | |
90 | ||
91 | ||
92 | General Rules - apply for all superblock formats | |
93 | ------------------------------------------------ | |
94 | ||
95 | An array is 'created' by writing appropriate superblocks to all | |
96 | devices. | |
97 | ||
98 | It is 'assembled' by associating each of these devices with an | |
99 | particular md virtual device. Once it is completely assembled, it can | |
100 | be accessed. | |
101 | ||
102 | An array should be created by a user-space tool. This will write | |
103 | superblocks to all devices. It will usually mark the array as | |
104 | 'unclean', or with some devices missing so that the kernel md driver | |
105 | can create appropriate redundancy (copying in raid1, parity | |
106 | calculation in raid4/5). | |
107 | ||
108 | When an array is assembled, it is first initialized with the | |
109 | SET_ARRAY_INFO ioctl. This contains, in particular, a major and minor | |
110 | version number. The major version number selects which superblock | |
111 | format is to be used. The minor number might be used to tune handling | |
112 | of the format, such as suggesting where on each device to look for the | |
113 | superblock. | |
114 | ||
115 | Then each device is added using the ADD_NEW_DISK ioctl. This | |
116 | provides, in particular, a major and minor number identifying the | |
117 | device to add. | |
118 | ||
119 | The array is started with the RUN_ARRAY ioctl. | |
120 | ||
121 | Once started, new devices can be added. They should have an | |
122 | appropriate superblock written to them, and then passed be in with | |
123 | ADD_NEW_DISK. | |
124 | ||
125 | Devices that have failed or are not yet active can be detached from an | |
126 | array using HOT_REMOVE_DISK. | |
127 | ||
128 | ||
129 | Specific Rules that apply to format-0 super block arrays, and | |
130 | arrays with no superblock (non-persistent). | |
131 | ------------------------------------------------------------- | |
132 | ||
133 | An array can be 'created' by describing the array (level, chunksize | |
134 | etc) in a SET_ARRAY_INFO ioctl. This must has major_version==0 and | |
135 | raid_disks != 0. | |
136 | ||
137 | Then uninitialized devices can be added with ADD_NEW_DISK. The | |
138 | structure passed to ADD_NEW_DISK must specify the state of the device | |
139 | and it's role in the array. | |
140 | ||
141 | Once started with RUN_ARRAY, uninitialized spares can be added with | |
142 | HOT_ADD_DISK. | |
bb636547 N |
143 | |
144 | ||
145 | ||
146 | MD devices in sysfs | |
147 | ------------------- | |
148 | md devices appear in sysfs (/sys) as regular block devices, | |
149 | e.g. | |
150 | /sys/block/md0 | |
151 | ||
152 | Each 'md' device will contain a subdirectory called 'md' which | |
153 | contains further md-specific information about the device. | |
154 | ||
155 | All md devices contain: | |
156 | level | |
d33a56d3 N |
157 | a text file indicating the 'raid level'. e.g. raid0, raid1, |
158 | raid5, linear, multipath, faulty. | |
bb636547 | 159 | If no raid level has been set yet (array is still being |
d33a56d3 N |
160 | assembled), the value will reflect whatever has been written |
161 | to it, which may be a name like the above, or may be a number | |
162 | such as '0', '5', etc. | |
bb636547 N |
163 | |
164 | raid_disks | |
165 | a text file with a simple number indicating the number of devices | |
166 | in a fully functional array. If this is not yet known, the file | |
167 | will be empty. If an array is being resized (not currently | |
168 | possible) this will contain the larger of the old and new sizes. | |
da943b99 N |
169 | Some raid level (RAID1) allow this value to be set while the |
170 | array is active. This will reconfigure the array. Otherwise | |
171 | it can only be set while assembling an array. | |
bb636547 | 172 | |
3b34380a N |
173 | chunk_size |
174 | This is the size if bytes for 'chunks' and is only relevant to | |
175 | raid levels that involve striping (1,4,5,6,10). The address space | |
176 | of the array is conceptually divided into chunks and consecutive | |
177 | chunks are striped onto neighbouring devices. | |
3f6dee9b | 178 | The size should be at least PAGE_SIZE (4k) and should be a power |
3b34380a N |
179 | of 2. This can only be set while assembling an array |
180 | ||
08a02ecd N |
181 | layout |
182 | The "layout" for the array for the particular level. This is | |
183 | simply a number that is interpretted differently by different | |
184 | levels. It can be written while assembling an array. | |
185 | ||
186 | reshape_position | |
187 | This is either "none" or a sector number within the devices of | |
188 | the array where "reshape" is up to. If this is set, the three | |
189 | attributes mentioned above (raid_disks, chunk_size, layout) can | |
190 | potentially have 2 values, an old and a new value. If these | |
191 | values differ, reading the attribute returns | |
192 | new (old) | |
193 | and writing will effect the 'new' value, leaving the 'old' | |
194 | unchanged. | |
195 | ||
a35b0d69 N |
196 | component_size |
197 | For arrays with data redundancy (i.e. not raid0, linear, faulty, | |
198 | multipath), all components must be the same size - or at least | |
199 | there must a size that they all provide space for. This is a key | |
200 | part or the geometry of the array. It is measured in sectors | |
201 | and can be read from here. Writing to this value may resize | |
202 | the array if the personality supports it (raid1, raid5, raid6), | |
203 | and if the component drives are large enough. | |
204 | ||
8bb93aac N |
205 | metadata_version |
206 | This indicates the format that is being used to record metadata | |
207 | about the array. It can be 0.90 (traditional format), 1.0, 1.1, | |
208 | 1.2 (newer format in varying locations) or "none" indicating that | |
209 | the kernel isn't managing metadata at all. | |
210 | ||
a94213b1 N |
211 | resync_start |
212 | The point at which resync should start. If no resync is needed, | |
213 | this will be a very large number. At array creation it will | |
214 | default to 0, though starting the array as 'clean' will | |
215 | set it much larger. | |
216 | ||
6d7ff738 N |
217 | new_dev |
218 | This file can be written but not read. The value written should | |
219 | be a block device number as major:minor. e.g. 8:0 | |
220 | This will cause that device to be attached to the array, if it is | |
221 | available. It will then appear at md/dev-XXX (depending on the | |
222 | name of the device) and further configuration is then possible. | |
223 | ||
16f17b39 N |
224 | safe_mode_delay |
225 | When an md array has seen no write requests for a certain period | |
226 | of time, it will be marked as 'clean'. When another write | |
fff9289b ML |
227 | request arrives, the array is marked as 'dirty' before the write |
228 | commences. This is known as 'safe_mode'. | |
16f17b39 N |
229 | The 'certain period' is controlled by this file which stores the |
230 | period as a number of seconds. The default is 200msec (0.200). | |
231 | Writing a value of 0 disables safemode. | |
232 | ||
9e653b63 N |
233 | array_state |
234 | This file contains a single word which describes the current | |
235 | state of the array. In many cases, the state can be set by | |
236 | writing the word for the desired state, however some states | |
237 | cannot be explicitly set, and some transitions are not allowed. | |
238 | ||
0fd62b86 NB |
239 | Select/poll works on this file. All changes except between |
240 | active_idle and active (which can be frequent and are not | |
241 | very interesting) are notified. active->active_idle is | |
242 | reported if the metadata is externally managed. | |
243 | ||
9e653b63 N |
244 | clear |
245 | No devices, no size, no level | |
246 | Writing is equivalent to STOP_ARRAY ioctl | |
247 | inactive | |
248 | May have some settings, but array is not active | |
249 | all IO results in error | |
250 | When written, doesn't tear down array, but just stops it | |
251 | suspended (not supported yet) | |
252 | All IO requests will block. The array can be reconfigured. | |
253 | Writing this, if accepted, will block until array is quiessent | |
254 | readonly | |
255 | no resync can happen. no superblocks get written. | |
256 | write requests fail | |
257 | read-auto | |
258 | like readonly, but behaves like 'clean' on a write request. | |
259 | ||
260 | clean - no pending writes, but otherwise active. | |
261 | When written to inactive array, starts without resync | |
262 | If a write request arrives then | |
263 | if metadata is known, mark 'dirty' and switch to 'active'. | |
264 | if not known, block and switch to write-pending | |
265 | If written to an active array that has pending writes, then fails. | |
266 | active | |
267 | fully active: IO and resync can be happening. | |
268 | When written to inactive array, starts with resync | |
269 | ||
270 | write-pending | |
271 | clean, but writes are blocked waiting for 'active' to be written. | |
272 | ||
273 | active-idle | |
274 | like active, but no writes have been seen for a while (safe_mode_delay). | |
275 | ||
276 | ||
bb636547 N |
277 | As component devices are added to an md array, they appear in the 'md' |
278 | directory as new directories named | |
279 | dev-XXX | |
280 | where XXX is a name that the kernel knows for the device, e.g. hdb1. | |
281 | Each directory contains: | |
282 | ||
283 | block | |
284 | a symlink to the block device in /sys/block, e.g. | |
285 | /sys/block/md0/md/dev-hdb1/block -> ../../../../block/hdb/hdb1 | |
286 | ||
287 | super | |
288 | A file containing an image of the superblock read from, or | |
289 | written to, that device. | |
290 | ||
291 | state | |
292 | A file recording the current state of the device in the array | |
293 | which can be a comma separated list of | |
294 | faulty - device has been kicked from active use due to | |
295 | a detected fault | |
296 | in_sync - device is a fully in-sync member of the array | |
f655675b N |
297 | writemostly - device will only be subject to read |
298 | requests if there are no other options. | |
299 | This applies only to raid1 arrays. | |
52664732 NB |
300 | blocked - device has failed, metadata is "external", |
301 | and the failure hasn't been acknowledged yet. | |
302 | Writes that would write to this device if | |
303 | it were not faulty are blocked. | |
bb636547 N |
304 | spare - device is working, but not a full member. |
305 | This includes spares that are in the process | |
d6bc8ac9 ML |
306 | of being recovered to |
307 | This list may grow in future. | |
45dc2de1 N |
308 | This can be written to. |
309 | Writing "faulty" simulates a failure on the device. | |
310 | Writing "remove" removes the device from the array. | |
f655675b N |
311 | Writing "writemostly" sets the writemostly flag. |
312 | Writing "-writemostly" clears the writemostly flag. | |
52664732 NB |
313 | Writing "blocked" sets the "blocked" flag. |
314 | Writing "-blocked" clear the "blocked" flag and allows writes | |
315 | to complete. | |
316 | ||
317 | This file responds to select/poll. Any change to 'faulty' | |
318 | or 'blocked' causes an event. | |
bb636547 | 319 | |
4dbcdc75 N |
320 | errors |
321 | An approximate count of read errors that have been detected on | |
322 | this device but have not caused the device to be evicted from | |
323 | the array (either because they were corrected or because they | |
324 | happened while the array was read-only). When using version-1 | |
325 | metadata, this value persists across restarts of the array. | |
326 | ||
327 | This value can be written while assembling an array thus | |
328 | providing an ongoing count for arrays with metadata managed by | |
329 | userspace. | |
330 | ||
014236d2 N |
331 | slot |
332 | This gives the role that the device has in the array. It will | |
333 | either be 'none' if the device is not active in the array | |
334 | (i.e. is a spare or has failed) or an integer less than the | |
992caacf | 335 | 'raid_disks' number for the array indicating which position |
014236d2 N |
336 | it currently fills. This can only be set while assembling an |
337 | array. A device for which this is set is assumed to be working. | |
338 | ||
93c8cad0 N |
339 | offset |
340 | This gives the location in the device (in sectors from the | |
341 | start) where data from the array will be stored. Any part of | |
342 | the device before this offset us not touched, unless it is | |
343 | used for storing metadata (Formats 1.1 and 1.2). | |
344 | ||
83303b61 N |
345 | size |
346 | The amount of the device, after the offset, that can be used | |
347 | for storage of data. This will normally be the same as the | |
348 | component_size. This can be written while assembling an | |
349 | array. If a value less than the current component_size is | |
d7027458 | 350 | written, it will be rejected. |
83303b61 | 351 | |
bb636547 N |
352 | |
353 | An active md device will also contain and entry for each active device | |
354 | in the array. These are named | |
355 | ||
356 | rdNN | |
357 | ||
992caacf | 358 | where 'NN' is the position in the array, starting from 0. |
bb636547 N |
359 | So for a 3 drive array there will be rd0, rd1, rd2. |
360 | These are symbolic links to the appropriate 'dev-XXX' entry. | |
361 | Thus, for example, | |
362 | cat /sys/block/md*/md/rd*/state | |
363 | will show 'in_sync' on every line. | |
364 | ||
365 | ||
366 | ||
367 | Active md devices for levels that support data redundancy (1,4,5,6) | |
368 | also have | |
369 | ||
370 | sync_action | |
371 | a text file that can be used to monitor and control the rebuild | |
372 | process. It contains one word which can be one of: | |
373 | resync - redundancy is being recalculated after unclean | |
374 | shutdown or creation | |
375 | recover - a hot spare is being built to replace a | |
376 | failed/missing device | |
377 | idle - nothing is happening | |
378 | check - A full check of redundancy was requested and is | |
379 | happening. This reads all block and checks | |
380 | them. A repair may also happen for some raid | |
381 | levels. | |
382 | repair - A full check and repair is happening. This is | |
383 | similar to 'resync', but was requested by the | |
384 | user, and the write-intent bitmap is NOT used to | |
385 | optimise the process. | |
386 | ||
387 | This file is writable, and each of the strings that could be | |
388 | read are meaningful for writing. | |
389 | ||
390 | 'idle' will stop an active resync/recovery etc. There is no | |
391 | guarantee that another resync/recovery may not be automatically | |
392 | started again, though some event will be needed to trigger | |
393 | this. | |
394 | 'resync' or 'recovery' can be used to restart the | |
395 | corresponding operation if it was stopped with 'idle'. | |
396 | 'check' and 'repair' will start the appropriate process | |
397 | providing the current state is 'idle'. | |
398 | ||
72a23c21 NB |
399 | This file responds to select/poll. Any important change in the value |
400 | triggers a poll event. Sometimes the value will briefly be | |
401 | "recover" if a recovery seems to be needed, but cannot be | |
402 | achieved. In that case, the transition to "recover" isn't | |
403 | notified, but the transition away is. | |
404 | ||
a99ac971 NB |
405 | degraded |
406 | This contains a count of the number of devices by which the | |
407 | arrays is degraded. So an optimal array with show '0'. A | |
408 | single failed/missing drive will show '1', etc. | |
409 | This file responds to select/poll, any increase or decrease | |
410 | in the count of missing devices will trigger an event. | |
411 | ||
bb636547 N |
412 | mismatch_count |
413 | When performing 'check' and 'repair', and possibly when | |
414 | performing 'resync', md will count the number of errors that are | |
415 | found. The count in 'mismatch_cnt' is the number of sectors | |
416 | that were re-written, or (for 'check') would have been | |
417 | re-written. As most raid levels work in units of pages rather | |
418 | than sectors, this my be larger than the number of actual errors | |
419 | by a factor of the number of sectors in a page. | |
420 | ||
9b1d1dac PC |
421 | bitmap_set_bits |
422 | If the array has a write-intent bitmap, then writing to this | |
423 | attribute can set bits in the bitmap, indicating that a resync | |
424 | would need to check the corresponding blocks. Either individual | |
425 | numbers or start-end pairs can be written. Multiple numbers | |
426 | can be separated by a space. | |
427 | Note that the numbers are 'bit' numbers, not 'block' numbers. | |
428 | They should be scaled by the bitmap_chunksize. | |
429 | ||
08a02ecd N |
430 | sync_speed_min |
431 | sync_speed_max | |
432 | This are similar to /proc/sys/dev/raid/speed_limit_{min,max} | |
433 | however they only apply to the particular array. | |
434 | If no value has been written to these, of if the word 'system' | |
435 | is written, then the system-wide value is used. If a value, | |
436 | in kibibytes-per-second is written, then it is used. | |
437 | When the files are read, they show the currently active value | |
438 | followed by "(local)" or "(system)" depending on whether it is | |
439 | a locally set or system-wide value. | |
440 | ||
441 | sync_completed | |
442 | This shows the number of sectors that have been completed of | |
443 | whatever the current sync_action is, followed by the number of | |
444 | sectors in total that could need to be processed. The two | |
445 | numbers are separated by a '/' thus effectively showing one | |
446 | value, a fraction of the process that is complete. | |
c6207277 N |
447 | A 'select' on this attribute will return when resync completes, |
448 | when it reaches the current sync_max (below) and possibly at | |
449 | other times. | |
450 | ||
451 | sync_max | |
452 | This is a number of sectors at which point a resync/recovery | |
453 | process will pause. When a resync is active, the value can | |
454 | only ever be increased, never decreased. The value of 'max' | |
455 | effectively disables the limit. | |
456 | ||
08a02ecd N |
457 | |
458 | sync_speed | |
459 | This shows the current actual speed, in K/sec, of the current | |
460 | sync_action. It is averaged over the last 30 seconds. | |
461 | ||
462 | suspend_lo | |
463 | suspend_hi | |
464 | The two values, given as numbers of sectors, indicate a range | |
465 | within the array where IO will be blocked. This is currently | |
466 | only supported for raid4/5/6. | |
467 | ||
468 | ||
bb636547 N |
469 | Each active md device may also have attributes specific to the |
470 | personality module that manages it. | |
471 | These are specific to the implementation of the module and could | |
472 | change substantially if the implementation changes. | |
473 | ||
474 | These currently include | |
475 | ||
476 | stripe_cache_size (currently raid5 only) | |
477 | number of entries in the stripe cache. This is writable, but | |
478 | there are upper and lower limits (32768, 16). Default is 128. | |
479 | strip_cache_active (currently raid5 only) | |
480 | number of active entries in the stripe cache | |
8b3e6cdc DW |
481 | preread_bypass_threshold (currently raid5 only) |
482 | number of times a stripe requiring preread will be bypassed by | |
483 | a stripe that does not require preread. For fairness defaults | |
484 | to 1. Setting this to 0 disables bypass accounting and | |
485 | requires preread stripes to wait until all full-width stripe- | |
486 | writes are complete. Valid values are 0 to stripe_cache_size. |