Commit | Line | Data |
---|---|---|
70c83e11 TH |
1 | |
2 | SCSI EH | |
3 | ====================================== | |
4 | ||
5 | This document describes SCSI midlayer error handling infrastructure. | |
6 | Please refer to Documentation/scsi/scsi_mid_low_api.txt for more | |
7 | information regarding SCSI midlayer. | |
8 | ||
9 | TABLE OF CONTENTS | |
10 | ||
11 | [1] How SCSI commands travel through the midlayer and to EH | |
12 | [1-1] struct scsi_cmnd | |
13 | [1-2] How do scmd's get completed? | |
14 | [1-2-1] Completing a scmd w/ scsi_done | |
15 | [1-2-2] Completing a scmd w/ timeout | |
16 | [1-3] How EH takes over | |
17 | [2] How SCSI EH works | |
18 | [2-1] EH through fine-grained callbacks | |
19 | [2-1-1] Overview | |
20 | [2-1-2] Flow of scmds through EH | |
21 | [2-1-3] Flow of control | |
9227c33d CH |
22 | [2-2] EH through transportt->eh_strategy_handler() |
23 | [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions | |
24 | [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions | |
70c83e11 TH |
25 | [2-2-3] Things to consider |
26 | ||
27 | ||
28 | [1] How SCSI commands travel through the midlayer and to EH | |
29 | ||
30 | [1-1] struct scsi_cmnd | |
31 | ||
32 | Each SCSI command is represented with struct scsi_cmnd (== scmd). A | |
33 | scmd has two list_head's to link itself into lists. The two are | |
34 | scmd->list and scmd->eh_entry. The former is used for free list or | |
35 | per-device allocated scmd list and not of much interest to this EH | |
36 | discussion. The latter is used for completion and EH lists and unless | |
37 | otherwise stated scmds are always linked using scmd->eh_entry in this | |
38 | discussion. | |
39 | ||
40 | ||
41 | [1-2] How do scmd's get completed? | |
42 | ||
43 | Once LLDD gets hold of a scmd, either the LLDD will complete the | |
44 | command by calling scsi_done callback passed from midlayer when | |
6ad55502 | 45 | invoking hostt->queuecommand() or the block layer will time it out. |
70c83e11 TH |
46 | |
47 | ||
48 | [1-2-1] Completing a scmd w/ scsi_done | |
49 | ||
50 | For all non-EH commands, scsi_done() is the completion callback. It | |
6ad55502 HR |
51 | just calls blk_complete_request() to delete the block layer timer and |
52 | raise SCSI_SOFTIRQ | |
70c83e11 TH |
53 | |
54 | SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to | |
55 | determine what to do with the command. scsi_decide_disposition() | |
56 | looks at the scmd->result value and sense data to determine what to do | |
57 | with the command. | |
58 | ||
59 | - SUCCESS | |
60 | scsi_finish_command() is invoked for the command. The | |
6ad55502 HR |
61 | function does some maintenance chores and then calls |
62 | scsi_io_completion() to finish the I/O. | |
63 | scsi_io_completion() then notifies the block layer on | |
64 | the completed request by calling blk_end_request and | |
65 | friends or figures out what to do with the remainder | |
66 | of the data in case of an error. | |
70c83e11 TH |
67 | |
68 | - NEEDS_RETRY | |
69 | - ADD_TO_MLQUEUE | |
70 | scmd is requeued to blk queue. | |
71 | ||
72 | - otherwise | |
a0658632 | 73 | scsi_eh_scmd_add(scmd) is invoked for the command. See |
5d3f083d | 74 | [1-3] for details of this function. |
70c83e11 TH |
75 | |
76 | ||
77 | [1-2-2] Completing a scmd w/ timeout | |
78 | ||
79 | The timeout handler is scsi_times_out(). When a timeout occurs, this | |
80 | function | |
81 | ||
8c0ae656 | 82 | 1. invokes optional hostt->eh_timed_out() callback. Return value can |
70c83e11 TH |
83 | be one of |
84 | ||
6ad55502 | 85 | - BLK_EH_RESET_TIMER |
70c83e11 TH |
86 | This indicates that more time is required to finish the |
87 | command. Timer is restarted. This action is counted as a | |
88 | retry and only allowed scmd->allowed + 1(!) times. Once the | |
6600593c | 89 | limit is reached, action for BLK_EH_DONE is taken instead. |
70c83e11 | 90 | |
6600593c | 91 | - BLK_EH_DONE |
6ad55502 | 92 | eh_timed_out() callback did not handle the command. |
70c83e11 TH |
93 | Step #2 is taken. |
94 | ||
a0658632 | 95 | 2. scsi_abort_command() is invoked to schedule an asynchrous abort. |
8e8c9d01 HR |
96 | Asynchronous abort are not invoked for commands which the |
97 | SCSI_EH_ABORT_SCHEDULED flag is set (this indicates that the command | |
98 | already had been aborted once, and this is a retry which failed), | |
99 | or when the EH deadline is expired. In these case Step #3 is taken. | |
6ad55502 | 100 | |
8e8c9d01 HR |
101 | 3. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the |
102 | command. See [1-4] for more information. | |
70c83e11 | 103 | |
6ad55502 HR |
104 | [1-3] Asynchronous command aborts |
105 | ||
106 | After a timeout occurs a command abort is scheduled from | |
107 | scsi_abort_command(). If the abort is successful the command | |
108 | will either be retried (if the number of retries is not exhausted) | |
109 | or terminated with DID_TIME_OUT. | |
110 | Otherwise scsi_eh_scmd_add() is invoked for the command. | |
111 | See [1-4] for more information. | |
70c83e11 | 112 | |
6ad55502 | 113 | [1-4] How EH takes over |
70c83e11 TH |
114 | |
115 | scmds enter EH via scsi_eh_scmd_add(), which does the following. | |
116 | ||
a0658632 | 117 | 1. Links scmd->eh_entry to shost->eh_cmd_q |
70c83e11 | 118 | |
a0658632 | 119 | 2. Sets SHOST_RECOVERY bit in shost->shost_state |
70c83e11 | 120 | |
a0658632 | 121 | 3. Increments shost->host_failed |
70c83e11 | 122 | |
a0658632 | 123 | 4. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed |
70c83e11 TH |
124 | |
125 | As can be seen above, once any scmd is added to shost->eh_cmd_q, | |
126 | SHOST_RECOVERY shost_state bit is turned on. This prevents any new | |
127 | scmd to be issued from blk queue to the host; eventually, all scmds on | |
128 | the host either complete normally, fail and get added to eh_cmd_q, or | |
129 | time out and get added to shost->eh_cmd_q. | |
130 | ||
131 | If all scmds either complete or fail, the number of in-flight scmds | |
132 | becomes equal to the number of failed scmds - i.e. shost->host_busy == | |
133 | shost->host_failed. This wakes up SCSI EH thread. So, once woken up, | |
134 | SCSI EH thread can expect that all in-flight commands have failed and | |
135 | are linked on shost->eh_cmd_q. | |
136 | ||
137 | Note that this does not mean lower layers are quiescent. If a LLDD | |
138 | completed a scmd with error status, the LLDD and lower layers are | |
139 | assumed to forget about the scmd at that point. However, if a scmd | |
8c0ae656 | 140 | has timed out, unless hostt->eh_timed_out() made lower layers forget |
70c83e11 TH |
141 | about the scmd, which currently no LLDD does, the command is still |
142 | active as long as lower layers are concerned and completion could | |
143 | occur at any time. Of course, all such completions are ignored as the | |
144 | timer has already expired. | |
145 | ||
146 | We'll talk about how SCSI EH takes actions to abort - make LLDD | |
147 | forget about - timed out scmds later. | |
148 | ||
149 | ||
150 | [2] How SCSI EH works | |
151 | ||
152 | LLDD's can implement SCSI EH actions in one of the following two | |
153 | ways. | |
154 | ||
155 | - Fine-grained EH callbacks | |
156 | LLDD can implement fine-grained EH callbacks and let SCSI | |
157 | midlayer drive error handling and call appropriate callbacks. | |
fff9289b | 158 | This will be discussed further in [2-1]. |
70c83e11 TH |
159 | |
160 | - eh_strategy_handler() callback | |
161 | This is one big callback which should perform whole error | |
542cb459 | 162 | handling. As such, it should do all chores the SCSI midlayer |
70c83e11 TH |
163 | performs during recovery. This will be discussed in [2-2]. |
164 | ||
165 | Once recovery is complete, SCSI EH resumes normal operation by | |
166 | calling scsi_restart_operations(), which | |
167 | ||
168 | 1. Checks if door locking is needed and locks door. | |
169 | ||
170 | 2. Clears SHOST_RECOVERY shost_state bit | |
171 | ||
172 | 3. Wakes up waiters on shost->host_wait. This occurs if someone | |
173 | calls scsi_block_when_processing_errors() on the host. | |
174 | (*QUESTION* why is it needed? All operations will be blocked | |
175 | anyway after it reaches blk queue.) | |
176 | ||
177 | 4. Kicks queues in all devices on the host in the asses | |
178 | ||
179 | ||
180 | [2-1] EH through fine-grained callbacks | |
181 | ||
182 | [2-1-1] Overview | |
183 | ||
184 | If eh_strategy_handler() is not present, SCSI midlayer takes charge | |
185 | of driving error handling. EH's goals are two - make LLDD, host and | |
186 | device forget about timed out scmds and make them ready for new | |
187 | commands. A scmd is said to be recovered if the scmd is forgotten by | |
188 | lower layers and lower layers are ready to process or fail the scmd | |
189 | again. | |
190 | ||
191 | To achieve these goals, EH performs recovery actions with increasing | |
2fe0ae78 | 192 | severity. Some actions are performed by issuing SCSI commands and |
70c83e11 TH |
193 | others are performed by invoking one of the following fine-grained |
194 | hostt EH callbacks. Callbacks may be omitted and omitted ones are | |
195 | considered to fail always. | |
196 | ||
197 | int (* eh_abort_handler)(struct scsi_cmnd *); | |
198 | int (* eh_device_reset_handler)(struct scsi_cmnd *); | |
199 | int (* eh_bus_reset_handler)(struct scsi_cmnd *); | |
200 | int (* eh_host_reset_handler)(struct scsi_cmnd *); | |
201 | ||
202 | Higher-severity actions are taken only when lower-severity actions | |
203 | cannot recover some of failed scmds. Also, note that failure of the | |
204 | highest-severity action means EH failure and results in offlining of | |
205 | all unrecovered devices. | |
206 | ||
207 | During recovery, the following rules are followed | |
208 | ||
209 | - Recovery actions are performed on failed scmds on the to do list, | |
210 | eh_work_q. If a recovery action succeeds for a scmd, recovered | |
211 | scmds are removed from eh_work_q. | |
212 | ||
213 | Note that single recovery action on a scmd can recover multiple | |
214 | scmds. e.g. resetting a device recovers all failed scmds on the | |
215 | device. | |
216 | ||
217 | - Higher severity actions are taken iff eh_work_q is not empty after | |
218 | lower severity actions are complete. | |
219 | ||
220 | - EH reuses failed scmds to issue commands for recovery. For | |
221 | timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd | |
222 | before reusing it for EH commands. | |
223 | ||
224 | When a scmd is recovered, the scmd is moved from eh_work_q to EH | |
225 | local eh_done_q using scsi_eh_finish_cmd(). After all scmds are | |
226 | recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to | |
227 | either retry or error-finish (notify upper layer of failure) recovered | |
228 | scmds. | |
229 | ||
230 | scmds are retried iff its sdev is still online (not offlined during | |
231 | EH), REQ_FAILFAST is not set and ++scmd->retries is less than | |
232 | scmd->allowed. | |
233 | ||
234 | ||
235 | [2-1-2] Flow of scmds through EH | |
236 | ||
237 | 1. Error completion / time out | |
238 | ACTION: scsi_eh_scmd_add() is invoked for scmd | |
70c83e11 TH |
239 | - add scmd to shost->eh_cmd_q |
240 | - set SHOST_RECOVERY | |
241 | - shost->host_failed++ | |
242 | LOCKING: shost->host_lock | |
243 | ||
244 | 2. EH starts | |
245 | ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q | |
246 | is cleared. | |
247 | LOCKING: shost->host_lock (not strictly necessary, just for | |
248 | consistency) | |
249 | ||
250 | 3. scmd recovered | |
251 | ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd | |
70c83e11 TH |
252 | - scsi_setup_cmd_retry() |
253 | - move from local eh_work_q to local eh_done_q | |
254 | LOCKING: none | |
72d8c36e WF |
255 | CONCURRENCY: at most one thread per separate eh_work_q to |
256 | keep queue manipulation lockless | |
70c83e11 TH |
257 | |
258 | 4. EH completes | |
259 | ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper | |
72d8c36e WF |
260 | layer of failure. May be called concurrently but must have |
261 | a no more than one thread per separate eh_work_q to | |
262 | manipulate the queue locklessly | |
70c83e11 TH |
263 | - scmd is removed from eh_done_q and scmd->eh_entry is cleared |
264 | - if retry is necessary, scmd is requeued using | |
265 | scsi_queue_insert() | |
266 | - otherwise, scsi_finish_command() is invoked for scmd | |
72d8c36e | 267 | - zero shost->host_failed |
70c83e11 TH |
268 | LOCKING: queue or finish function performs appropriate locking |
269 | ||
270 | ||
271 | [2-1-3] Flow of control | |
272 | ||
273 | EH through fine-grained callbacks start from scsi_unjam_host(). | |
274 | ||
275 | <<scsi_unjam_host>> | |
276 | ||
277 | 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local | |
278 | eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is | |
279 | cleared by this action. | |
280 | ||
281 | 2. Invoke scsi_eh_get_sense. | |
282 | ||
283 | <<scsi_eh_get_sense>> | |
284 | ||
285 | This action is taken for each error-completed | |
286 | (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most | |
287 | SCSI transports/LLDDs automatically acquire sense data on | |
288 | command failures (autosense). Autosense is recommended for | |
289 | performance reasons and as sense information could get out of | |
25985edc | 290 | sync between occurrence of CHECK CONDITION and this action. |
70c83e11 TH |
291 | |
292 | Note that if autosense is not supported, scmd->sense_buffer | |
293 | contains invalid sense data when error-completing the scmd | |
294 | with scsi_done(). scsi_decide_disposition() always returns | |
295 | FAILED in such cases thus invoking SCSI EH. When the scmd | |
296 | reaches here, sense data is acquired and | |
297 | scsi_decide_disposition() is called again. | |
298 | ||
299 | 1. Invoke scsi_request_sense() which issues REQUEST_SENSE | |
300 | command. If fails, no action. Note that taking no action | |
301 | causes higher-severity recovery to be taken for the scmd. | |
302 | ||
303 | 2. Invoke scsi_decide_disposition() on the scmd | |
304 | ||
305 | - SUCCESS | |
306 | scmd->retries is set to scmd->allowed preventing | |
307 | scsi_eh_flush_done_q() from retrying the scmd and | |
308 | scsi_eh_finish_cmd() is invoked. | |
309 | ||
310 | - NEEDS_RETRY | |
311 | scsi_eh_finish_cmd() invoked | |
312 | ||
313 | - otherwise | |
314 | No action. | |
315 | ||
316 | 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds(). | |
317 | ||
318 | <<scsi_eh_abort_cmds>> | |
319 | ||
6ad55502 HR |
320 | This action is taken for each timed out command when |
321 | no_async_abort is enabled in the host template. | |
70c83e11 TH |
322 | hostt->eh_abort_handler() is invoked for each scmd. The |
323 | handler returns SUCCESS if it has succeeded to make LLDD and | |
324 | all related hardware forget about the scmd. | |
325 | ||
326 | If a timedout scmd is successfully aborted and the sdev is | |
327 | either offline or ready, scsi_eh_finish_cmd() is invoked for | |
328 | the scmd. Otherwise, the scmd is left in eh_work_q for | |
329 | higher-severity actions. | |
330 | ||
331 | Note that both offline and ready status mean that the sdev is | |
332 | ready to process new scmds, where processing also implies | |
333 | immediate failing; thus, if a sdev is in one of the two | |
334 | states, no further recovery action is needed. | |
335 | ||
336 | Device readiness is tested using scsi_eh_tur() which issues | |
337 | TEST_UNIT_READY command. Note that the scmd must have been | |
338 | aborted successfully before reusing it for TEST_UNIT_READY. | |
339 | ||
340 | 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs() | |
341 | ||
342 | <<scsi_eh_ready_devs>> | |
343 | ||
344 | This function takes four increasingly more severe measures to | |
345 | make failed sdevs ready for new commands. | |
346 | ||
347 | 1. Invoke scsi_eh_stu() | |
348 | ||
349 | <<scsi_eh_stu>> | |
350 | ||
351 | For each sdev which has failed scmds with valid sense data | |
352 | of which scsi_check_sense()'s verdict is FAILED, | |
353 | START_STOP_UNIT command is issued w/ start=1. Note that | |
354 | as we explicitly choose error-completed scmds, it is known | |
355 | that lower layers have forgotten about the scmd and we can | |
356 | reuse it for STU. | |
357 | ||
358 | If STU succeeds and the sdev is either offline or ready, | |
359 | all failed scmds on the sdev are EH-finished with | |
360 | scsi_eh_finish_cmd(). | |
361 | ||
362 | *NOTE* If hostt->eh_abort_handler() isn't implemented or | |
363 | failed, we may still have timed out scmds at this point | |
364 | and STU doesn't make lower layers forget about those | |
365 | scmds. Yet, this function EH-finish all scmds on the sdev | |
366 | if STU succeeds leaving lower layers in an inconsistent | |
367 | state. It seems that STU action should be taken only when | |
368 | a sdev has no timed out scmd. | |
369 | ||
370 | 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset(). | |
371 | ||
372 | <<scsi_eh_bus_device_reset>> | |
373 | ||
374 | This action is very similar to scsi_eh_stu() except that, | |
375 | instead of issuing STU, hostt->eh_device_reset_handler() | |
376 | is used. Also, as we're not issuing SCSI commands and | |
377 | resetting clears all scmds on the sdev, there is no need | |
378 | to choose error-completed scmds. | |
379 | ||
380 | 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset() | |
381 | ||
382 | <<scsi_eh_bus_reset>> | |
383 | ||
384 | hostt->eh_bus_reset_handler() is invoked for each channel | |
385 | with failed scmds. If bus reset succeeds, all failed | |
386 | scmds on all ready or offline sdevs on the channel are | |
387 | EH-finished. | |
388 | ||
389 | 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset() | |
390 | ||
391 | <<scsi_eh_host_reset>> | |
392 | ||
393 | This is the last resort. hostt->eh_host_reset_handler() | |
394 | is invoked. If host reset succeeds, all failed scmds on | |
395 | all ready or offline sdevs on the host are EH-finished. | |
396 | ||
397 | 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs() | |
398 | ||
399 | <<scsi_eh_offline_sdevs>> | |
400 | ||
401 | Take all sdevs which still have unrecovered scmds offline | |
402 | and EH-finish the scmds. | |
403 | ||
404 | 5. Invoke scsi_eh_flush_done_q(). | |
405 | ||
406 | <<scsi_eh_flush_done_q>> | |
407 | ||
408 | At this point all scmds are recovered (or given up) and | |
409 | put on eh_done_q by scsi_eh_finish_cmd(). This function | |
410 | flushes eh_done_q by either retrying or notifying upper | |
411 | layer of failure of the scmds. | |
412 | ||
413 | ||
9227c33d | 414 | [2-2] EH through transportt->eh_strategy_handler() |
70c83e11 | 415 | |
9227c33d | 416 | transportt->eh_strategy_handler() is invoked in the place of |
70c83e11 TH |
417 | scsi_unjam_host() and it is responsible for whole recovery process. |
418 | On completion, the handler should have made lower layers forget about | |
419 | all failed scmds and either ready for new commands or offline. Also, | |
542cb459 | 420 | it should perform SCSI EH maintenance chores to maintain integrity of |
70c83e11 TH |
421 | SCSI midlayer. IOW, of the steps described in [2-1-2], all steps |
422 | except for #1 must be implemented by eh_strategy_handler(). | |
423 | ||
424 | ||
9227c33d | 425 | [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions |
70c83e11 TH |
426 | |
427 | The following conditions are true on entry to the handler. | |
428 | ||
429 | - Each failed scmd's eh_flags field is set appropriately. | |
430 | ||
431 | - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry. | |
432 | ||
433 | - SHOST_RECOVERY is set. | |
434 | ||
435 | - shost->host_failed == shost->host_busy | |
436 | ||
437 | ||
9227c33d | 438 | [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions |
70c83e11 TH |
439 | |
440 | The following conditions must be true on exit from the handler. | |
441 | ||
442 | - shost->host_failed is zero. | |
443 | ||
70c83e11 TH |
444 | - Each scmd is in such a state that scsi_setup_cmd_retry() on the |
445 | scmd doesn't make any difference. | |
446 | ||
447 | - shost->eh_cmd_q is cleared. | |
448 | ||
449 | - Each scmd->eh_entry is cleared. | |
450 | ||
451 | - Either scsi_queue_insert() or scsi_finish_command() is called on | |
452 | each scmd. Note that the handler is free to use scmd->retries and | |
453 | ->allowed to limit the number of retries. | |
454 | ||
455 | ||
456 | [2-2-3] Things to consider | |
457 | ||
458 | - Know that timed out scmds are still active on lower layers. Make | |
459 | lower layers forget about them before doing anything else with | |
460 | those scmds. | |
461 | ||
462 | - For consistency, when accessing/modifying shost data structure, | |
463 | grab shost->host_lock. | |
464 | ||
465 | - On completion, each failed sdev must have forgotten about all | |
466 | active scmds. | |
467 | ||
468 | - On completion, each failed sdev must be ready for new commands or | |
469 | offline. | |
470 | ||
471 | ||
472 | -- | |
473 | Tejun Heo | |
474 | htejun@gmail.com | |
475 | 11th September 2005 |