Commit | Line | Data |
---|---|---|
a88dc3ec MCC |
1 | .. SPDX-License-Identifier: GPL-2.0 |
2 | ||
3 | ========= | |
2908d778 | 4 | SAS Layer |
a88dc3ec | 5 | ========= |
2908d778 JB |
6 | |
7 | The SAS Layer is a management infrastructure which manages | |
8 | SAS LLDDs. It sits between SCSI Core and SAS LLDDs. The | |
9 | layout is as follows: while SCSI Core is concerned with | |
10 | SAM/SPC issues, and a SAS LLDD+sequencer is concerned with | |
11 | phy/OOB/link management, the SAS layer is concerned with: | |
12 | ||
13 | * SAS Phy/Port/HA event management (LLDD generates, | |
14 | SAS Layer processes), | |
15 | * SAS Port management (creation/destruction), | |
16 | * SAS Domain discovery and revalidation, | |
17 | * SAS Domain device management, | |
18 | * SCSI Host registration/unregistration, | |
19 | * Device registration with SCSI Core (SAS) or libata | |
20 | (SATA), and | |
21 | * Expander management and exporting expander control | |
22 | to user space. | |
23 | ||
24 | A SAS LLDD is a PCI device driver. It is concerned with | |
25 | phy/OOB management, and vendor specific tasks and generates | |
26 | events to the SAS layer. | |
27 | ||
28 | The SAS Layer does most SAS tasks as outlined in the SAS 1.1 | |
29 | spec. | |
30 | ||
31 | The sas_ha_struct describes the SAS LLDD to the SAS layer. | |
32 | Most of it is used by the SAS Layer but a few fields need to | |
33 | be initialized by the LLDDs. | |
34 | ||
35 | After initializing your hardware, from the probe() function | |
36 | you call sas_register_ha(). It will register your LLDD with | |
37 | the SCSI subsystem, creating a SCSI host and it will | |
38 | register your SAS driver with the sysfs SAS tree it creates. | |
39 | It will then return. Then you enable your phys to actually | |
40 | start OOB (at which point your driver will start calling the | |
41 | notify_* event callbacks). | |
42 | ||
a88dc3ec MCC |
43 | Structure descriptions |
44 | ====================== | |
45 | ||
46 | ``struct sas_phy`` | |
47 | ------------------ | |
2908d778 | 48 | |
2908d778 | 49 | Normally this is statically embedded to your driver's |
a88dc3ec MCC |
50 | phy structure:: |
51 | ||
52 | struct my_phy { | |
53 | blah; | |
54 | struct sas_phy sas_phy; | |
55 | bleh; | |
56 | }; | |
57 | ||
2908d778 JB |
58 | And then all the phys are an array of my_phy in your HA |
59 | struct (shown below). | |
60 | ||
61 | Then as you go along and initialize your phys you also | |
62 | initialize the sas_phy struct, along with your own | |
63 | phy structure. | |
64 | ||
65 | In general, the phys are managed by the LLDD and the ports | |
66 | are managed by the SAS layer. So the phys are initialized | |
67 | and updated by the LLDD and the ports are initialized and | |
68 | updated by the SAS layer. | |
69 | ||
70 | There is a scheme where the LLDD can RW certain fields, | |
71 | and the SAS layer can only read such ones, and vice versa. | |
72 | The idea is to avoid unnecessary locking. | |
73 | ||
a88dc3ec MCC |
74 | enabled |
75 | - must be set (0/1) | |
76 | ||
77 | id | |
78 | - must be set [0,MAX_PHYS)] | |
79 | ||
80 | class, proto, type, role, oob_mode, linkrate | |
81 | - must be set | |
82 | ||
83 | oob_mode | |
84 | - you set this when OOB has finished and then notify | |
85 | the SAS Layer. | |
86 | ||
87 | sas_addr | |
88 | - this normally points to an array holding the sas | |
89 | address of the phy, possibly somewhere in your my_phy | |
90 | struct. | |
91 | ||
92 | attached_sas_addr | |
93 | - set this when you (LLDD) receive an | |
94 | IDENTIFY frame or a FIS frame, _before_ notifying the SAS | |
95 | layer. The idea is that sometimes the LLDD may want to fake | |
96 | or provide a different SAS address on that phy/port and this | |
97 | allows it to do this. At best you should copy the sas | |
98 | address from the IDENTIFY frame or maybe generate a SAS | |
99 | address for SATA directly attached devices. The Discover | |
100 | process may later change this. | |
101 | ||
102 | frame_rcvd | |
103 | - this is where you copy the IDENTIFY/FIS frame | |
104 | when you get it; you lock, copy, set frame_rcvd_size and | |
105 | unlock the lock, and then call the event. It is a pointer | |
106 | since there's no way to know your hw frame size _exactly_, | |
107 | so you define the actual array in your phy struct and let | |
108 | this pointer point to it. You copy the frame from your | |
109 | DMAable memory to that area holding the lock. | |
110 | ||
111 | sas_prim | |
112 | - this is where primitives go when they're | |
113 | received. See sas.h. Grab the lock, set the primitive, | |
114 | release the lock, notify. | |
115 | ||
116 | port | |
117 | - this points to the sas_port if the phy belongs | |
118 | to a port -- the LLDD only reads this. It points to the | |
119 | sas_port this phy is part of. Set by the SAS Layer. | |
120 | ||
121 | ha | |
122 | - may be set; the SAS layer sets it anyway. | |
123 | ||
124 | lldd_phy | |
125 | - you should set this to point to your phy so you | |
126 | can find your way around faster when the SAS layer calls one | |
127 | of your callbacks and passes you a phy. If the sas_phy is | |
128 | embedded you can also use container_of -- whatever you | |
129 | prefer. | |
130 | ||
131 | ||
132 | ``struct sas_port`` | |
133 | ------------------- | |
134 | ||
2908d778 JB |
135 | The LLDD doesn't set any fields of this struct -- it only |
136 | reads them. They should be self explanatory. | |
137 | ||
138 | phy_mask is 32 bit, this should be enough for now, as I | |
139 | haven't heard of a HA having more than 8 phys. | |
140 | ||
a88dc3ec MCC |
141 | lldd_port |
142 | - I haven't found use for that -- maybe other | |
143 | LLDD who wish to have internal port representation can make | |
144 | use of this. | |
2908d778 | 145 | |
a88dc3ec MCC |
146 | ``struct sas_ha_struct`` |
147 | ------------------------ | |
2908d778 | 148 | |
2908d778 | 149 | It normally is statically declared in your own LLDD |
a88dc3ec MCC |
150 | structure describing your adapter:: |
151 | ||
152 | struct my_sas_ha { | |
153 | blah; | |
154 | struct sas_ha_struct sas_ha; | |
155 | struct my_phy phys[MAX_PHYS]; | |
156 | struct sas_port sas_ports[MAX_PHYS]; /* (1) */ | |
157 | bleh; | |
158 | }; | |
2908d778 | 159 | |
a88dc3ec | 160 | (1) If your LLDD doesn't have its own port representation. |
2908d778 JB |
161 | |
162 | What needs to be initialized (sample function given below). | |
163 | ||
164 | pcidev | |
a88dc3ec MCC |
165 | ^^^^^^ |
166 | ||
167 | sas_addr | |
168 | - since the SAS layer doesn't want to mess with | |
2908d778 JB |
169 | memory allocation, etc, this points to statically |
170 | allocated array somewhere (say in your host adapter | |
171 | structure) and holds the SAS address of the host | |
172 | adapter as given by you or the manufacturer, etc. | |
a88dc3ec | 173 | |
2908d778 | 174 | sas_port |
a88dc3ec MCC |
175 | ^^^^^^^^ |
176 | ||
177 | sas_phy | |
178 | - an array of pointers to structures. (see | |
2908d778 JB |
179 | note above on sas_addr). |
180 | These must be set. See more notes below. | |
a88dc3ec MCC |
181 | |
182 | num_phys | |
183 | - the number of phys present in the sas_phy array, | |
2908d778 JB |
184 | and the number of ports present in the sas_port |
185 | array. There can be a maximum num_phys ports (one per | |
186 | port) so we drop the num_ports, and only use | |
187 | num_phys. | |
188 | ||
a88dc3ec | 189 | The event interface:: |
2908d778 JB |
190 | |
191 | /* LLDD calls these to notify the class of an event. */ | |
5d6a75a1 AD |
192 | void sas_notify_port_event(struct sas_phy *, enum port_event, gfp_t); |
193 | void sas_notify_phy_event(struct sas_phy *, enum phy_event, gfp_t); | |
2908d778 | 194 | |
a88dc3ec | 195 | The port notification:: |
2908d778 JB |
196 | |
197 | /* The class calls these to notify the LLDD of an event. */ | |
198 | void (*lldd_port_formed)(struct sas_phy *); | |
199 | void (*lldd_port_deformed)(struct sas_phy *); | |
200 | ||
201 | If the LLDD wants notification when a port has been formed | |
202 | or deformed it sets those to a function satisfying the type. | |
203 | ||
204 | A SAS LLDD should also implement at least one of the Task | |
a88dc3ec | 205 | Management Functions (TMFs) described in SAM:: |
2908d778 JB |
206 | |
207 | /* Task Management Functions. Must be called from process context. */ | |
208 | int (*lldd_abort_task)(struct sas_task *); | |
209 | int (*lldd_abort_task_set)(struct domain_device *, u8 *lun); | |
2908d778 JB |
210 | int (*lldd_clear_task_set)(struct domain_device *, u8 *lun); |
211 | int (*lldd_I_T_nexus_reset)(struct domain_device *); | |
212 | int (*lldd_lu_reset)(struct domain_device *, u8 *lun); | |
213 | int (*lldd_query_task)(struct sas_task *); | |
214 | ||
215 | For more information please read SAM from T10.org. | |
216 | ||
a88dc3ec | 217 | Port and Adapter management:: |
2908d778 JB |
218 | |
219 | /* Port and Adapter management */ | |
220 | int (*lldd_clear_nexus_port)(struct sas_port *); | |
221 | int (*lldd_clear_nexus_ha)(struct sas_ha_struct *); | |
222 | ||
223 | A SAS LLDD should implement at least one of those. | |
224 | ||
a88dc3ec | 225 | Phy management:: |
2908d778 JB |
226 | |
227 | /* Phy management */ | |
228 | int (*lldd_control_phy)(struct sas_phy *, enum phy_func); | |
229 | ||
a88dc3ec MCC |
230 | lldd_ha |
231 | - set this to point to your HA struct. You can also | |
232 | use container_of if you embedded it as shown above. | |
2908d778 JB |
233 | |
234 | A sample initialization and registration function | |
235 | can look like this (called last thing from probe()) | |
a88dc3ec | 236 | *but* before you enable the phys to do OOB:: |
2908d778 | 237 | |
a88dc3ec MCC |
238 | static int register_sas_ha(struct my_sas_ha *my_ha) |
239 | { | |
240 | int i; | |
241 | static struct sas_phy *sas_phys[MAX_PHYS]; | |
242 | static struct sas_port *sas_ports[MAX_PHYS]; | |
2908d778 | 243 | |
a88dc3ec | 244 | my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0]; |
2908d778 | 245 | |
a88dc3ec MCC |
246 | for (i = 0; i < MAX_PHYS; i++) { |
247 | sas_phys[i] = &my_ha->phys[i].sas_phy; | |
248 | sas_ports[i] = &my_ha->sas_ports[i]; | |
249 | } | |
2908d778 | 250 | |
a88dc3ec MCC |
251 | my_ha->sas_ha.sas_phy = sas_phys; |
252 | my_ha->sas_ha.sas_port = sas_ports; | |
253 | my_ha->sas_ha.num_phys = MAX_PHYS; | |
2908d778 | 254 | |
a88dc3ec | 255 | my_ha->sas_ha.lldd_port_formed = my_port_formed; |
2908d778 | 256 | |
a88dc3ec MCC |
257 | my_ha->sas_ha.lldd_dev_found = my_dev_found; |
258 | my_ha->sas_ha.lldd_dev_gone = my_dev_gone; | |
2908d778 | 259 | |
a88dc3ec | 260 | my_ha->sas_ha.lldd_execute_task = my_execute_task; |
2908d778 | 261 | |
a88dc3ec MCC |
262 | my_ha->sas_ha.lldd_abort_task = my_abort_task; |
263 | my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set; | |
a88dc3ec MCC |
264 | my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set; |
265 | my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2) | |
266 | my_ha->sas_ha.lldd_lu_reset = my_lu_reset; | |
267 | my_ha->sas_ha.lldd_query_task = my_query_task; | |
2908d778 | 268 | |
a88dc3ec MCC |
269 | my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port; |
270 | my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha; | |
2908d778 | 271 | |
a88dc3ec | 272 | my_ha->sas_ha.lldd_control_phy = my_control_phy; |
2908d778 | 273 | |
a88dc3ec MCC |
274 | return sas_register_ha(&my_ha->sas_ha); |
275 | } | |
2908d778 | 276 | |
2908d778 JB |
277 | (2) SAS 1.1 does not define I_T Nexus Reset TMF. |
278 | ||
279 | Events | |
a88dc3ec | 280 | ====== |
2908d778 | 281 | |
a88dc3ec | 282 | Events are **the only way** a SAS LLDD notifies the SAS layer |
2908d778 JB |
283 | of anything. There is no other method or way a LLDD to tell |
284 | the SAS layer of anything happening internally or in the SAS | |
285 | domain. | |
286 | ||
a88dc3ec MCC |
287 | Phy events:: |
288 | ||
2908d778 JB |
289 | PHYE_LOSS_OF_SIGNAL, (C) |
290 | PHYE_OOB_DONE, | |
291 | PHYE_OOB_ERROR, (C) | |
292 | PHYE_SPINUP_HOLD. | |
293 | ||
a88dc3ec MCC |
294 | Port events, passed on a _phy_:: |
295 | ||
2908d778 JB |
296 | PORTE_BYTES_DMAED, (M) |
297 | PORTE_BROADCAST_RCVD, (E) | |
298 | PORTE_LINK_RESET_ERR, (C) | |
299 | PORTE_TIMER_EVENT, (C) | |
300 | PORTE_HARD_RESET. | |
301 | ||
302 | Host Adapter event: | |
303 | HAE_RESET | |
304 | ||
305 | A SAS LLDD should be able to generate | |
a88dc3ec | 306 | |
2908d778 JB |
307 | - at least one event from group C (choice), |
308 | - events marked M (mandatory) are mandatory (only one), | |
309 | - events marked E (expander) if it wants the SAS layer | |
310 | to handle domain revalidation (only one such). | |
311 | - Unmarked events are optional. | |
312 | ||
313 | Meaning: | |
314 | ||
a88dc3ec MCC |
315 | HAE_RESET |
316 | - when your HA got internal error and was reset. | |
317 | ||
318 | PORTE_BYTES_DMAED | |
319 | - on receiving an IDENTIFY/FIS frame | |
320 | ||
321 | PORTE_BROADCAST_RCVD | |
322 | - on receiving a primitive | |
323 | ||
324 | PORTE_LINK_RESET_ERR | |
325 | - timer expired, loss of signal, loss of DWS, etc. [1]_ | |
2908d778 | 326 | |
a88dc3ec MCC |
327 | PORTE_TIMER_EVENT |
328 | - DWS reset timeout timer expired [1]_ | |
2908d778 | 329 | |
a88dc3ec MCC |
330 | PORTE_HARD_RESET |
331 | - Hard Reset primitive received. | |
2908d778 | 332 | |
a88dc3ec MCC |
333 | PHYE_LOSS_OF_SIGNAL |
334 | - the device is gone [1]_ | |
2908d778 | 335 | |
a88dc3ec MCC |
336 | PHYE_OOB_DONE |
337 | - OOB went fine and oob_mode is valid | |
338 | ||
339 | PHYE_OOB_ERROR | |
340 | - Error while doing OOB, the device probably | |
341 | got disconnected. [1]_ | |
342 | ||
343 | PHYE_SPINUP_HOLD | |
344 | - SATA is present, COMWAKE not sent. | |
345 | ||
346 | .. [1] should set/clear the appropriate fields in the phy, | |
347 | or alternatively call the inlined sas_phy_disconnected() | |
348 | which is just a helper, from their tasklet. | |
349 | ||
350 | The Execute Command SCSI RPC:: | |
2908d778 | 351 | |
79855d17 | 352 | int (*lldd_execute_task)(struct sas_task *, gfp_t gfp_flags); |
2908d778 | 353 | |
79855d17 CH |
354 | Used to queue a task to the SAS LLDD. @task is the task to be executed. |
355 | @gfp_mask is the gfp_mask defining the context of the caller. | |
2908d778 JB |
356 | |
357 | This function should implement the Execute Command SCSI RPC, | |
2908d778 | 358 | |
79855d17 | 359 | That is, when lldd_execute_task() is called, the command |
2908d778 JB |
360 | go out on the transport *immediately*. There is *no* |
361 | queuing of any sort and at any level in a SAS LLDD. | |
362 | ||
a88dc3ec | 363 | Returns: |
2908d778 | 364 | |
a88dc3ec MCC |
365 | * -SAS_QUEUE_FULL, -ENOMEM, nothing was queued; |
366 | * 0, the task(s) were queued. | |
2908d778 | 367 | |
a88dc3ec MCC |
368 | :: |
369 | ||
370 | struct sas_task { | |
371 | dev -- the device this task is destined to | |
372 | task_proto -- _one_ of enum sas_proto | |
373 | scatter -- pointer to scatter gather list array | |
374 | num_scatter -- number of elements in scatter | |
375 | total_xfer_len -- total number of bytes expected to be transferred | |
376 | data_dir -- PCI_DMA_... | |
377 | task_done -- callback when the task has finished execution | |
378 | }; | |
379 | ||
380 | Discovery | |
381 | ========= | |
2908d778 JB |
382 | |
383 | The sysfs tree has the following purposes: | |
a88dc3ec | 384 | |
2908d778 JB |
385 | a) It shows you the physical layout of the SAS domain at |
386 | the current time, i.e. how the domain looks in the | |
387 | physical world right now. | |
388 | b) Shows some device parameters _at_discovery_time_. | |
389 | ||
390 | This is a link to the tree(1) program, very useful in | |
391 | viewing the SAS domain: | |
392 | ftp://mama.indstate.edu/linux/tree/ | |
a88dc3ec | 393 | |
2908d778 JB |
394 | I expect user space applications to actually create a |
395 | graphical interface of this. | |
396 | ||
397 | That is, the sysfs domain tree doesn't show or keep state if | |
398 | you e.g., change the meaning of the READY LED MEANING | |
399 | setting, but it does show you the current connection status | |
400 | of the domain device. | |
401 | ||
402 | Keeping internal device state changes is responsibility of | |
403 | upper layers (Command set drivers) and user space. | |
404 | ||
405 | When a device or devices are unplugged from the domain, this | |
406 | is reflected in the sysfs tree immediately, and the device(s) | |
407 | removed from the system. | |
408 | ||
409 | The structure domain_device describes any device in the SAS | |
410 | domain. It is completely managed by the SAS layer. A task | |
411 | points to a domain device, this is how the SAS LLDD knows | |
412 | where to send the task(s) to. A SAS LLDD only reads the | |
413 | contents of the domain_device structure, but it never creates | |
414 | or destroys one. | |
415 | ||
416 | Expander management from User Space | |
a88dc3ec | 417 | =================================== |
2908d778 JB |
418 | |
419 | In each expander directory in sysfs, there is a file called | |
420 | "smp_portal". It is a binary sysfs attribute file, which | |
421 | implements an SMP portal (Note: this is *NOT* an SMP port), | |
422 | to which user space applications can send SMP requests and | |
423 | receive SMP responses. | |
424 | ||
425 | Functionality is deceptively simple: | |
426 | ||
427 | 1. Build the SMP frame you want to send. The format and layout | |
428 | is described in the SAS spec. Leave the CRC field equal 0. | |
a88dc3ec | 429 | |
2908d778 | 430 | open(2) |
a88dc3ec | 431 | |
2908d778 | 432 | 2. Open the expander's SMP portal sysfs file in RW mode. |
a88dc3ec | 433 | |
2908d778 | 434 | write(2) |
a88dc3ec | 435 | |
2908d778 | 436 | 3. Write the frame you built in 1. |
a88dc3ec | 437 | |
2908d778 | 438 | read(2) |
a88dc3ec | 439 | |
2908d778 JB |
440 | 4. Read the amount of data you expect to receive for the frame you built. |
441 | If you receive different amount of data you expected to receive, | |
442 | then there was some kind of error. | |
a88dc3ec | 443 | |
2908d778 | 444 | close(2) |
a88dc3ec | 445 | |
2908d778 JB |
446 | All this process is shown in detail in the function do_smp_func() |
447 | and its callers, in the file "expander_conf.c". | |
448 | ||
449 | The kernel functionality is implemented in the file | |
450 | "sas_expander.c". | |
451 | ||
452 | The program "expander_conf.c" implements this. It takes one | |
453 | argument, the sysfs file name of the SMP portal to the | |
454 | expander, and gives expander information, including routing | |
455 | tables. | |
456 | ||
457 | The SMP portal gives you complete control of the expander, | |
458 | so please be careful. |