Commit | Line | Data |
---|---|---|
ce876851 AG |
1 | Contents: |
2 | ||
3 | 1) TCM Userspace Design | |
4 | a) Background | |
5 | b) Benefits | |
6 | c) Design constraints | |
7 | d) Implementation overview | |
8 | i. Mailbox | |
9 | ii. Command ring | |
10 | iii. Data Area | |
11 | e) Device discovery | |
12 | f) Device events | |
13 | g) Other contingencies | |
14 | 2) Writing a user pass-through handler | |
15 | a) Discovering and configuring TCMU uio devices | |
16 | b) Waiting for events on the device(s) | |
17 | c) Managing the command ring | |
9c1cd1b6 | 18 | 3) A final note |
ce876851 AG |
19 | |
20 | ||
21 | TCM Userspace Design | |
22 | -------------------- | |
23 | ||
24 | TCM is another name for LIO, an in-kernel iSCSI target (server). | |
25 | Existing TCM targets run in the kernel. TCMU (TCM in Userspace) | |
26 | allows userspace programs to be written which act as iSCSI targets. | |
27 | This document describes the design. | |
28 | ||
29 | The existing kernel provides modules for different SCSI transport | |
30 | protocols. TCM also modularizes the data storage. There are existing | |
31 | modules for file, block device, RAM or using another SCSI device as | |
32 | storage. These are called "backstores" or "storage engines". These | |
33 | built-in modules are implemented entirely as kernel code. | |
34 | ||
35 | Background: | |
36 | ||
37 | In addition to modularizing the transport protocol used for carrying | |
38 | SCSI commands ("fabrics"), the Linux kernel target, LIO, also modularizes | |
39 | the actual data storage as well. These are referred to as "backstores" | |
40 | or "storage engines". The target comes with backstores that allow a | |
41 | file, a block device, RAM, or another SCSI device to be used for the | |
42 | local storage needed for the exported SCSI LUN. Like the rest of LIO, | |
43 | these are implemented entirely as kernel code. | |
44 | ||
45 | These backstores cover the most common use cases, but not all. One new | |
46 | use case that other non-kernel target solutions, such as tgt, are able | |
47 | to support is using Gluster's GLFS or Ceph's RBD as a backstore. The | |
48 | target then serves as a translator, allowing initiators to store data | |
49 | in these non-traditional networked storage systems, while still only | |
50 | using standard protocols themselves. | |
51 | ||
52 | If the target is a userspace process, supporting these is easy. tgt, | |
53 | for example, needs only a small adapter module for each, because the | |
54 | modules just use the available userspace libraries for RBD and GLFS. | |
55 | ||
56 | Adding support for these backstores in LIO is considerably more | |
57 | difficult, because LIO is entirely kernel code. Instead of undertaking | |
58 | the significant work to port the GLFS or RBD APIs and protocols to the | |
59 | kernel, another approach is to create a userspace pass-through | |
60 | backstore for LIO, "TCMU". | |
61 | ||
62 | ||
63 | Benefits: | |
64 | ||
65 | In addition to allowing relatively easy support for RBD and GLFS, TCMU | |
66 | will also allow easier development of new backstores. TCMU combines | |
67 | with the LIO loopback fabric to become something similar to FUSE | |
68 | (Filesystem in Userspace), but at the SCSI layer instead of the | |
69 | filesystem layer. A SUSE, if you will. | |
70 | ||
71 | The disadvantage is there are more distinct components to configure, and | |
72 | potentially to malfunction. This is unavoidable, but hopefully not | |
73 | fatal if we're careful to keep things as simple as possible. | |
74 | ||
75 | Design constraints: | |
76 | ||
77 | - Good performance: high throughput, low latency | |
78 | - Cleanly handle if userspace: | |
79 | 1) never attaches | |
80 | 2) hangs | |
81 | 3) dies | |
82 | 4) misbehaves | |
83 | - Allow future flexibility in user & kernel implementations | |
84 | - Be reasonably memory-efficient | |
85 | - Simple to configure & run | |
86 | - Simple to write a userspace backend | |
87 | ||
88 | ||
89 | Implementation overview: | |
90 | ||
91 | The core of the TCMU interface is a memory region that is shared | |
92 | between kernel and userspace. Within this region is: a control area | |
93 | (mailbox); a lockless producer/consumer circular buffer for commands | |
94 | to be passed up, and status returned; and an in/out data buffer area. | |
95 | ||
96 | TCMU uses the pre-existing UIO subsystem. UIO allows device driver | |
97 | development in userspace, and this is conceptually very close to the | |
98 | TCMU use case, except instead of a physical device, TCMU implements a | |
99 | memory-mapped layout designed for SCSI commands. Using UIO also | |
100 | benefits TCMU by handling device introspection (e.g. a way for | |
101 | userspace to determine how large the shared region is) and signaling | |
102 | mechanisms in both directions. | |
103 | ||
104 | There are no embedded pointers in the memory region. Everything is | |
105 | expressed as an offset from the region's starting address. This allows | |
106 | the ring to still work if the user process dies and is restarted with | |
107 | the region mapped at a different virtual address. | |
108 | ||
109 | See target_core_user.h for the struct definitions. | |
110 | ||
111 | The Mailbox: | |
112 | ||
113 | The mailbox is always at the start of the shared memory region, and | |
114 | contains a version, details about the starting offset and size of the | |
115 | command ring, and head and tail pointers to be used by the kernel and | |
116 | userspace (respectively) to put commands on the ring, and indicate | |
117 | when the commands are completed. | |
118 | ||
119 | version - 1 (userspace should abort if otherwise) | |
32c76de3 SY |
120 | flags: |
121 | - TCMU_MAILBOX_FLAG_CAP_OOOC: indicates out-of-order completion is | |
122 | supported. See "The Command Ring" for details. | |
ce876851 AG |
123 | cmdr_off - The offset of the start of the command ring from the start |
124 | of the memory region, to account for the mailbox size. | |
125 | cmdr_size - The size of the command ring. This does *not* need to be a | |
126 | power of two. | |
127 | cmd_head - Modified by the kernel to indicate when a command has been | |
128 | placed on the ring. | |
129 | cmd_tail - Modified by userspace to indicate when it has completed | |
130 | processing of a command. | |
131 | ||
132 | The Command Ring: | |
133 | ||
134 | Commands are placed on the ring by the kernel incrementing | |
135 | mailbox.cmd_head by the size of the command, modulo cmdr_size, and | |
136 | then signaling userspace via uio_event_notify(). Once the command is | |
137 | completed, userspace updates mailbox.cmd_tail in the same way and | |
138 | signals the kernel via a 4-byte write(). When cmd_head equals | |
139 | cmd_tail, the ring is empty -- no commands are currently waiting to be | |
140 | processed by userspace. | |
141 | ||
0ad46af8 AG |
142 | TCMU commands are 8-byte aligned. They start with a common header |
143 | containing "len_op", a 32-bit value that stores the length, as well as | |
144 | the opcode in the lowest unused bits. It also contains cmd_id and | |
145 | flags fields for setting by the kernel (kflags) and userspace | |
146 | (uflags). | |
147 | ||
148 | Currently only two opcodes are defined, TCMU_OP_CMD and TCMU_OP_PAD. | |
149 | ||
150 | When the opcode is CMD, the entry in the command ring is a struct | |
151 | tcmu_cmd_entry. Userspace finds the SCSI CDB (Command Data Block) via | |
152 | tcmu_cmd_entry.req.cdb_off. This is an offset from the start of the | |
153 | overall shared memory region, not the entry. The data in/out buffers | |
154 | are accessible via tht req.iov[] array. iov_cnt contains the number of | |
155 | entries in iov[] needed to describe either the Data-In or Data-Out | |
156 | buffers. For bidirectional commands, iov_cnt specifies how many iovec | |
e4648b01 | 157 | entries cover the Data-Out area, and iov_bidi_cnt specifies how many |
0ad46af8 AG |
158 | iovec entries immediately after that in iov[] cover the Data-In |
159 | area. Just like other fields, iov.iov_base is an offset from the start | |
160 | of the region. | |
ce876851 AG |
161 | |
162 | When completing a command, userspace sets rsp.scsi_status, and | |
163 | rsp.sense_buffer if necessary. Userspace then increments | |
164 | mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the | |
165 | kernel via the UIO method, a 4-byte write to the file descriptor. | |
166 | ||
32c76de3 SY |
167 | If TCMU_MAILBOX_FLAG_CAP_OOOC is set for mailbox->flags, kernel is |
168 | capable of handling out-of-order completions. In this case, userspace can | |
169 | handle command in different order other than original. Since kernel would | |
170 | still process the commands in the same order it appeared in the command | |
171 | ring, userspace need to update the cmd->id when completing the | |
172 | command(a.k.a steal the original command's entry). | |
173 | ||
0ad46af8 AG |
174 | When the opcode is PAD, userspace only updates cmd_tail as above -- |
175 | it's a no-op. (The kernel inserts PAD entries to ensure each CMD entry | |
176 | is contiguous within the command ring.) | |
177 | ||
178 | More opcodes may be added in the future. If userspace encounters an | |
179 | opcode it does not handle, it must set UNKNOWN_OP bit (bit 0) in | |
180 | hdr.uflags, update cmd_tail, and proceed with processing additional | |
181 | commands, if any. | |
182 | ||
ce876851 AG |
183 | The Data Area: |
184 | ||
185 | This is shared-memory space after the command ring. The organization | |
186 | of this area is not defined in the TCMU interface, and userspace | |
187 | should access only the parts referenced by pending iovs. | |
188 | ||
189 | ||
190 | Device Discovery: | |
191 | ||
192 | Other devices may be using UIO besides TCMU. Unrelated user processes | |
193 | may also be handling different sets of TCMU devices. TCMU userspace | |
194 | processes must find their devices by scanning sysfs | |
195 | class/uio/uio*/name. For TCMU devices, these names will be of the | |
196 | format: | |
197 | ||
198 | tcm-user/<hba_num>/<device_name>/<subtype>/<path> | |
199 | ||
200 | where "tcm-user" is common for all TCMU-backed UIO devices. <hba_num> | |
201 | and <device_name> allow userspace to find the device's path in the | |
202 | kernel target's configfs tree. Assuming the usual mount point, it is | |
203 | found at: | |
204 | ||
205 | /sys/kernel/config/target/core/user_<hba_num>/<device_name> | |
206 | ||
207 | This location contains attributes such as "hw_block_size", that | |
208 | userspace needs to know for correct operation. | |
209 | ||
210 | <subtype> will be a userspace-process-unique string to identify the | |
211 | TCMU device as expecting to be backed by a certain handler, and <path> | |
212 | will be an additional handler-specific string for the user process to | |
213 | configure the device, if needed. The name cannot contain ':', due to | |
214 | LIO limitations. | |
215 | ||
216 | For all devices so discovered, the user handler opens /dev/uioX and | |
217 | calls mmap(): | |
218 | ||
219 | mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0) | |
220 | ||
221 | where size must be equal to the value read from | |
222 | /sys/class/uio/uioX/maps/map0/size. | |
223 | ||
224 | ||
225 | Device Events: | |
226 | ||
227 | If a new device is added or removed, a notification will be broadcast | |
228 | over netlink, using a generic netlink family name of "TCM-USER" and a | |
229 | multicast group named "config". This will include the UIO name as | |
230 | described in the previous section, as well as the UIO minor | |
231 | number. This should allow userspace to identify both the UIO device and | |
232 | the LIO device, so that after determining the device is supported | |
233 | (based on subtype) it can take the appropriate action. | |
234 | ||
235 | ||
236 | Other contingencies: | |
237 | ||
238 | Userspace handler process never attaches: | |
239 | ||
240 | - TCMU will post commands, and then abort them after a timeout period | |
241 | (30 seconds.) | |
242 | ||
243 | Userspace handler process is killed: | |
244 | ||
245 | - It is still possible to restart and re-connect to TCMU | |
246 | devices. Command ring is preserved. However, after the timeout period, | |
247 | the kernel will abort pending tasks. | |
248 | ||
249 | Userspace handler process hangs: | |
250 | ||
251 | - The kernel will abort pending tasks after a timeout period. | |
252 | ||
253 | Userspace handler process is malicious: | |
254 | ||
255 | - The process can trivially break the handling of devices it controls, | |
256 | but should not be able to access kernel memory outside its shared | |
257 | memory areas. | |
258 | ||
259 | ||
260 | Writing a user pass-through handler (with example code) | |
261 | ------------------------------------------------------- | |
262 | ||
263 | A user process handing a TCMU device must support the following: | |
264 | ||
265 | a) Discovering and configuring TCMU uio devices | |
266 | b) Waiting for events on the device(s) | |
267 | c) Managing the command ring: Parsing operations and commands, | |
268 | performing work as needed, setting response fields (scsi_status and | |
269 | possibly sense_buffer), updating cmd_tail, and notifying the kernel | |
270 | that work has been finished | |
271 | ||
272 | First, consider instead writing a plugin for tcmu-runner. tcmu-runner | |
273 | implements all of this, and provides a higher-level API for plugin | |
274 | authors. | |
275 | ||
276 | TCMU is designed so that multiple unrelated processes can manage TCMU | |
277 | devices separately. All handlers should make sure to only open their | |
278 | devices, based opon a known subtype string. | |
279 | ||
280 | a) Discovering and configuring TCMU UIO devices: | |
281 | ||
282 | (error checking omitted for brevity) | |
283 | ||
284 | int fd, dev_fd; | |
285 | char buf[256]; | |
286 | unsigned long long map_len; | |
287 | void *map; | |
288 | ||
289 | fd = open("/sys/class/uio/uio0/name", O_RDONLY); | |
290 | ret = read(fd, buf, sizeof(buf)); | |
291 | close(fd); | |
292 | buf[ret-1] = '\0'; /* null-terminate and chop off the \n */ | |
293 | ||
294 | /* we only want uio devices whose name is a format we expect */ | |
295 | if (strncmp(buf, "tcm-user", 8)) | |
296 | exit(-1); | |
297 | ||
298 | /* Further checking for subtype also needed here */ | |
299 | ||
300 | fd = open(/sys/class/uio/%s/maps/map0/size, O_RDONLY); | |
301 | ret = read(fd, buf, sizeof(buf)); | |
302 | close(fd); | |
303 | str_buf[ret-1] = '\0'; /* null-terminate and chop off the \n */ | |
304 | ||
305 | map_len = strtoull(buf, NULL, 0); | |
306 | ||
307 | dev_fd = open("/dev/uio0", O_RDWR); | |
308 | map = mmap(NULL, map_len, PROT_READ|PROT_WRITE, MAP_SHARED, dev_fd, 0); | |
309 | ||
310 | ||
311 | b) Waiting for events on the device(s) | |
312 | ||
313 | while (1) { | |
314 | char buf[4]; | |
315 | ||
316 | int ret = read(dev_fd, buf, 4); /* will block */ | |
317 | ||
318 | handle_device_events(dev_fd, map); | |
319 | } | |
320 | ||
321 | ||
322 | c) Managing the command ring | |
323 | ||
324 | #include <linux/target_core_user.h> | |
325 | ||
326 | int handle_device_events(int fd, void *map) | |
327 | { | |
328 | struct tcmu_mailbox *mb = map; | |
329 | struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; | |
330 | int did_some_work = 0; | |
331 | ||
332 | /* Process events from cmd ring until we catch up with cmd_head */ | |
333 | while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) { | |
334 | ||
cf87edc6 | 335 | if (tcmu_hdr_get_op(ent->hdr.len_op) == TCMU_OP_CMD) { |
ce876851 AG |
336 | uint8_t *cdb = (void *)mb + ent->req.cdb_off; |
337 | bool success = true; | |
338 | ||
339 | /* Handle command here. */ | |
340 | printf("SCSI opcode: 0x%x\n", cdb[0]); | |
341 | ||
342 | /* Set response fields */ | |
343 | if (success) | |
344 | ent->rsp.scsi_status = SCSI_NO_SENSE; | |
345 | else { | |
346 | /* Also fill in rsp->sense_buffer here */ | |
347 | ent->rsp.scsi_status = SCSI_CHECK_CONDITION; | |
348 | } | |
349 | } | |
cf87edc6 AG |
350 | else if (tcmu_hdr_get_op(ent->hdr.len_op) != TCMU_OP_PAD) { |
351 | /* Tell the kernel we didn't handle unknown opcodes */ | |
352 | ent->hdr.uflags |= TCMU_UFLAG_UNKNOWN_OP; | |
353 | } | |
ce876851 | 354 | else { |
cf87edc6 | 355 | /* Do nothing for PAD entries except update cmd_tail */ |
ce876851 AG |
356 | } |
357 | ||
358 | /* update cmd_tail */ | |
359 | mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size; | |
360 | ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; | |
361 | did_some_work = 1; | |
362 | } | |
363 | ||
364 | /* Notify the kernel that work has been finished */ | |
365 | if (did_some_work) { | |
366 | uint32_t buf = 0; | |
367 | ||
368 | write(fd, &buf, 4); | |
369 | } | |
370 | ||
371 | return 0; | |
372 | } | |
373 | ||
374 | ||
ce876851 AG |
375 | A final note |
376 | ------------ | |
377 | ||
378 | Please be careful to return codes as defined by the SCSI | |
379 | specifications. These are different than some values defined in the | |
380 | scsi/scsi.h include file. For example, CHECK CONDITION's status code | |
381 | is 2, not 1. |