Commit | Line | Data |
---|---|---|
4ca9bc22 MCC |
1 | ==================== |
2 | TCM Userspace Design | |
3 | ==================== | |
4 | ||
5 | ||
6 | .. Contents: | |
7 | ||
c44166fe | 8 | 1) Design |
4ca9bc22 MCC |
9 | a) Background |
10 | b) Benefits | |
11 | c) Design constraints | |
12 | d) Implementation overview | |
13 | i. Mailbox | |
14 | ii. Command ring | |
15 | iii. Data Area | |
16 | e) Device discovery | |
17 | f) Device events | |
18 | g) Other contingencies | |
19 | 2) Writing a user pass-through handler | |
20 | a) Discovering and configuring TCMU uio devices | |
21 | b) Waiting for events on the device(s) | |
22 | c) Managing the command ring | |
23 | 3) A final note | |
ce876851 AG |
24 | |
25 | ||
c44166fe MCC |
26 | Design |
27 | ====== | |
ce876851 AG |
28 | |
29 | TCM is another name for LIO, an in-kernel iSCSI target (server). | |
30 | Existing TCM targets run in the kernel. TCMU (TCM in Userspace) | |
31 | allows userspace programs to be written which act as iSCSI targets. | |
32 | This document describes the design. | |
33 | ||
34 | The existing kernel provides modules for different SCSI transport | |
35 | protocols. TCM also modularizes the data storage. There are existing | |
36 | modules for file, block device, RAM or using another SCSI device as | |
37 | storage. These are called "backstores" or "storage engines". These | |
38 | built-in modules are implemented entirely as kernel code. | |
39 | ||
4ca9bc22 MCC |
40 | Background |
41 | ---------- | |
ce876851 AG |
42 | |
43 | In addition to modularizing the transport protocol used for carrying | |
44 | SCSI commands ("fabrics"), the Linux kernel target, LIO, also modularizes | |
45 | the actual data storage as well. These are referred to as "backstores" | |
46 | or "storage engines". The target comes with backstores that allow a | |
47 | file, a block device, RAM, or another SCSI device to be used for the | |
48 | local storage needed for the exported SCSI LUN. Like the rest of LIO, | |
49 | these are implemented entirely as kernel code. | |
50 | ||
51 | These backstores cover the most common use cases, but not all. One new | |
52 | use case that other non-kernel target solutions, such as tgt, are able | |
53 | to support is using Gluster's GLFS or Ceph's RBD as a backstore. The | |
54 | target then serves as a translator, allowing initiators to store data | |
55 | in these non-traditional networked storage systems, while still only | |
56 | using standard protocols themselves. | |
57 | ||
58 | If the target is a userspace process, supporting these is easy. tgt, | |
59 | for example, needs only a small adapter module for each, because the | |
60 | modules just use the available userspace libraries for RBD and GLFS. | |
61 | ||
62 | Adding support for these backstores in LIO is considerably more | |
63 | difficult, because LIO is entirely kernel code. Instead of undertaking | |
64 | the significant work to port the GLFS or RBD APIs and protocols to the | |
65 | kernel, another approach is to create a userspace pass-through | |
66 | backstore for LIO, "TCMU". | |
67 | ||
68 | ||
4ca9bc22 MCC |
69 | Benefits |
70 | -------- | |
ce876851 AG |
71 | |
72 | In addition to allowing relatively easy support for RBD and GLFS, TCMU | |
73 | will also allow easier development of new backstores. TCMU combines | |
74 | with the LIO loopback fabric to become something similar to FUSE | |
75 | (Filesystem in Userspace), but at the SCSI layer instead of the | |
76 | filesystem layer. A SUSE, if you will. | |
77 | ||
78 | The disadvantage is there are more distinct components to configure, and | |
79 | potentially to malfunction. This is unavoidable, but hopefully not | |
80 | fatal if we're careful to keep things as simple as possible. | |
81 | ||
4ca9bc22 MCC |
82 | Design constraints |
83 | ------------------ | |
ce876851 AG |
84 | |
85 | - Good performance: high throughput, low latency | |
86 | - Cleanly handle if userspace: | |
4ca9bc22 | 87 | |
ce876851 AG |
88 | 1) never attaches |
89 | 2) hangs | |
90 | 3) dies | |
91 | 4) misbehaves | |
4ca9bc22 | 92 | |
ce876851 AG |
93 | - Allow future flexibility in user & kernel implementations |
94 | - Be reasonably memory-efficient | |
95 | - Simple to configure & run | |
96 | - Simple to write a userspace backend | |
97 | ||
98 | ||
4ca9bc22 MCC |
99 | Implementation overview |
100 | ----------------------- | |
ce876851 AG |
101 | |
102 | The core of the TCMU interface is a memory region that is shared | |
103 | between kernel and userspace. Within this region is: a control area | |
104 | (mailbox); a lockless producer/consumer circular buffer for commands | |
105 | to be passed up, and status returned; and an in/out data buffer area. | |
106 | ||
107 | TCMU uses the pre-existing UIO subsystem. UIO allows device driver | |
108 | development in userspace, and this is conceptually very close to the | |
109 | TCMU use case, except instead of a physical device, TCMU implements a | |
110 | memory-mapped layout designed for SCSI commands. Using UIO also | |
111 | benefits TCMU by handling device introspection (e.g. a way for | |
112 | userspace to determine how large the shared region is) and signaling | |
113 | mechanisms in both directions. | |
114 | ||
115 | There are no embedded pointers in the memory region. Everything is | |
116 | expressed as an offset from the region's starting address. This allows | |
117 | the ring to still work if the user process dies and is restarted with | |
118 | the region mapped at a different virtual address. | |
119 | ||
120 | See target_core_user.h for the struct definitions. | |
121 | ||
4ca9bc22 MCC |
122 | The Mailbox |
123 | ----------- | |
ce876851 AG |
124 | |
125 | The mailbox is always at the start of the shared memory region, and | |
126 | contains a version, details about the starting offset and size of the | |
127 | command ring, and head and tail pointers to be used by the kernel and | |
128 | userspace (respectively) to put commands on the ring, and indicate | |
129 | when the commands are completed. | |
130 | ||
131 | version - 1 (userspace should abort if otherwise) | |
4ca9bc22 | 132 | |
32c76de3 | 133 | flags: |
4ca9bc22 MCC |
134 | - TCMU_MAILBOX_FLAG_CAP_OOOC: |
135 | indicates out-of-order completion is supported. | |
136 | See "The Command Ring" for details. | |
137 | ||
138 | cmdr_off | |
139 | The offset of the start of the command ring from the start | |
140 | of the memory region, to account for the mailbox size. | |
141 | cmdr_size | |
142 | The size of the command ring. This does *not* need to be a | |
143 | power of two. | |
144 | cmd_head | |
145 | Modified by the kernel to indicate when a command has been | |
146 | placed on the ring. | |
147 | cmd_tail | |
148 | Modified by userspace to indicate when it has completed | |
149 | processing of a command. | |
150 | ||
151 | The Command Ring | |
152 | ---------------- | |
ce876851 AG |
153 | |
154 | Commands are placed on the ring by the kernel incrementing | |
155 | mailbox.cmd_head by the size of the command, modulo cmdr_size, and | |
156 | then signaling userspace via uio_event_notify(). Once the command is | |
157 | completed, userspace updates mailbox.cmd_tail in the same way and | |
158 | signals the kernel via a 4-byte write(). When cmd_head equals | |
159 | cmd_tail, the ring is empty -- no commands are currently waiting to be | |
160 | processed by userspace. | |
161 | ||
0ad46af8 AG |
162 | TCMU commands are 8-byte aligned. They start with a common header |
163 | containing "len_op", a 32-bit value that stores the length, as well as | |
164 | the opcode in the lowest unused bits. It also contains cmd_id and | |
165 | flags fields for setting by the kernel (kflags) and userspace | |
166 | (uflags). | |
167 | ||
168 | Currently only two opcodes are defined, TCMU_OP_CMD and TCMU_OP_PAD. | |
169 | ||
170 | When the opcode is CMD, the entry in the command ring is a struct | |
171 | tcmu_cmd_entry. Userspace finds the SCSI CDB (Command Data Block) via | |
172 | tcmu_cmd_entry.req.cdb_off. This is an offset from the start of the | |
173 | overall shared memory region, not the entry. The data in/out buffers | |
c57ac574 | 174 | are accessible via the req.iov[] array. iov_cnt contains the number of |
0ad46af8 AG |
175 | entries in iov[] needed to describe either the Data-In or Data-Out |
176 | buffers. For bidirectional commands, iov_cnt specifies how many iovec | |
e4648b01 | 177 | entries cover the Data-Out area, and iov_bidi_cnt specifies how many |
0ad46af8 AG |
178 | iovec entries immediately after that in iov[] cover the Data-In |
179 | area. Just like other fields, iov.iov_base is an offset from the start | |
180 | of the region. | |
ce876851 AG |
181 | |
182 | When completing a command, userspace sets rsp.scsi_status, and | |
183 | rsp.sense_buffer if necessary. Userspace then increments | |
184 | mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the | |
185 | kernel via the UIO method, a 4-byte write to the file descriptor. | |
186 | ||
32c76de3 SY |
187 | If TCMU_MAILBOX_FLAG_CAP_OOOC is set for mailbox->flags, kernel is |
188 | capable of handling out-of-order completions. In this case, userspace can | |
189 | handle command in different order other than original. Since kernel would | |
190 | still process the commands in the same order it appeared in the command | |
191 | ring, userspace need to update the cmd->id when completing the | |
192 | command(a.k.a steal the original command's entry). | |
193 | ||
0ad46af8 AG |
194 | When the opcode is PAD, userspace only updates cmd_tail as above -- |
195 | it's a no-op. (The kernel inserts PAD entries to ensure each CMD entry | |
196 | is contiguous within the command ring.) | |
197 | ||
198 | More opcodes may be added in the future. If userspace encounters an | |
199 | opcode it does not handle, it must set UNKNOWN_OP bit (bit 0) in | |
200 | hdr.uflags, update cmd_tail, and proceed with processing additional | |
201 | commands, if any. | |
202 | ||
4ca9bc22 MCC |
203 | The Data Area |
204 | ------------- | |
ce876851 AG |
205 | |
206 | This is shared-memory space after the command ring. The organization | |
207 | of this area is not defined in the TCMU interface, and userspace | |
208 | should access only the parts referenced by pending iovs. | |
209 | ||
210 | ||
4ca9bc22 MCC |
211 | Device Discovery |
212 | ---------------- | |
ce876851 AG |
213 | |
214 | Other devices may be using UIO besides TCMU. Unrelated user processes | |
215 | may also be handling different sets of TCMU devices. TCMU userspace | |
216 | processes must find their devices by scanning sysfs | |
217 | class/uio/uio*/name. For TCMU devices, these names will be of the | |
4ca9bc22 | 218 | format:: |
ce876851 | 219 | |
4ca9bc22 | 220 | tcm-user/<hba_num>/<device_name>/<subtype>/<path> |
ce876851 AG |
221 | |
222 | where "tcm-user" is common for all TCMU-backed UIO devices. <hba_num> | |
223 | and <device_name> allow userspace to find the device's path in the | |
224 | kernel target's configfs tree. Assuming the usual mount point, it is | |
4ca9bc22 | 225 | found at:: |
ce876851 | 226 | |
4ca9bc22 | 227 | /sys/kernel/config/target/core/user_<hba_num>/<device_name> |
ce876851 AG |
228 | |
229 | This location contains attributes such as "hw_block_size", that | |
230 | userspace needs to know for correct operation. | |
231 | ||
232 | <subtype> will be a userspace-process-unique string to identify the | |
233 | TCMU device as expecting to be backed by a certain handler, and <path> | |
234 | will be an additional handler-specific string for the user process to | |
235 | configure the device, if needed. The name cannot contain ':', due to | |
236 | LIO limitations. | |
237 | ||
238 | For all devices so discovered, the user handler opens /dev/uioX and | |
4ca9bc22 | 239 | calls mmap():: |
ce876851 | 240 | |
4ca9bc22 | 241 | mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0) |
ce876851 AG |
242 | |
243 | where size must be equal to the value read from | |
244 | /sys/class/uio/uioX/maps/map0/size. | |
245 | ||
246 | ||
4ca9bc22 MCC |
247 | Device Events |
248 | ------------- | |
ce876851 AG |
249 | |
250 | If a new device is added or removed, a notification will be broadcast | |
251 | over netlink, using a generic netlink family name of "TCM-USER" and a | |
252 | multicast group named "config". This will include the UIO name as | |
253 | described in the previous section, as well as the UIO minor | |
254 | number. This should allow userspace to identify both the UIO device and | |
255 | the LIO device, so that after determining the device is supported | |
256 | (based on subtype) it can take the appropriate action. | |
257 | ||
258 | ||
4ca9bc22 MCC |
259 | Other contingencies |
260 | ------------------- | |
ce876851 AG |
261 | |
262 | Userspace handler process never attaches: | |
263 | ||
264 | - TCMU will post commands, and then abort them after a timeout period | |
265 | (30 seconds.) | |
266 | ||
267 | Userspace handler process is killed: | |
268 | ||
269 | - It is still possible to restart and re-connect to TCMU | |
270 | devices. Command ring is preserved. However, after the timeout period, | |
271 | the kernel will abort pending tasks. | |
272 | ||
273 | Userspace handler process hangs: | |
274 | ||
275 | - The kernel will abort pending tasks after a timeout period. | |
276 | ||
277 | Userspace handler process is malicious: | |
278 | ||
279 | - The process can trivially break the handling of devices it controls, | |
280 | but should not be able to access kernel memory outside its shared | |
281 | memory areas. | |
282 | ||
283 | ||
284 | Writing a user pass-through handler (with example code) | |
4ca9bc22 | 285 | ======================================================= |
ce876851 AG |
286 | |
287 | A user process handing a TCMU device must support the following: | |
288 | ||
289 | a) Discovering and configuring TCMU uio devices | |
290 | b) Waiting for events on the device(s) | |
291 | c) Managing the command ring: Parsing operations and commands, | |
292 | performing work as needed, setting response fields (scsi_status and | |
293 | possibly sense_buffer), updating cmd_tail, and notifying the kernel | |
294 | that work has been finished | |
295 | ||
296 | First, consider instead writing a plugin for tcmu-runner. tcmu-runner | |
297 | implements all of this, and provides a higher-level API for plugin | |
298 | authors. | |
299 | ||
300 | TCMU is designed so that multiple unrelated processes can manage TCMU | |
301 | devices separately. All handlers should make sure to only open their | |
302 | devices, based opon a known subtype string. | |
303 | ||
4ca9bc22 | 304 | a) Discovering and configuring TCMU UIO devices:: |
ce876851 | 305 | |
4ca9bc22 | 306 | /* error checking omitted for brevity */ |
ce876851 | 307 | |
4ca9bc22 MCC |
308 | int fd, dev_fd; |
309 | char buf[256]; | |
310 | unsigned long long map_len; | |
311 | void *map; | |
ce876851 | 312 | |
4ca9bc22 MCC |
313 | fd = open("/sys/class/uio/uio0/name", O_RDONLY); |
314 | ret = read(fd, buf, sizeof(buf)); | |
315 | close(fd); | |
316 | buf[ret-1] = '\0'; /* null-terminate and chop off the \n */ | |
ce876851 | 317 | |
4ca9bc22 MCC |
318 | /* we only want uio devices whose name is a format we expect */ |
319 | if (strncmp(buf, "tcm-user", 8)) | |
ce876851 AG |
320 | exit(-1); |
321 | ||
4ca9bc22 | 322 | /* Further checking for subtype also needed here */ |
ce876851 | 323 | |
4ca9bc22 MCC |
324 | fd = open(/sys/class/uio/%s/maps/map0/size, O_RDONLY); |
325 | ret = read(fd, buf, sizeof(buf)); | |
326 | close(fd); | |
327 | str_buf[ret-1] = '\0'; /* null-terminate and chop off the \n */ | |
ce876851 | 328 | |
4ca9bc22 | 329 | map_len = strtoull(buf, NULL, 0); |
ce876851 | 330 | |
4ca9bc22 MCC |
331 | dev_fd = open("/dev/uio0", O_RDWR); |
332 | map = mmap(NULL, map_len, PROT_READ|PROT_WRITE, MAP_SHARED, dev_fd, 0); | |
ce876851 | 333 | |
ce876851 | 334 | |
4ca9bc22 | 335 | b) Waiting for events on the device(s) |
ce876851 | 336 | |
4ca9bc22 MCC |
337 | while (1) { |
338 | char buf[4]; | |
ce876851 | 339 | |
4ca9bc22 | 340 | int ret = read(dev_fd, buf, 4); /* will block */ |
ce876851 | 341 | |
4ca9bc22 | 342 | handle_device_events(dev_fd, map); |
ce876851 | 343 | } |
ce876851 | 344 | |
ce876851 | 345 | |
4ca9bc22 MCC |
346 | c) Managing the command ring:: |
347 | ||
348 | #include <linux/target_core_user.h> | |
349 | ||
350 | int handle_device_events(int fd, void *map) | |
351 | { | |
352 | struct tcmu_mailbox *mb = map; | |
353 | struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; | |
354 | int did_some_work = 0; | |
355 | ||
356 | /* Process events from cmd ring until we catch up with cmd_head */ | |
357 | while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) { | |
358 | ||
359 | if (tcmu_hdr_get_op(ent->hdr.len_op) == TCMU_OP_CMD) { | |
360 | uint8_t *cdb = (void *)mb + ent->req.cdb_off; | |
361 | bool success = true; | |
362 | ||
363 | /* Handle command here. */ | |
364 | printf("SCSI opcode: 0x%x\n", cdb[0]); | |
365 | ||
366 | /* Set response fields */ | |
367 | if (success) | |
368 | ent->rsp.scsi_status = SCSI_NO_SENSE; | |
369 | else { | |
370 | /* Also fill in rsp->sense_buffer here */ | |
371 | ent->rsp.scsi_status = SCSI_CHECK_CONDITION; | |
372 | } | |
373 | } | |
374 | else if (tcmu_hdr_get_op(ent->hdr.len_op) != TCMU_OP_PAD) { | |
375 | /* Tell the kernel we didn't handle unknown opcodes */ | |
376 | ent->hdr.uflags |= TCMU_UFLAG_UNKNOWN_OP; | |
377 | } | |
378 | else { | |
379 | /* Do nothing for PAD entries except update cmd_tail */ | |
380 | } | |
381 | ||
382 | /* update cmd_tail */ | |
383 | mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size; | |
384 | ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; | |
385 | did_some_work = 1; | |
386 | } | |
387 | ||
388 | /* Notify the kernel that work has been finished */ | |
389 | if (did_some_work) { | |
390 | uint32_t buf = 0; | |
391 | ||
392 | write(fd, &buf, 4); | |
393 | } | |
394 | ||
395 | return 0; | |
396 | } | |
ce876851 AG |
397 | |
398 | ||
ce876851 | 399 | A final note |
4ca9bc22 | 400 | ============ |
ce876851 AG |
401 | |
402 | Please be careful to return codes as defined by the SCSI | |
403 | specifications. These are different than some values defined in the | |
404 | scsi/scsi.h include file. For example, CHECK CONDITION's status code | |
405 | is 2, not 1. |