Commit | Line | Data |
---|---|---|
1da177e4 LT |
1 | USERSPACE MAD ACCESS |
2 | ||
3 | Device files | |
4 | ||
5 | Each port of each InfiniBand device has a "umad" device and an | |
6 | "issm" device attached. For example, a two-port HCA will have two | |
7 | umad devices and two issm devices, while a switch will have one | |
8 | device of each type (for switch port 0). | |
9 | ||
10 | Creating MAD agents | |
11 | ||
12 | A MAD agent can be created by filling in a struct ib_user_mad_reg_req | |
13 | and then calling the IB_USER_MAD_REGISTER_AGENT ioctl on a file | |
14 | descriptor for the appropriate device file. If the registration | |
15 | request succeeds, a 32-bit id will be returned in the structure. | |
16 | For example: | |
17 | ||
18 | struct ib_user_mad_reg_req req = { /* ... */ }; | |
19 | ret = ioctl(fd, IB_USER_MAD_REGISTER_AGENT, (char *) &req); | |
20 | if (!ret) | |
21 | my_agent = req.id; | |
22 | else | |
23 | perror("agent register"); | |
24 | ||
25 | Agents can be unregistered with the IB_USER_MAD_UNREGISTER_AGENT | |
26 | ioctl. Also, all agents registered through a file descriptor will | |
27 | be unregistered when the descriptor is closed. | |
28 | ||
29 | Receiving MADs | |
30 | ||
3f75dadd HR |
31 | MADs are received using read(). The receive side now supports |
32 | RMPP. The buffer passed to read() must be at least one | |
33 | struct ib_user_mad + 256 bytes. For example: | |
34 | ||
35 | If the buffer passed is not large enough to hold the received | |
36 | MAD (RMPP), the errno is set to ENOSPC and the length of the | |
37 | buffer needed is set in mad.length. | |
38 | ||
39 | Example for normal MAD (non RMPP) reads: | |
40 | struct ib_user_mad *mad; | |
41 | mad = malloc(sizeof *mad + 256); | |
42 | ret = read(fd, mad, sizeof *mad + 256); | |
43 | if (ret != sizeof mad + 256) { | |
44 | perror("read"); | |
45 | free(mad); | |
46 | } | |
47 | ||
48 | Example for RMPP reads: | |
49 | struct ib_user_mad *mad; | |
50 | mad = malloc(sizeof *mad + 256); | |
51 | ret = read(fd, mad, sizeof *mad + 256); | |
52 | if (ret == -ENOSPC)) { | |
53 | length = mad.length; | |
54 | free(mad); | |
55 | mad = malloc(sizeof *mad + length); | |
56 | ret = read(fd, mad, sizeof *mad + length); | |
57 | } | |
58 | if (ret < 0) { | |
1da177e4 | 59 | perror("read"); |
3f75dadd HR |
60 | free(mad); |
61 | } | |
1da177e4 LT |
62 | |
63 | In addition to the actual MAD contents, the other struct ib_user_mad | |
64 | fields will be filled in with information on the received MAD. For | |
65 | example, the remote LID will be in mad.lid. | |
66 | ||
67 | If a send times out, a receive will be generated with mad.status set | |
68 | to ETIMEDOUT. Otherwise when a MAD has been successfully received, | |
69 | mad.status will be 0. | |
70 | ||
71 | poll()/select() may be used to wait until a MAD can be read. | |
72 | ||
73 | Sending MADs | |
74 | ||
75 | MADs are sent using write(). The agent ID for sending should be | |
76 | filled into the id field of the MAD, the destination LID should be | |
3f75dadd HR |
77 | filled into the lid field, and so on. The send side does support |
78 | RMPP so arbitrary length MAD can be sent. For example: | |
79 | ||
80 | struct ib_user_mad *mad; | |
1da177e4 | 81 | |
3f75dadd | 82 | mad = malloc(sizeof *mad + mad_length); |
1da177e4 | 83 | |
3f75dadd | 84 | /* fill in mad->data */ |
1da177e4 | 85 | |
3f75dadd HR |
86 | mad->hdr.id = my_agent; /* req.id from agent registration */ |
87 | mad->hdr.lid = my_dest; /* in network byte order... */ | |
1da177e4 LT |
88 | /* etc. */ |
89 | ||
3f75dadd HR |
90 | ret = write(fd, &mad, sizeof *mad + mad_length); |
91 | if (ret != sizeof *mad + mad_length) | |
1da177e4 LT |
92 | perror("write"); |
93 | ||
94 | Setting IsSM Capability Bit | |
95 | ||
96 | To set the IsSM capability bit for a port, simply open the | |
97 | corresponding issm device file. If the IsSM bit is already set, | |
98 | then the open call will block until the bit is cleared (or return | |
99 | immediately with errno set to EAGAIN if the O_NONBLOCK flag is | |
100 | passed to open()). The IsSM bit will be cleared when the issm file | |
101 | is closed. No read, write or other operations can be performed on | |
102 | the issm file. | |
103 | ||
104 | /dev files | |
105 | ||
106 | To create the appropriate character device files automatically with | |
107 | udev, a rule like | |
108 | ||
109 | KERNEL="umad*", NAME="infiniband/%k" | |
110 | KERNEL="issm*", NAME="infiniband/%k" | |
111 | ||
112 | can be used. This will create device nodes named | |
113 | ||
114 | /dev/infiniband/umad0 | |
115 | /dev/infiniband/issm0 | |
116 | ||
117 | for the first port, and so on. The InfiniBand device and port | |
118 | associated with these devices can be determined from the files | |
119 | ||
120 | /sys/class/infiniband_mad/umad0/ibdev | |
121 | /sys/class/infiniband_mad/umad0/port | |
122 | ||
123 | and | |
124 | ||
125 | /sys/class/infiniband_mad/issm0/ibdev | |
126 | /sys/class/infiniband_mad/issm0/port |