Commit | Line | Data |
---|---|---|
97162a1e MCC |
1 | ==================== |
2 | Userspace MAD access | |
3 | ==================== | |
1da177e4 LT |
4 | |
5 | Device files | |
97162a1e | 6 | ============ |
1da177e4 LT |
7 | |
8 | Each port of each InfiniBand device has a "umad" device and an | |
9 | "issm" device attached. For example, a two-port HCA will have two | |
10 | umad devices and two issm devices, while a switch will have one | |
11 | device of each type (for switch port 0). | |
12 | ||
13 | Creating MAD agents | |
97162a1e | 14 | =================== |
1da177e4 LT |
15 | |
16 | A MAD agent can be created by filling in a struct ib_user_mad_reg_req | |
17 | and then calling the IB_USER_MAD_REGISTER_AGENT ioctl on a file | |
18 | descriptor for the appropriate device file. If the registration | |
19 | request succeeds, a 32-bit id will be returned in the structure. | |
97162a1e | 20 | For example:: |
1da177e4 LT |
21 | |
22 | struct ib_user_mad_reg_req req = { /* ... */ }; | |
23 | ret = ioctl(fd, IB_USER_MAD_REGISTER_AGENT, (char *) &req); | |
24 | if (!ret) | |
25 | my_agent = req.id; | |
26 | else | |
27 | perror("agent register"); | |
28 | ||
29 | Agents can be unregistered with the IB_USER_MAD_UNREGISTER_AGENT | |
30 | ioctl. Also, all agents registered through a file descriptor will | |
31 | be unregistered when the descriptor is closed. | |
32 | ||
97162a1e MCC |
33 | 2014 |
34 | a new registration ioctl is now provided which allows additional | |
0f29b46d IW |
35 | fields to be provided during registration. |
36 | Users of this registration call are implicitly setting the use of | |
37 | pkey_index (see below). | |
38 | ||
1da177e4 | 39 | Receiving MADs |
97162a1e | 40 | ============== |
1da177e4 | 41 | |
3f75dadd HR |
42 | MADs are received using read(). The receive side now supports |
43 | RMPP. The buffer passed to read() must be at least one | |
44 | struct ib_user_mad + 256 bytes. For example: | |
45 | ||
46 | If the buffer passed is not large enough to hold the received | |
47 | MAD (RMPP), the errno is set to ENOSPC and the length of the | |
48 | buffer needed is set in mad.length. | |
49 | ||
97162a1e MCC |
50 | Example for normal MAD (non RMPP) reads:: |
51 | ||
3f75dadd HR |
52 | struct ib_user_mad *mad; |
53 | mad = malloc(sizeof *mad + 256); | |
54 | ret = read(fd, mad, sizeof *mad + 256); | |
55 | if (ret != sizeof mad + 256) { | |
56 | perror("read"); | |
57 | free(mad); | |
58 | } | |
59 | ||
97162a1e MCC |
60 | Example for RMPP reads:: |
61 | ||
3f75dadd HR |
62 | struct ib_user_mad *mad; |
63 | mad = malloc(sizeof *mad + 256); | |
64 | ret = read(fd, mad, sizeof *mad + 256); | |
65 | if (ret == -ENOSPC)) { | |
66 | length = mad.length; | |
67 | free(mad); | |
68 | mad = malloc(sizeof *mad + length); | |
69 | ret = read(fd, mad, sizeof *mad + length); | |
70 | } | |
71 | if (ret < 0) { | |
1da177e4 | 72 | perror("read"); |
3f75dadd HR |
73 | free(mad); |
74 | } | |
1da177e4 LT |
75 | |
76 | In addition to the actual MAD contents, the other struct ib_user_mad | |
77 | fields will be filled in with information on the received MAD. For | |
78 | example, the remote LID will be in mad.lid. | |
79 | ||
80 | If a send times out, a receive will be generated with mad.status set | |
81 | to ETIMEDOUT. Otherwise when a MAD has been successfully received, | |
82 | mad.status will be 0. | |
83 | ||
84 | poll()/select() may be used to wait until a MAD can be read. | |
85 | ||
86 | Sending MADs | |
97162a1e | 87 | ============ |
1da177e4 LT |
88 | |
89 | MADs are sent using write(). The agent ID for sending should be | |
90 | filled into the id field of the MAD, the destination LID should be | |
3f75dadd | 91 | filled into the lid field, and so on. The send side does support |
97162a1e | 92 | RMPP so arbitrary length MAD can be sent. For example:: |
3f75dadd HR |
93 | |
94 | struct ib_user_mad *mad; | |
1da177e4 | 95 | |
3f75dadd | 96 | mad = malloc(sizeof *mad + mad_length); |
1da177e4 | 97 | |
3f75dadd | 98 | /* fill in mad->data */ |
1da177e4 | 99 | |
3f75dadd HR |
100 | mad->hdr.id = my_agent; /* req.id from agent registration */ |
101 | mad->hdr.lid = my_dest; /* in network byte order... */ | |
1da177e4 LT |
102 | /* etc. */ |
103 | ||
3f75dadd HR |
104 | ret = write(fd, &mad, sizeof *mad + mad_length); |
105 | if (ret != sizeof *mad + mad_length) | |
1da177e4 LT |
106 | perror("write"); |
107 | ||
bd8031b4 | 108 | Transaction IDs |
97162a1e | 109 | =============== |
bd8031b4 HR |
110 | |
111 | Users of the umad devices can use the lower 32 bits of the | |
112 | transaction ID field (that is, the least significant half of the | |
113 | field in network byte order) in MADs being sent to match | |
114 | request/response pairs. The upper 32 bits are reserved for use by | |
115 | the kernel and will be overwritten before a MAD is sent. | |
116 | ||
2be8e3ee | 117 | P_Key Index Handling |
97162a1e | 118 | ==================== |
2be8e3ee RD |
119 | |
120 | The old ib_umad interface did not allow setting the P_Key index for | |
121 | MADs that are sent and did not provide a way for obtaining the P_Key | |
122 | index of received MADs. A new layout for struct ib_user_mad_hdr | |
0f29b46d IW |
123 | with a pkey_index member has been defined; however, to preserve binary |
124 | compatibility with older applications, this new layout will not be used | |
125 | unless one of IB_USER_MAD_ENABLE_PKEY or IB_USER_MAD_REGISTER_AGENT2 ioctl's | |
126 | are called before a file descriptor is used for anything else. | |
2be8e3ee RD |
127 | |
128 | In September 2008, the IB_USER_MAD_ABI_VERSION will be incremented | |
129 | to 6, the new layout of struct ib_user_mad_hdr will be used by | |
130 | default, and the IB_USER_MAD_ENABLE_PKEY ioctl will be removed. | |
131 | ||
1da177e4 | 132 | Setting IsSM Capability Bit |
97162a1e | 133 | =========================== |
1da177e4 LT |
134 | |
135 | To set the IsSM capability bit for a port, simply open the | |
136 | corresponding issm device file. If the IsSM bit is already set, | |
137 | then the open call will block until the bit is cleared (or return | |
138 | immediately with errno set to EAGAIN if the O_NONBLOCK flag is | |
139 | passed to open()). The IsSM bit will be cleared when the issm file | |
140 | is closed. No read, write or other operations can be performed on | |
141 | the issm file. | |
142 | ||
143 | /dev files | |
97162a1e | 144 | ========== |
1da177e4 LT |
145 | |
146 | To create the appropriate character device files automatically with | |
97162a1e | 147 | udev, a rule like:: |
1da177e4 | 148 | |
aa07a994 BVA |
149 | KERNEL=="umad*", NAME="infiniband/%k" |
150 | KERNEL=="issm*", NAME="infiniband/%k" | |
1da177e4 | 151 | |
97162a1e | 152 | can be used. This will create device nodes named:: |
1da177e4 LT |
153 | |
154 | /dev/infiniband/umad0 | |
155 | /dev/infiniband/issm0 | |
156 | ||
157 | for the first port, and so on. The InfiniBand device and port | |
97162a1e | 158 | associated with these devices can be determined from the files:: |
1da177e4 LT |
159 | |
160 | /sys/class/infiniband_mad/umad0/ibdev | |
161 | /sys/class/infiniband_mad/umad0/port | |
162 | ||
97162a1e | 163 | and:: |
1da177e4 LT |
164 | |
165 | /sys/class/infiniband_mad/issm0/ibdev | |
166 | /sys/class/infiniband_mad/issm0/port |