Commit | Line | Data |
---|---|---|
9f1585cb SW |
1 | Glock internal locking rules |
2 | ------------------------------ | |
3 | ||
4 | This documents the basic principles of the glock state machine | |
5 | internals. Each glock (struct gfs2_glock in fs/gfs2/incore.h) | |
6 | has two main (internal) locks: | |
7 | ||
f3dd1649 | 8 | 1. A spinlock (gl_lockref.lock) which protects the internal state such |
9f1585cb SW |
9 | as gl_state, gl_target and the list of holders (gl_holders) |
10 | 2. A non-blocking bit lock, GLF_LOCK, which is used to prevent other | |
11 | threads from making calls to the DLM, etc. at the same time. If a | |
12 | thread takes this lock, it must then call run_queue (usually via the | |
13 | workqueue) when it releases it in order to ensure any pending tasks | |
14 | are completed. | |
15 | ||
16 | The gl_holders list contains all the queued lock requests (not | |
17 | just the holders) associated with the glock. If there are any | |
18 | held locks, then they will be contiguous entries at the head | |
19 | of the list. Locks are granted in strictly the order that they | |
20 | are queued, except for those marked LM_FLAG_PRIORITY which are | |
21 | used only during recovery, and even then only for journal locks. | |
22 | ||
23 | There are three lock states that users of the glock layer can request, | |
24 | namely shared (SH), deferred (DF) and exclusive (EX). Those translate | |
25 | to the following DLM lock modes: | |
26 | ||
27 | Glock mode | DLM lock mode | |
28 | ------------------------------ | |
29 | UN | IV/NL Unlocked (no DLM lock associated with glock) or NL | |
30 | SH | PR (Protected read) | |
31 | DF | CW (Concurrent write) | |
32 | EX | EX (Exclusive) | |
33 | ||
34 | Thus DF is basically a shared mode which is incompatible with the "normal" | |
35 | shared lock mode, SH. In GFS2 the DF mode is used exclusively for direct I/O | |
36 | operations. The glocks are basically a lock plus some routines which deal | |
37 | with cache management. The following rules apply for the cache: | |
38 | ||
39 | Glock mode | Cache data | Cache Metadata | Dirty Data | Dirty Metadata | |
40 | -------------------------------------------------------------------------- | |
41 | UN | No | No | No | No | |
42 | SH | Yes | Yes | No | No | |
43 | DF | No | Yes | No | No | |
44 | EX | Yes | Yes | Yes | Yes | |
45 | ||
46 | These rules are implemented using the various glock operations which | |
47 | are defined for each type of glock. Not all types of glocks use | |
48 | all the modes. Only inode glocks use the DF mode for example. | |
49 | ||
50 | Table of glock operations and per type constants: | |
51 | ||
52 | Field | Purpose | |
53 | ---------------------------------------------------------------------------- | |
54 | go_xmote_th | Called before remote state change (e.g. to sync dirty data) | |
55 | go_xmote_bh | Called after remote state change (e.g. to refill cache) | |
56 | go_inval | Called if remote state change requires invalidating the cache | |
57 | go_demote_ok | Returns boolean value of whether its ok to demote a glock | |
58 | | (e.g. checks timeout, and that there is no cached data) | |
59 | go_lock | Called for the first local holder of a lock | |
60 | go_unlock | Called on the final local unlock of a lock | |
61 | go_dump | Called to print content of object for debugfs file, or on | |
62 | | error to dump glock to the log. | |
e9ccb73a | 63 | go_type | The type of the glock, LM_TYPE_..... |
2ebc3f8b SW |
64 | go_callback | Called if the DLM sends a callback to drop this lock |
65 | go_flags | GLOF_ASPACE is set, if the glock has an address space | |
66 | | associated with it | |
9f1585cb SW |
67 | |
68 | The minimum hold time for each lock is the time after a remote lock | |
69 | grant for which we ignore remote demote requests. This is in order to | |
70 | prevent a situation where locks are being bounced around the cluster | |
71 | from node to node with none of the nodes making any progress. This | |
72 | tends to show up most with shared mmaped files which are being written | |
73 | to by multiple nodes. By delaying the demotion in response to a | |
74 | remote callback, that gives the userspace program time to make | |
75 | some progress before the pages are unmapped. | |
76 | ||
77 | There is a plan to try and remove the go_lock and go_unlock callbacks | |
78 | if possible, in order to try and speed up the fast path though the locking. | |
79 | Also, eventually we hope to make the glock "EX" mode locally shared | |
80 | such that any local locking will be done with the i_mutex as required | |
81 | rather than via the glock. | |
82 | ||
83 | Locking rules for glock operations: | |
84 | ||
f3dd1649 AG |
85 | Operation | GLF_LOCK bit lock held | gl_lockref.lock spinlock held |
86 | ------------------------------------------------------------------------- | |
9f1585cb SW |
87 | go_xmote_th | Yes | No |
88 | go_xmote_bh | Yes | No | |
89 | go_inval | Yes | No | |
90 | go_demote_ok | Sometimes | Yes | |
91 | go_lock | Yes | No | |
92 | go_unlock | Yes | No | |
93 | go_dump | Sometimes | Yes | |
2ebc3f8b | 94 | go_callback | Sometimes (N/A) | Yes |
9f1585cb SW |
95 | |
96 | N.B. Operations must not drop either the bit lock or the spinlock | |
97 | if its held on entry. go_dump and do_demote_ok must never block. | |
98 | Note that go_dump will only be called if the glock's state | |
99 | indicates that it is caching uptodate data. | |
100 | ||
101 | Glock locking order within GFS2: | |
102 | ||
103 | 1. i_mutex (if required) | |
104 | 2. Rename glock (for rename only) | |
105 | 3. Inode glock(s) | |
106 | (Parents before children, inodes at "same level" with same parent in | |
107 | lock number order) | |
108 | 4. Rgrp glock(s) (for (de)allocation operations) | |
109 | 5. Transaction glock (via gfs2_trans_begin) for non-read operations | |
110 | 6. Page lock (always last, very important!) | |
111 | ||
112 | There are two glocks per inode. One deals with access to the inode | |
113 | itself (locking order as above), and the other, known as the iopen | |
114 | glock is used in conjunction with the i_nlink field in the inode to | |
115 | determine the lifetime of the inode in question. Locking of inodes | |
116 | is on a per-inode basis. Locking of rgrps is on a per rgrp basis. | |
2ebc3f8b SW |
117 | In general we prefer to lock local locks prior to cluster locks. |
118 | ||
119 | Glock Statistics | |
120 | ------------------ | |
121 | ||
122 | The stats are divided into two sets: those relating to the | |
123 | super block and those relating to an individual glock. The | |
124 | super block stats are done on a per cpu basis in order to | |
125 | try and reduce the overhead of gathering them. They are also | |
126 | further divided by glock type. All timings are in nanoseconds. | |
127 | ||
128 | In the case of both the super block and glock statistics, | |
129 | the same information is gathered in each case. The super | |
130 | block timing statistics are used to provide default values for | |
131 | the glock timing statistics, so that newly created glocks | |
132 | should have, as far as possible, a sensible starting point. | |
133 | The per-glock counters are initialised to zero when the | |
134 | glock is created. The per-glock statistics are lost when | |
135 | the glock is ejected from memory. | |
136 | ||
137 | The statistics are divided into three pairs of mean and | |
138 | variance, plus two counters. The mean/variance pairs are | |
139 | smoothed exponential estimates and the algorithm used is | |
140 | one which will be very familiar to those used to calculation | |
141 | of round trip times in network code. See "TCP/IP Illustrated, | |
142 | Volume 1", W. Richard Stevens, sect 21.3, "Round-Trip Time Measurement", | |
143 | p. 299 and onwards. Also, Volume 2, Sect. 25.10, p. 838 and onwards. | |
144 | Unlike the TCP/IP Illustrated case, the mean and variance are | |
145 | not scaled, but are in units of integer nanoseconds. | |
146 | ||
147 | The three pairs of mean/variance measure the following | |
148 | things: | |
149 | ||
150 | 1. DLM lock time (non-blocking requests) | |
151 | 2. DLM lock time (blocking requests) | |
152 | 3. Inter-request time (again to the DLM) | |
153 | ||
154 | A non-blocking request is one which will complete right | |
155 | away, whatever the state of the DLM lock in question. That | |
156 | currently means any requests when (a) the current state of | |
157 | the lock is exclusive, i.e. a lock demotion (b) the requested | |
158 | state is either null or unlocked (again, a demotion) or (c) the | |
159 | "try lock" flag is set. A blocking request covers all the other | |
160 | lock requests. | |
161 | ||
162 | There are two counters. The first is there primarily to show | |
163 | how many lock requests have been made, and thus how much data | |
164 | has gone into the mean/variance calculations. The other counter | |
165 | is counting queuing of holders at the top layer of the glock | |
166 | code. Hopefully that number will be a lot larger than the number | |
167 | of dlm lock requests issued. | |
168 | ||
169 | So why gather these statistics? There are several reasons | |
170 | we'd like to get a better idea of these timings: | |
171 | ||
172 | 1. To be able to better set the glock "min hold time" | |
173 | 2. To spot performance issues more easily | |
174 | 3. To improve the algorithm for selecting resource groups for | |
175 | allocation (to base it on lock wait time, rather than blindly | |
176 | using a "try lock") | |
177 | ||
178 | Due to the smoothing action of the updates, a step change in | |
179 | some input quantity being sampled will only fully be taken | |
180 | into account after 8 samples (or 4 for the variance) and this | |
181 | needs to be carefully considered when interpreting the | |
182 | results. | |
183 | ||
184 | Knowing both the time it takes a lock request to complete and | |
185 | the average time between lock requests for a glock means we | |
186 | can compute the total percentage of the time for which the | |
187 | node is able to use a glock vs. time that the rest of the | |
188 | cluster has its share. That will be very useful when setting | |
189 | the lock min hold time. | |
190 | ||
191 | Great care has been taken to ensure that we | |
192 | measure exactly the quantities that we want, as accurately | |
193 | as possible. There are always inaccuracies in any | |
194 | measuring system, but I hope this is as accurate as we | |
195 | can reasonably make it. | |
196 | ||
197 | Per sb stats can be found here: | |
198 | /sys/kernel/debug/gfs2/<fsname>/sbstats | |
199 | Per glock stats can be found here: | |
200 | /sys/kernel/debug/gfs2/<fsname>/glstats | |
201 | ||
202 | Assuming that debugfs is mounted on /sys/kernel/debug and also | |
203 | that <fsname> is replaced with the name of the gfs2 filesystem | |
204 | in question. | |
205 | ||
206 | The abbreviations used in the output as are follows: | |
207 | ||
208 | srtt - Smoothed round trip time for non-blocking dlm requests | |
209 | srttvar - Variance estimate for srtt | |
210 | srttb - Smoothed round trip time for (potentially) blocking dlm requests | |
211 | srttvarb - Variance estimate for srttb | |
212 | sirt - Smoothed inter-request time (for dlm requests) | |
213 | sirtvar - Variance estimate for sirt | |
214 | dlm - Number of dlm requests made (dcnt in glstats file) | |
215 | queue - Number of glock requests queued (qcnt in glstats file) | |
216 | ||
217 | The sbstats file contains a set of these stats for each glock type (so 8 lines | |
218 | for each type) and for each cpu (one column per cpu). The glstats file contains | |
219 | a set of these stats for each glock in a similar format to the glocks file, but | |
220 | using the format mean/variance for each of the timing stats. | |
221 | ||
222 | The gfs2_glock_lock_time tracepoint prints out the current values of the stats | |
223 | for the glock in question, along with some addition information on each dlm | |
224 | reply that is received: | |
225 | ||
226 | status - The status of the dlm request | |
227 | flags - The dlm request flags | |
228 | tdiff - The time taken by this specific request | |
229 | (remaining fields as per above list) | |
230 | ||
9f1585cb | 231 |