Commit | Line | Data |
---|---|---|
9f1585cb SW |
1 | Glock internal locking rules |
2 | ------------------------------ | |
3 | ||
4 | This documents the basic principles of the glock state machine | |
5 | internals. Each glock (struct gfs2_glock in fs/gfs2/incore.h) | |
6 | has two main (internal) locks: | |
7 | ||
8 | 1. A spinlock (gl_spin) which protects the internal state such | |
9 | as gl_state, gl_target and the list of holders (gl_holders) | |
10 | 2. A non-blocking bit lock, GLF_LOCK, which is used to prevent other | |
11 | threads from making calls to the DLM, etc. at the same time. If a | |
12 | thread takes this lock, it must then call run_queue (usually via the | |
13 | workqueue) when it releases it in order to ensure any pending tasks | |
14 | are completed. | |
15 | ||
16 | The gl_holders list contains all the queued lock requests (not | |
17 | just the holders) associated with the glock. If there are any | |
18 | held locks, then they will be contiguous entries at the head | |
19 | of the list. Locks are granted in strictly the order that they | |
20 | are queued, except for those marked LM_FLAG_PRIORITY which are | |
21 | used only during recovery, and even then only for journal locks. | |
22 | ||
23 | There are three lock states that users of the glock layer can request, | |
24 | namely shared (SH), deferred (DF) and exclusive (EX). Those translate | |
25 | to the following DLM lock modes: | |
26 | ||
27 | Glock mode | DLM lock mode | |
28 | ------------------------------ | |
29 | UN | IV/NL Unlocked (no DLM lock associated with glock) or NL | |
30 | SH | PR (Protected read) | |
31 | DF | CW (Concurrent write) | |
32 | EX | EX (Exclusive) | |
33 | ||
34 | Thus DF is basically a shared mode which is incompatible with the "normal" | |
35 | shared lock mode, SH. In GFS2 the DF mode is used exclusively for direct I/O | |
36 | operations. The glocks are basically a lock plus some routines which deal | |
37 | with cache management. The following rules apply for the cache: | |
38 | ||
39 | Glock mode | Cache data | Cache Metadata | Dirty Data | Dirty Metadata | |
40 | -------------------------------------------------------------------------- | |
41 | UN | No | No | No | No | |
42 | SH | Yes | Yes | No | No | |
43 | DF | No | Yes | No | No | |
44 | EX | Yes | Yes | Yes | Yes | |
45 | ||
46 | These rules are implemented using the various glock operations which | |
47 | are defined for each type of glock. Not all types of glocks use | |
48 | all the modes. Only inode glocks use the DF mode for example. | |
49 | ||
50 | Table of glock operations and per type constants: | |
51 | ||
52 | Field | Purpose | |
53 | ---------------------------------------------------------------------------- | |
54 | go_xmote_th | Called before remote state change (e.g. to sync dirty data) | |
55 | go_xmote_bh | Called after remote state change (e.g. to refill cache) | |
56 | go_inval | Called if remote state change requires invalidating the cache | |
57 | go_demote_ok | Returns boolean value of whether its ok to demote a glock | |
58 | | (e.g. checks timeout, and that there is no cached data) | |
59 | go_lock | Called for the first local holder of a lock | |
60 | go_unlock | Called on the final local unlock of a lock | |
61 | go_dump | Called to print content of object for debugfs file, or on | |
62 | | error to dump glock to the log. | |
e9ccb73a | 63 | go_type | The type of the glock, LM_TYPE_..... |
9f1585cb SW |
64 | go_min_hold_time | The minimum hold time |
65 | ||
66 | The minimum hold time for each lock is the time after a remote lock | |
67 | grant for which we ignore remote demote requests. This is in order to | |
68 | prevent a situation where locks are being bounced around the cluster | |
69 | from node to node with none of the nodes making any progress. This | |
70 | tends to show up most with shared mmaped files which are being written | |
71 | to by multiple nodes. By delaying the demotion in response to a | |
72 | remote callback, that gives the userspace program time to make | |
73 | some progress before the pages are unmapped. | |
74 | ||
75 | There is a plan to try and remove the go_lock and go_unlock callbacks | |
76 | if possible, in order to try and speed up the fast path though the locking. | |
77 | Also, eventually we hope to make the glock "EX" mode locally shared | |
78 | such that any local locking will be done with the i_mutex as required | |
79 | rather than via the glock. | |
80 | ||
81 | Locking rules for glock operations: | |
82 | ||
83 | Operation | GLF_LOCK bit lock held | gl_spin spinlock held | |
84 | ----------------------------------------------------------------- | |
85 | go_xmote_th | Yes | No | |
86 | go_xmote_bh | Yes | No | |
87 | go_inval | Yes | No | |
88 | go_demote_ok | Sometimes | Yes | |
89 | go_lock | Yes | No | |
90 | go_unlock | Yes | No | |
91 | go_dump | Sometimes | Yes | |
92 | ||
93 | N.B. Operations must not drop either the bit lock or the spinlock | |
94 | if its held on entry. go_dump and do_demote_ok must never block. | |
95 | Note that go_dump will only be called if the glock's state | |
96 | indicates that it is caching uptodate data. | |
97 | ||
98 | Glock locking order within GFS2: | |
99 | ||
100 | 1. i_mutex (if required) | |
101 | 2. Rename glock (for rename only) | |
102 | 3. Inode glock(s) | |
103 | (Parents before children, inodes at "same level" with same parent in | |
104 | lock number order) | |
105 | 4. Rgrp glock(s) (for (de)allocation operations) | |
106 | 5. Transaction glock (via gfs2_trans_begin) for non-read operations | |
107 | 6. Page lock (always last, very important!) | |
108 | ||
109 | There are two glocks per inode. One deals with access to the inode | |
110 | itself (locking order as above), and the other, known as the iopen | |
111 | glock is used in conjunction with the i_nlink field in the inode to | |
112 | determine the lifetime of the inode in question. Locking of inodes | |
113 | is on a per-inode basis. Locking of rgrps is on a per rgrp basis. | |
114 |