Commit | Line | Data |
---|---|---|
f0ba4377 MCC |
1 | ===== |
2 | Cache | |
3 | ===== | |
4 | ||
c6b4fcba JT |
5 | Introduction |
6 | ============ | |
7 | ||
8 | dm-cache is a device mapper target written by Joe Thornber, Heinz | |
9 | Mauelshagen, and Mike Snitzer. | |
10 | ||
11 | It aims to improve performance of a block device (eg, a spindle) by | |
12 | dynamically migrating some of its data to a faster, smaller device | |
13 | (eg, an SSD). | |
14 | ||
15 | This device-mapper solution allows us to insert this caching at | |
16 | different levels of the dm stack, for instance above the data device for | |
17 | a thin-provisioning pool. Caching solutions that are integrated more | |
18 | closely with the virtual memory system should give better performance. | |
19 | ||
20 | The target reuses the metadata library used in the thin-provisioning | |
21 | library. | |
22 | ||
23 | The decision as to what data to migrate and when is left to a plug-in | |
24 | policy module. Several of these have been written as we experiment, | |
25 | and we hope other people will contribute others for specific io | |
26 | scenarios (eg. a vm image server). | |
27 | ||
28 | Glossary | |
29 | ======== | |
30 | ||
f0ba4377 MCC |
31 | Migration |
32 | Movement of the primary copy of a logical block from one | |
c6b4fcba | 33 | device to the other. |
f0ba4377 MCC |
34 | Promotion |
35 | Migration from slow device to fast device. | |
36 | Demotion | |
37 | Migration from fast device to slow device. | |
c6b4fcba JT |
38 | |
39 | The origin device always contains a copy of the logical block, which | |
40 | may be out of date or kept in sync with the copy on the cache device | |
41 | (depending on policy). | |
42 | ||
43 | Design | |
44 | ====== | |
45 | ||
46 | Sub-devices | |
47 | ----------- | |
48 | ||
49 | The target is constructed by passing three devices to it (along with | |
50 | other parameters detailed later): | |
51 | ||
52 | 1. An origin device - the big, slow one. | |
53 | ||
54 | 2. A cache device - the small, fast one. | |
55 | ||
56 | 3. A small metadata device - records which blocks are in the cache, | |
57 | which are dirty, and extra hints for use by the policy object. | |
58 | This information could be put on the cache device, but having it | |
59 | separate allows the volume manager to configure it differently, | |
66bb2644 MS |
60 | e.g. as a mirror for extra robustness. This metadata device may only |
61 | be used by a single cache device. | |
c6b4fcba JT |
62 | |
63 | Fixed block size | |
64 | ---------------- | |
65 | ||
66 | The origin is divided up into blocks of a fixed size. This block size | |
67 | is configurable when you first create the cache. Typically we've been | |
05473044 | 68 | using block sizes of 256KB - 1024KB. The block size must be between 64 |
1346638e | 69 | sectors (32KB) and 2097152 sectors (1GB) and a multiple of 64 sectors (32KB). |
c6b4fcba JT |
70 | |
71 | Having a fixed block size simplifies the target a lot. But it is | |
72 | something of a compromise. For instance, a small part of a block may be | |
73 | getting hit a lot, yet the whole block will be promoted to the cache. | |
74 | So large block sizes are bad because they waste cache space. And small | |
75 | block sizes are bad because they increase the amount of metadata (both | |
76 | in core and on disk). | |
77 | ||
2ee57d58 JT |
78 | Cache operating modes |
79 | --------------------- | |
c6b4fcba | 80 | |
2ee57d58 JT |
81 | The cache has three operating modes: writeback, writethrough and |
82 | passthrough. | |
c6b4fcba JT |
83 | |
84 | If writeback, the default, is selected then a write to a block that is | |
85 | cached will go only to the cache and the block will be marked dirty in | |
86 | the metadata. | |
87 | ||
88 | If writethrough is selected then a write to a cached block will not | |
89 | complete until it has hit both the origin and cache devices. Clean | |
90 | blocks should remain clean. | |
91 | ||
2ee57d58 JT |
92 | If passthrough is selected, useful when the cache contents are not known |
93 | to be coherent with the origin device, then all reads are served from | |
94 | the origin device (all reads miss the cache) and all writes are | |
95 | forwarded to the origin device; additionally, write hits cause cache | |
7b6b2bc9 MS |
96 | block invalidates. To enable passthrough mode the cache must be clean. |
97 | Passthrough mode allows a cache device to be activated without having to | |
98 | worry about coherency. Coherency that exists is maintained, although | |
99 | the cache will gradually cool as writes take place. If the coherency of | |
100 | the cache can later be verified, or established through use of the | |
101 | "invalidate_cblocks" message, the cache device can be transitioned to | |
102 | writethrough or writeback mode while still warm. Otherwise, the cache | |
103 | contents can be discarded prior to transitioning to the desired | |
104 | operating mode. | |
2ee57d58 | 105 | |
c6b4fcba | 106 | A simple cleaner policy is provided, which will clean (write back) all |
7b6b2bc9 MS |
107 | dirty blocks in a cache. Useful for decommissioning a cache or when |
108 | shrinking a cache. Shrinking the cache's fast device requires all cache | |
109 | blocks, in the area of the cache being removed, to be clean. If the | |
110 | area being removed from the cache still contains dirty blocks the resize | |
111 | will fail. Care must be taken to never reduce the volume used for the | |
112 | cache's fast device until the cache is clean. This is of particular | |
113 | importance if writeback mode is used. Writethrough and passthrough | |
114 | modes already maintain a clean cache. Future support to partially clean | |
115 | the cache, above a specified threshold, will allow for keeping the cache | |
116 | warm and in writeback mode during resize. | |
c6b4fcba JT |
117 | |
118 | Migration throttling | |
119 | -------------------- | |
120 | ||
121 | Migrating data between the origin and cache device uses bandwidth. | |
122 | The user can set a throttle to prevent more than a certain amount of | |
f884ab15 | 123 | migration occurring at any one time. Currently we're not taking any |
c6b4fcba JT |
124 | account of normal io traffic going to the devices. More work needs |
125 | doing here to avoid migrating during those peak io moments. | |
126 | ||
127 | For the time being, a message "migration_threshold <#sectors>" | |
128 | can be used to set the maximum number of sectors being migrated, | |
9614e2ba | 129 | the default being 2048 sectors (1MB). |
c6b4fcba JT |
130 | |
131 | Updating on-disk metadata | |
132 | ------------------------- | |
133 | ||
07f2b6e0 MS |
134 | On-disk metadata is committed every time a FLUSH or FUA bio is written. |
135 | If no such requests are made then commits will occur every second. This | |
136 | means the cache behaves like a physical disk that has a volatile write | |
137 | cache. If power is lost you may lose some recent writes. The metadata | |
138 | should always be consistent in spite of any crash. | |
c6b4fcba JT |
139 | |
140 | The 'dirty' state for a cache block changes far too frequently for us | |
141 | to keep updating it on the fly. So we treat it as a hint. In normal | |
142 | operation it will be written when the dm device is suspended. If the | |
143 | system crashes all cache blocks will be assumed dirty when restarted. | |
144 | ||
145 | Per-block policy hints | |
146 | ---------------------- | |
147 | ||
148 | Policy plug-ins can store a chunk of data per cache block. It's up to | |
149 | the policy how big this chunk is, but it should be kept small. Like the | |
150 | dirty flags this data is lost if there's a crash so a safe fallback | |
151 | value should always be possible. | |
152 | ||
c6b4fcba JT |
153 | Policy hints affect performance, not correctness. |
154 | ||
155 | Policy messaging | |
156 | ---------------- | |
157 | ||
158 | Policies will have different tunables, specific to each one, so we | |
159 | need a generic way of getting and setting these. Device-mapper | |
160 | messages are used. Refer to cache-policies.txt. | |
161 | ||
162 | Discard bitset resolution | |
163 | ------------------------- | |
164 | ||
165 | We can avoid copying data during migration if we know the block has | |
166 | been discarded. A prime example of this is when mkfs discards the | |
167 | whole block device. We store a bitset tracking the discard state of | |
168 | blocks. However, we allow this bitset to have a different block size | |
169 | from the cache blocks. This is because we need to track the discard | |
170 | state for all of the origin device (compare with the dirty bitset | |
171 | which is just for the smaller cache device). | |
172 | ||
173 | Target interface | |
174 | ================ | |
175 | ||
176 | Constructor | |
177 | ----------- | |
178 | ||
f0ba4377 MCC |
179 | :: |
180 | ||
181 | cache <metadata dev> <cache dev> <origin dev> <block size> | |
182 | <#feature args> [<feature arg>]* | |
183 | <policy> <#policy args> [policy args]* | |
c6b4fcba | 184 | |
f0ba4377 MCC |
185 | ================ ======================================================= |
186 | metadata dev fast device holding the persistent metadata | |
187 | cache dev fast device holding cached data blocks | |
188 | origin dev slow device holding original data blocks | |
189 | block size cache unit size in sectors | |
c6b4fcba | 190 | |
f0ba4377 MCC |
191 | #feature args number of feature arguments passed |
192 | feature args writethrough or passthrough (The default is writeback.) | |
c6b4fcba | 193 | |
f0ba4377 MCC |
194 | policy the replacement policy to use |
195 | #policy args an even number of arguments corresponding to | |
196 | key/value pairs passed to the policy | |
197 | policy args key/value pairs passed to the policy | |
198 | E.g. 'sequential_threshold 1024' | |
199 | See cache-policies.txt for details. | |
200 | ================ ======================================================= | |
c6b4fcba JT |
201 | |
202 | Optional feature arguments are: | |
f0ba4377 MCC |
203 | |
204 | ||
205 | ==================== ======================================================== | |
206 | writethrough write through caching that prohibits cache block | |
207 | content from being different from origin block content. | |
208 | Without this argument, the default behaviour is to write | |
209 | back cache block contents later for performance reasons, | |
210 | so they may differ from the corresponding origin blocks. | |
211 | ||
212 | passthrough a degraded mode useful for various cache coherency | |
213 | situations (e.g., rolling back snapshots of | |
214 | underlying storage). Reads and writes always go to | |
215 | the origin. If a write goes to a cached origin | |
216 | block, then the cache block is invalidated. | |
217 | To enable passthrough mode the cache must be clean. | |
218 | ||
219 | metadata2 use version 2 of the metadata. This stores the dirty | |
220 | bits in a separate btree, which improves speed of | |
221 | shutting down the cache. | |
222 | ||
223 | no_discard_passdown disable passing down discards from the cache | |
224 | to the origin's data device. | |
225 | ==================== ======================================================== | |
de7180ff | 226 | |
c6b4fcba JT |
227 | A policy called 'default' is always registered. This is an alias for |
228 | the policy we currently think is giving best all round performance. | |
229 | ||
230 | As the default policy could vary between kernels, if you are relying on | |
231 | the characteristics of a specific policy, always request it by name. | |
232 | ||
233 | Status | |
234 | ------ | |
235 | ||
f0ba4377 MCC |
236 | :: |
237 | ||
238 | <metadata block size> <#used metadata blocks>/<#total metadata blocks> | |
239 | <cache block size> <#used cache blocks>/<#total cache blocks> | |
240 | <#read hits> <#read misses> <#write hits> <#write misses> | |
241 | <#demotions> <#promotions> <#dirty> <#features> <features>* | |
242 | <#core args> <core args>* <policy name> <#policy args> <policy args>* | |
243 | <cache metadata mode> | |
244 | ||
245 | ||
246 | ========================= ===================================================== | |
247 | metadata block size Fixed block size for each metadata block in | |
248 | sectors | |
249 | #used metadata blocks Number of metadata blocks used | |
250 | #total metadata blocks Total number of metadata blocks | |
251 | cache block size Configurable block size for the cache device | |
252 | in sectors | |
253 | #used cache blocks Number of blocks resident in the cache | |
254 | #total cache blocks Total number of cache blocks | |
255 | #read hits Number of times a READ bio has been mapped | |
256 | to the cache | |
257 | #read misses Number of times a READ bio has been mapped | |
258 | to the origin | |
259 | #write hits Number of times a WRITE bio has been mapped | |
260 | to the cache | |
261 | #write misses Number of times a WRITE bio has been | |
262 | mapped to the origin | |
263 | #demotions Number of times a block has been removed | |
264 | from the cache | |
265 | #promotions Number of times a block has been moved to | |
266 | the cache | |
267 | #dirty Number of blocks in the cache that differ | |
268 | from the origin | |
269 | #feature args Number of feature args to follow | |
270 | feature args 'writethrough' (optional) | |
271 | #core args Number of core arguments (must be even) | |
272 | core args Key/value pairs for tuning the core | |
273 | e.g. migration_threshold | |
274 | policy name Name of the policy | |
275 | #policy args Number of policy arguments to follow (must be even) | |
276 | policy args Key/value pairs e.g. sequential_threshold | |
277 | cache metadata mode ro if read-only, rw if read-write | |
278 | ||
279 | In serious cases where even a read-only mode is | |
280 | deemed unsafe no further I/O will be permitted and | |
281 | the status will just contain the string 'Fail'. | |
282 | The userspace recovery tools should then be used. | |
283 | needs_check 'needs_check' if set, '-' if not set | |
284 | A metadata operation has failed, resulting in the | |
285 | needs_check flag being set in the metadata's | |
286 | superblock. The metadata device must be | |
287 | deactivated and checked/repaired before the | |
288 | cache can be made fully operational again. | |
289 | '-' indicates needs_check is not set. | |
290 | ========================= ===================================================== | |
c6b4fcba JT |
291 | |
292 | Messages | |
293 | -------- | |
294 | ||
295 | Policies will have different tunables, specific to each one, so we | |
296 | need a generic way of getting and setting these. Device-mapper | |
297 | messages are used. (A sysfs interface would also be possible.) | |
298 | ||
f0ba4377 | 299 | The message format is:: |
c6b4fcba JT |
300 | |
301 | <key> <value> | |
302 | ||
f0ba4377 MCC |
303 | E.g.:: |
304 | ||
c6b4fcba JT |
305 | dmsetup message my_cache 0 sequential_threshold 1024 |
306 | ||
65790ff9 JT |
307 | |
308 | Invalidation is removing an entry from the cache without writing it | |
309 | back. Cache blocks can be invalidated via the invalidate_cblocks | |
7b6b2bc9 | 310 | message, which takes an arbitrary number of cblock ranges. Each cblock |
83f539e1 MS |
311 | range's end value is "one past the end", meaning 5-10 expresses a range |
312 | of values from 5 to 9. Each cblock must be expressed as a decimal | |
313 | value, in the future a variant message that takes cblock ranges | |
3f816bac | 314 | expressed in hexadecimal may be needed to better support efficient |
83f539e1 | 315 | invalidation of larger caches. The cache must be in passthrough mode |
f0ba4377 | 316 | when invalidate_cblocks is used:: |
65790ff9 JT |
317 | |
318 | invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]* | |
319 | ||
f0ba4377 MCC |
320 | E.g.:: |
321 | ||
65790ff9 JT |
322 | dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789 |
323 | ||
c6b4fcba JT |
324 | Examples |
325 | ======== | |
326 | ||
327 | The test suite can be found here: | |
328 | ||
65790ff9 | 329 | https://github.com/jthornber/device-mapper-test-suite |
c6b4fcba | 330 | |
f0ba4377 MCC |
331 | :: |
332 | ||
333 | dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ | |
334 | /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0' | |
335 | dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ | |
336 | /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \ | |
337 | mq 4 sequential_threshold 1024 random_threshold 8' |