Commit | Line | Data |
---|---|---|
a907c907 N |
1 | Written by: Neil Brown |
2 | Please see MAINTAINERS file for where to send questions. | |
7c37fbda NB |
3 | |
4 | Overlay Filesystem | |
5 | ================== | |
6 | ||
7 | This document describes a prototype for a new approach to providing | |
8 | overlay-filesystem functionality in Linux (sometimes referred to as | |
9 | union-filesystems). An overlay-filesystem tries to present a | |
10 | filesystem which is the result over overlaying one filesystem on top | |
11 | of the other. | |
12 | ||
13 | The result will inevitably fail to look exactly like a normal | |
14 | filesystem for various technical reasons. The expectation is that | |
15 | many use cases will be able to ignore these differences. | |
16 | ||
16149013 AG |
17 | |
18 | Overlay objects | |
19 | --------------- | |
20 | ||
21 | The overlay filesystem approach is 'hybrid', because the objects that | |
22 | appear in the filesystem do not always appear to belong to that filesystem. | |
23 | In many cases, an object accessed in the union will be indistinguishable | |
7c37fbda NB |
24 | from accessing the corresponding object from the original filesystem. |
25 | This is most obvious from the 'st_dev' field returned by stat(2). | |
26 | ||
27 | While directories will report an st_dev from the overlay-filesystem, | |
65f26738 | 28 | non-directory objects may report an st_dev from the lower filesystem or |
7c37fbda NB |
29 | upper filesystem that is providing the object. Similarly st_ino will |
30 | only be unique when combined with st_dev, and both of these can change | |
31 | over the lifetime of a non-directory object. Many applications and | |
32 | tools ignore these values and will not be affected. | |
33 | ||
65f26738 AG |
34 | In the special case of all overlay layers on the same underlying |
35 | filesystem, all objects will report an st_dev from the overlay | |
36 | filesystem and st_ino from the underlying filesystem. This will | |
37 | make the overlay mount more compliant with filesystem scanners and | |
38 | overlay objects will be distinguishable from the corresponding | |
39 | objects in the original filesystem. | |
40 | ||
16149013 AG |
41 | On 64bit systems, even if all overlay layers are not on the same |
42 | underlying filesystem, the same compliant behavior could be achieved | |
43 | with the "xino" feature. The "xino" feature composes a unique object | |
44 | identifier from the real object st_ino and an underlying fsid index. | |
45 | If all underlying filesystems support NFS file handles and export file | |
46 | handles with 32bit inode number encoding (e.g. ext4), overlay filesystem | |
47 | will use the high inode number bits for fsid. Even when the underlying | |
48 | filesystem uses 64bit inode numbers, users can still enable the "xino" | |
49 | feature with the "-o xino=on" overlay mount option. That is useful for the | |
50 | case of underlying filesystems like xfs and tmpfs, which use 64bit inode | |
51 | numbers, but are very unlikely to use the high inode number bit. | |
52 | ||
53 | ||
7c37fbda NB |
54 | Upper and Lower |
55 | --------------- | |
56 | ||
57 | An overlay filesystem combines two filesystems - an 'upper' filesystem | |
58 | and a 'lower' filesystem. When a name exists in both filesystems, the | |
59 | object in the 'upper' filesystem is visible while the object in the | |
60 | 'lower' filesystem is either hidden or, in the case of directories, | |
61 | merged with the 'upper' object. | |
62 | ||
63 | It would be more correct to refer to an upper and lower 'directory | |
64 | tree' rather than 'filesystem' as it is quite possible for both | |
65 | directory trees to be in the same filesystem and there is no | |
66 | requirement that the root of a filesystem be given for either upper or | |
67 | lower. | |
68 | ||
69 | The lower filesystem can be any filesystem supported by Linux and does | |
70 | not need to be writable. The lower filesystem can even be another | |
71 | overlayfs. The upper filesystem will normally be writable and if it | |
72 | is it must support the creation of trusted.* extended attributes, and | |
73 | must provide valid d_type in readdir responses, so NFS is not suitable. | |
74 | ||
75 | A read-only overlay of two read-only filesystems may use any | |
76 | filesystem type. | |
77 | ||
78 | Directories | |
79 | ----------- | |
80 | ||
81 | Overlaying mainly involves directories. If a given name appears in both | |
82 | upper and lower filesystems and refers to a non-directory in either, | |
83 | then the lower object is hidden - the name refers only to the upper | |
84 | object. | |
85 | ||
86 | Where both upper and lower objects are directories, a merged directory | |
87 | is formed. | |
88 | ||
89 | At mount time, the two directories given as mount options "lowerdir" and | |
90 | "upperdir" are combined into a merged directory: | |
91 | ||
ef94b186 | 92 | mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\ |
c3c86996 | 93 | workdir=/work /merged |
7c37fbda NB |
94 | |
95 | The "workdir" needs to be an empty directory on the same filesystem | |
96 | as upperdir. | |
97 | ||
98 | Then whenever a lookup is requested in such a merged directory, the | |
99 | lookup is performed in each actual directory and the combined result | |
100 | is cached in the dentry belonging to the overlay filesystem. If both | |
101 | actual lookups find directories, both are stored and a merged | |
102 | directory is created, otherwise only one is stored: the upper if it | |
103 | exists, else the lower. | |
104 | ||
105 | Only the lists of names from directories are merged. Other content | |
106 | such as metadata and extended attributes are reported for the upper | |
107 | directory only. These attributes of the lower directory are hidden. | |
108 | ||
109 | whiteouts and opaque directories | |
110 | -------------------------------- | |
111 | ||
112 | In order to support rm and rmdir without changing the lower | |
113 | filesystem, an overlay filesystem needs to record in the upper filesystem | |
114 | that files have been removed. This is done using whiteouts and opaque | |
115 | directories (non-directories are always opaque). | |
116 | ||
117 | A whiteout is created as a character device with 0/0 device number. | |
118 | When a whiteout is found in the upper level of a merged directory, any | |
119 | matching name in the lower level is ignored, and the whiteout itself | |
120 | is also hidden. | |
121 | ||
122 | A directory is made opaque by setting the xattr "trusted.overlay.opaque" | |
123 | to "y". Where the upper filesystem contains an opaque directory, any | |
124 | directory in the lower filesystem with the same name is ignored. | |
125 | ||
126 | readdir | |
127 | ------- | |
128 | ||
129 | When a 'readdir' request is made on a merged directory, the upper and | |
130 | lower directories are each read and the name lists merged in the | |
131 | obvious way (upper is read first, then lower - entries that already | |
132 | exist are not re-added). This merged name list is cached in the | |
133 | 'struct file' and so remains as long as the file is kept open. If the | |
134 | directory is opened and read by two processes at the same time, they | |
135 | will each have separate caches. A seekdir to the start of the | |
136 | directory (offset 0) followed by a readdir will cause the cache to be | |
137 | discarded and rebuilt. | |
138 | ||
139 | This means that changes to the merged directory do not appear while a | |
140 | directory is being read. This is unlikely to be noticed by many | |
141 | programs. | |
142 | ||
143 | seek offsets are assigned sequentially when the directories are read. | |
144 | Thus if | |
c3c86996 | 145 | |
7c37fbda NB |
146 | - read part of a directory |
147 | - remember an offset, and close the directory | |
148 | - re-open the directory some time later | |
149 | - seek to the remembered offset | |
150 | ||
151 | there may be little correlation between the old and new locations in | |
152 | the list of filenames, particularly if anything has changed in the | |
153 | directory. | |
154 | ||
155 | Readdir on directories that are not merged is simply handled by the | |
156 | underlying directory (upper or lower). | |
157 | ||
a6c60655 MS |
158 | renaming directories |
159 | -------------------- | |
160 | ||
161 | When renaming a directory that is on the lower layer or merged (i.e. the | |
162 | directory was not created on the upper layer to start with) overlayfs can | |
163 | handle it in two different ways: | |
164 | ||
c3c86996 | 165 | 1. return EXDEV error: this error is returned by rename(2) when trying to |
a6c60655 MS |
166 | move a file or directory across filesystem boundaries. Hence |
167 | applications are usually prepared to hande this error (mv(1) for example | |
168 | recursively copies the directory tree). This is the default behavior. | |
169 | ||
c3c86996 | 170 | 2. If the "redirect_dir" feature is enabled, then the directory will be |
a6c60655 MS |
171 | copied up (but not the contents). Then the "trusted.overlay.redirect" |
172 | extended attribute is set to the path of the original location from the | |
173 | root of the overlay. Finally the directory is moved to the new | |
174 | location. | |
7c37fbda | 175 | |
438c84c2 MS |
176 | There are several ways to tune the "redirect_dir" feature. |
177 | ||
178 | Kernel config options: | |
179 | ||
180 | - OVERLAY_FS_REDIRECT_DIR: | |
181 | If this is enabled, then redirect_dir is turned on by default. | |
182 | - OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW: | |
183 | If this is enabled, then redirects are always followed by default. Enabling | |
184 | this results in a less secure configuration. Enable this option only when | |
185 | worried about backward compatibility with kernels that have the redirect_dir | |
186 | feature and follow redirects even if turned off. | |
187 | ||
188 | Module options (can also be changed through /sys/module/overlay/parameters/*): | |
189 | ||
190 | - "redirect_dir=BOOL": | |
191 | See OVERLAY_FS_REDIRECT_DIR kernel config option above. | |
192 | - "redirect_always_follow=BOOL": | |
193 | See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kernel config option above. | |
194 | - "redirect_max=NUM": | |
195 | The maximum number of bytes in an absolute redirect (default is 256). | |
196 | ||
197 | Mount options: | |
198 | ||
199 | - "redirect_dir=on": | |
200 | Redirects are enabled. | |
201 | - "redirect_dir=follow": | |
202 | Redirects are not created, but followed. | |
203 | - "redirect_dir=off": | |
204 | Redirects are not created and only followed if "redirect_always_follow" | |
205 | feature is enabled in the kernel/module config. | |
206 | - "redirect_dir=nofollow": | |
207 | Redirects are not created and not followed (equivalent to "redirect_dir=off" | |
208 | if "redirect_always_follow" feature is not enabled). | |
209 | ||
f168f109 AG |
210 | When the NFS export feature is enabled, every copied up directory is |
211 | indexed by the file handle of the lower inode and a file handle of the | |
212 | upper directory is stored in a "trusted.overlay.upper" extended attribute | |
213 | on the index entry. On lookup of a merged directory, if the upper | |
214 | directory does not match the file handle stores in the index, that is an | |
215 | indication that multiple upper directories may be redirected to the same | |
216 | lower directory. In that case, lookup returns an error and warns about | |
217 | a possible inconsistency. | |
218 | ||
219 | Because lower layer redirects cannot be verified with the index, enabling | |
220 | NFS export support on an overlay filesystem with no upper layer requires | |
221 | turning off redirect follow (e.g. "redirect_dir=nofollow"). | |
222 | ||
f168f109 | 223 | |
7c37fbda NB |
224 | Non-directories |
225 | --------------- | |
226 | ||
227 | Objects that are not directories (files, symlinks, device-special | |
228 | files etc.) are presented either from the upper or lower filesystem as | |
229 | appropriate. When a file in the lower filesystem is accessed in a way | |
230 | the requires write-access, such as opening for write access, changing | |
231 | some metadata etc., the file is first copied from the lower filesystem | |
232 | to the upper filesystem (copy_up). Note that creating a hard-link | |
233 | also requires copy_up, though of course creation of a symlink does | |
234 | not. | |
235 | ||
236 | The copy_up may turn out to be unnecessary, for example if the file is | |
237 | opened for read-write but the data is not modified. | |
238 | ||
239 | The copy_up process first makes sure that the containing directory | |
240 | exists in the upper filesystem - creating it and any parents as | |
241 | necessary. It then creates the object with the same metadata (owner, | |
242 | mode, mtime, symlink-target etc.) and then if the object is a file, the | |
243 | data is copied from the lower to the upper filesystem. Finally any | |
244 | extended attributes are copied up. | |
245 | ||
246 | Once the copy_up is complete, the overlay filesystem simply | |
247 | provides direct access to the newly created file in the upper | |
248 | filesystem - future operations on the file are barely noticed by the | |
249 | overlay filesystem (though an operation on the name of the file such as | |
250 | rename or unlink will of course be noticed and handled). | |
251 | ||
252 | ||
a78d9f0d MS |
253 | Multiple lower layers |
254 | --------------------- | |
255 | ||
256 | Multiple lower layers can now be given using the the colon (":") as a | |
257 | separator character between the directory names. For example: | |
258 | ||
259 | mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged | |
260 | ||
6d900f5a MS |
261 | As the example shows, "upperdir=" and "workdir=" may be omitted. In |
262 | that case the overlay will be read-only. | |
263 | ||
264 | The specified lower directories will be stacked beginning from the | |
265 | rightmost one and going left. In the above example lower1 will be the | |
266 | top, lower2 the middle and lower3 the bottom layer. | |
a78d9f0d MS |
267 | |
268 | ||
9412812e AG |
269 | Sharing and copying layers |
270 | -------------------------- | |
271 | ||
272 | Lower layers may be shared among several overlay mounts and that is indeed | |
273 | a very common practice. An overlay mount may use the same lower layer | |
274 | path as another overlay mount and it may use a lower layer path that is | |
275 | beneath or above the path of another overlay lower layer path. | |
276 | ||
277 | Using an upper layer path and/or a workdir path that are already used by | |
85fdee1e | 278 | another overlay mount is not allowed and may fail with EBUSY. Using |
9412812e | 279 | partially overlapping paths is not allowed but will not fail with EBUSY. |
85fdee1e AG |
280 | If files are accessed from two overlayfs mounts which share or overlap the |
281 | upper layer and/or workdir path the behavior of the overlay is undefined, | |
282 | though it will not result in a crash or deadlock. | |
9412812e AG |
283 | |
284 | Mounting an overlay using an upper layer path, where the upper layer path | |
285 | was previously used by another mounted overlay in combination with a | |
286 | different lower layer path, is allowed, unless the "inodes index" feature | |
287 | is enabled. | |
288 | ||
289 | With the "inodes index" feature, on the first time mount, an NFS file | |
290 | handle of the lower layer root directory, along with the UUID of the lower | |
291 | filesystem, are encoded and stored in the "trusted.overlay.origin" extended | |
292 | attribute on the upper layer root directory. On subsequent mount attempts, | |
293 | the lower root directory file handle and lower filesystem UUID are compared | |
294 | to the stored origin in upper root directory. On failure to verify the | |
295 | lower root origin, mount will fail with ESTALE. An overlayfs mount with | |
296 | "inodes index" enabled will fail with EOPNOTSUPP if the lower filesystem | |
297 | does not support NFS export, lower filesystem does not have a valid UUID or | |
298 | if the upper filesystem does not support extended attributes. | |
299 | ||
300 | It is quite a common practice to copy overlay layers to a different | |
301 | directory tree on the same or different underlying filesystem, and even | |
302 | to a different machine. With the "inodes index" feature, trying to mount | |
303 | the copied layers will fail the verification of the lower root file handle. | |
304 | ||
305 | ||
7c37fbda NB |
306 | Non-standard behavior |
307 | --------------------- | |
308 | ||
309 | The copy_up operation essentially creates a new, identical file and | |
16149013 AG |
310 | moves it over to the old name. Any open files referring to this inode |
311 | will access the old data. | |
312 | ||
313 | The new file may be on a different filesystem, so both st_dev and st_ino | |
314 | of the real file may change. The values of st_dev and st_ino returned by | |
315 | stat(2) on an overlay object are often not the same as the real file | |
316 | stat(2) values to prevent the values from changing on copy_up. | |
7c37fbda | 317 | |
16149013 AG |
318 | Unless "xino" feature is enabled, when overlay layers are not all on the |
319 | same underlying filesystem, the value of st_dev may be different for two | |
320 | non-directory objects in the same overlay filesystem and the value of | |
321 | st_ino for directory objects may be non persistent and could change even | |
322 | while the overlay filesystem is still mounted. | |
7c37fbda | 323 | |
60b86642 AG |
324 | Unless "inode index" feature is enabled, if a file with multiple hard |
325 | links is copied up, then this will "break" the link. Changes will not be | |
326 | propagated to other names referring to the same inode. | |
7c37fbda | 327 | |
a6c60655 MS |
328 | Unless "redirect_dir" feature is enabled, rename(2) on a lower or merged |
329 | directory will fail with EXDEV. | |
2d8f2908 | 330 | |
16149013 | 331 | |
7c37fbda NB |
332 | Changes to underlying filesystems |
333 | --------------------------------- | |
334 | ||
335 | Offline changes, when the overlay is not mounted, are allowed to either | |
336 | the upper or the lower trees. | |
337 | ||
338 | Changes to the underlying filesystems while part of a mounted overlay | |
339 | filesystem are not allowed. If the underlying filesystem is changed, | |
340 | the behavior of the overlay is undefined, though it will not result in | |
341 | a crash or deadlock. | |
2b7a8f36 | 342 | |
f168f109 AG |
343 | When the overlay NFS export feature is enabled, overlay filesystems |
344 | behavior on offline changes of the underlying lower layer is different | |
345 | than the behavior when NFS export is disabled. | |
346 | ||
347 | On every copy_up, an NFS file handle of the lower inode, along with the | |
348 | UUID of the lower filesystem, are encoded and stored in an extended | |
349 | attribute "trusted.overlay.origin" on the upper inode. | |
350 | ||
351 | When the NFS export feature is enabled, a lookup of a merged directory, | |
352 | that found a lower directory at the lookup path or at the path pointed | |
353 | to by the "trusted.overlay.redirect" extended attribute, will verify | |
354 | that the found lower directory file handle and lower filesystem UUID | |
355 | match the origin file handle that was stored at copy_up time. If a | |
356 | found lower directory does not match the stored origin, that directory | |
357 | will not be merged with the upper directory. | |
358 | ||
359 | ||
a01f64b5 AG |
360 | |
361 | NFS export | |
362 | ---------- | |
363 | ||
364 | When the underlying filesystems supports NFS export and the "nfs_export" | |
365 | feature is enabled, an overlay filesystem may be exported to NFS. | |
366 | ||
367 | With the "nfs_export" feature, on copy_up of any lower object, an index | |
368 | entry is created under the index directory. The index entry name is the | |
369 | hexadecimal representation of the copy up origin file handle. For a | |
370 | non-directory object, the index entry is a hard link to the upper inode. | |
371 | For a directory object, the index entry has an extended attribute | |
372 | "trusted.overlay.upper" with an encoded file handle of the upper | |
373 | directory inode. | |
374 | ||
375 | When encoding a file handle from an overlay filesystem object, the | |
376 | following rules apply: | |
377 | ||
378 | 1. For a non-upper object, encode a lower file handle from lower inode | |
379 | 2. For an indexed object, encode a lower file handle from copy_up origin | |
380 | 3. For a pure-upper object and for an existing non-indexed upper object, | |
381 | encode an upper file handle from upper inode | |
382 | ||
383 | The encoded overlay file handle includes: | |
384 | - Header including path type information (e.g. lower/upper) | |
385 | - UUID of the underlying filesystem | |
386 | - Underlying filesystem encoding of underlying inode | |
387 | ||
388 | This encoding format is identical to the encoding format file handles that | |
389 | are stored in extended attribute "trusted.overlay.origin". | |
390 | ||
391 | When decoding an overlay file handle, the following steps are followed: | |
392 | ||
393 | 1. Find underlying layer by UUID and path type information. | |
394 | 2. Decode the underlying filesystem file handle to underlying dentry. | |
395 | 3. For a lower file handle, lookup the handle in index directory by name. | |
396 | 4. If a whiteout is found in index, return ESTALE. This represents an | |
397 | overlay object that was deleted after its file handle was encoded. | |
398 | 5. For a non-directory, instantiate a disconnected overlay dentry from the | |
399 | decoded underlying dentry, the path type and index inode, if found. | |
400 | 6. For a directory, use the connected underlying decoded dentry, path type | |
401 | and index, to lookup a connected overlay dentry. | |
402 | ||
403 | Decoding a non-directory file handle may return a disconnected dentry. | |
404 | copy_up of that disconnected dentry will create an upper index entry with | |
405 | no upper alias. | |
406 | ||
407 | When overlay filesystem has multiple lower layers, a middle layer | |
408 | directory may have a "redirect" to lower directory. Because middle layer | |
409 | "redirects" are not indexed, a lower file handle that was encoded from the | |
410 | "redirect" origin directory, cannot be used to find the middle or upper | |
411 | layer directory. Similarly, a lower file handle that was encoded from a | |
412 | descendant of the "redirect" origin directory, cannot be used to | |
413 | reconstruct a connected overlay path. To mitigate the cases of | |
414 | directories that cannot be decoded from a lower file handle, these | |
415 | directories are copied up on encode and encoded as an upper file handle. | |
416 | On an overlay filesystem with no upper layer this mitigation cannot be | |
417 | used NFS export in this setup requires turning off redirect follow (e.g. | |
418 | "redirect_dir=nofollow"). | |
419 | ||
420 | The overlay filesystem does not support non-directory connectable file | |
421 | handles, so exporting with the 'subtree_check' exportfs configuration will | |
422 | cause failures to lookup files over NFS. | |
423 | ||
424 | When the NFS export feature is enabled, all directory index entries are | |
425 | verified on mount time to check that upper file handles are not stale. | |
426 | This verification may cause significant overhead in some cases. | |
427 | ||
428 | ||
2b7a8f36 MS |
429 | Testsuite |
430 | --------- | |
431 | ||
432 | There's testsuite developed by David Howells at: | |
433 | ||
434 | git://git.infradead.org/users/dhowells/unionmount-testsuite.git | |
435 | ||
436 | Run as root: | |
437 | ||
438 | # cd unionmount-testsuite | |
439 | # ./run --ov |