Commit | Line | Data |
---|---|---|
95ec8dab MW |
1 | Direct Access for files |
2 | ----------------------- | |
3 | ||
4 | Motivation | |
5 | ---------- | |
6 | ||
7 | The page cache is usually used to buffer reads and writes to files. | |
8 | It is also used to provide the pages which are mapped into userspace | |
9 | by a call to mmap. | |
10 | ||
11 | For block devices that are memory-like, the page cache pages would be | |
12 | unnecessary copies of the original storage. The DAX code removes the | |
13 | extra copy by performing reads and writes directly to the storage device. | |
14 | For file mappings, the storage device is mapped directly into userspace. | |
15 | ||
16 | ||
17 | Usage | |
18 | ----- | |
19 | ||
20 | If you have a block device which supports DAX, you can make a filesystem | |
44f4c054 MW |
21 | on it as usual. The DAX code currently only supports files with a block |
22 | size equal to your kernel's PAGE_SIZE, so you may need to specify a block | |
23 | size when creating the filesystem. When mounting it, use the "-o dax" | |
24 | option on the command line or add 'dax' to the options in /etc/fstab. | |
95ec8dab MW |
25 | |
26 | ||
27 | Implementation Tips for Block Driver Writers | |
28 | -------------------------------------------- | |
29 | ||
30 | To support DAX in your block driver, implement the 'direct_access' | |
31 | block device operation. It is used to translate the sector number | |
32 | (expressed in units of 512-byte sectors) to a page frame number (pfn) | |
33 | that identifies the physical page for the memory. It also returns a | |
34 | kernel virtual address that can be used to access the memory. | |
35 | ||
36 | The direct_access method takes a 'size' parameter that indicates the | |
37 | number of bytes being requested. The function should return the number | |
38 | of bytes that can be contiguously accessed at that offset. It may also | |
39 | return a negative errno if an error occurs. | |
40 | ||
41 | In order to support this method, the storage must be byte-accessible by | |
42 | the CPU at all times. If your device uses paging techniques to expose | |
43 | a large amount of memory through a smaller window, then you cannot | |
44 | implement direct_access. Equally, if your device can occasionally | |
45 | stall the CPU for an extended period, you should also not attempt to | |
46 | implement direct_access. | |
47 | ||
48 | These block devices may be used for inspiration: | |
95ec8dab MW |
49 | - brd: RAM backed block device driver |
50 | - dcssblk: s390 dcss block device driver | |
221c7dc8 | 51 | - pmem: NVDIMM persistent memory driver |
95ec8dab MW |
52 | |
53 | ||
54 | Implementation Tips for Filesystem Writers | |
55 | ------------------------------------------ | |
56 | ||
57 | Filesystem support consists of | |
58 | - adding support to mark inodes as being DAX by setting the S_DAX flag in | |
59 | i_flags | |
dd936e43 JK |
60 | - implementing ->read_iter and ->write_iter operations which use dax_iomap_rw() |
61 | when inode has S_DAX flag set | |
95ec8dab | 62 | - implementing an mmap file operation for DAX files which sets the |
844f35db | 63 | VM_MIXEDMAP and VM_HUGEPAGE flags on the VMA, and setting the vm_ops to |
dd936e43 | 64 | include handlers for fault, pmd_fault, page_mkwrite, pfn_mkwrite. These |
91d25ba8 RZ |
65 | handlers should probably call dax_iomap_fault() passing the appropriate |
66 | fault size and iomap operations. | |
dd936e43 JK |
67 | - calling iomap_zero_range() passing appropriate iomap operations instead of |
68 | block_truncate_page() for DAX files | |
95ec8dab MW |
69 | - ensuring that there is sufficient locking between reads, writes, |
70 | truncates and page faults | |
71 | ||
dd936e43 JK |
72 | The iomap handlers for allocating blocks must make sure that allocated blocks |
73 | are zeroed out and converted to written extents before being returned to avoid | |
74 | exposure of uninitialized data through mmap. | |
95ec8dab MW |
75 | |
76 | These filesystems may be used for inspiration: | |
221c7dc8 | 77 | - ext2: see Documentation/filesystems/ext2.txt |
93fb7f19 | 78 | - ext4: see Documentation/filesystems/ext4/ |
89b408a6 | 79 | - xfs: see Documentation/admin-guide/xfs.rst |
95ec8dab MW |
80 | |
81 | ||
4b0228fa VV |
82 | Handling Media Errors |
83 | --------------------- | |
84 | ||
85 | The libnvdimm subsystem stores a record of known media error locations for | |
86 | each pmem block device (in gendisk->badblocks). If we fault at such location, | |
87 | or one with a latent error not yet discovered, the application can expect | |
88 | to receive a SIGBUS. Libnvdimm also allows clearing of these errors by simply | |
89 | writing the affected sectors (through the pmem driver, and if the underlying | |
90 | NVDIMM supports the clear_poison DSM defined by ACPI). | |
91 | ||
92 | Since DAX IO normally doesn't go through the driver/bio path, applications or | |
93 | sysadmins have an option to restore the lost data from a prior backup/inbuilt | |
94 | redundancy in the following ways: | |
95 | ||
96 | 1. Delete the affected file, and restore from a backup (sysadmin route): | |
97 | This will free the file system blocks that were being used by the file, | |
98 | and the next time they're allocated, they will be zeroed first, which | |
99 | happens through the driver, and will clear bad sectors. | |
100 | ||
101 | 2. Truncate or hole-punch the part of the file that has a bad-block (at least | |
102 | an entire aligned sector has to be hole-punched, but not necessarily an | |
103 | entire filesystem block). | |
104 | ||
105 | These are the two basic paths that allow DAX filesystems to continue operating | |
106 | in the presence of media errors. More robust error recovery mechanisms can be | |
107 | built on top of this in the future, for example, involving redundancy/mirroring | |
108 | provided at the block layer through DM, or additionally, at the filesystem | |
109 | level. These would have to rely on the above two tenets, that error clearing | |
110 | can happen either by sending an IO through the driver, or zeroing (also through | |
111 | the driver). | |
112 | ||
113 | ||
95ec8dab MW |
114 | Shortcomings |
115 | ------------ | |
116 | ||
117 | Even if the kernel or its modules are stored on a filesystem that supports | |
118 | DAX on a block device that supports DAX, they will still be copied into RAM. | |
119 | ||
d92576f1 MW |
120 | The DAX code does not work correctly on architectures which have virtually |
121 | mapped caches such as ARM, MIPS and SPARC. | |
122 | ||
95ec8dab | 123 | Calling get_user_pages() on a range of user memory that has been mmaped |
9ff2dc56 SB |
124 | from a DAX file will fail when there are no 'struct page' to describe |
125 | those pages. This problem has been addressed in some device drivers | |
126 | by adding optional struct page support for pages under the control of | |
127 | the driver (see CONFIG_NVDIMM_PFN in drivers/nvdimm for an example of | |
128 | how to do this). In the non struct page cases O_DIRECT reads/writes to | |
129 | those memory ranges from a non-DAX file will fail (note that O_DIRECT | |
130 | reads/writes _of a DAX file_ do work, it is the memory that is being | |
131 | accessed that is key here). Other things that will not work in the | |
132 | non struct page case include RDMA, sendfile() and splice(). |