From: Jens Axboe <axboe@kernel.dk>
Date: Fri, 9 Aug 2013 20:31:06 +0000 (-0600)
Subject: verify: fix problem with hole punching on newer Linux kernels
X-Git-Tag: fio-2.1.3~12
X-Git-Url: https://git.kernel.dk/?p=fio.git;a=commitdiff_plain;h=3e10fb832645e3ab3ef006f589f0459dc567cb53;hp=ddf24e42b5f801e6a66f8976c97edcb88080ecdf

verify: fix problem with hole punching on newer Linux kernels

Jeff Moyer writes:

Our QE team noticed fio failures on recent kernels.  I simplified the
job file to the following:

[global]
name=fio-mmap
rw=write
bs=4K
direct=1
end_fsync=1
verify=crc32

[file3]
size=100M
ioengine=mmap
mem=malloc
direct=1

After fio completes (and returns verify errors), the file is completely
full of zeroes.

Here is what the verify logic is doing:

static void do_verify(struct thread_data *td, uint64_t verify_bytes)
{
...
        /*
         * sync io first and invalidate cache, to make sure we really
         * read from disk.
         */
        for_each_file(td, f, i) {
                if (!fio_file_open(f))
                        continue;
                if (fio_io_sync(td, f))
                        break;
                if (file_invalidate_cache(td, f)) <--------
                        break;
        }

That invalidate cache call looks like so:

static int __file_invalidate_cache(struct thread_data *td, struct fio_file *f,
                                   unsigned long long off,
                                   unsigned long long len)
{
...
        /*
         * FIXME: add blockdev flushing too
         */
        if (f->mmap_ptr) {
                ret = posix_madvise(f->mmap_ptr, f->mmap_sz, POSIX_MADV_DONTNEED);
                (void) posix_madvise(f->mmap_ptr, f->mmap_sz, FIO_MADV_FREE); <-------

FIO_MADV_FREE can be defined as MADV_REMOVE, which will actually punch a
hole in the file (a hole the size of the entire file, btw).  Of course,
unallocated blocks are returned as zeroes by the file system, so that
explains that!

---

For now, only do the MADV_FREE on non-files to avoid punching holes.
Further investigation is needed on the blockdev side, but at least
this should cure the immediate problem.

Reported-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---

diff --git a/filesetup.c b/filesetup.c
index 6427f3e4..7d3a0613 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -388,7 +388,8 @@ static int __file_invalidate_cache(struct thread_data *td, struct fio_file *f,
 	if (f->mmap_ptr) {
 		ret = posix_madvise(f->mmap_ptr, f->mmap_sz, POSIX_MADV_DONTNEED);
 #ifdef FIO_MADV_FREE
-		(void) posix_madvise(f->mmap_ptr, f->mmap_sz, FIO_MADV_FREE);
+		if (f->filetype == FIO_TYPE_BD)
+			(void) posix_madvise(f->mmap_ptr, f->mmap_sz, FIO_MADV_FREE);
 #endif
 	} else if (f->filetype == FIO_TYPE_FILE) {
 		ret = posix_fadvise(f->fd, off, len, POSIX_FADV_DONTNEED);