ioengines: don't call munmap unless size boundary is exceeded

In ioengines using mmap() and munmap(), don't call munmap() when doing an
IO that accesses the last byte of a file. This affects the dev-dax,
libpmem, and mmap ioengines.

Example:
io       12987 DEBUG fio_libpmem_prep
io       12987  io_u->offset 7516061696 : fdd->libpmem_off 0 : io_u->buflen 131072 : fdd->libpmem_sz 7516192768
io       12987 munmap addr=0x7f5600000000 size=0x1c0000000

That accesses the last 128 KiB of a 7 GiB file, triggering an munmap
because the comparison to 7 GiB fails:
	7516061696 + 131072 < 7516192768

Change < to <= so that doesn't occur unless an overflow really
happens.

In one test in linux, this increases performance from 5 GB/s to 20 GB/s
as fio stops spending time in mmap() page table mapping spinlocks and
instead spends its time in memmove().  The perf top results improve from:
  36.81%  [kernel]             [k] queued_spin_lock_slowpath
  24.26%  libc-2.26.so         [.] __memmove_avx_unaligned_erms
   4.09%  [kernel]             [k] __radix_tree_lookup
   3.85%  [kernel]             [k] find_next_iomem_res
   3.70%  [kernel]             [k] _raw_spin_lock_irq
   3.48%  [kernel]             [k] down_read

to:
  79.76%  libc-2.26.so         [.] __memmove_avx_unaligned_erms
   3.23%  libc-2.26.so         [.] vfprintf

Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/engines/dev-dax.c b/engines/dev-dax.c
index b1f91a40f739d3f549592434d269b0c859982a50..caae1e09b5f8112d1725a5300b43ed4d1a9a56da 100644 (file)
--- a/engines/dev-dax.c
+++ b/engines/dev-dax.c
@@ -157,7 +157,7 @@ static int fio_devdax_prep(struct thread_data *td, struct io_u *io_u)
         * It fits within existing mapping, use it
         */
        if (io_u->offset >= fdd->devdax_off &&
-           io_u->offset + io_u->buflen < fdd->devdax_off + fdd->devdax_sz)
+           io_u->offset + io_u->buflen <= fdd->devdax_off + fdd->devdax_sz)
                goto done;
 
        /*

diff --git a/engines/libpmem.c b/engines/libpmem.c
index 3f4e44fb96471936686f5806600505c138896825..3038784512474440e42bb8ea8bf31aeba3da523f 100644 (file)
--- a/engines/libpmem.c
+++ b/engines/libpmem.c
@@ -430,7 +430,7 @@ static int fio_libpmem_prep(struct thread_data *td, struct io_u *io_u)
                        io_u->buflen, fdd->libpmem_sz);
 
        if (io_u->offset >= fdd->libpmem_off &&
-           (io_u->offset + io_u->buflen <
+           (io_u->offset + io_u->buflen <=
             fdd->libpmem_off + fdd->libpmem_sz))
                goto done;
 

diff --git a/engines/mmap.c b/engines/mmap.c
index 51606e12ff316c3ff537ec328103b17e4c3b1ba0..77556588e1a21218ead2006d5d73ee1bd41087c5 100644 (file)
--- a/engines/mmap.c
+++ b/engines/mmap.c
@@ -137,7 +137,7 @@ static int fio_mmapio_prep(struct thread_data *td, struct io_u *io_u)
         * It fits within existing mapping, use it
         */
        if (io_u->offset >= fmd->mmap_off &&
-           io_u->offset + io_u->buflen < fmd->mmap_off + fmd->mmap_sz)
+           io_u->offset + io_u->buflen <= fmd->mmap_off + fmd->mmap_sz)
                goto done;
 
        /*