Retry bdev cache invalidation for EAGAIN errors

I've hit a couple test failures where fio quickly died thusly:

    fio 2.0.7
    Starting 1 thread
    fio: pid=6196, err=11/file:filesetup.c:404, func=invalidate_cache, error=Resource temporarily unavailable

(Yeah, we're a little behind. But I think the relevant code is similar.)

We're testing against Linux (RHEL 6.6) multipath block devices, and it
appears that at the time the fio test starts, the multipath maps for
those devices may still be in flux. I poked around a bit with systemtap,
and it appears that while updating the maps, multipathd "suspends" its
multipath device for a few tens of milliseconds, and the kernel
multipath code rejects ioctl calls with EAGAIN if the device is
suspended. It's a small window, but we've managed to hit it multiple
times, though it's not reliably reproducible.

I know the current sources treat failure here as non-fatal, but if we're
using fio for performance tests, trying a little harder to do the
invalidation seems like a good idea. My approach is to add a retry loop
in __file_invalidate_cache; a patch is attached.

It could also be pushed down into blockdev_invalidate_cache, where it
could be local to the Linux (and Android?) implementation, since none of
the others actually do anything. (They return either EAGAIN or zero,
both of which are taken as success indicators.)

Signed-off-by: Jens Axboe <axboe@fb.com>

diff --git a/filesetup.c b/filesetup.c
index 881aeee8f8bf2b94649046181d9892d7050face8..e1dedcd8bdec6d074d37f2ea07d6bde45007f9e5 100644 (file)
--- a/filesetup.c
+++ b/filesetup.c
@@ -413,7 +413,19 @@ static int __file_invalidate_cache(struct thread_data *td, struct fio_file *f,
        else if (f->filetype == FIO_TYPE_FILE)
                ret = posix_fadvise(f->fd, off, len, POSIX_FADV_DONTNEED);
        else if (f->filetype == FIO_TYPE_BD) {
+               int retry_count = 0;
+
                ret = blockdev_invalidate_cache(f);
+               while (ret < 0 && errno == EAGAIN && retry_count++ < 25) {
+                       /*
+                        * Linux multipath devices reject ioctl while
+                        * the maps are being updated. That window can
+                        * last tens of milliseconds; we'll try up to
+                        * a quarter of a second.
+                        */
+                       usleep(10000);
+                       ret = blockdev_invalidate_cache(f);
+               }
                if (ret < 0 && errno == EACCES && geteuid()) {
                        if (!root_warn) {
                                log_err("fio: only root may flush block "