From: Ken Raeburn <raeburn@permabit.com>
Date: Fri, 4 Sep 2015 19:30:58 +0000 (-0600)
Subject: Retry bdev cache invalidation for EAGAIN errors
X-Git-Tag: fio-2.2.10~10
X-Git-Url: https://git.kernel.dk/?p=fio.git;a=commitdiff_plain;h=2b9f9141f2b56ec74f27ef8a82d75859dc3af04c

Retry bdev cache invalidation for EAGAIN errors

I've hit a couple test failures where fio quickly died thusly:

    fio 2.0.7
    Starting 1 thread
    fio: pid=6196, err=11/file:filesetup.c:404, func=invalidate_cache, error=Resource temporarily unavailable

(Yeah, we're a little behind. But I think the relevant code is similar.)

We're testing against Linux (RHEL 6.6) multipath block devices, and it
appears that at the time the fio test starts, the multipath maps for
those devices may still be in flux. I poked around a bit with systemtap,
and it appears that while updating the maps, multipathd "suspends" its
multipath device for a few tens of milliseconds, and the kernel
multipath code rejects ioctl calls with EAGAIN if the device is
suspended. It's a small window, but we've managed to hit it multiple
times, though it's not reliably reproducible.

I know the current sources treat failure here as non-fatal, but if we're
using fio for performance tests, trying a little harder to do the
invalidation seems like a good idea. My approach is to add a retry loop
in __file_invalidate_cache; a patch is attached.

It could also be pushed down into blockdev_invalidate_cache, where it
could be local to the Linux (and Android?) implementation, since none of
the others actually do anything. (They return either EAGAIN or zero,
both of which are taken as success indicators.)

Signed-off-by: Jens Axboe <axboe@fb.com>
---

diff --git a/filesetup.c b/filesetup.c
index 881aeee8..e1dedcd8 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -413,7 +413,19 @@ static int __file_invalidate_cache(struct thread_data *td, struct fio_file *f,
 	else if (f->filetype == FIO_TYPE_FILE)
 		ret = posix_fadvise(f->fd, off, len, POSIX_FADV_DONTNEED);
 	else if (f->filetype == FIO_TYPE_BD) {
+		int retry_count = 0;
+
 		ret = blockdev_invalidate_cache(f);
+		while (ret < 0 && errno == EAGAIN && retry_count++ < 25) {
+			/*
+			 * Linux multipath devices reject ioctl while
+			 * the maps are being updated. That window can
+			 * last tens of milliseconds; we'll try up to
+			 * a quarter of a second.
+			 */
+			usleep(10000);
+			ret = blockdev_invalidate_cache(f);
+		}
 		if (ret < 0 && errno == EACCES && geteuid()) {
 			if (!root_warn) {
 				log_err("fio: only root may flush block "