io_uring: fix race condition when sq threads goes sleeping

Reading the SQ tail needs to come after setting IORING_SQ_NEED_WAKEUP in
flags; there is no cheap barrier for ordering a store before a load, a
full memory barrier is required.

Userspace needs a full memory barrier between updating SQ tail and
checking for the IORING_SQ_NEED_WAKEUP too.

Signed-off-by: Stefan Bühler <source@stbuehler.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 41e3a6f6a096afe3b70320a1d67b6420aef78001..69910fd9ccca03840fa155f8f1f3c855457866fd 100644 (file)
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -1865,7 +1865,8 @@ static int io_sq_thread(void *data)
 
                        /* Tell userspace we may need a wakeup call */
                        ctx->sq_ring->flags |= IORING_SQ_NEED_WAKEUP;
-                       smp_wmb();
+                       /* make sure to read SQ tail after writing flags */
+                       smp_mb();
 
                        if (!io_get_sqring(ctx, &sqes[0])) {
                                if (kthread_should_stop()) {