engines/io_uring: set COOP_TASKRUN for ring setup

If available, this will generally improve performance quite a bit.
Fio can easily use it, we just need to set the flag.
IORING_SETUP_COOP_TASKRUN tells the kernel that the application doesn't
need to get rescheduled to handle task_work, it's fine to defer this
to when we transition anyway.

For a QD=8 random read workload, this increases performance from
680K to 870K IOPS.

Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/engines/io_uring.c b/engines/io_uring.c
index 94376efa7f79ad91894830250e059312f28ed798..a51468f5b0dd528c3a27b738e9ad9821d4860ca8 100644 (file)
--- a/engines/io_uring.c
+++ b/engines/io_uring.c
@@ -809,9 +809,19 @@ static int fio_ioring_queue_init(struct thread_data *td)
        p.flags |= IORING_SETUP_CQSIZE;
        p.cq_entries = depth;
 
+       /*
+        * Setup COOP_TASKRUN as we don't need to get IPI interrupted for
+        * completing IO operations.
+        */
+       p.flags |= IORING_SETUP_COOP_TASKRUN;
+
 retry:
        ret = syscall(__NR_io_uring_setup, depth, &p);
        if (ret < 0) {
+               if (errno == EINVAL && p.flags & IORING_SETUP_COOP_TASKRUN) {
+                       p.flags &= ~IORING_SETUP_COOP_TASKRUN;
+                       goto retry;
+               }
                if (errno == EINVAL && p.flags & IORING_SETUP_CQSIZE) {
                        p.flags &= ~IORING_SETUP_CQSIZE;
                        goto retry;