io_uring: fix overflow resched cqe reordering

Leaving the CQ critical section in the middle of a overflow flushing
can cause cqe reordering since the cache cq pointers are reset and any
new cqe emitters that might get called in between are not going to be
forced into io_cqe_cache_refill().

Fixes: eac2ca2d682f9 ("io_uring: check if we need to reschedule during overflow flush")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/90ba817f1a458f091f355f407de1c911d2b93bbf.1747483784.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 541e65a1eebfdea76832c3b9f052e00a1c83c0d8..edda31a15c6e659ce5f3aa4c80fbf4a92942015c 100644 (file)
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -636,6 +636,7 @@ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool dying)
                 * to care for a non-real case.
                 */
                if (need_resched()) {
+                       ctx->cqe_sentinel = ctx->cqe_cached;
                        io_cq_unlock_post(ctx);
                        mutex_unlock(&ctx->uring_lock);
                        cond_resched();