io_uring: optimise io_req_local_work_add
authorPavel Begunkov <asml.silence@gmail.com>
Thu, 6 Apr 2023 13:20:14 +0000 (14:20 +0100)
committerJens Axboe <axboe@kernel.dk>
Thu, 6 Apr 2023 22:24:36 +0000 (16:24 -0600)
commit360cd42c4e95ff06d8d7b0a54e42236c7e7c187f
tree97807acf0f2bcb3ada1ad005bbaa8d95396aceab
parentc66ae3ec38f946edb1776d25c1c8cd63803b8ec3
io_uring: optimise io_req_local_work_add

Chains of memory accesses are never good for performance.
The req->task->io_uring->in_cancel in io_req_local_work_add() is there
so that when a task is exiting via io_uring_try_cancel_requests() and
starts waiting for completions, it gets woken up by every new task_work
item queued.

Do a little trick by announcing waiting in io_uring_try_cancel_requests(),
making io_req_local_work_add() wake us up. We also need to check for
deferred tw items after prepare_to_wait(TASK_INTERRUPTIBLE);

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/fb11597e9bbcb365901824f8c5c2cf0d6ee100d0.1680782017.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_uring/io_uring.c