io_uring: optimise putting task struct
authorPavel Begunkov <asml.silence@gmail.com>
Mon, 9 Aug 2021 12:04:20 +0000 (13:04 +0100)
committerJens Axboe <axboe@kernel.dk>
Tue, 10 Aug 2021 19:52:07 +0000 (13:52 -0600)
commit8aca6d1ccf3b5aa54c0eb0f255a6df8455fcc090
tree9642cdf5333ab7740b8a6ea8524cffea5a7d929f
parent00fedb5797f39fae17cbf62db7e63e0fff4a0bb1
io_uring: optimise putting task struct

We cache all the reference to task + tctx, so if io_put_task() is
called by the corresponding task itself, we can save on atomics and
return the refs right back into the cache.

It's beneficial for all inline completions, and also iopolling, when
polling and submissions are done by the same task, including
SQPOLL|IOPOLL.

Note: io_uring_cancel_generic() can return refs to the cache as well,
so those should be flushed in the loop for tctx_inflight() to work
right.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/6fe9646b3cb70e46aca1f58426776e368c8926b3.1628471125.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c