io_uring: optimise putting task struct
authorPavel Begunkov <asml.silence@gmail.com>
Mon, 9 Aug 2021 12:04:20 +0000 (13:04 +0100)
committerJens Axboe <axboe@kernel.dk>
Tue, 10 Aug 2021 23:51:41 +0000 (17:51 -0600)
commit96b0fdd60ffea4052600bac377a010968f4a6827
tree1c228c1f59d9365d5166d898143eaea5c5a79852
parent4ddc28d9773afc57c0579103ece93d134b594d79
io_uring: optimise putting task struct

We cache all the reference to task + tctx, so if io_put_task() is
called by the corresponding task itself, we can save on atomics and
return the refs right back into the cache.

It's beneficial for all inline completions, and also iopolling, when
polling and submissions are done by the same task, including
SQPOLL|IOPOLL.

Note: io_uring_cancel_generic() can return refs to the cache as well,
so those should be flushed in the loop for tctx_inflight() to work
right.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/6fe9646b3cb70e46aca1f58426776e368c8926b3.1628471125.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c