io_uring: optimise putting task struct
authorPavel Begunkov <asml.silence@gmail.com>
Mon, 9 Aug 2021 12:04:20 +0000 (13:04 +0100)
committerJens Axboe <axboe@kernel.dk>
Mon, 9 Aug 2021 15:10:28 +0000 (09:10 -0600)
commitae1973250363c85c2e4c411fa09ae0944a35b0b3
tree38347da9501f7242d9f75ed326bbd4741f554fb0
parentb196fcb709488ef6ee3ddb1caa66e3211bc0b9a7
io_uring: optimise putting task struct

We cache all the reference to task + tctx, so if io_put_task() is
called by the corresponding task itself, we can save on atomics and
return the refs right back into the cache.

It's beneficial for all inline completions, and also iopolling, when
polling and submissions are done by the same task, including
SQPOLL|IOPOLL.

Note: io_uring_cancel_generic() can return refs to the cache as well,
so those should be flushed in the loop for tctx_inflight() to work
right.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/6fe9646b3cb70e46aca1f58426776e368c8926b3.1628471125.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c