io_uring: optimise io_cqring_wait() hot path
authorPavel Begunkov <asml.silence@gmail.com>
Mon, 9 Aug 2021 15:07:32 +0000 (09:07 -0600)
committerJens Axboe <axboe@kernel.dk>
Tue, 10 Aug 2021 23:51:39 +0000 (17:51 -0600)
commitce67fd47eb7572036679238dab12d2910f114069
tree1a7367116d9ea1b566207e9adea4866b82496af6
parent4ec2c18f7a103f381adef7bc52f653f17680c424
io_uring: optimise io_cqring_wait() hot path

Turns out we always init struct io_wait_queue in io_cqring_wait(), even
if it's not used after, i.e. there are already enough of CQEs. And often
it's exactly what happens, for instance, requests may have been
completed inline, or in case of io_uring_enter(submit=N, wait=1).

It shows up in my profiler, so optimise it by delaying the struct init.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/6f1b81c60b947d165583dc333947869c3d85d037.1628471125.git.asml.silence@gmail.com
[axboe: fixed up for new cqring wait]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c