io_uring: optimise io_cqring_wait() hot path
authorPavel Begunkov <asml.silence@gmail.com>
Mon, 9 Aug 2021 15:07:32 +0000 (09:07 -0600)
committerJens Axboe <axboe@kernel.dk>
Tue, 10 Aug 2021 19:52:06 +0000 (13:52 -0600)
commitd593cae47d9c0cd0a940bea9ac36354a7d783f5d
tree5c237f6a6b94f36c882ea95c1e956455cbfd6321
parent1eb5509312ffeb573f9e353775ffcd0fe12c96bc
io_uring: optimise io_cqring_wait() hot path

Turns out we always init struct io_wait_queue in io_cqring_wait(), even
if it's not used after, i.e. there are already enough of CQEs. And often
it's exactly what happens, for instance, requests may have been
completed inline, or in case of io_uring_enter(submit=N, wait=1).

It shows up in my profiler, so optimise it by delaying the struct init.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/6f1b81c60b947d165583dc333947869c3d85d037.1628471125.git.asml.silence@gmail.com
[axboe: fixed up for new cqring wait]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c