io_uring: optimise io_cqring_wait() hot path
authorPavel Begunkov <asml.silence@gmail.com>
Mon, 9 Aug 2021 15:07:32 +0000 (09:07 -0600)
committerJens Axboe <axboe@kernel.dk>
Mon, 9 Aug 2021 15:07:32 +0000 (09:07 -0600)
commit59d942afd228592cefec73b05b78e15f21dd2c0d
tree783b6e97e3001d9c4f2ecac297d9df0fee3adc59
parente54da48cb4f56023e1d897aecd690feea6737871
io_uring: optimise io_cqring_wait() hot path

Turns out we always init struct io_wait_queue in io_cqring_wait(), even
if it's not used after, i.e. there are already enough of CQEs. And often
it's exactly what happens, for instance, requests may have been
completed inline, or in case of io_uring_enter(submit=N, wait=1).

It shows up in my profiler, so optimise it by delaying the struct init.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/6f1b81c60b947d165583dc333947869c3d85d037.1628471125.git.asml.silence@gmail.com
[axboe: fixed up for new cqring wait]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c