io_uring: support bio caching for non-polled IO io_uring-bio-cache.6
authorJens Axboe <axboe@kernel.dk>
Thu, 19 Aug 2021 01:38:58 +0000 (19:38 -0600)
committerJens Axboe <axboe@kernel.dk>
Thu, 19 Aug 2021 03:22:14 +0000 (21:22 -0600)
commit89ad7923d6702d0c8fc2b0ec97998b395f2fa03a
treebbed8f751ae00fc13f7f4d7c8c666026598b5288
parentfacf46c9a78c920229e4ad242910cd9c4688ced8
io_uring: support bio caching for non-polled IO

Mark the kiocb with IOCB_ALLOC_CACHE even for non-polled IO, in case
the lower layer participates in per-cpu bio caching. If it does, then
IOCB_PUT_CACHE will be set upon kiocb->ki_complete() invocation,
passing ownership to io_uring.

io_uring doesn't complete even IRQ based requests from IRQ context,
so we can safely put the bio when we run the actual io_kiocb completion.

This provides a 5-10% boost in IOPS with IRQ driven IO.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c