io_uring: optimise io_get_cqe()
authorPavel Begunkov <asml.silence@gmail.com>
Tue, 12 Apr 2022 14:09:51 +0000 (15:09 +0100)
committerJens Axboe <axboe@kernel.dk>
Mon, 25 Apr 2022 00:02:46 +0000 (18:02 -0600)
commitd8da428b7a9a71bc2ee6bf628a8c0f9beb96a195
tree24e83fa14dea9343d98fc5227421e610a0a4befc
parent1cd15904b6e836f46e925229da8950785252b315
io_uring: optimise io_get_cqe()

io_get_cqe() is expensive because of a bunch of loads, masking, etc.
However, most of the time we should have enough of entries in the CQ,
so we can cache two pointers representing a range of contiguous CQE
memory we can use. When the range is exhausted we'll go through a slower
path to set up a new range. When there are no CQEs avaliable, pointers
will naturally point to the same address.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/487eeef00f3146537b3d9c1a9cef2fc0b9a86f81.1649771823.git.asml.silence@gmail.com
[axboe: santinel -> sentinel]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c