io_uring: optimise submission side poll_refs
authorPavel Begunkov <asml.silence@gmail.com>
Thu, 23 Jun 2022 13:24:49 +0000 (14:24 +0100)
committerJens Axboe <axboe@kernel.dk>
Thu, 21 Jul 2022 14:46:53 +0000 (08:46 -0600)
commit617b1092abc7c2eaf5e5f6ce77366a0dc2157ace
tree7be6f72bfe81a85734fd7c37365af66ab96aff40
parent4a615dbc8db60b8350c22550ef706066c7708a3f
io_uring: optimise submission side poll_refs

The final poll_refs put in __io_arm_poll_handler() takes quite some
cycles. When we're arming from the original task context task_work won't
be run, so in this case we can assume that we won't race with task_works
and so not take the initial ownership ref.

One caveat is that after arming a poll we may race with it, so we have
to add a bunch of io_poll_get_ownership() hidden inside of
io_poll_can_finish_inline() whenever we want to complete arming inline.
For the same reason we can't just set REQ_F_DOUBLE_POLL in
__io_queue_proc() and so need to sync with the first poll entry by
taking its wq head lock.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/8825315d7f5e182ac1578a031e546f79b1c97d01.1655990418.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_uring/poll.c