io_uring/zcrx: implement zerocopy receive pp memory provider
authorPavel Begunkov <asml.silence@gmail.com>
Sat, 15 Feb 2025 00:09:39 +0000 (16:09 -0800)
committerJens Axboe <axboe@kernel.dk>
Mon, 17 Feb 2025 12:41:09 +0000 (05:41 -0700)
commit34a3e60821ab9f335a58d43a88cccdbefdebdec3
tree63404433d0cc217653e56050380c0b95262e400c
parent035af94b39fd13751abf5f0a2948c9eddede55d0
io_uring/zcrx: implement zerocopy receive pp memory provider

Implement a page pool memory provider for io_uring to receieve in a
zero copy fashion. For that, the provider allocates user pages wrapped
around into struct net_iovs, that are stored in a previously registered
struct net_iov_area.

Unlike the traditional receive, that frees pages and returns them back
to the page pool right after data was copied to the user, e.g. inside
recv(2), we extend the lifetime until the user space confirms that it's
done processing the data. That's done by taking a net_iov reference.
When the user is done with the buffer, it must return it back to the
kernel by posting an entry into the refill ring, which is usually polled
off the io_uring memory provider callback in the page pool's netmem
allocation path.

There is also a separate set of per net_iov "user" references accounting
whether a buffer is currently given to the user (including possible
fragmentation).

Reviewed-by: Jens Axboe <axboe@kernel.dk>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: David Wei <dw@davidwei.uk>
Acked-by: Jakub Kicinski <kuba@kernel.org>
Link: https://lore.kernel.org/r/20250215000947.789731-5-dw@davidwei.uk
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_uring/zcrx.c
io_uring/zcrx.h