blk-mq: move srcu from blk_mq_hw_ctx to request_queue
authorMing Lei <ming.lei@redhat.com>
Fri, 3 Dec 2021 13:15:32 +0000 (21:15 +0800)
committerJens Axboe <axboe@kernel.dk>
Fri, 3 Dec 2021 21:51:29 +0000 (14:51 -0700)
commit704b914f15fb7daaf517e3acc4bed472b50ca19e
tree5fffc1140fef4ddceeaae58e44fbb65333c33861
parent2a904d00855f94cb85751e45fa494f225d44ae0d
blk-mq: move srcu from blk_mq_hw_ctx to request_queue

In case of BLK_MQ_F_BLOCKING, per-hctx srcu is used to protect dispatch
critical area. However, this srcu instance stays at the end of hctx, and
it often takes standalone cacheline, often cold.

Inside srcu_read_lock() and srcu_read_unlock(), WRITE is always done on
the indirect percpu variable which is allocated from heap instead of
being embedded, srcu->srcu_idx is read only in srcu_read_lock(). It
doesn't matter if srcu structure stays in hctx or request queue.

So switch to per-request-queue srcu for protecting dispatch, and this
way simplifies quiesce a lot, not mention quiesce is always done on the
request queue wide.

Signed-off-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/20211203131534.3668411-3-ming.lei@redhat.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
block/blk-core.c
block/blk-mq-sysfs.c
block/blk-mq.c
block/blk-mq.h
block/blk-sysfs.c
block/blk.h
block/genhd.c
include/linux/blk-mq.h
include/linux/blkdev.h