block, bfq: set next_rq to waker_bfqq->next_rq in waker injection
authorJia Cheng Hu <jia.jiachenghu@gmail.com>
Fri, 22 Jan 2021 18:19:44 +0000 (19:19 +0100)
committerJens Axboe <axboe@kernel.dk>
Mon, 25 Jan 2021 01:18:24 +0000 (18:18 -0700)
commitd4fc3640ff361a09e359867e0bca898abd2b7ecb
tree32cfec179ba748356172ef15519e85b23da2a3c0
parentb5f74ecacc3139ef873e69acc3aba28083ecc416
block, bfq: set next_rq to waker_bfqq->next_rq in waker injection

Since commit c5089591c3ba ("block, bfq: detect wakers and
unconditionally inject their I/O"), when the in-service bfq_queue, say
Q, is temporarily empty, BFQ checks whether there are I/O requests to
inject (also) from the waker bfq_queue for Q. To this goal, the value
pointed by bfqq->waker_bfqq->next_rq must be controlled. However, the
current implementation mistakenly looks at bfqq->next_rq, which
instead points to the next request of the currently served queue.

This mistake evidently causes losses of throughput in scenarios with
waker bfq_queues.

This commit corrects this mistake.

Fixes: c5089591c3ba ("block, bfq: detect wakers and unconditionally inject their I/O")
Signed-off-by: Jia Cheng Hu <jia.jiachenghu@gmail.com>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Paolo Valente <paolo.valente@linaro.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
block/bfq-iosched.c