sched: Fix balance_callback()
authorPeter Zijlstra <peterz@infradead.org>
Mon, 11 May 2020 12:13:00 +0000 (14:13 +0200)
committerPeter Zijlstra <peterz@infradead.org>
Tue, 10 Nov 2020 17:38:57 +0000 (18:38 +0100)
commit565790d28b1e33ee2f77bad5348b99f6dfc366fd
tree179b55e6e483c0daf2975f95ccf57401fad0e7d6
parenta8b62fd0850503cf1e557d7e5a98d3f1f5c25eef
sched: Fix balance_callback()

The intent of balance_callback() has always been to delay executing
balancing operations until the end of the current rq->lock section.
This is because balance operations must often drop rq->lock, and that
isn't safe in general.

However, as noted by Scott, there were a few holes in that scheme;
balance_callback() was called after rq->lock was dropped, which means
another CPU can interleave and touch the callback list.

Rework code to call the balance callbacks before dropping rq->lock
where possible, and otherwise splice the balance list onto a local
stack.

This guarantees that the balance list must be empty when we take
rq->lock. IOW, we'll only ever run our own balance callbacks.

Reported-by: Scott Wood <swood@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Valentin Schneider <valentin.schneider@arm.com>
Reviewed-by: Daniel Bristot de Oliveira <bristot@redhat.com>
Link: https://lkml.kernel.org/r/20201023102346.203901269@infradead.org
kernel/sched/core.c
kernel/sched/sched.h