sched/fair: Trigger nohz.next_balance updates when a CPU goes NOHZ-idle

Consider a system with some NOHZ-idle CPUs, such that

  nohz.idle_cpus_mask = S
  nohz.next_balance = T

When a new CPU k goes NOHZ idle (nohz_balance_enter_idle()), we end up
with:

  nohz.idle_cpus_mask = S \U {k}
  nohz.next_balance = T

Note that the nohz.next_balance hasn't changed - it won't be updated until
a NOHZ balance is triggered. This is problematic if the newly NOHZ idle CPU
has an earlier rq.next_balance than the other NOHZ idle CPUs, IOW if:

  cpu_rq(k).next_balance < nohz.next_balance

In such scenarios, the existing nohz.next_balance will prevent any NOHZ
balance from happening, which itself will prevent nohz.next_balance from
being updated to this new cpu_rq(k).next_balance. Unnecessary load balance
delays of over 12ms caused by this were observed on an arm64 RB5 board.

Use the new nohz.needs_update flag to mark the presence of newly-idle CPUs
that need their rq->next_balance to be collated into
nohz.next_balance. Trigger a NOHZ_NEXT_KICK when the flag is set.

Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://lkml.kernel.org/r/20210823111700.2842997-3-valentin.schneider@arm.com

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index f4de7f555cbe0865c4e97353a6523d50a1843184..6cc958e0584db23ad1ced8cf7b1f034d14dc5084 100644 (file)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5787,6 +5787,7 @@ static struct {
        cpumask_var_t idle_cpus_mask;
        atomic_t nr_cpus;
        int has_blocked;                /* Idle CPUS has blocked load */
+       int needs_update;               /* Newly idle CPUs need their next_balance collated */
        unsigned long next_balance;     /* in jiffy units */
        unsigned long next_blocked;     /* Next update of blocked load in jiffies */
 } nohz ____cacheline_aligned;
@@ -10450,6 +10451,9 @@ static void nohz_balancer_kick(struct rq *rq)
 unlock:
        rcu_read_unlock();
 out:
+       if (READ_ONCE(nohz.needs_update))
+               flags |= NOHZ_NEXT_KICK;
+
        if (flags)
                kick_ilb(flags);
 }
@@ -10546,12 +10550,13 @@ void nohz_balance_enter_idle(int cpu)
        /*
         * Ensures that if nohz_idle_balance() fails to observe our
         * @idle_cpus_mask store, it must observe the @has_blocked
-        * store.
+        * and @needs_update stores.
         */
        smp_mb__after_atomic();
 
        set_cpu_sd_state_idle(cpu);
 
+       WRITE_ONCE(nohz.needs_update, 1);
 out:
        /*
         * Each time a cpu enter idle, we assume that it has blocked load and
@@ -10600,13 +10605,17 @@ static void _nohz_idle_balance(struct rq *this_rq, unsigned int flags,
        /*
         * We assume there will be no idle load after this update and clear
         * the has_blocked flag. If a cpu enters idle in the mean time, it will
-        * set the has_blocked flag and trig another update of idle load.
+        * set the has_blocked flag and trigger another update of idle load.
         * Because a cpu that becomes idle, is added to idle_cpus_mask before
         * setting the flag, we are sure to not clear the state and not
         * check the load of an idle cpu.
+        *
+        * Same applies to idle_cpus_mask vs needs_update.
         */
        if (flags & NOHZ_STATS_KICK)
                WRITE_ONCE(nohz.has_blocked, 0);
+       if (flags & NOHZ_NEXT_KICK)
+               WRITE_ONCE(nohz.needs_update, 0);
 
        /*
         * Ensures that if we miss the CPU, we must see the has_blocked
@@ -10630,6 +10639,8 @@ static void _nohz_idle_balance(struct rq *this_rq, unsigned int flags,
                if (need_resched()) {
                        if (flags & NOHZ_STATS_KICK)
                                has_blocked_load = true;
+                       if (flags & NOHZ_NEXT_KICK)
+                               WRITE_ONCE(nohz.needs_update, 1);
                        goto abort;
                }