cpuidle, sched: Use smp_mb__after_atomic() in current_clr_polling()

In architectures that use the polling bit, current_clr_polling() employs
smp_mb() to ensure that the clearing of the polling bit is visible to
other cores before checking TIF_NEED_RESCHED.

However, smp_mb() can be costly. Given that clear_bit() is an atomic
operation, replacing smp_mb() with smp_mb__after_atomic() is appropriate.

Many architectures implement smp_mb__after_atomic() as a lighter-weight
barrier compared to smp_mb(), leading to performance improvements.
For instance, on x86, smp_mb__after_atomic() is a no-op. This change
eliminates a smp_mb() instruction in the cpuidle wake-up path, saving
several CPU cycles and thereby reducing wake-up latency.

Architectures that do not use the polling bit will retain the original
smp_mb() behavior to ensure that existing dependencies remain unaffected.

Signed-off-by: Yujun Dong <yujundong@pascal-lab.net>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://lore.kernel.org/r/20241230141624.155356-1-yujundong@pascal-lab.net

diff --git a/include/linux/sched/idle.h b/include/linux/sched/idle.h
index e670ac282333e912a9982642928dd956df5b4424..439f6029d3b94ecd7fdf662ba6ad4e4da05341e9 100644 (file)
--- a/include/linux/sched/idle.h
+++ b/include/linux/sched/idle.h
@@ -79,6 +79,21 @@ static __always_inline bool __must_check current_clr_polling_and_test(void)
        return unlikely(tif_need_resched());
 }
 
+static __always_inline void current_clr_polling(void)
+{
+       __current_clr_polling();
+
+       /*
+        * Ensure we check TIF_NEED_RESCHED after we clear the polling bit.
+        * Once the bit is cleared, we'll get IPIs with every new
+        * TIF_NEED_RESCHED and the IPI handler, scheduler_ipi(), will also
+        * fold.
+        */
+       smp_mb__after_atomic(); /* paired with resched_curr() */
+
+       preempt_fold_need_resched();
+}
+
 #else
 static inline void __current_set_polling(void) { }
 static inline void __current_clr_polling(void) { }
@@ -91,21 +106,15 @@ static inline bool __must_check current_clr_polling_and_test(void)
 {
        return unlikely(tif_need_resched());
 }
-#endif
 
 static __always_inline void current_clr_polling(void)
 {
        __current_clr_polling();
 
-       /*
-        * Ensure we check TIF_NEED_RESCHED after we clear the polling bit.
-        * Once the bit is cleared, we'll get IPIs with every new
-        * TIF_NEED_RESCHED and the IPI handler, scheduler_ipi(), will also
-        * fold.
-        */
        smp_mb(); /* paired with resched_curr() */
 
        preempt_fold_need_resched();
 }
+#endif
 
 #endif /* _LINUX_SCHED_IDLE_H */