rcu: Make cond_resched_rcu_qs() apply to normal RCU flavors
authorPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Sun, 14 Dec 2014 04:32:04 +0000 (20:32 -0800)
committerPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Fri, 16 Jan 2015 07:33:14 +0000 (23:33 -0800)
commit5cd37193ce8539be1e6ef76be226f4bcc984e0f5
tree7bba4e3fd57b669d60e37fd32011db4860016505
parenta94844b22a2e2b9155bbc0878c507850477221c2
rcu: Make cond_resched_rcu_qs() apply to normal RCU flavors

Although cond_resched_rcu_qs() only applies to TASKS_RCU, it is used
in places where it would be useful for it to apply to the normal RCU
flavors, rcu_preempt, rcu_sched, and rcu_bh.  This is especially the
case for workloads that aggressively overload the system, particularly
those that generate large numbers of RCU updates on systems running
NO_HZ_FULL CPUs.  This commit therefore communicates quiescent states
from cond_resched_rcu_qs() to the normal RCU flavors.

Note that it is unfortunately necessary to leave the old ->passed_quiesce
mechanism in place to allow quiescent states that apply to only one
flavor to be recorded.  (Yes, we could decrement ->rcu_qs_ctr_snap in
that case, but that is not so good for debugging of RCU internals.)
In addition, if one of the RCU flavor's grace period has stalled, this
will invoke rcu_momentary_dyntick_idle(), resulting in a heavy-weight
quiescent state visible from other CPUs.

Reported-by: Sasha Levin <sasha.levin@oracle.com>
Reported-by: Dave Jones <davej@redhat.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
[ paulmck: Merge commit from Sasha Levin fixing a bug where __this_cpu()
  was used in preemptible code. ]
Documentation/RCU/trace.txt
include/linux/rcupdate.h
include/linux/rcutiny.h
include/linux/rcutree.h
kernel/rcu/tree.c
kernel/rcu/tree.h
kernel/rcu/tree_trace.c