locking/rwsem: Enable reader optimistic lock stealing
authorWaiman Long <longman@redhat.com>
Sat, 21 Nov 2020 04:14:14 +0000 (23:14 -0500)
committerPeter Zijlstra <peterz@infradead.org>
Wed, 9 Dec 2020 16:08:48 +0000 (17:08 +0100)
commit1a728dff855a318bb58bcc1259b1826a7ad9f0bd
treefc773e33ed49257132e2123517323574c41b8e48
parent2f06f702925b512a95b95dca3855549c047eef58
locking/rwsem: Enable reader optimistic lock stealing

If the optimistic spinning queue is empty and the rwsem does not have
the handoff or write-lock bits set, it is actually not necessary to
call rwsem_optimistic_spin() to spin on it. Instead, it can steal the
lock directly as its reader bias is in the count already.  If it is
the first reader in this state, it will try to wake up other readers
in the wait queue.

With this patch applied, the following were the lock event counts
after rebooting a 2-socket system and a "make -j96" kernel rebuild.

  rwsem_opt_rlock=4437
  rwsem_rlock=29
  rwsem_rlock_steal=19

So lock stealing represents about 0.4% of all the read locks acquired
in the slow path.

Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Davidlohr Bueso <dbueso@suse.de>
Link: https://lkml.kernel.org/r/20201121041416.12285-4-longman@redhat.com
kernel/locking/lock_events_list.h
kernel/locking/rwsem.c