[linux-block.git] / Documentation / vm / split_page_table_lock.rst

.. _split_page_table_lock:

=====================
Split page table lock
=====================

Originally, mm->page_table_lock spinlock protected all page tables of the
mm_struct. But this approach leads to poor page fault scalability of
multi-threaded applications due high contention on the lock. To improve
scalability, split page table lock was introduced.

With split page table lock we have separate per-table lock to serialize
access to the table. At the moment we use split lock for PTE and PMD
tables. Access to higher level tables protected by mm->page_table_lock.

There are helpers to lock/unlock a table and other accessor functions:

 - pte_offset_map_lock()
	maps pte and takes PTE table lock, returns pointer to the taken
	lock;
 - pte_unmap_unlock()
	unlocks and unmaps PTE table;
 - pte_alloc_map_lock()
	allocates PTE table if needed and take the lock, returns pointer
	to taken lock or NULL if allocation failed;
 - pte_lockptr()
	returns pointer to PTE table lock;
 - pmd_lock()
	takes PMD table lock, returns pointer to taken lock;
 - pmd_lockptr()
	returns pointer to PMD table lock;

Split page table lock for PTE tables is enabled compile-time if
CONFIG_SPLIT_PTLOCK_CPUS (usually 4) is less or equal to NR_CPUS.
If split lock is disabled, all tables guaded by mm->page_table_lock.

Split page table lock for PMD tables is enabled, if it's enabled for PTE
tables and the architecture supports it (see below).

Hugetlb and split page table lock
=================================

Hugetlb can support several page sizes. We use split lock only for PMD
level, but not for PUD.

Hugetlb-specific helpers:

 - huge_pte_lock()
	takes pmd split lock for PMD_SIZE page, mm->page_table_lock
	otherwise;
 - huge_pte_lockptr()
	returns pointer to table lock;

Support of split page table lock by an architecture
===================================================

There's no need in special enabling of PTE split page table lock: everything
required is done by pgtable_pte_page_ctor() and pgtable_pte_page_dtor(), which
must be called on PTE table allocation / freeing.

Make sure the architecture doesn't use slab allocator for page table
allocation: slab uses page->slab_cache for its pages.
This field shares storage with page->ptl.

PMD split lock only makes sense if you have more than two page table
levels.

PMD split lock enabling requires pgtable_pmd_page_ctor() call on PMD table
allocation and pgtable_pmd_page_dtor() on freeing.

Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and
pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing
paths: i.e X86_PAE preallocate few PMDs on pgd_alloc().

With everything in place you can set CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK.

NOTE: pgtable_pte_page_ctor() and pgtable_pmd_page_ctor() can fail -- it must
be handled properly.

page->ptl
=========

page->ptl is used to access split page table lock, where 'page' is struct
page of page containing the table. It shares storage with page->private
(and few other fields in union).

To avoid increasing size of struct page and have best performance, we use a
trick:

 - if spinlock_t fits into long, we use page->ptr as spinlock, so we
   can avoid indirect access and save a cache line.
 - if size of spinlock_t is bigger then size of long, we use page->ptl as
   pointer to spinlock_t and allocate it dynamically. This allows to use
   split lock with enabled DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC, but costs
   one more cache line for indirect access;

The spinlock_t allocated in pgtable_pte_page_ctor() for PTE table and in
pgtable_pmd_page_ctor() for PMD table.

Please, never access page->ptl directly -- use appropriate helper.
Commit	Line	Data
d18edf52 MR	1	.. _split_page_table_lock:
	2
	3	=====================
49076ec2 KS	4	Split page table lock
	5	=====================
	6
	7	Originally, mm->page_table_lock spinlock protected all page tables of the
	8	mm_struct. But this approach leads to poor page fault scalability of
	9	multi-threaded applications due high contention on the lock. To improve
	10	scalability, split page table lock was introduced.
	11
	12	With split page table lock we have separate per-table lock to serialize
	13	access to the table. At the moment we use split lock for PTE and PMD
	14	tables. Access to higher level tables protected by mm->page_table_lock.
	15
	16	There are helpers to lock/unlock a table and other accessor functions:
d18edf52	17
49076ec2 KS	18	- pte_offset_map_lock()
	19	maps pte and takes PTE table lock, returns pointer to the taken
	20	lock;
	21	- pte_unmap_unlock()
	22	unlocks and unmaps PTE table;
	23	- pte_alloc_map_lock()
	24	allocates PTE table if needed and take the lock, returns pointer
	25	to taken lock or NULL if allocation failed;
	26	- pte_lockptr()
	27	returns pointer to PTE table lock;
	28	- pmd_lock()
	29	takes PMD table lock, returns pointer to taken lock;
	30	- pmd_lockptr()
	31	returns pointer to PMD table lock;
	32
	33	Split page table lock for PTE tables is enabled compile-time if
	34	CONFIG_SPLIT_PTLOCK_CPUS (usually 4) is less or equal to NR_CPUS.
	35	If split lock is disabled, all tables guaded by mm->page_table_lock.
	36
	37	Split page table lock for PMD tables is enabled, if it's enabled for PTE
	38	tables and the architecture supports it (see below).
	39
	40	Hugetlb and split page table lock
d18edf52	41	=================================
49076ec2 KS	42
	43	Hugetlb can support several page sizes. We use split lock only for PMD
	44	level, but not for PUD.
	45
	46	Hugetlb-specific helpers:
d18edf52	47
49076ec2 KS	48	- huge_pte_lock()
	49	takes pmd split lock for PMD_SIZE page, mm->page_table_lock
	50	otherwise;
	51	- huge_pte_lockptr()
	52	returns pointer to table lock;
	53
	54	Support of split page table lock by an architecture
d18edf52	55	===================================================
49076ec2	56
b4ed71f5 MR	57	There's no need in special enabling of PTE split page table lock: everything
	58	required is done by pgtable_pte_page_ctor() and pgtable_pte_page_dtor(), which
	59	must be called on PTE table allocation / freeing.
49076ec2 KS	60
49076ec2 KS	61	Make sure the architecture doesn't use slab allocator for page table
1d798ca3 KS	62	allocation: slab uses page->slab_cache for its pages.
1d798ca3 KS	63	This field shares storage with page->ptl.
49076ec2 KS	64
	65	PMD split lock only makes sense if you have more than two page table
	66	levels.
	67
	68	PMD split lock enabling requires pgtable_pmd_page_ctor() call on PMD table
	69	allocation and pgtable_pmd_page_dtor() on freeing.
	70
c283610e KS	71	Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and
	72	pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing
	73	paths: i.e X86_PAE preallocate few PMDs on pgd_alloc().
49076ec2 KS	74
	75	With everything in place you can set CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK.
	76
b4ed71f5	77	NOTE: pgtable_pte_page_ctor() and pgtable_pmd_page_ctor() can fail -- it must
49076ec2 KS	78	be handled properly.
	79
	80	page->ptl
d18edf52	81	=========
49076ec2 KS	82
	83	page->ptl is used to access split page table lock, where 'page' is struct
	84	page of page containing the table. It shares storage with page->private
	85	(and few other fields in union).
	86
	87	To avoid increasing size of struct page and have best performance, we use a
	88	trick:
d18edf52	89
49076ec2 KS	90	- if spinlock_t fits into long, we use page->ptr as spinlock, so we
	91	can avoid indirect access and save a cache line.
	92	- if size of spinlock_t is bigger then size of long, we use page->ptl as
	93	pointer to spinlock_t and allocate it dynamically. This allows to use
	94	split lock with enabled DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC, but costs
	95	one more cache line for indirect access;
	96
b4ed71f5	97	The spinlock_t allocated in pgtable_pte_page_ctor() for PTE table and in
49076ec2 KS	98	pgtable_pmd_page_ctor() for PMD table.
	99
	100	Please, never access page->ptl directly -- use appropriate helper.