[linux-block.git] / Documentation / x86 / topology.rst

.. SPDX-License-Identifier: GPL-2.0

============
x86 Topology
============

This documents and clarifies the main aspects of x86 topology modelling and
representation in the kernel. Update/change when doing changes to the
respective code.

The architecture-agnostic topology definitions are in
Documentation/cputopology.txt. This file holds x86-specific
differences/specialities which must not necessarily apply to the generic
definitions. Thus, the way to read up on Linux topology on x86 is to start
with the generic one and look at this one in parallel for the x86 specifics.

Needless to say, code should use the generic functions - this file is *only*
here to *document* the inner workings of x86 topology.

Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.

The main aim of the topology facilities is to present adequate interfaces to
code which needs to know/query/use the structure of the running system wrt
threads, cores, packages, etc.

The kernel does not care about the concept of physical sockets because a
socket has no relevance to software. It's an electromechanical component. In
the past a socket always contained a single package (see below), but with the
advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
there might be still references to sockets in the code, but they are of
historical nature and should be cleaned up.

The topology of a system is described in the units of:

    - packages
    - cores
    - threads

Package
=======
Packages contain a number of cores plus shared resources, e.g. DRAM
controller, shared caches etc.

AMD nomenclature for package is 'Node'.

Package-related topology information in the kernel:

  - cpuinfo_x86.x86_max_cores:

    The number of cores in a package. This information is retrieved via CPUID.

  - cpuinfo_x86.phys_proc_id:

    The physical ID of the package. This information is retrieved via CPUID
    and deduced from the APIC IDs of the cores in the package.

  - cpuinfo_x86.logical_proc_id:

    The logical ID of the package. As we do not trust BIOSes to enumerate the
    packages in a consistent way, we introduced the concept of logical package
    ID so we can sanely calculate the number of maximum possible packages in
    the system and have the packages enumerated linearly.

  - topology_max_packages():

    The maximum possible number of packages in the system. Helpful for per
    package facilities to preallocate per package information.

  - cpu_llc_id:

    A per-CPU variable containing:

      - On Intel, the first APIC ID of the list of CPUs sharing the Last Level
        Cache

      - On AMD, the Node ID or Core Complex ID containing the Last Level
        Cache. In general, it is a number identifying an LLC uniquely on the
        system.

Cores
=====
A core consists of 1 or more threads. It does not matter whether the threads
are SMT- or CMT-type threads.

AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
"core".

Core-related topology information in the kernel:

  - smp_num_siblings:

    The number of threads in a core. The number of threads in a package can be
    calculated by::

	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings


Threads
=======
A thread is a single scheduling unit. It's the equivalent to a logical Linux
CPU.

AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
uses "thread".

Thread-related topology information in the kernel:

  - topology_core_cpumask():

    The cpumask contains all online threads in the package to which a thread
    belongs.

    The number of online threads is also printed in /proc/cpuinfo "siblings."

  - topology_sibling_cpumask():

    The cpumask contains all online threads in the core to which a thread
    belongs.

  - topology_logical_package_id():

    The logical package ID to which a thread belongs.

  - topology_physical_package_id():

    The physical package ID to which a thread belongs.

  - topology_core_id();

    The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
    "core_id."


System topology examples
========================

.. note::
  The alternative Linux CPU enumeration depends on how the BIOS enumerates the
  threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
  That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
  the same whether threads are enabled or not. That's merely an implementation
  detail and has no practical impact.

1) Single Package, Single Core::

   [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0

2) Single Package, Dual Core

   a) One thread per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
		    -> [core 1] -> [thread 0] -> Linux CPU 1

   b) Two threads per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 1
		    -> [core 1] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 3

      Alternative enumeration::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 2
		    -> [core 1] -> [thread 0] -> Linux CPU 1
				-> [thread 1] -> Linux CPU 3

      AMD nomenclature for CMT systems::

	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
				     -> [Compute Unit Core 1] -> Linux CPU 1
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
				     -> [Compute Unit Core 1] -> Linux CPU 3

4) Dual Package, Dual Core

   a) One thread per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
		    -> [core 1] -> [thread 0] -> Linux CPU 1

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
		    -> [core 1] -> [thread 0] -> Linux CPU 3

   b) Two threads per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 1
		    -> [core 1] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 3

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
				-> [thread 1] -> Linux CPU 5
		    -> [core 1] -> [thread 0] -> Linux CPU 6
				-> [thread 1] -> Linux CPU 7

      Alternative enumeration::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 4
		    -> [core 1] -> [thread 0] -> Linux CPU 1
				-> [thread 1] -> Linux CPU 5

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 6
		    -> [core 1] -> [thread 0] -> Linux CPU 3
				-> [thread 1] -> Linux CPU 7

      AMD nomenclature for CMT systems::

	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
				     -> [Compute Unit Core 1] -> Linux CPU 1
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
				     -> [Compute Unit Core 1] -> Linux CPU 3

	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
				     -> [Compute Unit Core 1] -> Linux CPU 5
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
				     -> [Compute Unit Core 1] -> Linux CPU 7
Commit	Line	Data
848942cb CD	1	.. SPDX-License-Identifier: GPL-2.0
	2
	3	============
f7be8610 BP	4	x86 Topology
	5	============
	6
	7	This documents and clarifies the main aspects of x86 topology modelling and
	8	representation in the kernel. Update/change when doing changes to the
	9	respective code.
	10
	11	The architecture-agnostic topology definitions are in
	12	Documentation/cputopology.txt. This file holds x86-specific
	13	differences/specialities which must not necessarily apply to the generic
	14	definitions. Thus, the way to read up on Linux topology on x86 is to start
	15	with the generic one and look at this one in parallel for the x86 specifics.
	16
	17	Needless to say, code should use the generic functions - this file is only
	18	here to document the inner workings of x86 topology.
	19
	20	Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.
	21
	22	The main aim of the topology facilities is to present adequate interfaces to
	23	code which needs to know/query/use the structure of the running system wrt
	24	threads, cores, packages, etc.
	25
	26	The kernel does not care about the concept of physical sockets because a
	27	socket has no relevance to software. It's an electromechanical component. In
	28	the past a socket always contained a single package (see below), but with the
	29	advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
	30	there might be still references to sockets in the code, but they are of
	31	historical nature and should be cleaned up.
	32
	33	The topology of a system is described in the units of:
	34
	35	- packages
	36	- cores
	37	- threads
	38
848942cb CD	39	Package
	40	=======
	41	Packages contain a number of cores plus shared resources, e.g. DRAM
	42	controller, shared caches etc.
f7be8610	43
848942cb	44	AMD nomenclature for package is 'Node'.
f7be8610	45
848942cb	46	Package-related topology information in the kernel:
f7be8610 BP	47
	48	- cpuinfo_x86.x86_max_cores:
	49
	50	The number of cores in a package. This information is retrieved via CPUID.
	51
	52	- cpuinfo_x86.phys_proc_id:
	53
	54	The physical ID of the package. This information is retrieved via CPUID
	55	and deduced from the APIC IDs of the cores in the package.
	56
ef7c7727	57	- cpuinfo_x86.logical_proc_id:
f7be8610 BP	58
	59	The logical ID of the package. As we do not trust BIOSes to enumerate the
	60	packages in a consistent way, we introduced the concept of logical package
	61	ID so we can sanely calculate the number of maximum possible packages in
	62	the system and have the packages enumerated linearly.
	63
	64	- topology_max_packages():
	65
	66	The maximum possible number of packages in the system. Helpful for per
	67	package facilities to preallocate per package information.
	68
a268b5f1 BP	69	- cpu_llc_id:
	70
	71	A per-CPU variable containing:
a268b5f1	72
848942cb CD	73	- On Intel, the first APIC ID of the list of CPUs sharing the Last Level
848942cb CD	74	Cache
f7be8610	75
848942cb CD	76	- On AMD, the Node ID or Core Complex ID containing the Last Level
	77	Cache. In general, it is a number identifying an LLC uniquely on the
	78	system.
f7be8610	79
848942cb CD	80	Cores
	81	=====
	82	A core consists of 1 or more threads. It does not matter whether the threads
	83	are SMT- or CMT-type threads.
f7be8610	84
848942cb CD	85	AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
848942cb CD	86	"core".
f7be8610	87
848942cb	88	Core-related topology information in the kernel:
f7be8610 BP	89
	90	- smp_num_siblings:
	91
	92	The number of threads in a core. The number of threads in a package can be
848942cb	93	calculated by::
f7be8610 BP	94
	95	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings
	96
	97
848942cb CD	98	Threads
	99	=======
	100	A thread is a single scheduling unit. It's the equivalent to a logical Linux
	101	CPU.
f7be8610	102
848942cb CD	103	AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
848942cb CD	104	uses "thread".
f7be8610	105
848942cb	106	Thread-related topology information in the kernel:
f7be8610 BP	107
	108	- topology_core_cpumask():
	109
	110	The cpumask contains all online threads in the package to which a thread
	111	belongs.
	112
	113	The number of online threads is also printed in /proc/cpuinfo "siblings."
	114
0c52f7c5	115	- topology_sibling_cpumask():
f7be8610 BP	116
	117	The cpumask contains all online threads in the core to which a thread
	118	belongs.
	119
848942cb	120	- topology_logical_package_id():
f7be8610 BP	121
	122	The logical package ID to which a thread belongs.
	123
848942cb	124	- topology_physical_package_id():
f7be8610 BP	125
	126	The physical package ID to which a thread belongs.
	127
848942cb	128	- topology_core_id();
f7be8610 BP	129
	130	The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
	131	"core_id."
	132
	133
	134
	135	System topology examples
848942cb	136	========================
f7be8610	137
848942cb CD	138	.. note::
	139	The alternative Linux CPU enumeration depends on how the BIOS enumerates the
	140	threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
	141	That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
	142	the same whether threads are enabled or not. That's merely an implementation
	143	detail and has no practical impact.
f7be8610	144
848942cb	145	1) Single Package, Single Core::
f7be8610 BP	146
	147	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	148
	149	2) Single Package, Dual Core
	150
848942cb	151	a) One thread per core::
f7be8610 BP	152
	153	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	154	-> [core 1] -> [thread 0] -> Linux CPU 1
	155
848942cb	156	b) Two threads per core::
f7be8610 BP	157
	158	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	159	-> [thread 1] -> Linux CPU 1
	160	-> [core 1] -> [thread 0] -> Linux CPU 2
	161	-> [thread 1] -> Linux CPU 3
	162
848942cb	163	Alternative enumeration::
f7be8610 BP	164
	165	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	166	-> [thread 1] -> Linux CPU 2
	167	-> [core 1] -> [thread 0] -> Linux CPU 1
	168	-> [thread 1] -> Linux CPU 3
	169
848942cb	170	AMD nomenclature for CMT systems::
f7be8610 BP	171
	172	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
	173	-> [Compute Unit Core 1] -> Linux CPU 1
	174	-> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
	175	-> [Compute Unit Core 1] -> Linux CPU 3
	176
	177	4) Dual Package, Dual Core
	178
848942cb	179	a) One thread per core::
f7be8610 BP	180
	181	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	182	-> [core 1] -> [thread 0] -> Linux CPU 1
	183
	184	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
	185	-> [core 1] -> [thread 0] -> Linux CPU 3
	186
848942cb	187	b) Two threads per core::
f7be8610 BP	188
	189	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	190	-> [thread 1] -> Linux CPU 1
	191	-> [core 1] -> [thread 0] -> Linux CPU 2
	192	-> [thread 1] -> Linux CPU 3
	193
	194	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
	195	-> [thread 1] -> Linux CPU 5
	196	-> [core 1] -> [thread 0] -> Linux CPU 6
	197	-> [thread 1] -> Linux CPU 7
	198
848942cb	199	Alternative enumeration::
f7be8610 BP	200
	201	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	202	-> [thread 1] -> Linux CPU 4
	203	-> [core 1] -> [thread 0] -> Linux CPU 1
	204	-> [thread 1] -> Linux CPU 5
	205
	206	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
	207	-> [thread 1] -> Linux CPU 6
	208	-> [core 1] -> [thread 0] -> Linux CPU 3
	209	-> [thread 1] -> Linux CPU 7
	210
848942cb	211	AMD nomenclature for CMT systems::
f7be8610 BP	212
	213	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
	214	-> [Compute Unit Core 1] -> Linux CPU 1
	215	-> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
	216	-> [Compute Unit Core 1] -> Linux CPU 3
	217
	218	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
	219	-> [Compute Unit Core 1] -> Linux CPU 5
	220	-> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
	221	-> [Compute Unit Core 1] -> Linux CPU 7