[linux-2.6-block.git] / Documentation / x86 / topology.rst

.. SPDX-License-Identifier: GPL-2.0

============
x86 Topology
============

This documents and clarifies the main aspects of x86 topology modelling and
representation in the kernel. Update/change when doing changes to the
respective code.

The architecture-agnostic topology definitions are in
Documentation/admin-guide/cputopology.rst. This file holds x86-specific
differences/specialities which must not necessarily apply to the generic
definitions. Thus, the way to read up on Linux topology on x86 is to start
with the generic one and look at this one in parallel for the x86 specifics.

Needless to say, code should use the generic functions - this file is *only*
here to *document* the inner workings of x86 topology.

Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.

The main aim of the topology facilities is to present adequate interfaces to
code which needs to know/query/use the structure of the running system wrt
threads, cores, packages, etc.

The kernel does not care about the concept of physical sockets because a
socket has no relevance to software. It's an electromechanical component. In
the past a socket always contained a single package (see below), but with the
advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
there might be still references to sockets in the code, but they are of
historical nature and should be cleaned up.

The topology of a system is described in the units of:

    - packages
    - cores
    - threads

Package
=======
Packages contain a number of cores plus shared resources, e.g. DRAM
controller, shared caches etc.

AMD nomenclature for package is 'Node'.

Package-related topology information in the kernel:

  - cpuinfo_x86.x86_max_cores:

    The number of cores in a package. This information is retrieved via CPUID.

  - cpuinfo_x86.x86_max_dies:

    The number of dies in a package. This information is retrieved via CPUID.

  - cpuinfo_x86.phys_proc_id:

    The physical ID of the package. This information is retrieved via CPUID
    and deduced from the APIC IDs of the cores in the package.

  - cpuinfo_x86.logical_proc_id:

    The logical ID of the package. As we do not trust BIOSes to enumerate the
    packages in a consistent way, we introduced the concept of logical package
    ID so we can sanely calculate the number of maximum possible packages in
    the system and have the packages enumerated linearly.

  - topology_max_packages():

    The maximum possible number of packages in the system. Helpful for per
    package facilities to preallocate per package information.

  - cpu_llc_id:

    A per-CPU variable containing:

      - On Intel, the first APIC ID of the list of CPUs sharing the Last Level
        Cache

      - On AMD, the Node ID or Core Complex ID containing the Last Level
        Cache. In general, it is a number identifying an LLC uniquely on the
        system.

Cores
=====
A core consists of 1 or more threads. It does not matter whether the threads
are SMT- or CMT-type threads.

AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
"core".

Core-related topology information in the kernel:

  - smp_num_siblings:

    The number of threads in a core. The number of threads in a package can be
    calculated by::

	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings


Threads
=======
A thread is a single scheduling unit. It's the equivalent to a logical Linux
CPU.

AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
uses "thread".

Thread-related topology information in the kernel:

  - topology_core_cpumask():

    The cpumask contains all online threads in the package to which a thread
    belongs.

    The number of online threads is also printed in /proc/cpuinfo "siblings."

  - topology_sibling_cpumask():

    The cpumask contains all online threads in the core to which a thread
    belongs.

  - topology_logical_package_id():

    The logical package ID to which a thread belongs.

  - topology_physical_package_id():

    The physical package ID to which a thread belongs.

  - topology_core_id();

    The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
    "core_id."


System topology examples
========================

.. note::
  The alternative Linux CPU enumeration depends on how the BIOS enumerates the
  threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
  That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
  the same whether threads are enabled or not. That's merely an implementation
  detail and has no practical impact.

1) Single Package, Single Core::

   [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0

2) Single Package, Dual Core

   a) One thread per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
		    -> [core 1] -> [thread 0] -> Linux CPU 1

   b) Two threads per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 1
		    -> [core 1] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 3

      Alternative enumeration::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 2
		    -> [core 1] -> [thread 0] -> Linux CPU 1
				-> [thread 1] -> Linux CPU 3

      AMD nomenclature for CMT systems::

	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
				     -> [Compute Unit Core 1] -> Linux CPU 1
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
				     -> [Compute Unit Core 1] -> Linux CPU 3

4) Dual Package, Dual Core

   a) One thread per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
		    -> [core 1] -> [thread 0] -> Linux CPU 1

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
		    -> [core 1] -> [thread 0] -> Linux CPU 3

   b) Two threads per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 1
		    -> [core 1] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 3

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
				-> [thread 1] -> Linux CPU 5
		    -> [core 1] -> [thread 0] -> Linux CPU 6
				-> [thread 1] -> Linux CPU 7

      Alternative enumeration::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 4
		    -> [core 1] -> [thread 0] -> Linux CPU 1
				-> [thread 1] -> Linux CPU 5

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 6
		    -> [core 1] -> [thread 0] -> Linux CPU 3
				-> [thread 1] -> Linux CPU 7

      AMD nomenclature for CMT systems::

	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
				     -> [Compute Unit Core 1] -> Linux CPU 1
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
				     -> [Compute Unit Core 1] -> Linux CPU 3

	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
				     -> [Compute Unit Core 1] -> Linux CPU 5
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
				     -> [Compute Unit Core 1] -> Linux CPU 7
Commit	Line	Data
848942cb CD	1	.. SPDX-License-Identifier: GPL-2.0
	2
	3	============
f7be8610 BP	4	x86 Topology
	5	============
	6
	7	This documents and clarifies the main aspects of x86 topology modelling and
	8	representation in the kernel. Update/change when doing changes to the
	9	respective code.
	10
	11	The architecture-agnostic topology definitions are in
4f4cfa6c	12	Documentation/admin-guide/cputopology.rst. This file holds x86-specific
f7be8610 BP	13	differences/specialities which must not necessarily apply to the generic
	14	definitions. Thus, the way to read up on Linux topology on x86 is to start
	15	with the generic one and look at this one in parallel for the x86 specifics.
	16
	17	Needless to say, code should use the generic functions - this file is only
	18	here to document the inner workings of x86 topology.
	19
	20	Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.
	21
	22	The main aim of the topology facilities is to present adequate interfaces to
	23	code which needs to know/query/use the structure of the running system wrt
	24	threads, cores, packages, etc.
	25
	26	The kernel does not care about the concept of physical sockets because a
	27	socket has no relevance to software. It's an electromechanical component. In
	28	the past a socket always contained a single package (see below), but with the
	29	advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
	30	there might be still references to sockets in the code, but they are of
	31	historical nature and should be cleaned up.
	32
	33	The topology of a system is described in the units of:
	34
	35	- packages
	36	- cores
	37	- threads
	38
848942cb CD	39	Package
	40	=======
	41	Packages contain a number of cores plus shared resources, e.g. DRAM
	42	controller, shared caches etc.
f7be8610	43
848942cb	44	AMD nomenclature for package is 'Node'.
f7be8610	45
848942cb	46	Package-related topology information in the kernel:
f7be8610 BP	47
	48	- cpuinfo_x86.x86_max_cores:
	49
	50	The number of cores in a package. This information is retrieved via CPUID.
	51
7745f03e LB	52	- cpuinfo_x86.x86_max_dies:
	53
	54	The number of dies in a package. This information is retrieved via CPUID.
	55
f7be8610 BP	56	- cpuinfo_x86.phys_proc_id:
	57
	58	The physical ID of the package. This information is retrieved via CPUID
	59	and deduced from the APIC IDs of the cores in the package.
	60
ef7c7727	61	- cpuinfo_x86.logical_proc_id:
f7be8610 BP	62
	63	The logical ID of the package. As we do not trust BIOSes to enumerate the
	64	packages in a consistent way, we introduced the concept of logical package
	65	ID so we can sanely calculate the number of maximum possible packages in
	66	the system and have the packages enumerated linearly.
	67
	68	- topology_max_packages():
	69
	70	The maximum possible number of packages in the system. Helpful for per
	71	package facilities to preallocate per package information.
	72
a268b5f1 BP	73	- cpu_llc_id:
	74
	75	A per-CPU variable containing:
a268b5f1	76
848942cb CD	77	- On Intel, the first APIC ID of the list of CPUs sharing the Last Level
848942cb CD	78	Cache
f7be8610	79
848942cb CD	80	- On AMD, the Node ID or Core Complex ID containing the Last Level
	81	Cache. In general, it is a number identifying an LLC uniquely on the
	82	system.
f7be8610	83
848942cb CD	84	Cores
	85	=====
	86	A core consists of 1 or more threads. It does not matter whether the threads
	87	are SMT- or CMT-type threads.
f7be8610	88
848942cb CD	89	AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
848942cb CD	90	"core".
f7be8610	91
848942cb	92	Core-related topology information in the kernel:
f7be8610 BP	93
	94	- smp_num_siblings:
	95
	96	The number of threads in a core. The number of threads in a package can be
848942cb	97	calculated by::
f7be8610 BP	98
	99	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings
	100
	101
848942cb CD	102	Threads
	103	=======
	104	A thread is a single scheduling unit. It's the equivalent to a logical Linux
	105	CPU.
f7be8610	106
848942cb CD	107	AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
848942cb CD	108	uses "thread".
f7be8610	109
848942cb	110	Thread-related topology information in the kernel:
f7be8610 BP	111
	112	- topology_core_cpumask():
	113
	114	The cpumask contains all online threads in the package to which a thread
	115	belongs.
	116
	117	The number of online threads is also printed in /proc/cpuinfo "siblings."
	118
0c52f7c5	119	- topology_sibling_cpumask():
f7be8610 BP	120
	121	The cpumask contains all online threads in the core to which a thread
	122	belongs.
	123
848942cb	124	- topology_logical_package_id():
f7be8610 BP	125
	126	The logical package ID to which a thread belongs.
	127
848942cb	128	- topology_physical_package_id():
f7be8610 BP	129
	130	The physical package ID to which a thread belongs.
	131
848942cb	132	- topology_core_id();
f7be8610 BP	133
	134	The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
	135	"core_id."
	136
	137
	138
	139	System topology examples
848942cb	140	========================
f7be8610	141
848942cb CD	142	.. note::
	143	The alternative Linux CPU enumeration depends on how the BIOS enumerates the
	144	threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
	145	That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
	146	the same whether threads are enabled or not. That's merely an implementation
	147	detail and has no practical impact.
f7be8610	148
848942cb	149	1) Single Package, Single Core::
f7be8610 BP	150
	151	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	152
	153	2) Single Package, Dual Core
	154
848942cb	155	a) One thread per core::
f7be8610 BP	156
	157	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	158	-> [core 1] -> [thread 0] -> Linux CPU 1
	159
848942cb	160	b) Two threads per core::
f7be8610 BP	161
	162	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	163	-> [thread 1] -> Linux CPU 1
	164	-> [core 1] -> [thread 0] -> Linux CPU 2
	165	-> [thread 1] -> Linux CPU 3
	166
848942cb	167	Alternative enumeration::
f7be8610 BP	168
	169	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	170	-> [thread 1] -> Linux CPU 2
	171	-> [core 1] -> [thread 0] -> Linux CPU 1
	172	-> [thread 1] -> Linux CPU 3
	173
848942cb	174	AMD nomenclature for CMT systems::
f7be8610 BP	175
	176	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
	177	-> [Compute Unit Core 1] -> Linux CPU 1
	178	-> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
	179	-> [Compute Unit Core 1] -> Linux CPU 3
	180
	181	4) Dual Package, Dual Core
	182
848942cb	183	a) One thread per core::
f7be8610 BP	184
	185	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	186	-> [core 1] -> [thread 0] -> Linux CPU 1
	187
	188	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
	189	-> [core 1] -> [thread 0] -> Linux CPU 3
	190
848942cb	191	b) Two threads per core::
f7be8610 BP	192
	193	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	194	-> [thread 1] -> Linux CPU 1
	195	-> [core 1] -> [thread 0] -> Linux CPU 2
	196	-> [thread 1] -> Linux CPU 3
	197
	198	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
	199	-> [thread 1] -> Linux CPU 5
	200	-> [core 1] -> [thread 0] -> Linux CPU 6
	201	-> [thread 1] -> Linux CPU 7
	202
848942cb	203	Alternative enumeration::
f7be8610 BP	204
	205	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
	206	-> [thread 1] -> Linux CPU 4
	207	-> [core 1] -> [thread 0] -> Linux CPU 1
	208	-> [thread 1] -> Linux CPU 5
	209
	210	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
	211	-> [thread 1] -> Linux CPU 6
	212	-> [core 1] -> [thread 0] -> Linux CPU 3
	213	-> [thread 1] -> Linux CPU 7
	214
848942cb	215	AMD nomenclature for CMT systems::
f7be8610 BP	216
	217	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
	218	-> [Compute Unit Core 1] -> Linux CPU 1
	219	-> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
	220	-> [Compute Unit Core 1] -> Linux CPU 3
	221
	222	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
	223	-> [Compute Unit Core 1] -> Linux CPU 5
	224	-> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
	225	-> [Compute Unit Core 1] -> Linux CPU 7