106 lines
4.6 KiB
ReStructuredText
106 lines
4.6 KiB
ReStructuredText
|
============================
|
|||
|
|
NUMA resource associativity
|
|||
|
|
============================
|
|||
|
|
|
|||
|
|
Associativity represents the groupings of the various platform resources into
|
|||
|
|
domains of substantially similar mean performance relative to resources outside
|
|||
|
|
of that domain. Resources subsets of a given domain that exhibit better
|
|||
|
|
performance relative to each other than relative to other resources subsets
|
|||
|
|
are represented as being members of a sub-grouping domain. This performance
|
|||
|
|
characteristic is presented in terms of NUMA node distance within the Linux kernel.
|
|||
|
|
From the platform view, these groups are also referred to as domains.
|
|||
|
|
|
|||
|
|
PAPR interface currently supports different ways of communicating these resource
|
|||
|
|
grouping details to the OS. These are referred to as Form 0, Form 1 and Form2
|
|||
|
|
associativity grouping. Form 0 is the oldest format and is now considered deprecated.
|
|||
|
|
|
|||
|
|
Hypervisor indicates the type/form of associativity used via "ibm,architecture-vec-5 property".
|
|||
|
|
Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of Form 0 or Form 1.
|
|||
|
|
A value of 1 indicates the usage of Form 1 associativity. For Form 2 associativity
|
|||
|
|
bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
|
|||
|
|
|
|||
|
|
Form 0
|
|||
|
|
------
|
|||
|
|
Form 0 associativity supports only two NUMA distances (LOCAL and REMOTE).
|
|||
|
|
|
|||
|
|
Form 1
|
|||
|
|
------
|
|||
|
|
With Form 1 a combination of ibm,associativity-reference-points, and ibm,associativity
|
|||
|
|
device tree properties are used to determine the NUMA distance between resource groups/domains.
|
|||
|
|
|
|||
|
|
The “ibm,associativity” property contains a list of one or more numbers (domainID)
|
|||
|
|
representing the resource’s platform grouping domains.
|
|||
|
|
|
|||
|
|
The “ibm,associativity-reference-points” property contains a list of one or more numbers
|
|||
|
|
(domainID index) that represents the 1 based ordinal in the associativity lists.
|
|||
|
|
The list of domainID indexes represents an increasing hierarchy of resource grouping.
|
|||
|
|
|
|||
|
|
ex:
|
|||
|
|
{ primary domainID index, secondary domainID index, tertiary domainID index.. }
|
|||
|
|
|
|||
|
|
Linux kernel uses the domainID at the primary domainID index as the NUMA node id.
|
|||
|
|
Linux kernel computes NUMA distance between two domains by recursively comparing
|
|||
|
|
if they belong to the same higher-level domains. For mismatch at every higher
|
|||
|
|
level of the resource group, the kernel doubles the NUMA distance between the
|
|||
|
|
comparing domains.
|
|||
|
|
|
|||
|
|
Form 2
|
|||
|
|
-------
|
|||
|
|
Form 2 associativity format adds separate device tree properties representing NUMA node distance
|
|||
|
|
thereby making the node distance computation flexible. Form 2 also allows flexible primary
|
|||
|
|
domain numbering. With numa distance computation now detached from the index value in
|
|||
|
|
"ibm,associativity-reference-points" property, Form 2 allows a large number of primary domain
|
|||
|
|
ids at the same domainID index representing resource groups of different performance/latency
|
|||
|
|
characteristics.
|
|||
|
|
|
|||
|
|
Hypervisor indicates the usage of FORM2 associativity using bit 2 of byte 5 in the
|
|||
|
|
"ibm,architecture-vec-5" property.
|
|||
|
|
|
|||
|
|
"ibm,numa-lookup-index-table" property contains a list of one or more numbers representing
|
|||
|
|
the domainIDs present in the system. The offset of the domainID in this property is
|
|||
|
|
used as an index while computing numa distance information via "ibm,numa-distance-table".
|
|||
|
|
|
|||
|
|
prop-encoded-array: The number N of the domainIDs encoded as with encode-int, followed by
|
|||
|
|
N domainID encoded as with encode-int
|
|||
|
|
|
|||
|
|
For ex:
|
|||
|
|
"ibm,numa-lookup-index-table" = {4, 0, 8, 250, 252}. The offset of domainID 8 (2) is used when
|
|||
|
|
computing the distance of domain 8 from other domains present in the system. For the rest of
|
|||
|
|
this document, this offset will be referred to as domain distance offset.
|
|||
|
|
|
|||
|
|
"ibm,numa-distance-table" property contains a list of one or more numbers representing the NUMA
|
|||
|
|
distance between resource groups/domains present in the system.
|
|||
|
|
|
|||
|
|
prop-encoded-array: The number N of the distance values encoded as with encode-int, followed by
|
|||
|
|
N distance values encoded as with encode-bytes. The max distance value we could encode is 255.
|
|||
|
|
The number N must be equal to the square of m where m is the number of domainIDs in the
|
|||
|
|
numa-lookup-index-table.
|
|||
|
|
|
|||
|
|
For ex:
|
|||
|
|
ibm,numa-lookup-index-table = <3 0 8 40>;
|
|||
|
|
ibm,numa-distace-table = <9>, /bits/ 8 < 10 20 80 20 10 160 80 160 10>;
|
|||
|
|
|
|||
|
|
::
|
|||
|
|
|
|||
|
|
| 0 8 40
|
|||
|
|
--|------------
|
|||
|
|
|
|
|||
|
|
0 | 10 20 80
|
|||
|
|
|
|
|||
|
|
8 | 20 10 160
|
|||
|
|
|
|
|||
|
|
40| 80 160 10
|
|||
|
|
|
|||
|
|
A possible "ibm,associativity" property for resources in node 0, 8 and 40
|
|||
|
|
|
|||
|
|
{ 3, 6, 7, 0 }
|
|||
|
|
{ 3, 6, 9, 8 }
|
|||
|
|
{ 3, 6, 7, 40}
|
|||
|
|
|
|||
|
|
With "ibm,associativity-reference-points" { 0x3 }
|
|||
|
|
|
|||
|
|
"ibm,lookup-index-table" helps in having a compact representation of distance matrix.
|
|||
|
|
Since domainID can be sparse, the matrix of distances can also be effectively sparse.
|
|||
|
|
With "ibm,lookup-index-table" we can achieve a compact representation of
|
|||
|
|
distance information.
|