PMU – Equivalence

Requirement

Overview

Performance counters are CPU hardware registers that count hardware events such as instructions executed, cache-misses suffered, or branches mispredicted. They form a basis for profiling applications to trace dynamic control flow and identify hotspots. Linux perf interface provides rich generalized abstractions over hardware specific capabilities. 

PMU Tools

PMU tools is a collection of tools for profiling and performance analysis on Intel CPUs on top of Linux perf. This uses performance counters in the CPU.  These tools are developed and maintained on https://github.com/andikleen/pmu-tools. In addition to a number of tools for profiling and performance analysis this package provides jevents library.

jevents library

jevents is a C library to use from C programs to make access to the kernel Linux perf interface easier. It also includes some examples to use the library. This library provides the following features:

• Resolving symbolic event names using downloaded event files
• Reading performance counters from ring 3 in C programs,
• Handling the perf ring buffer (for example to read memory addresses)

For more information on jevents see https://github.com/andikleen/pmu-tools/tree/master/jevents.

Design

intel_pmu plugin

The intel_pmu plugin collects information provided by Linux perf interface. It is not done directly, but through jevents API. Using this interface, the intel_pmu plugin collects the hardware specific metrics defined in event list file which should contain definitions of PMU events. The list of events to monitor is configurable.

Plugin configuration

The following configuration options should be supported by intel_pmu collectd plugin:  

Here is an example of the plugin configuration section of collectd.conf file:

  <Plugin intel_pmu>

    EventList "/var/cache/pmu/GenuineIntel-6-55-core.json"

    HardwareEvents "L2_RQSTS.CODE_RD_HIT,L2_RQSTS.CODE_RD_MISS" "L2_RQSTS.ALL_CODE_RD"

    Cores ""
    HardwareEvents "L2_RQSTS.PF_MISS"
    Cores "1"
    DispatchMultiPmu false
  </Plugin>

In above example events L2_RQSTS.CODE_RD_HIT,L2_RQSTS.CODE_RD_MISS and L2_RQSTS.ALL_CODE_RD are going to be monitored on
all available cores, and event L2_RQSTS.PF_MISS is going to be monitored on core 1.

Another example with only uncore events set:

  <Plugin intel_pmu>

    EventList "/var/cache/pmu/GenuineIntel-6-55-uncore.json"

    HardwareEvents "UNC_CHA_TOR_INSERTS.IA_MISS:config1=0x4043200000000" "UNC_IIO_TXN_REQ_BY_CPU.MEM_WRITE.PART0"

    Cores "0" "18"
    DispatchMultiPmu false

  </Plugin>

Implementation details

 intel_pmu plugin does not introduce its own layer of functionality. It just reads configuration provided by user and prepares all needed parameters/data structures for jevents API. This table shows the correspondence between plugin’s API and jevents API that is used to configure Linux perf monitoring.

plugin API

jevents API

Description

pmu_config

Parse events groups to monitor provided by user in collectd.conf

pmu_init

alloc_eventlist

Allocate memory for new eventlist

resolve_event_extra

Resolve hardware specific events names to perf events (perf_event_attr)

jevent_pmu_uncore

Check if event is uncore event

jevent_next_pmu

Expand event into multiple PMU if neccessary (in use for uncore events)

setup_event

Setup perf events for monitoring

pmu_read

read_all_events

Read values of all monitored events

pmu_shutdown

free_eventlist

Free memory allocated for eventlist (recursively including all events) 

For more details on plugin API see collectd plugin implementation guide https://collectd.org/wiki/index.php/Plugin_architecture.

Hardware Specific Events

The intel_pmu plugin allows to monitor hardware specific events. To support this functionality plugin will use feature provided by jevents library – resolving symbolic event names using downloaded event files. To be able to use hardware specific event names in configuration file user will have to download events list file for current CPU before using intel_pmu plugin. This can be done using event_download.py script which is part of pmu-tools package.

Note: For uncore events values can be collected only for first core of every socket e.g. '0' '18' etc.

Time based multiplexing

If there are more events than counters, the kernel uses time multiplexing to give each event a chance to access the monitoring hardware. With multiplexing, an event is not measured all the time. At the end of the run, the tool scales the count based on total time enabled vs time running. The actual formula is:
scaled_count = raw_count * time_enabled / time_running

This provides an estimate of what the count would have been, had the event been measured during the entire run. Note that this is an estimate not an actual count. Depending on the workload, there will be blind spots which can introduce errors during scaling.
The plugin dispatches all the four values, that is scaled, raw, time enabled & running, to the user. The values type is COUNTER.

SNMP Support

All metrics collected by intel_pmu plugin should be available through SNMP.  This will be achieved by creating proper configuration for snmp_agent collectd plugin. No additional functionality needed in intel_pmu plugin to support SNMP. See description of SNMP feature for more details on snmp_agent plugin.

Considerations

Configuration Considerations

When using intel_pmu plugin number of reading threads in collectd should be increased. The value should be more than a half of configured cores, so for
60 monitored cores the recommendation is to set ReadThreads > 30 (e.g. 35).

Deployment Considerations

By leveraging the core configuration for the PMU plugin, it’s necessary to taskset and isolate cores for specific applications that you would like to monitor until the process support is implemented.

API/GUI/CLI Considerations

Equivalence Considerations

The SNMP MIB used for this plugin is a newly Defined MIB.

Security Considerations

Alarms, events, statistics considerations

Certain platform generations will not support all the metrics intended to be collected by the plugin. Unsupported metrics will not be reported.

Redundancy Considerations

Performance Considerations

Not part of Telemetry so performance is Not Applicable

Testing Consideration

The timing interval requirement needs to be taken into consideration when conducting tests.

The Tests should be carried out on a system under load as well as a relatively idle system.

Other Considerations

Impact

The following table outlines possible impact(s) the deployment of this deliverable may have on the current system.

Key Assumptions

The following assumptions apply to the scope specified in this document.

Key Exclusions

The following exclusions apply to the scope discussed in this document.

Key Dependencies

The following table outlines the key dependencies associated with this deliverable.