Collectd Metrics and Events

CPU

(A read plugin that retrieves CPU usage in Nanoseconds of as a percentage)

Interface

(A read plugin that retrieves Linux Interface statistics)

Memory

(A read plugin that retrieves memory usage statistics)

disk

(A read plugin that retrieves disk usage statistics)

Ping

(A read plugin that retrieves the RTT for a ping)

if pkg_recv > 1

latency_stddev = sqrt (((((double) pkg_recv) * latency_squared) - (latency_total * latency_total)) / ((double) (pkg_recv * (pkg_recv - 1))));

load

(A read plugin that retrieves system load for 1, 5 and 15 mins.)

OVS events

(A read plugin that retrieves events (like link status changes) from OVS.)

OVS Stats

(A read plugin that retrieves interface stats from OVS.)

The total number of packets received that were less than 64 octets in length (excluding framing bits but including FCS octets) and had either a bad Frame Check Sequence (FCS) with an integral number of octets (FCS Error) or a bad FCS with a non-integral number of octets (Alignment Error).

Hugepages

(A read plugin that retrieves the number of available and free hugepages on a platform as well as what is available in terms of hugepages per socket.)

Virtual memory makes it easy for several processes to share memory [TM1] .

Each process has its own virtual address space, which is mapped to physical memory by the operating system” [TM2] . The process views the virtual memory address space as a contiguous/linear address space, but in reality – the virtual addresses need to be mapped to physical addresses, this is typically done by the Memory Management Unit (MMU) on the CPU.

“There are two ways to enable the system to manage large amounts of memory:

• Increase the number of page table entries in the hardware memory management unit
• Increase the page size/use huge pages à to reduce the number of lookups

The first method is expensive, since the hardware memory management unit in a modern processor only supports hundreds or thousands of page table entries. Additionally, hardware and memory management algorithms that work well with thousands of pages (megabytes of memory) may have difficulty performing well with millions (or even billions) of pages. This results in performance issues: when an application needs to use more memory pages than the memory management unit supports, the system falls back to slower, software-based memory management, which causes the entire system to run more slowly.

Huge pages are blocks of memory that come in 2MB and 1GB sizes. The page tables used by the 2MB pages are suitable for managing multiple gigabytes of memory, whereas the page tables of 1GB pages are best for scaling to terabytes of memory” [TM3] 

More info on virtual memory and TLB lookups can be found:

1. http://www.tldp.org/LDP/tlk/mm/memory.html
2. https://lwn.net/Articles/253361/

processes

(A read plugin that  collects the number of processes, grouped by their state (e. g. running, sleeping, zombies, etc.). In addition to that, it can select detailed statistics about selected processes, grouped by name.)

virt

(A read plugin that uses virtualization API libvirt to gather statistics about virtualized guests on a system directly from the hypervisor, without a need to install collectd instance on the guest.)

Amount of memory which can be reclaimed by balloon without causing host swapping

RDT

(A read plugin that provides the last level cache utilization and memory bandwidth utilization)

NOTE: it's recommended to configure the interval for this plugin as 1 second

A higher IPC means that the processor can get more work done per unit time, which generally translates to faster application performance [0]

"Ideally every instruction a CPU gets should be read, executed and finished in one cycle, however that is never the case. The processor has to take the instruction, decode the instruction, gather the data (depends on where the data is), perform work on the data, then decide what to do with the result. Moving has never been more complicated, and the ability for a processor to hide latency, pre-prepare data by predicting future events or keeping hold of previous events for potential future use is all part of the plan. All the meanwhile there is an external focus on making sure power consumption is low and the frequency of the processor can scale depending on what the target device actually is". [1]

0 https://www.nextplatform.com/2016/03/31/examining-potential-hpc-benefits-new-intel-xeon-processors/

1 http://www.anandtech.com/show/9482/intel-broadwell-pt2-overclocking-ipc/3

Local Memory Bandwidth in Bytes per second, for a specified processor (socket), Resource Monitoring ID (RMID), and a given measurement time over the preceding measurement interval.

RMID may be mapped to Threads, Application processes, or VMs by the host OS or hypervisor.

Local refers to Memory Bandwidth used by RMID within the specified processor (socket).

https://software.intel.com/en-us/articles/memory-bandwidth-monitoring-proof-points

Remote Memory Bandwidth in Bytes per second, for a specified processor (socket), Resource Monitoring ID (RMID), and a given measurement time over the preceding measurement interval.

RMID may be mapped to Threads, Application processes, or VMs by the host OS or hypervisor.

RMID and Class of Service (CLOS) limits may be mapped to Threads, Application processes, or VMs by the host OS or hypervisor.

Question if units are in bytes or kibibits.

Al Morton after looking into this, I found that kilobytes is what is used for the cache.
In collectd we are reporting it simply in bytes. I updated the 4th column to reflect that as well as the definition.

dpdkstats

(A read plugin that retrieve stats from the DPDK extended

NIC stats API.)

mcelog (RAS memory)

A read plugin that uses mcelog to check for memory Machine Check Exceptions and sends the stats for reported exceptions

IPMI

A read plugin that reports platform thermals, voltages, fan speed, current, flow, power etc.

(specific per BMC) so these will change depending on what's supported by the BMC.
This is en example for S2600WT2R platform

 IPMI defines many types of sensors, but groups them into two
main categories: Threshold and discrete.

Threshold sensors are “analog”, they have continuous (or mostly continuous) readings. Things like fans speed, voltage, or temperature.

Discrete sensors have a set of binary readings that may each be independently zero or one. In some sensors, these may be independent. For instance, a power supply may have both an external power failure and a predictive failure at the same time. In other cases they may be mutually exclusive. For instance, each
bit may represent the initialization state of a piece of software.

"Memory Thermal Throttling" is related to memory thermal management system. Based on the DIMM thermal conditions it may restrict read and write traffic/bandwidth to main memory as a means of controlling power consumption. This metric is measured as a percentage and 0% means no memory throttling occurs. When thermal conditions are going high, the memory management system enables throttling and restricts the read or write traffic (e.g. 50%).

intel_pmu

A read plugin that collects performance monitoring events supported by Intel Performance Monitoring Units (PMUs). The PMU is hardware built inside a processor to measure its performance parameters such as instruction cycles, cache hits, cache misses, branch misses and many others. Performance monitoring events provide facilities to characterize the interaction between programmed sequences of instructions and microarchitectural sub-systems.

main thread

emc hits

Number of packets hitting Exact Match Cache.

Plugin instance: pmd thread can be different combinations of <numa_id> and <core_id>.

e.g: ovs_pmd_stats-pmd_thread_numa_id_0_core_id_5

megaflow hits

Average number of subtable lookups for every hit.

Number of packets not matching any existing flow.

Number of packets destined for user space process but subsequently dropped before reaching userspace.

polling cycles

Number of cycles used for polling packets.

processing cycles

Number of cycles used for processing incoming packets.

Average number of cycles per packet.

avg processing cycles per packet

The realtime speed of the CPU reported from:

/sys/devices/system/cpu/cpu<id>/cpufreq/scaling_cur_freq

Host

CLOCK_MONOTONIC

Clock that cannot be set and represents monotonic time since some unspecified starting point. This clock is not affected by discontinuous jumps in the system time (e.g., if the system administrator manually changes the clock), but is affected by the incremental adjustments performed by adjtime(3) and NTP.

CLOCK_BOOTTIME (since Linux 2.6.39; Linux-specific)Identical to CLOCK_MONOTONIC, except it also includes any time that the system is suspended. This allows applications to get a suspend-aware monotonic clock without having to deal with the complications of CLOCK_REALTIME, which may have discontinuities if the time is changed usingsettimeofday(2).

Reference: https://linux.die.net/man/2/clock_gettime

dpdk_events

A read plugin that retrieves events (like link status changes) from OVS.

pcie

A read plugin that monitors PCIe standard and advanced errors and sends notifications about those errors

Correctable Errors include:
Receiver Error Status
Bad TLP Status
Bad DLLP Status
REPLAY_NUM Rollover
Replay Timer Timeout
Advisory Non-Fatal
Corrected Internal
Header Log Overflow

Uncorrectable Errors include:
Data Link Protocol
Surprise Down
Poisoned TLP
Flow Control Protocol
Completion Timeout
Completer Abort
Unexpected Completion
Receiver Overflow
Malformed TLP
ECRC Error Status
Unsupported Request
ACS Violation
Internal
MC blocked TLP
Atomic egress blocked
TLP prefix blocked

mcelog (RAS memory)

A read plugin that uses mcelog to check for memory Machine Check Exceptions and sends notifications for reported exceptions

Warning for Corrected Memory Errors

Failure for Uncorrected Memory Errors

Failure on failure to connect to the mcelog socket/ if connection is lost

OK on connection to mcelog socket

IPMI

A read plugin that (IPMI) System Event Log (SEL) and sends appropriate notifications based on monitored SEL events and threshold events.

mcelog RAS System, CPU, QPI, OI

(specific to a Platform) so these will change depending on what's supported by the Platform.

A read plugin that uses mcelog to check for cpu, IO, QPI or system Machine Check Exceptions and sends the stats for reported exceptions

virt

A read plugin that uses virtualization API libvirt to gather events about virtualized guests on a system directly from the hypervisor, without a need to install collectd instance on the guest.

OKAY:

• VIR_DOMAIN_NOSTATE
• VIR_DOMAIN_RUNNING
• VIR_DOMAIN_SHUTDOWN
• VIR_DOMAIN_SHUTOFF

WARNING:

• VIR_DOMAIN_BLOCKED
• VIR_DOMAIN_PAUSED
• VIR_DOMAIN_PMSUSPENDED

FAILURE:

• VIR_DOMAIN_CRASHED