Testing and Testcases

The CNTT Reference Model specification contains some information and requirements where VSPERF (and other projects) need to respond and enhance their current capabilities to address the implied testing more fully.

For example, the current (9/11) reference model master .MD has:

Per VNF-C, per vNIC, and per vCPU have implications on VSPERF testing features, and may require new methods in TST 009

For example, the Test Setups in TST 009 begin to address the needed Per-V* testing, in Section 6.2,

where the Multiple PVP configuration can be used to assess performance of 1, 2, 4, 8, 12, 16, parallel VMs (or Pods), and a characteristic curve can be provided using the same resources and TST001 workload profile as the VNF of interest (if this specialization is needed). This setup assumes that the Benchmarking Goals and Use cases (Section 6.1 of TST009) require pairs of ports and bidirectional testing. Single port unidirectional testing is also possible with Multiple PVP.

The Test Device/Traffic Generator needs to generate flows designed to pass through the vSwitch and reach the VM, where the processing needed to return the flows takes place.

vNICs themselves can be monitored for octets transferred, and ultimately the limiting bottleneck of each multiple parallel VM setup should be determined.

Below are some considerations and questions, where "the present document" refers to TST009:

1. The number of VMs in the setup is a parameter for Throughput, Latency and Delay Variation.
2. The number of VMs in series on each parallel path through the DUT (PVVP means 2, see Figure 6.2-1 of the present document).
3. The number of simultaneous L2/L3/L4 flows is a parameter of Throughput, Latency, and Delay Variation benchmark testing. This implies a fourth dimension. Good to clarify this in the present doc.
4. All of the Test setups in clause 6.2 employ 2 physical NICs. Are there any single NIC or single vNIC VNFs envisioned (for “telco” deployment) that are critical/need to be tested immediately?
5. For the VMs themselves, should the testing employ reference VNFs/VNF-Cs with specific workload profiles, as described in ETSI NFV TST001, Clause 6? Table 6.2 has VNF to workload mapping, See table 6.3 below:

• ETSI NFV INF010 recommended several metrics for the Infrastructure “service quality” provided to the VNFs instantiated there. This was an early ETSI NFV Group Specification (December 2014), and it summarizes the service quality metrics envisioned at that time in Table 1. One of the key metrics is VM Stalls, with event Frequency and Duration, which would be characterized using the methods described in clause 12.4 (Long Duration Testing) in the present document, but there is no(?) current CNTT RM requirement that matches this metric. Both ETSI NFV GS INF010 and GS TST001 envision metrics to capture this area of infrastructure behaviour.
• Power Efficiency appears in CNTT RM Table 4-10. There was IETF Benchmarking Methodology Working Group work on this topic, but it did not advance beyond the proposal stage: https://tools.ietf.org/html/draft-manral-bmwg-power-usage-04
• what else?

Containers - CNF

Networking Performance Benchmarking

Intern Project: Container Networking Testing and Benchmarking 

VSPERF-Container Networking Benchmarking and Testing

Workload Characterization

Understanding the performance of containerized applications under various scenarios is critical for cloud native migration. One option is a black-box approach for profiling various (compute/network intensive) containerized applications under different CPU and Memory constraints. Profiling, refers to study how sensitive the container is to the shared resource availability. The approach is black-box because it does not rely on any specific adaptions or any agents within containers. The main components of this approach are Resource Limiter, Metrics collection, Data Analytics, and Traffic/Load-Gens. For every combinations of CPU and Memory, which are varied in pre-defined steps (10%, 25%, 50%, 75%, 90%), the sensitivity of the application container to this each combination is calculated. A model will be built to categorize the sensitivity of the application.