Demo:

The Jupyter-Notebook File: testresult-analysis.ipynb

Summary:

1. ~27 Visitors to the booth. ~75% were not aware of VSPERF
2. Attendees' affiliations:
   1. Intel, AT&T, Verizon, Ericsson, Iconnectiv, Nokia, Ixia/Keysight, BigSwitch, 6Wind, UNH, Lumina, Pantheon, AtsGen, Delhi Univ., DoD, Wipro, Subex, CISA.
3. Lot of appreciation from all the visitors. 
4. Inputs from 6Wind: 
   
   1. Consider the context-switches - /proc/pid/status/
   2. libvirt: CPU Stealing.
   3. Per-Queue statistics, multi-flow analysis.
   4. QPI Transaction information from Intel UPI/PMU
5. Inputs from Verizon:
   1. It is hard to obtain Interface metrics when SRIOV is used - Need to explore a solution.
6. Query: Why it is difficult to do what VSPERF does through Openstack?
7. Request: Define use-cases for VSPERF.
   1. Ex: Benchmarking: after the cloud is deployed and before the application is turned up.
8. Request: Consider Container networking.
9. Request: Make notebook public, notebook to consume other data-sources.
10. Offer: Help to explore better correlation and improve analytics with ML.
11. Request: Support to plugin commercial switches.
12. Request: Consider benchmarking smart-NICs
13. Request: Consider BESS integration with VSPERF.
14. Issue: Could not plot second by second packet losses and use them in analysis (T-rex does not provide).

A Key Initiative discussed at Sessions & Meetings:

The topic of specifying a small number of Reference NFVI platforms to support tests of VNF Certification, and the VNF Certification process itself, was covered in a presentation and panel discussion on Wednesday. 

Creating a Common Telco NFVi

This talk and panel announced the formation of a Common NFVI Telco Task Force with 10 Service Providers, and which will probably be chaired by Mark Cottrell, AT&T.

At the same time, the GSMA (another group of mobile network operators) shared the following Draft report with the Linux Foundation and ETSI NFV:

These slides/documents and the meetings that referenced them revealed a new role for the testing projects at OPNFV. For example, most views of different common platforms include performance characteristics which need to be measured objectively and compared with acceptance thresholds for the measurements. Below is an example from the GSMA draft:

Clearly, the VSPERF project has a role to play in the platform profile specification, selection, and qualification process. We currently measure Throughput and Latency benchmarks, as well as compute, memory and networking metrics from the platform itself (barometer & collectd).

Beyond the NFVI measurement and qualification, there is a role for OPNFV Dovetail project to follow-up with additional certification testing, and perhaps an ongoing role for VSPERF if multiple NFVI vendors submit their platforms for testing against the profile specifications.  Further, the profiles must be revisited periodically in order to keep up with hardware upgrades (every 3 years) and software upgrades (every 6 months).

Ultimately, VNFs will be certified as compliant with the platform, in a key group of On-boarding and Life-cycle operations (also to be expanded over time).