Category

A brief discription

Hardware

Chassis from Hardware test-equipment vendor.

Baremetal

BM-A, BM-B and BM-C are packet generators built on top of DPDK.

VM

Traffic generator as a Virtual-Machine.

Environment

Benchmarking Standard

VSPERF Test Code

Traffic Configuration

Traffic Generators

Results-Plot Reference

VSPERF with OVS as the vswitch

RFC2544, Throughput

phy2phy_tput

Bi-Directional

Multistream=False

All.

Fig: 3, 4, 9, 10, 15, 17, 18.

VSPERF with VPP as the vswitch

RFC2544, Throughput

phy2phy_tput_vpp

Bi-Directional

Multistream=False

All

VSPERF with OVS as the vswitch

RFC2544, Throughput.

phy2phy_tput

Bi-Directional

Multistream=True

Number of streams – 4096, 1M, 16.8M(BM-A Only)

All

Figs: 5, 6,7, 8, 11, 12, 13, 14, 16, 19, 20

VSPERF with VPP as the vswitch

RFC2544, Throughput

phy2phy_tput_vpp

Bi-Directional

Multistream=True

Streams – 4096, 1M, 16.8M(BM-A Only)

All

VSPERF with OVS as vSwitch and a VLoop VM.

One/Two Stress VMs.

RFC2544,

Throughput

pvp_tput

Bi-Directional

Multistream=False

Hardware Only

Figs: 21, 22

Category

Details

Environment of the DUT

OS: CentOS Linux 7 Core

Kernel Version: 3.10.0-327.36.3.el7.x86_64

- NIC(s): 5

    - Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Board: Intel Corporation S2600WT2R [2 sockets]

CPU:  Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

CPU cores: 88

Memory: 65687516 kB

Hugepages: 1G, 32 pages. iommu-enabled.

Environment of the Traffic generators

OS: CentOS Linux 7 Core

Kernel Version: 3.10.0-327.36.3.el7.x86_64

- NIC(s): 5

    - Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Board: Intel Corporation S2600WT2R [2 sockets]

CPU:  Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

CPU cores: 44

Memory: 65687516 kB

Hugepages: 1G, 32 pages. iommu-enabled.

VSPERF

GIT tag: 0498b5fb3893e4331191808e3501d00b684af9b4

OVS

OvsDpdkVhost, Version: 2.6.90, GIT tag: ed26e3ea9995ba632e681d5990af5ee9814f650e

VPP

VppDpdkVhost, Version: v17.01-release

DPDK

Version: 16.07.0, GIT tag: 20e2b6eba13d9eb61b23ea75f09f2aa966fa6325

Hardware traffic Generator Configuration

Traffic ports: 10G

BM-A

Version: v0.33

CPU-Pinning: YES

Hugepages:

DPDK: 2.2.0

Cores/Interface: 1

Txqueues/port:1 (max 128)

RxQueues/port:1 (max 128)

BM-B Details

CPU-Pinning: YES

RxQueues/Port:1

TxQueues/Port:2

Hugepages:

DPDK: 2.2.0

Traffic-Generator as a VM

Version: 4.73

RxQueues/Port: 1

VM Configurations: 2-Port, 6 vCPUs.

CPU-Pinning: YES. 1:1

Other Details:

VLOOP VNF

QemuDpdkVhostUser, Version:2.5.0,

GIT tag: a8c40fa2d667e585382080db36ac44e216b37a1c

Image: vloop-vnf-ubuntu-14.04_20160823.qcow2

Loopback apps: testpmd, Version: 16.07.0,

(GIT tag: 20e2b6eba13d9eb61b23ea75f09f2aa966fa6325)

Noisy-VM

A Stressor-VM

Instances: 2

Noise-levels

Noiselevel-0

CPU-100%

Network Load-NIL

Readsize: 32K

Writesize:32K

BufferSize:64K

Read-Rade:5000

Write-rate:2000

Noise-Level-1

CPU-100%

Network Load – NIL

Read-Size: 4MB

Write-Size:4MB

Buffer-Size: 33MB

Read-rate:10000

Write-rate: 5000

Noise-Level-2

CPU-100%

Network-load - NIL

Read-Size: 8MB

Write-Size: 8MB

Buffer-Size: 64MB

Read-rate:10000

Write-rate:5000

Test/Test-Category

Inference

Ref. Figures

Reasoning|Explanation|Justification

Hardware Traffic Generator

For 64-Bytes, with the hardware traffic generator, and for both OVS and VPP, the maximum forwarding achieved is only 80% of line-rate.

Fig-3

Inherent limitations - memory bandwidth or PCI bandwidth or transactions per second on the PCI bus.

For single flow, Avg. latency for OVS and VPP varies from 10-90us with minimal (1-9%) difference between them. Average latency jumps significantly after 128 B.

Fig-4

The packet-processing architecture could be one possible reason for the differences.

Note: One should take into consideration of the inconsistencies in latency measurements - along with lack of delay-variations information.

With multistream (4096 flows and 1M Flows) the throughput performance of OVS is lower compared to VPP for lesser packet sizes (64 and 128).

*Inconsistency*

Fig-5 and Fig-7

OVS could be doing more processing on the packets compared to the VPP. Other reasons could be:

• Difference in packet-handling architectures

For multi-stream, latency variation are:

Inconsistency for 256B with OVS vs VPP.

With Multistream (4096 flows and 1M flows) the latency of VPP is higher compared to OVS for lesser packet sizes (64, 128 and 256).

Fig-6 and Fig-8

The inconsistencies should be taken into consideration before making any conclusions.

The packet-processing architecture could be one possible reason for the differences.

BM-A as Baremetal Traffic Generator

For Single flow, The throughput achieved with BM-A for both OVS and VPP is consistent with the hardware traffic generators

Figs: 9 and 3.

We should be take into consideration of the inconsistencies with lower-packet sizes. However, for larger packet size software generators can match hardware counterpart for this scenario.

For multistream, the throughput performance difference between OVS and VPP is lesser (approximately 50%, 30% and 0% for 64bytes, 128 bytes and 256 bytes respectively) compared to hardware traffic generator (approximately 70%, 50% and 5%)

Figs: 11 and 5

Possible reasons:

There are issues with latency computations in BM-A.

Figs 12 and 14.

Resource requirement for latency measurements are not satisfied by the configurations - Work to fix this issue is in progress.

BM-B as Baremetal Traffic Generator

The throughput results for VPP, with BM-B is consistent with the hardware traffic generator. It is able to send and receive at the same rate as hardware traffic generator

Fig 15 and Fig3.

Fig 16 and Fig 6.

We should be take into consideration of the inconsistencies with lower-packet sizes. However, for larger packet size software generators can match hardware counterpart for this scenario.

OVS throughput performance with BM-B is totally different – lesser for single flow vs VPP, and same as VPP for multistream – in comparison with other traffic generators.

Fig 15

The way the packet is formed by BM-B could be one possible reason.

BM-B yet to support latency measurements in VSPERF

N/A

The work is in progress to support latency measurements.

Traffic generator as VM.

Traffic generator as VM is yet to match the traffic generation capabilities of Baremetal traffic-generators. The throughput for 64-bytes is almost 50% that of BM-A or BM-B.

Figs: 17 and 19

Inherent overhead of running in VM vs running baremetal could be one possible reason. This could be confirmed if we run BM-A|BM-B in a VM.

TGen-VM may be doing extra work per packet.

TGen-VM is doing both port-level statistics and stream-level statistics and it may not apply to others.

TGen-VM with improved configuration - multiple rx-queues and CPU-cores - such as that of BM-A|BM-B should improve its performance.

Among the software generators the latency measures of TGen-VM is consistent with the hardware latency measurements.

However, the latency values (min and avg) can be 10x times the values provided by the hardware traffic-generator.

Figs: 18 and 20.

TGen-VM does port level statistics and stream level statistics including checking for packet sequencing errors and dropped packets, etc.

Possible reason for high/inconsistent latency measurements could be Configuration of NTP servers.

With TG as a VM there was no ‘significant’ performance difference between OVS and VPP – neither the throughput nor the latency,

Figs 17, 18, 19 and 20.

Possibly due to the lower-throughput rate. At that rate, both the switches should be able to perform similarly.

The average latency significantly increases for larger-packet sizes for both single flow and 4096Flows.

Figs 18 and 20.

This trend is seen for all cases.

Comparison of Baremetal Traffic generators.

There is lack of consistency of OVS throughput results between the two traffic generators for both Single flow and multi-stream.  However, with VPP, they are very consistent.

Figs: 9 and 15.

And, Figures 11 and 16

It needs additional experiments to understand and explain this aspect.

Both the traffic generators lack proper latency measurements – as of now.

Fig 10 and 12.

Latency measurements are expected to improve by next-release.

Virtual Switches

VPP can forward packets at a faster rate compared to OVS for multi-stream case and for lower-packet size. The %ge of difference can range from 80% to 30%.

Figs 5, 7, 11, 13, and 15.

Configuration of the OVS needs to be analysed in detail to explain the performance difference.

There still exists many inconsistencies and results differ across traffic generators.

The cache miss of VPP is 6% lower compared to the cache-misses for OVS.

Fig 26.

Processing architecture of VPP could be one possible reason.

For both OVS and VPP the latency jumps significantly for larger packet-sizes.

Figs: 4, 6, 8, 18 and 20.

A thorough analysis of latency measurement is required.

Noisy Neighbor Tests

The performance (throughput and latency) with PVP topology is extremely poor compared to P2P topology.

Figs: 22 and Fig 22.

The L2FWD module configuration and virtual-interface could be possible reason for degradation in performance.

If a noisy neighbor consumes the L3 cache fully and overload the memory bus (noiselevel 1 and 2), it can create sufficient noise and degrade the performance of the VNF.

Fig 21.

The Last-level cache is a shared resource - among all CPUs. Hence, a single process consuming L3-cache completely will affect the performance other processes.

CPU affinity configuration and NUMA configuration can protect from majority of Noise. 

Consumption of Last-level cache (L3) is key to creating noise.

It maybe be worth studying the use of tools such as cache-allocation-technology (Libpqos) to manage noisy-neighbors 

The effect on latency due to noisy-neighbors is minimal

Fig-22

The low-throughput performance could be a reason for this trend.

Hardware Vs Software Traffic generators.

Fig 27.

TG characteristics can impact measurements