Vsperf Multi-Flow testing

Introduction

The purpose of this series of testing is to examine the impact of sending multiple streams thought the OVS bridge on the flow throughput and performance. Testing 2544 zero loss on both PVP and Phy2Phy2 modes.

Test Goals:

To document any disparity in flow processing between Layer 2,Layer 3 and Layer 4 for flows ranging between 1,000 through 1 million.

To determine any any significant limitations or anomalies in processing in the handling of flows based on ranges that would affect performance when deployed in specific situations. Or when using in qualifications labs.

To characterize the Multiflow performance for VSPERF that can be used to compare for future benchmarking.

Test Non-Goals:

This is not a maximum performance testing setup. The system has been tuned to provide it’s best performance to ensure consistent results for this test. These results should not be used as a set of limitations for the software as different hardware configurations will be able to give different results.

These results are are to be used to compare within this dataset to each other.

This testing does not reflect performance limitations of the traffic generator used in this case. Other tools will have different abilities. One generate was used for consistent results.

Test Configuration:

Figure 1: Test Setup

System Configuration

Isolated all NUMA 1 CPU’s using tuned cpu parameters.

DPDK_SOCKET_MEM = ['4096', '4096']

VSWITCH_VHOST_CPU_MAP = [5,7,13,15]

VSWITCH_PMD_CPU_MASK = 'a0a0'

GUEST_SMP = ['3']

GUEST_CORE_BINDING = [('1', '9', '11')]

GUEST_TESTPMD_PARAMS = ['-l 0,1,2 -n 1 --socket-mem 2048 -- '

'--burst=64 -i --txqflags=0xf00 '

'--disable-hw-vlan --nb-cores=2, --txq=1 --rxq=1 --rxd=1024 --txd=1024']

Data Sets:

Test Data was determined by using the max number of each of the 5 iterations. This was to rule out one aberrations in the zero loss test that causes lower numbers infrequalty but impacted a flow here and there. 5 iterations were used to determine we had a level of confidence in the test numbers reported.

Test Cases:

Test Cases Performed
Run	Test Name	Description	Test Condition	Frame Size	Flow Counts
1-5	Baseline-001	Establish baseline metrics using 1 flow for Phy2Phy2_TPUT	Phy2Phy_TPUT	64,128,256,512,1024,1500	1
6-10	Baseline-002	Establish baseline metrics using 1 flow for PVP_TPUT	PVP_TPUT	64,128,256,512,1024,1500	1
11-15	Multi-Src-001	Run 2544 Zero Loss Test using modifying L2 Source Address only	Phy2Phy_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
16-20	Multi-Src-002	Run 2544 Zero Loss Test using modifying L2 Source Address only	PVP_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
21-25	Multi-Src-003	Run 2544 Zero Loss Test using modifying L3 Source Address only	Phy2Phy_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
26-30	Multi-Src-004	Run 2544 Zero Loss Test using modifying L3 Source Address only	PVP_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
31-35	Multi-Src-005	Run 2544 Zero Loss Test using modifying L4 Socket Source Address only	Phy2Phy_TPUT	64,128,256,512,1024,1500	1,000 10,000 64,000
36-40	Multi-Src-006	Run 2544 Zero Loss Test using modifying L4 Socket Source Address only	PVP_TPUT	64,128,256,512,1024,1500	1,000 10,000 64,000
41-45	Multi-Dst-007	Run 2544 Zero Loss Test using modifying L2 Dest Address only	Phy2Phy_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
46-50	Multi-Dst-008	Run 2544 Zero Loss Test using modifying L2 Dest Address only	PVP_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
51-55	Multi-Dst-009	Run 2544 Zero Loss Test using modifying L3 DestAddress only	Phy2Phy_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
56-60	Multi-Dst-010	Run 2544 Zero Loss Test using modifying L3 Dest Address only	PVP_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
61-65	Multi-Dst-011	Run 2544 Zero Loss Test using modifying L4 Dest Socket Address only	Phy2Phy_TPUT	64,128,256,512,1024,1500	1,000 10,000 64,000
66-70	Multi-Dst-012	Run 2544 Zero Loss Test using modifying L4 Dest Socket Address only	PVP_TPUT	64,128,256,512,1024,1500	1,000 10,000 64,000
71-75	Multi-Both-013	Run 2544 Zero Loss Test using modifying L2 Both src and Dest	Phy2Phy_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
76-80	Multi-Both-014	Run 2544 Zero Loss Test using modifying L2 Both src and Dest	PVP_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
81-85	Multi-Both-015	Run 2544 Zero Loss Test using modifying L3 Both src and Dest	Phy2Phy_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
86-90	Multi-Both-016	Run 2544 Zero Loss Test using modifying L3 Both src and Dest	PVP_TPUT	64,128,256,512,1024,1500	1,000 10,000 1,000,000
91-95	Multi-Both-017	Run 2544 Zero Loss Test using modifying L4 Both src and Dest	Phy2Phy_TPUT	64,128,256,512,1024,1500	1,000 10,000 64,000
96-100	Multi-Both-018	Run 2544 Zero Loss Test using modifying L4 Both src and Dest	PVP_TPUT	64,128,256,512,1024,1500	1,000 10,000 64,000

Analysis of Test Results:

Test Results

Baseline

Figure 2: 1 Flow Phy2Phy RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 3: 1 Flow PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Modify Both Source and Destination Address

Figure 4: Layer 2 Src & Des PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 5: Layer 3 Src & Des PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 6: Layer 4 Src & Des PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 7: Layer 2 Src & Dst Phy2Phy_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 8: Layer 3 Src & Dst Phy2Phy_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 9: Layer 4 Src & Dst Phy2Phy_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Flows Rotating Source address only

Figure 10: Layer 2 Src only PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 11: Layer 3 Src only PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 12: Layer 4 Src only PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 13: Layer 2 Src only Phy2Phy_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 14: Layer 3 Src only Phy2Phy_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 15: Layer 4 Src only Phy2Phy_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Flows Rotating Destination address only

Figure 16: Layer 2 Dest only PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 17: Layer 3 Dest only PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 18: Layer 4 Dest only PVP_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 19: Layer 2 Dest only Phy2Phy_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 20: Layer 3 Dest only Phy2Phy_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Figure 21: Layer 4 Dest only Phy2Phy_TPUT RFC 2544 Zero Loss. Verify Duration 20 Minutes

Flow Cross Analysis

PVP_TPUT Cross Flow Analysis

Rx fps PVP_TPUT L2 1,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000	5,706,159	5,618,036	5,181,619	4,574,736	1,636,512	L2	PVP_TPUT
Rotate_SRC	1,000	5,706,160	5,592,918	5,181,618	4,624,379	1,636,512	L2	PVP_TPUT
Rotate_DST	1,000	5,286,931	5,558,547	5,213,515	4,607,830	1,636,512	L2	PVP_TPUT

Rx fps PVP_TPUT L2 10,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	10,000	5,286,930	5,677,521	5,149,722	4,591,304	1,636,512	L2	PVP_TPUT
Rotate_SRC	10,000	5,033,066	5,558,550	5,213,517	4,574,736	1,636,512	L2	PVP_TPUT
Rotate_DST	10,000	1,849,888	4,528,837	4,990,231	4,475,443	1,636,512	L2	PVP_TPUT


Rx fps PVP_TPUT L2 1,000,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000,000	5,182,123	5,439,577	5,022,131	4,525,090	1,636,512	L2	PVP_TPUT
Rotate_SRC	1,000,000	5,810,968	5,380,094	4,990,232	4,458,897	1,636,512	L2	PVP_TPUT
Rotate_DST	1,000,000	6,020,580	5,737,004	5,245,413	4,425,798	1,636,512	L2	PVP_TPUT

Rx fps PVP_TPUT L3 1,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000	7,697,494	7,105,165	6,361,837	4,683,586	1,636,512	L3	PVP_TPUT
Rotate_SRC	1,000	8,011,913	7,343,100	6,457,529	4,683,587	1,636,512	L3	PVP_TPUT
Rotate_DST	1,000	4,553,281	4,428,330	4,160,887	3,747,308	1,636,512	L3	PVP_TPUT

Rx fps PVP_TPUT L3 10,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000	7,697,494	7,105,165	6,361,837	4,683,586	1,636,512	L3	PVP_TPUT
Rotate_SRC	1,000	8,011,913	7,343,100	6,457,529	4,683,587	1,636,512	L3	PVP_TPUT
Rotate_DST	1,000	5,182,125	5,023,182	4,735,049	4,161,022	1,636,512	L4	PVP_TPUT
Flows	64	128	256	512	1500	Layer	Test	Test

Rx fps PVP L3 1,000,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	10,000	6,859,037	6,153,395	5,691,980	4,683,587	1,636,512	L3	PVP_TPUT
Rotate_SRC	10,000	6,754,230	6,272,369	5,660,085	4,683,587	1,636,512	L3	PVP_TPUT
Rotate_DST	10,000	4,972,508	4,963,703	4,671,252	4,061,729	1,636,512	L3	PVP_TPUT

Rx fps PVP L4 1,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000	5,182,125	5,023,182	4,735,049	4,161,022	1,636,512	L4	PVP_TPUT
Rotate_SRC	1,000	7,907,108	7,105,161	6,393,736	4,683,587	1,636,512	L4	PVP_TPUT
Rotate_DST	1,000	5,182,125	5,023,182	4,735,049	4,161,022	1,636,512	L4	PVP_TPUT

Rx fps PVP L4 65,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	10,000	4,972,510	4,606,788	4,256,582	3,846,602	1,636,512	L4	PVP_TPUT
Rotate_SRC	10,000	6,439,812	5,796,489	5,564,392	4,657,478	1,636,512	L4	PVP_TPUT
Rotate_DST	10,000	4,972,510	4,606,788	4,256,582	3,846,602	1,636,512	L4	PVP_TPUT

Rx fps PVP_TPUT L4 64,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	65,000	5,182,125	5,023,182	4,607,458	4,028,634	1,636,512	L4	PVP_TPUT
Rotate_SRC	65,000	5,182,124	5,023,182	4,607,457	4,028,633	1,636,512	L4	PVP_TPUT
Rotate_DST	65,000	5,182,125	5,023,182	4,607,458	4,028,634	1,636,512	L4	PVP_TPUT

Phy2Phy_TPUT Cross Flow Analysis

Rx fps Phy2Phy_TPUT L2 1,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000	12,833,037	14,897,700	9,041,256	4,683,588	1,636,512	L2	Phy2Phy_TPUT
Rotate_SRC	1,000	12,204,196	14,897,706	9,041,252	4,683,587	1,636,512	L2	Phy2Phy_TPUT
Rotate_DST	1,000	12,204,196	14,897,706	9,041,252	4,683,587	1,636,512	L2	Phy2Phy_TPUT

Rx fps Phy2Phy_TPUT L2 10,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	10,000	12,937,839	14,897,714	9,041,247	4,683,587	1,636,512	L2	Phy2Phy_TPUT
Rotate_SRC	10,000	12,204,195	14,897,705	9,041,250	4,683,587	1,636,512	L2	Phy2Phy_TPUT
Rotate_DST	10,000	12,204,195	14,897,705	9,041,250	4,683,587	1,636,512	L2	Phy2Phy_TPUT


Rx fps Phy2Phy_TPUT L2 1,000,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000,000	14,824,362	14,957,195	9,041,252	4,683,588	1,636,512	L2	Phy2Phy_TPUT
Rotate_SRC	1,000,000	7,487,881	8,592,289	8,626,578	4,683,587	1,636,512	L2	Phy2Phy_TPUT
Rotate_DST	1,000,000	7,487,881	8,592,289	8,626,578	4,683,587	1,636,512	L2	Phy2Phy_TPUT

Rx fps Phy2Phy_TPUT L3 1,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000	23,628,158	16,716,624	9,041,251	4,683,588	1,636,512	L3	Phy2Phy_TPUT
Rotate_SRC	1,000	23,523,356	16,533,085	9,041,248	4,683,585	1,636,512	L3	Phy2Phy_TPUT
Rotate_DST	1,000	23,628,142	16,741,734	9,041,252	4,683,584	1,636,512	L3	Phy2Phy_TPUT

Rx fps Phy2Phy_TPUT L3 10,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	10,000	17,549,343	16,860,704	9,041,252	4,683,586	1,636,512	L3	Phy2Phy_TPUT
Rotate_SRC	10,000	10,317,666	10,317,354	9,041,249	4,683,586	1,636,512	L3	Phy2Phy_TPUT
Rotate_DST	10,000	10,946,502	10,946,570	9,041,246	4,683,584	1,636,512	L3	Phy2Phy_TPUT
Flows	64	128	256	512	1500	Layer	Test	Test

Rx fps Phy2Phy L3 1,000,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000,000	12,833,030	12,815,721	9,041,245	4,683,586	1,636,512	L3	Phy2Phy_TPUT
Rotate_SRC	1,000,000	12,728,228	12,731,122	9,041,249	4,683,585	1,636,512	L3	Phy2Phy_TPUT
Rotate_DST	1,000,000	12,728,225	12,756,236	9,041,247	4,683,584	1,636,512	L3	Phy2Phy_TPUT

Rx fps Phy2Phy L4 1,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	1,000	23,523,355	16,860,711	9,041,250	4,683,586	1,636,512	L4	Phy2Phy_TPUT
Rotate_SRC	1,000	15,243,599	15,314,099	9,041,253	4,683,587	1,636,512	L4	Phy2Phy_TPUT
Rotate_DST	1,000	23,523,355	16,860,711	9,041,250	4,683,586	1,636,512	L4	Phy2Phy_TPUT

Rx fps Phy2Phy L4 65,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	10,000	19,540,676	16,801,222	9,041,250	4,683,587	1,636,512	L4	Phy2Phy_TPUT
Rotate_SRC	10,000	19,435,885	16,801,230	9,041,253	4,683,587	1,636,512	L4	Phy2Phy_TPUT
Rotate_DST	10,000	19,540,676	16,801,222	9,041,250	4,683,587	1,636,512	L4	Phy2Phy_TPUT

Rx fps Phy2Phy_TPUT L4 64,000 Flows
Test	Flows	64	128	256	512	1500	Layer	Test
Rotate_Both	65,000	13,461,878	13,410,574	9,041,247	4,683,585	1,636,512	L4	Phy2Phy_TPUT
Rotate_SRC	65,000	12,518,611	12,552,666	9,041,249	4,683,586	1,636,512	L4	Phy2Phy_TPUT
Rotate_DST	65,000	13,461,878	13,410,574	9,041,247	4,683,585	1,636,512	L4	Phy2Phy_TPUT

Anuket

Multi-Flow Testing