Long Duration Testing

Introduction

As part of the collaboration between VSPERF and ETSI NFV TST working group, we developed and tested new methods to characterize the limits of data plane resources. The new methods are comprised of new search algorithm(s) to discover true resource limits (RFC 2544 Throughput, for example) while avoiding the effects of transient loss in the NFVI platform. The current search algorithm is called Binary Search with Loss Verification. A key aspect of each measurement completed during the search (a trial) is the duration of the test. The Trial Duration must be considerably less than the average time interval between transient events that cause frame loss. The details are described in ETSI GS NFV-TST 009 .

The Long Duration Testing compliments the Binary Search with Loss Verification algorithm by helping to determine (or confirm) the trial duration, and to identify the processes and/or interrupts that cause transient loss. Once the sources of transients have been identified, they can be optimized (for example, by reducing their rate of occurrence) or mitigated (by increasing strategic buffer sizes to absorb more frames during intervals of suspended forwarding).

Testing at Gambia Plugfest

We will be using an expanded set of traffic generators for Long Duration Testing, and primarily iPerf3 during the Plugfest. This means we will use VSPERF in --mode trafficgen-off and operating iPerf3 manually, developing tools to automate data collection as we go.

We have already created the network namespace environment needed to run both iPerf3 Client and Server on the same host, and ensure that traffic follows the external path as shown below.

IPerf 3 was sending at low packet rates (~40K pps) in initial iPerf3 testing - this means we are only capturing BIG interrupts with this stream (see sample output below).

In future, using DPDK to operate at higher rates and buffer size set on per packet basis, traffic will fill the buffers more easily. T-rex generator can only achieve about 5 Gbps (based on recent testing, with T-rex core assignment = 4 cores in current configurations).

During the Plugfest, it was recommended to examine PROX tool's ability to characterize processor Interrupts. The PROX tool was installed on Pod12, Node 4. The PROX tool can examine interrupt effects on a heartbeat on each core. The current master can be accessed on gitub: https://github.com/opnfv/samplevnf/tree/master/VNFs/DPPD-PROX/

We obtained a PROX status display snapshot output for VSPERF Intel Pod 12, Node 4 (the Device Under Test, DUT, for the Long Duration and Cross-NUMA Node testing). This test used the configuration file in [opnfv@pod12-node4 ~]$ sudo ./prox -k -f ../config/irq.cfg

where -k : Log statistics to file "stats_dump" in current directory

From PROX, which essentially measures the duration of a 1 nanosecond heartbeat, it is clear that some process is producing fairly frequent interrupts that cause delays in the 10-50 usec and 50-100usec range on core 6. Also, Core 11 has some interrupts in the 10-50usec range: it is important to re-check this result when the VSPERF deployment is running. In the iPerf3 testing below, the Max MTU packets used have 1.2144 usec serialization delay on the 10 Gbps links, so a 10 usec interruption would affect approximately 12 frames.

For example, the BSwLV OVS-Vanilla config has:

GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')] and

VSWITCH_VHOST_NET_AFFINITIZATION = False with VSWITCH_VHOST_CPU_MAP = [4,5,8,11]

We can get these statistics on a per-second basis, from a TCP socket - there is syntax to request these stats. (also -k command, Xavier will share scripts which are better).

Next steps:

OVS-DPDK with isolcpu rcu_nocbs
and taskset the iPerf process to qa core with no interrupts.

As an example of the iPerf3 loss counting capability, we have the results of one 60 second test below (where each second would be a trial):

-----------------------------------------------------------

Server listening on 5201

-----------------------------------------------------------

Accepted connection from 10.10.124.25, port 39851

[ 5] local 10.10.122.25 port 5201 connected to 10.10.124.25 port 35449

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 5] 0.00-1.00 sec 55.7 MBytes 467 Mbits/sec 0.004 ms 0/40351 (0%)

[ 5] 1.00-2.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43161 (0%)

[ 5] 2.00-3.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43152 (0%)

[ 5] 3.00-4.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43166 (0%)

[ 5] 4.00-5.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43139 (0%)

[ 5] 5.00-6.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43181 (0%)

[ 5] 6.00-7.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43157 (0%)

[ 5] 7.00-8.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43172 (0%)

[ 5] 8.00-9.00 sec 59.6 MBytes 500 Mbits/sec 0.006 ms 46/43211 (0.11%)

[ 5] 9.00-10.00 sec 59.6 MBytes 500 Mbits/sec 0.006 ms 0/43128 (0%)

[ 5] 10.00-11.00 sec 59.5 MBytes 499 Mbits/sec 0.005 ms 124/43196 (0.29%)

[ 5] 11.00-12.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43126 (0%)

[ 5] 12.00-13.00 sec 59.7 MBytes 501 Mbits/sec 0.011 ms 0/43214 (0%)

[ 5] 13.00-14.00 sec 59.5 MBytes 499 Mbits/sec 0.004 ms 0/43106 (0%)

[ 5] 14.00-15.00 sec 59.7 MBytes 500 Mbits/sec 0.005 ms 32/43237 (0.074%)

[ 5] 15.00-16.00 sec 59.5 MBytes 499 Mbits/sec 0.005 ms 0/43095 (0%)

[ 5] 16.00-17.00 sec 59.7 MBytes 501 Mbits/sec 0.005 ms 0/43210 (0%)

[ 5] 17.00-18.00 sec 59.6 MBytes 500 Mbits/sec 0.006 ms 0/43127 (0%)

[ 5] 18.00-19.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43189 (0%)

[ 5] 19.00-20.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43140 (0%)

[ 5] 20.00-21.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43158 (0%)

[ 5] 21.00-22.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43142 (0%)

[ 5] 22.00-23.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43185 (0%)

[ 5] 23.00-24.00 sec 59.6 MBytes 500 Mbits/sec 0.006 ms 0/43173 (0%)

[ 5] 24.00-25.00 sec 59.6 MBytes 500 Mbits/sec 0.006 ms 0/43159 (0%)

[ 5] 25.00-26.00 sec 59.6 MBytes 500 Mbits/sec 0.006 ms 0/43195 (0%)

[ 5] 26.00-27.00 sec 59.6 MBytes 500 Mbits/sec 0.006 ms 0/43152 (0%)

[ 5] 27.00-28.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43155 (0%)

[ 5] 28.00-29.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43143 (0%)

[ 5] 29.00-30.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43160 (0%)

[ 5] 30.00-31.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43141 (0%)

[ 5] 31.00-32.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43172 (0%)

[ 5] 32.00-33.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 19/43178 (0.044%)

[ 5] 33.00-34.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43166 (0%)

[ 5] 34.00-35.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 7/43155 (0.016%)

[ 5] 35.00-36.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43171 (0%)

[ 5] 36.00-37.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43131 (0%)

[ 5] 37.00-38.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43168 (0%)

[ 5] 38.00-39.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43185 (0%)

[ 5] 39.00-40.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43169 (0%)

[ 5] 40.00-41.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43160 (0%)

[ 5] 41.00-42.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43148 (0%)

[ 5] 42.00-43.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43162 (0%)

[ 5] 43.00-44.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43176 (0%)

[ 5] 44.00-45.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43178 (0%)

[ 5] 45.00-46.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43141 (0%)

[ 5] 46.00-47.00 sec 59.6 MBytes 500 Mbits/sec 0.006 ms 0/43156 (0%)

[ 5] 47.00-48.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43189 (0%)

[ 5] 48.00-49.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43142 (0%)

[ 5] 49.00-50.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43153 (0%)

[ 5] 50.00-51.00 sec 59.5 MBytes 499 Mbits/sec 0.005 ms 45/43150 (0.1%)

[ 5] 51.00-52.00 sec 59.7 MBytes 501 Mbits/sec 0.005 ms 0/43210 (0%)

[ 5] 52.00-53.00 sec 59.6 MBytes 500 Mbits/sec 0.005 ms 0/43168 (0%)

[ 5] 53.00-54.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43158 (0%)

[ 5] 54.00-55.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43160 (0%)

[ 5] 55.00-56.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43177 (0%)

[ 5] 56.00-57.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43150 (0%)

[ 5] 57.00-58.00 sec 59.6 MBytes 500 Mbits/sec 0.004 ms 0/43152 (0%)

[ 5] 58.00-59.00 sec 59.5 MBytes 499 Mbits/sec 0.003 ms 0/43055 (0%)

[ 5] 59.00-60.00 sec 59.7 MBytes 501 Mbits/sec 0.004 ms 0/43267 (0%)

[ 5] 60.00-60.04 sec 0.00 Bytes 0.00 bits/sec 0.004 ms 0/0 (0%)

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 5] 0.00-60.04 sec 0.00 Bytes 0.00 bits/sec 0.004 ms 273/2586968 (0.011%)