LTD Test Spec Overview

(This page is always Under Construction; it describes all tests as of the Colorado release)

The Level Test Design (LTD) Specification is one of the products of the OPNFV vsperf project. The LTD spec is formatted in ReStructured Text (RST), and presented in HTML form with the project Artifacts. This wiki page provides an overview of the LTD spec in several forms:

The LTD Spec and its goals are Summarized in a 23 page Internet Draft (now approved)
Short descriptions of each of the tests in the LTD Spec (below, also see this complete version, with LTP, News, etc.)
Example Device Under Test (DUT) Set-ups, called Deployment Scenarios at Test Methodology page.

The approach VSPERF has adopted is to take existing tests that are relevant to performance benchmarking of physical switches, and applying them to benchmarking virtual switches (this is to allow for a fair comparison with their physical counterparts). Once this is done, tests that are relevant to benchmarking virtual switches only, will be developed.

List of Tests and Descriptions

Speed of Activation

   o  Activation.RFC2889.AddressLearningRate

This test measures the address learning rate of the DUT, in accordance with section 5.8 of RFC 2889. A minimum of 3 DUT ports are required for this test, a learning port where frames first arrive at the DUT, a test port where frames are forwarded and returned to the source, and a monitoring port where the test device listens for flooded or miss-forwarded frames.

A prerequisite for this test is Throughput.RFC2889.AddressCachingCapacity, where the cache capacity is determined and must not be exceeded here.

   o  PacketLatency.InitialPacketProcessingLatency

In some virtual switch architectures, the first packets of a flow will take the system longer to process than subsequent packets in the flow. This test determines the latency for these packets. The test will measure the latency of the packets as they are processed by the flow-setup-path of the DUT.

Accuracy of Activation

   o  CPDP.Coupling.Flow.Addition

This Control path Data path Coupling (CPDP) test is a variation on a RFC 2544 Throughput test, where the vswitch controller adds and removes flow entries while measuring the data-path performance. The test will determine if activity on the vswitch control (north-bound) interface affects performance in the data-path. The test also assesses average latency during control path activity.

It is possible to compare the CPDP results with a RFC 2544 Throughput test conducted without the control activity, to more clearly quantify the effects, if any. Throughput.RFC2544.PacketLossRatio is not a prerequisite test.

Reliability of Activation

   o  Throughput.RFC2544.SystemRecoveryTime

This test measures the DUT's time to recover from exposure to a traffic overload. The recommended overload level is 150% of the offered load corresponding to the Throughput level as determined using section 26.1 of RFC 2544. The test also assess the packet latency at the Throughput level, and checks to assure that the latency has returned to the pre-overload average (within a 10% tolerance).

Therefore, the Throughput.RFC2544.PacketLossRatio is a prerequisite test.

   o  Throughput.RFC2544.ResetTime

Occasionally, network functions need to be reset to restore normal operation. This test described in RFC 6201 updates the procedures in section 26.2 of RFC 2544, defining a new benchmark, Reset Time.

Scale of Activation

   o  Activation.RFC2889.AddressCachingCapacity

This test measures the address cache capacity of the DUT, in accordance with section 5.7 of RFC 2889. A minimum of 3 DUT ports are required for this test, a learning port where frames first arrive at the DUT, a test port where frames are forwarded and returned to the source, and a monitoring port where the test device listens for flooded or miss-forwarded frames.

Speed of Operation

   o  Throughput.RFC2544.PacketLossRatio

This is the fundamental data path speed test for networking devices, assessing the maximum offered load for the DUT under the constraint that no frames or packets are lost. section 26.1 of RFC 2544 specifies the method for this test (earlier sections of RFC 2544 describe the various test conditions). The term "Throughput" refers to the maximum loss-less sending rate after allowing the DUT queues to drain at the end of the trial. We have expanded this test to allow characterizing the sending rate at target loss ratios other than zero, and 10||-7 is the non-zero default. A complete range of loss ratios could be assessed, as described in section 26.3 of RFC 2544.

This test also assesses the average one-way latency for successfully received packets (note that the average may be misleading if the latency distribution is bi-modal). All Latency tests called for deviate from section 26.2 of RFC 2544, and assess the metric more thoroughly.

   o  CPU.RFC2544.0PacketLoss

This CPU intensive test is a variation on a RFC 2544 Throughput test (0% loss), where a CPU-intensive application is running on the same DUT as the vswitch. The test will determine if stressful CPU activity on all non-vswitch Cores affects performance in the data-path.

Although Throughput.RFC2544.PacketLossRatio is not a prerequisite test, the results may provide a baseline for unmeasurable affect of a CPU intensive application.

   o  Throughput.RFC2544.PacketLossRatioFrameModification

This test assesses the RFC 2544 Throughput level with additional processes of packet inspection and modification. The Recommended modification is to change or add a VLAN tag, but many more possibilities exist.

Although Throughput.RFC2544.PacketLossRatio is not a prerequisite test, that Throughput level will normally represent an upper bound on Throughput with Frame Modification.

   o  Throughput.RFC2544.BackToBackFrames

This test attempts to characterize the longest "burst" of back-to-back frames that the DUT can process without loss, as per section 26.4 of RFC 2544. This benchmark should be repeated and result consistency examined, as results with physical devices have been unstable in some cases.

   o  Throughput.RFC2889.MaxForwardingRate

When operating beyond the RFC 2544 Throughput level, the DUT may be able to achieve higher a frame rate transmitted on all egress ports, albeit with frame losses occurring. The Maximum Forwarding Rate is sometimes measured between the Throughput level and the Maximum Offered Load (maximum load on all ingress interfaces) when the transmitted frame rate degrades below the Maximum Offered Load. Section 5.1 of RFC 2889 describes this test.

The Throughput.RFC2544.PacketLossRatio is a prerequisite test.

   o  Throughput.RFC2889.ForwardPressure

Section 5.6 of RFC 2889 describes the Forward Pressure test, where the specified limit on Ethernet Inter-Frame Gap (12 octets) is exceeded by sending Frames with 11 octet Inter-Frame Gaps. The Maximum Forwarding Rate is assessed and reported for the Forward Pressure condition.

The Throughput.RFC2889.MaxForwardingRate is a prerequisite test.

   o  Throughput.RFC2889.BroadcastFrameForwarding

The forwarding rates of Broadcast Frames are assessed in this test, where at least 4 ports are used and 3 are monitored for broadcast transmissions. Section 5.10 of RFC 2889 describes the test. It is not intended to overload ingress or egress ports during this test. An error condition occurs if any broadcast frames appear on the paired egress side of their ingress port. The Frame Loss Ratio of RFC 2544 should also be reported.

This test also assesses the one-way latency for successfully transmitted broadcast frames, averaged over the latency for each packet broadcasted to all egress ports. Section 5.10 of RFC 2889 describes this aspect of the test, as well.

Accuracy of Operation

   o  Throughput.RFC2889.ErrorFramesFiltering

Another test of abnormal condition handling (beyond Forward Pressure) is a test of the DUT's ability to correctly filter the abnormal frames, or if it simply transfers such frames to an egress port. Section 5.9 of RFC 2889 describes the test, including the five categories of illegal frames (Oversize, Undersize, CRC errors, Dribble bit errors, and Alignment errors).

Although Section 5.9 of RFC 2889 indicates this test is a pass or fail outcome, it is appropriate to report the offered load of correct frames and the resulting transmitted frame rate.

   o Throughput.RFC2544.Profile

This is a test to characterize the loss ratio and latency in the overload region, beyond the RFC 2544 Throughput level (with 0% loss ratio). The offered load to achieve the Maximum Forwarding Rate in Section 5.1 of RFC 2889 testing prescribes the maximum load for this test.

Increased traffic expressed as a percentage of the difference between the Throughput level and the Maximum Offered Load (a physical line rate limitation) is applied to the DUT, in values of -50%, -10%, 0%, +10% and +50%. The Forwarding Rate (frames and Mbps), Latency, CPU & Memory Utilization, and any failures observed should be recorded for all frame sizes tested.

Reliability of Operation

   o  Throughput.RFC2889.Soak

This is the long duration, or "Soak" test to assess the stability of conditions measured in the Throughput.RFC2544.PacketLossRatio test. Ideally, the Soak test duration is 24 hours.

   o  Throughput.RFC2889.SoakFrameModification

This is the long duration, or "Soak" test to assess the stability of conditions measured in the Throughput.RFC2544.PacketLossRateFrameModification test. Ideally, the Soak test duration is 24 hours.

   o  PacketDelayVariation.RFC3393.Soak

This is the long duration, or "Soak" test to assess the stability of delay distributions measured in the Throughput.RFC2544.PacketLossRatio test. Ideally, the Soak test duration is 24 hours. The RFC 5481 PDV form of delay variation is measured on the traffic, using the 99th percentile, for each 60 second interval during the test. This extended period test will determine of there are any long delays and their frequency of occurrence.

The prerequisite test is the Throughput.RFC2544.PacketLossRatio, which sets the Throughput level for each frame size.

Scalability of Operation

   o  Scalability.RFC2544.0PacketLoss

This test assesses the RFC 2544 Throughput level with a variable number of flows present in the offered load, as recommended in section 12 of RFC 2544, where changing the destination address constitutes a new flow. The number of flows range from 1000 to the Maximum number of supported flows.

   o  MemoryBandwidth.RFC2544.0PacketLoss.Scalability

This test assesses the RFC 2544 Throughput level while a memory intensive application is running on the same DUT as the vswitch. The test will determine if stressful memory and cache activity, with corresponding memory bus activity, affects performance in the data-path. The test description provides specific requirements for the memory intensive application, including read:write ratio, NUMA architecture considerations, local/remote memory considerations, etc.

Although Throughput.RFC2544.PacketLossRatio is not a prerequisite test, the results may provide a baseline for unmeasurable affect of a memory/cache intensive application.