This page is intended as a working draft of how we will look to leverage and develop test cases for the dovetail project.

The dovetail test suite

The dovetail test suite is intended to provide a method for validating the interfaces and behaviors of an NFVi platform according to the expected capabilities exposed in an OPNFV NVFi realization. All dovetail tests will be available in open source and will be developed on readily available open source test frameworks.

Working with the ETSI NFV TST 001 reference: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
Where I would see our dovetail project focus on SUT tests validating a SUT scope associated with Chapter 4.9 - NFV Infrastructure + VIM Under Test, as adapted for OPNFV see figure below.  While we may also look at interesting use cases like SFC, I would suggest we have clear plan on pre-deployment validation to ensure that any platform we are evaluating is first evaluated to be in a steady state of readiness.

Aside from the very basic requirements we would have on a test case to be used in dovetail:

• Test cases should implement a published standard interface for validation
  • Where no standard is available provide API support references
  • If a standard exists is not followed an exemption method needs to be established
• Test cases be documented
  • Use case specification
  • Test preconditions
  • Basic test flow execution descriptor
  • Post conditions and pass fail criteria
  • Parameter border test cases descriptions
  • Fault/Error test case descriptions  (this feels optional at this time)
• Test cases must pass on OPNFV reference deployments
  • Tests must pass with deployments with at least two installers
  • Tests must pass with at least two deployment scenarios involving different SDN controllers
  • (hongbo: this needs to be discussed further. there are several SDN controllers. some test cases from dedicated SDN controller can not be used for other controller.)
• Test documentation/implementation file and directory structure (per supported framework)
• Test result storage, structure and test result information management (these should be able to be run publically or privately)

Test suite structure

A dovetail test suite should have the following overall components and structure:  (stolen, if simplified a little, from IEEE)

• Test Plan
  • The test plan should describe how the test will proceed, who will do the testing and what will be tested.
• Test Design Specification
  • The design spec outlines what needs to be tested including such information as applicable standards, requirements, and tooling.
• Test Case Specification
  • The Test Case spec describes the purpose of a test, required inputs and expected results, including step-by-step procedures pass/fail criteria.
• Test Procedure Specification
  • Describes how to run the test, preconditions and procedural steps to be followed.
• Test Log
  • A log of the tests run, by whom, when, and the results of the tests.
• Test Report
  • The test report describes the tests that were run, any failures and associated bugs, detailing deviations from expectation.
• Test Summary Report
  • Summary of all results including a test result assessment (to be defined).

Phasing the dovetail development effort

While not all tests will be possible to develop at once the following approach is proposed for the development of dovetail test suites in a structured manner.

Dovetail phase 1

Dovetail should initially set out to provide validation of interfaces and behaviors common to an OPNFV NVFi.  This can be seen as a set of test cases that evaluate if a NVFi implementation is able to achieve a steady operational state covering the common behaviors expected of an OPNFV NFVi.  In this case the dovetail tests will focus on a SUT definition of VNFi & VIM as described in 4.9 of the ETSI NFV TST 001 specification.

Dovetail phase 2

Dovetail should further establish a set of test suites that validate additional desired OPNFV VNFi behaviours.  This may include for instance, deployment specific capabilities for edge or remote installations.  It may include the validation of functionality that is not yet common to all OPNFV VNFi scenario's.

In phase 2 it may also be possible that dovetail provides such services as application test suites to validate the behavior of applications in preparation for deployment on an OPNFV VNFi.  This may result in the definition of new SUT scopes for dovetail as described in of the ETSI NFV TST 001 specification.

Dovetail phase 3

hongbo: the same as those defined in the phase 1 and phase 2

Out of scope (I left this for now, but it does not belong on this page)

Dovetail is not a testing problem with a goal of identifying gaps in the platform. Its goal is to validate the conformance of OPNFV "stacks" to a minimum set of capabilities common to all OPNFV implementations. As such, it is not a goal of the Dovetal project to write failing tests, or to engage in the development of APIs to fill perceived gaps in the platform. While formulating the Dovetail test suite, it is possible that we will encounter limits of the commonalities of OPNFV implementations in areas which we would like to test conformance. In this case, efforts to create alignment, or to implement missing features, should happen in the context of other OPNFV projects. Dovetail will always be primarily concerned with documenting and verifying common platform capabilities after they exist.

hongbo: as we discussed in the dovetail meeting, the goal of the dovetail is to clearly articulates meaning of the brand/value proposition from C&C . in this case, the dovetail needs to implements the requirements from C&C by outputting the deliverable in terns of testcases and scripts. in order to do that, the dovetail can choose the test cases and scripts from the OPNFV test projects but not limited to the test projects. it is obviously that the certification defined by the dovetail will show the value of OPNFV which exist in the OPNFV platform.