Since dovetail project differ from those testing projects, such as (FUNCTEST, YARDSTICK), it is necessary to list terms used in project and give a detailed description to avoid confusion.

Here is the table  showing current terms in project.

If you feel any word is unclear , you can put it into this table, so we can review and discuss these terms constantly

Welcome recommendations and comments.

...

Recommend not using the word certification, nor cert, in our tools or documents.

(to use "compliance" instead of "certification"? MatthewLi)

...

Lets not use the term scenario.  Can we adopt "test suite" instead?

(agree, should think out name to replace this "scenario", since in OPNFV, "scenario" has special meaning MatthewLi)

...

how does this differ from "test suite", is this a "test area"?

(I'm not sure the initial idea of this name, but in current implementation, it means test case from other projects; for example, in dovetail.ipv6.tc001.yml , script test case is 'tempest_smoke_serial' which is smoke tier test case in functest )

...

"SUT_IP" may be a candidate

(agree SUT_IP, MatthewLi)

...

OPNFV Verified Program (OVP)

The OVP is a program initiated by the OPNFV Board of Directors with the following key objectives:

• • Help build the market for
    • OPNFV-based infrastructure
    • applications designed to run on that infrastructure
  • Reduce adoption risks for end-users
  • Decrease testing costs by verifying hardware and software platform interfaces and components
  • Enhance interoperability

Dovetail is the project that delivers all the open source software toolchain, documentation, and test suite for OVP. Dovetail project meets OVP testing requirements by leveraging many OPNFV projects and upstream projects. The Certification and Compliance committee (C&C committee) serves as the program's administrator on behalf the OPNFV Board.

Products that pass the OVP compliance verification suite may be labeled as “OPNFV Verified”. The OPNFV C&C Committee on behalf of the Board of Directors can award a product “OPNFV Verified” status.

More details of OVP can be found in the OVP document.

System Under Test (SUT)

The System Under Test (SUT) are products from vendors, not OPNFV release artifacts themselves. OPNFV development tooling, such as CI/CD, is also not part of the SUT. The vendors can bring up the SUT to a pre-test state in any way they choose. Dovetail will provide documentation to help the testers to prepare and configure the SUT.

In the current OVP version (Danube release time of Dovetail), the SUT consists of NFVI software, VIM software, and necessary hardware in one System Under Test. The hardware should follow Pharos guidelines (but the guideline is generalized in the Dovetail documentation to remove specifics tied to OPNFV CI/CD environment). Vendors can use their own hardware, or third party or white box hardware to be tested as a whole. And if the combined whole passes the Test Suite, the OVP label applies to the software.

In the Danube cycle, OVP does not plan to test hardware-only systems.

Test Suite

A Test Suite is a collection of test cases grouped together for a specific purpose, for example, OVP. A Test Suite can be named and versioned for easy and accurate reference. The Dovetail test tool primarily supports the Compliance Test Suite for OVP, but can in general support multiple test suites.

Compliance Test Suite

A Compliance Test Suite is the Test Suite used in the OPNFV OVP. By default, this is the Test Suite we are referring to. The Compliance Test Suite is to be versioned with MAJOR.MINOR.PATCH together with the Dovetail tool.

Test Area

A Test Area is a collection of test cases grouped together by functional areas or upstream projects. For example, Openstack Refstack/Defcore, IPv6, HA, SDNVPN.

Test Case

A Test Case is an instance of test unit within a Test Area. In some implementation of the test cases, we may see additional sub-test-cases as smaller units under a test case.

OVP (or Dovetail) Testing

The following diagram help illustrates the main components of the testing system for OVP.

Image Added

Tester

The tester is a knowledgeable engineer of the vendor that has admin right to the SUT within its lab or administrative domain. We use the term in abstract sense, not necessarily meaning one individual. The tester must have sufficient admin privilege to make all hardware and software configurations according to the Dovetail Tester's Guide.

The tester must also have login/access rights established with the Linux Foundation (OPNFV) to access the OVP server to review test results.

Reviewer

The reviewer is a knowledgeable engineer to review test results submitted by vendors to determine if OVP requirements have been met and OPNFV Verified label should be granted. Again  we use the term in abstract sense, not necessarily meaning one individual. The C&C committee administers review process.

The reviewer must have login/access ID established with the Linux Foundation (OPNFV) to access the OVP server to review test results.

OVP Server

The OVP server is the backend system (currently consists of a web frontend and a database backend) hosted by the Linux Foundation on behalf of OPNFV. When a test suite is executed, test results are submitted and saved in the OVP Server's database. The OVP Server can be accessed via a Test API using Linux Foundation credentials (to be finalized).

OVP Client (Dovetail client or command-line interface)

The OVP Client, or Dovetail client or Dovetail CLI, is the software that runs in a vendor's host to drive the automated OVP test suite. the Tester who executes the test suite must have the valid credentials of the OVP server in order to store and review test results.

The OVP Client comes with two forms: either as a Dorker container, or as a software install (based on Python). Dovetail Tester's Guide contains detailed guides on how to execute the test suite using the Dovetail client CLI.