Since dovetail project differ from those testing projects, such as (FUNCTEST, YARDSTICK), it is necessary to list terms used in project and give a detailed description to avoid confusion.
Here is the table showing current terms in project.
If you feel any word is unclear , you can put it into this table, so we can review and discuss these terms constantly
Welcome recommendations and comments.
| term | reference | description | recommend | status | comments |
|---|---|---|---|---|---|
| certification | certification file name in directory cert/ | management unit, collection of test case. dovetail tool runs only one certification at a time. | Recommend not using the word certification, nor cert, in our tools or documents. (to use "compliance" instead of "certification"? MatthewLi) | ||
| scenario | input parameter for dovetail tool | actually the same meaning as certification, seems like a redundant term | Lets not use the term scenario. Can we adopt "test suite" instead? (agree, should think out name to replace this "scenario", since in OPNFV, "scenario" has special meaning MatthewLi) | ||
| test case | test case file in directory testcase/ | running unit, definition of a test case in dovetail, may contains multiple sub test cases | |||
| sub test case | test case definition files | represent test cases from upstream projects (FUNCTEST/YARDSTICK/...) | propose removing this term. "test activity" maybe. | ||
| scripts | test case definition files | dovetail leverages some upstream projects to run tests, such as (FUNCTEST/YARDSTICK/...), scripts includes a list of attributes concerning a certain upstream project, like 'type', 'testcase' and 'sub_testcase_list' | propose using "test source" | ||
| script test case | parameter name in conf/dovetail_config.yml | a sub set of test cases from upstream projects ( FUNCTEST/YARDSTICK/... ), one script test case may contain several sub test cases | how does this differ from "test suite", is this a "test area"? (I'm not sure the initial idea of this name, but in current implementation, it means test case from other projects; for example, in dovetail.ipv6.tc001.yml , script test case is 'tempest_smoke_serial' which is smoke tier test case in functest ) | ||
| System Under Test (SUT) | The target of the test cases. | The defined functional entity to be evaluated during the execution of the test suite described. The SUT should be consistently scoped for any given test suite. | |||
| INSTALLER_IP | config item in functest_config.yml | Any ip address that can access the OPENSTACK APIs is OK, could be the external ip address of OPENSTACK | "SUT_IP" may be a candidate (agree SUT_IP, MatthewLi) | ||
| INSTALLER_TYPE | config item in functest_config.yml | It's used to indicated that which installer is used to deploy the SUT, dovetail may get rid of it until testing projects do not depend on it any more | ("SUT_TYPE" maybe MatthewLi) | ||
| compliance_set | |||||
| proposed_tests | |||||
| test area |
compliance_set
...
OPNFV Verified Program (OVP)
The OVP is a program initiated by the OPNFV Board of Directors with the following key objectives:
- Help build the market for
- OPNFV-based infrastructure
- applications designed to run on that infrastructure
- Reduce adoption risks for end-users
- Decrease testing costs by verifying hardware and software platform interfaces and components
- Enhance interoperability
- Help build the market for
Dovetail is the project that delivers all the open source software toolchain, documentation, and test suite for OVP. Dovetail project meets OVP testing requirements by leveraging many OPNFV projects and upstream projects. The Certification and Compliance committee (C&C committee) serves as the program's administrator on behalf the OPNFV Board.
Products that pass the OVP compliance verification suite may be labeled as “OPNFV Verified”. The OPNFV C&C Committee on behalf of the Board of Directors can award a product “OPNFV Verified” status.
More details of OVP can be found in the OVP document.
System Under Test (SUT)
The System Under Test (SUT) are products from vendors, not OPNFV release artifacts themselves. OPNFV development tooling, such as CI/CD, is also not part of the SUT. The vendors can bring up the SUT to a pre-test state in any way they choose. Dovetail will provide documentation to help the testers to prepare and configure the SUT.
In the current OVP version (Danube release time of Dovetail), the SUT consists of NFVI software, VIM software, and necessary hardware in one System Under Test. The hardware should follow Pharos guidelines (but the guideline is generalized in the Dovetail documentation to remove specifics tied to OPNFV CI/CD environment). Vendors can use their own hardware, or third party or white box hardware to be tested as a whole. And if the combined whole passes the Test Suite, the OVP label applies to the software.
In the Danube cycle, OVP does not plan to test hardware-only systems.
Test Suite
A Test Suite is a collection of test cases grouped together for a specific purpose, for example, OVP. A Test Suite can be named and versioned for easy and accurate reference. The Dovetail test tool primarily supports the Compliance Test Suite for OVP, but can in general support multiple test suites.
Compliance Test Suite
A Compliance Test Suite is the Test Suite used in the OPNFV OVP. By default, this is the Test Suite we are referring to. The Compliance Test Suite is to be versioned with MAJOR.MINOR.PATCH together with the Dovetail tool.
Test Area
A Test Area is a collection of test cases grouped together by functional areas or upstream projects. For example, Openstack Refstack/Defcore, IPv6, HA, SDNVPN.
Test Case
A Test Case is an instance of test unit within a Test Area. In some implementation of the test cases, we may see additional sub-test-cases as smaller units under a test case.
OVP (or Dovetail) Testing
The following diagram help illustrates the main components of the testing system for OVP.
Tester
The tester is a knowledgeable engineer of the vendor that has admin right to the SUT within its lab or administrative domain. We use the term in abstract sense, not necessarily meaning one individual. The tester must have sufficient admin privilege to make all hardware and software configurations according to the Dovetail Tester's Guide.
The tester must also have login/access rights established with the Linux Foundation (OPNFV) to access the OVP server to review test results.
Reviewer
The reviewer is a knowledgeable engineer to review test results submitted by vendors to determine if OVP requirements have been met and OPNFV Verified label should be granted. Again we use the term in abstract sense, not necessarily meaning one individual. The C&C committee administers review process.
The reviewer must have login/access ID established with the Linux Foundation (OPNFV) to access the OVP server to review test results.
OVP Server
The OVP server is the backend system (currently consists of a web frontend and a database backend) hosted by the Linux Foundation on behalf of OPNFV. When a test suite is executed, test results are submitted and saved in the OVP Server's database. The OVP Server can be accessed via a Test API using Linux Foundation credentials (to be finalized).
OVP Client (Dovetail client or command-line interface)
The OVP Client, or Dovetail client or Dovetail CLI, is the software that runs in a vendor's host to drive the automated OVP test suite. the Tester who executes the test suite must have the valid credentials of the OVP server in order to store and review test results.
The OVP Client comes with two forms: either as a Dorker container, or as a software install (based on Python). Dovetail Tester's Guide contains detailed guides on how to execute the test suite using the Dovetail client CLI.