Process
- assignee.
- Describe the main specifications in a rst file in order to have a further technical discussion on Gerrit.
Overview Illustration
System Under Test (SUT)
The SUT are products from vendors (not OPNFV release artifacts themselves). Development tooling, such as CI/CD, is not part of the SUT. The vendors can bring up the SUT to a pre-Dovetail-test state in any way they choose. Dovetail will provide documentation to help do so.
The SUT consists of NFVI software, VIM software, and necessary hardware in one System Under Test. The hardware should follow Pharos guidelines. Vendors can use their own hardware, or third party or white box hardware to be tested as a whole. And if the combined whole passes the test suite, the CVP label applies to the software.
In the Danube cycle, Dovetail does not plan to test hardware-only systems.
Test Suite
Dovetail Danube scope is a subset of functionalities contained in the OPNFV Danube release. The C&C Committee has further reduced the scope for Danube cycle to be NFVI+VIM on functions and interfaces CVP OVP document. And for example, performance testing is currently out of scope in Danube.
All test cases in the Dovetail test suite must meet Dovetail Test Case Requirements.
Proposals for inclusion of test areas can be made by completing this worksheet Dovetail Test Area / Test Case Worksheet and the associated analysis work. These proposals will then be reviewed in Dovetail project weekly, following the scope CVP OVP document and Dovetail Test Case Requirements.
The work of constructing the Dovetail test suite is tracked in JIRA ticket:
We may consider working with other upstream communities in Euphrates in a similar fashion.
Desired features for Dovetail, if not possible or not accepted in Danube, should be road-mapped using Jira enhancement tickets. Features targeted for E release should start immediately in order to influence E release planning that is ongoing in the community.
EUAG and other user inputs should be incorporated into the road map using the same Jira
Dovetail's E cycle planning will be based on these Jira
Deliverables and JIRA Tasks (fixVersion "Dovetail Danube"):
- A stand-alone Dovetail client software, to be delivered to and used by testers/vendors as
- Dorker container
- Git clone python source
Tracked by JIRA tickets: tbd
- A User Guide for the client software. Tracked by JIRA ticket: DOVETAIL-73.
- Backend DB and web server software, and web UI software, for administrators and for reviewers and others to access test results. This item is TBD pending on what falls under LF and what falls under Dovetail.
- All backend server software and instructions for someone to create and administer the server
- Web UI for test result viewers (tbd)
Tracked by JIRA tickets: tbd
- Test strategy document: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-meeting/2017/opnfv-meeting.2017-03-17-13.00.log.html / http://artifacts.opnfv.org/dovetail/review/30811/testing_user_teststrategy/index.html. Tracked by JIRA tickets:
Jira Legacy server OPNFV JIRASystem Jira serverId 96acfcf21afe526e-db1a48e5-385933b1-891e8ed7-03a53e9315b04f559eac1ef8 key DOVETAIL-352 - Finalized Dovetail test suite:
- test requirements wiki
- test area/test case list wiki
- compliance_set.yml;
- Test suite Document.
Tracked by JIRA tickets: DOVETAIL-345.
- QA: testing Dovetail client and server for meeting feature and quality requirements. Tracked by JIRA tickets;
Jira Legacy server OPNFV JIRASystem Jira serverId 96acfcf21afe526e-db1a48e5-385933b1-891e8ed7-03a53e9315b04f559eac1ef8 key DOVETAIL-180 - Bug fixes.
- Beta testing
- Release.
To be updated. It is not accurate nor complete at this time.
Release "Dovetail Danube": https://jira.opnfv.org/projects/DOVETAIL/versions/11001
Kanban board for the overall Dovetail Danube release: https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=149&selectedIssue=DOVETAIL-345&quickFilter=371
Kanban board for tasks related test suite (Testarea & Tesecase): https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=147&quickFilter=357
Project Milestones and Due Dates
Proposed rough timeline :
- Project planning complete March 10. TSC review on March
2128. Submit to TSC and Board for review/feedback. Test strategy doc by end of April. - Launch announcement in OPNFV Summit (
- Danube release: end of June (~Danube + 3 mon).
- Launch the CVP program: ~July.
Milestones | Feb 2017 | March 2017 | April 2017 | May 2017 | June 2017 | July 2017 |
---|---|---|---|---|---|---|
NOTE: Milestones below do NOT mean project tasks are sequential. Most will be in progress in parallel. | Preparation | |||||
(M1) High level plan of Dovetail completed & reviewed by TSC/Board,
| March 3&10 - completed in Dovetail March 6 - review with C&C March 14 - TSC review March 20 - Final C&C, to Board | |||||
(M2) Plan in Jira with epics defined & owners assigned. Expected completion dates committed. | March 24 | |||||
(M3) Dovetail tool software available for Alpha testing by volunteers in Pharos and/or vendor and/or third party tester labs. (Note: this can be using Colorado without dependency with Danube or finalized test plan.) | March 31 | |||||
(M4) CVP review/approval | April 3-6 (ONS) | |||||
(M5) Test strategy details reviewed, finalized and documented. Test tool migrated to Danube, all design decisions finalized, Tool software completed in a CI for validation. (2 installers) Server side workfow draft system ready for test Demo/review in PlugFest. All open decision points (except test case review) closed/finalized. | April 24-28 (PlugFest/HackFest) | |||||
(M6) Test areas and Test cases list ( i.e. the CVP test suite) finalized. Dovetail software linked with the right test suite and frozen. Draft user docs completed. Bug fixes only from this point on. C&C addendum completed, reviewed, ready for decision. | May 26 | |||||
(M7) Beta ready. Test suite/software/doc beta ready. LF/C&C/workflow and tools in place. The test suite and tool presentation/demo in OPNFV Summit. CVP announcement. Tutorials. | June 12-16 (OPNFV Summit) | |||||
(M8) User trial by vendors and test labs completed. Feedbacks/bugs collected. Final bug JIRA tickets. | June 30 | |||||
(M9) Final release of all deliverable | July 14 | |||||
(M10) CVP launch | end of July | |||||
Proposed JIRA Tasks (fixVersion "Dovetail Danube") for deliverables:
To be updated on March 10. It is not accurate nor complete at this time.
...
---------------------------
Brainstorming
...
- Timeline:
- Project planning complete March 10. TSC review on March 21. Submit to TSC and Board for review/feedback. Test strategy doc by end of April.
- Launch announcement in OPNFV Summit (
- Danube release: end of June (~Danube + 3 mon). Launch the CVP program: ~July.
- Draft milestones timetable: see below
- General strategic approach to "Danube Qualification Testing" for CVP:
Danube qualification testing will include interface compliance testing, capability compliance testing, feature compliance testing of the OPNFV platform.
The OPNFV Danube platform is defined as the sum of all the scenarios participating in OPNFV Danube release. Note that simple interface compliance testing is not seen as sufficient for Danube platform qualification.- Controlled process
- The SUT for Danube cycle CVP is:
- Products from vendors (not OPNFV release artifacts themselves). Development tooling, such as CI/CD, is not part of the SUT. The vendors can bring up the SUT to a pre-Dovetail-test state in any way they choose (and Dovetail will provide documentation to help do so)
- The SUT can consist of hardware, NFVI software, and VIM software in one System Under Test. Or, a software only vendor can use third party or white box hardware to be tested as a whole and if the combined whole passes the test suite, the CVP label applies to the software. The Danube cycle does not plan to test hardware-only systems.
To conduct the test (self test or third party test), the tester needs to compose the System Under Test (hardware, NFVI, and VIM) and bring it to the pre-Dovetail-test state. The tester then conducts the Dovetail tests using its automation tool which will report the result directly to a database managed by OPNFV.(see bullet #4)
Action: agree on SUT
- Test strategy document - blueprint of what to test, how to test
links to Bryan's etherpad- publish by end of April
- Defines scope of what should be tested, and what is out of scope
- Refers to how tests are executed, what types of tests are executed
- Defines the test environment, and prerequisites for SUT being tested
- Specifies tools to be used for the test process
- How (and why) to create one:
[Wenjing]: in order to finalize a plan by March 10, we need the main outline of the strategy document (and the main issues) agreed to by March 10 also. Not all details, but main ones that can impact the credibility of the plan.
Action 1: Summarize the brainstorm information and update criteria in wiki:
- Scope and precise details of the Danube test suite are to be determined by the following process:
- Dovetail scope is a subset of functionalities contained in the OPNFV reference platform(s). When we say Danube based, it further narrows down to the functionalities found in Danube release cycle. In other words, this sets the upper bound.
- C&C has further reduced the scope for Danube cycle to be NFVI+VIM, and performance testing is currently out of scope.
- Plug-in C&C addendum on Danube scope. 2/27?
A set of criteria is to be met by the test cases to be included in the Dovetail test suite. A draft of this set of criteria is in the wiki:- Nfv network basic validation: vPing, vRouter, ...
- Nfv use case: vIMS as integrated in functest
- HA : as in HA project
- IPv6 : as in IPv6 project
- others: we can call for proposal, once criteria published
- This process is open to all in the community and consensus based. Participation is on Jira (and aided by etherpads as needed) and dovetail calls publicized in the tech-discuss mailing list. In specific cases where consensus can't be reached by this process, requirement/scope issues go to C&C committee/Board, and technical issues go to TSC for determination.
Action: General agreement on starting point, supporting data needed, and extended review sessions for DOVETAIL-345 until resolved (do we need more than the weekly call time?)
/* next week march 17, we will continue on 6, 7, 8 */
- Dovetail will deliver all toolings, documentations and the test suite based on Danube, with the end of June as the final release date. Deliverables include:
- standalone test automation tooling
- an official test suite for CVP (as the result of 2d process), an extended test suite for feedback or experimental review (ie useful/desirable
- documentation that includes detailed guides on how to prepare and conduct the Dovetail test, pass criteria, report/review process. Documentation that includes a summary description of test areas and test cases with links to online details.
- the above test tooling and test suites and docs must be validated using OPNFV and product references. (ci
- hosting the dovetail test result database, and any additional work needed to enable CVP administrative process (eg UI). test the hosting.
need to make quick design decisions on: (i) test result in databases (vs. in files) (ii) use refstack directly is feasible in Danube (vs. later in E). (already in progress)
Action: agreement on the list and put into JIRA
- Working with upstream
- We are currently evaluating to work with OpenStack Interop WG/Defcore by using refstack tooling for OpenStack testing needs within dovetail. Jose/Howard had a meeting during Atlanta PTG on this topic and OpenStack community seems to be very supportive. Working with upstream helps streamline work and leverage expertise. The current outstanding action item is to have the quick evaluation to help us determine (i) if this is the right direction (ii) if yes, can we deliver this in time for Danube cycle? If time does not permit us doing so in Danube cycle, we will roadmap it to E.
- Longer term plan for other upstream projects should follow the roadmap process below.
Action: pursue Openstack for Euphrates(?)
- Long term planning of roadmap
- Desired features for dovetail, if not possible/accepted in Danube, should be road mapped using Jira enhancement tickets. Features targeted for E release should start immediately in order to influence E release planning that is ongoing in the community.
- EUAG and other user inputs should be incorporated into the roadmap using the same Jira
- Dovetail's E cycle planning will be based on these Jira