Overview Illustration

System Under Test (SUT)

The SUT are products from vendors (not OPNFV release artifacts themselves). Development tooling, such as CI/CD, is not part of the SUT. The vendors can bring up the SUT to a pre-Dovetail-test state in any way they choose. Dovetail will provide documentation to help do so.

The SUT consists of NFVI software, VIM software, and necessary hardware in one System Under Test. The hardware should follow Pharos guidelines. Vendors can use their own hardware, or third party or white box hardware to be tested as a whole. And if the combined whole passes the test suite, the CVP label applies to the software.

In the Danube cycle, Dovetail does not plan to test hardware-only systems.

Test Suite

Dovetail Danube scope is a subset of functionalities contained in the OPNFV Danube release. The C&C Committee has further reduced the scope for Danube cycle to be NFVI+VIM on functions and interfaces OVP document. And for example, performance testing is currently out of scope in Danube.

All test cases in the Dovetail test suite must meet Dovetail Test Case Requirements.

Proposals for inclusion of test areas can be made by completing this worksheet Dovetail Test Area / Test Case Worksheet and the associated analysis work. These proposals will then be reviewed in Dovetail project weekly, following the scope OVP document and Dovetail Test Case Requirements.

The work of constructing the Dovetail test suite is tracked in JIRA ticket:  https://jira.opnfv.org/browse/DOVETAIL-345.

The resulting test suite will be defined by compliance_set.yml file, and maintained in wiki for reader's convenience: Dovetail Test Areas and Test Cases. (Warning: the current wiki page ls not yet review and inaccurate.)

Working with Upstream

Openstack Refstack/Defcore https://refstack.openstack.org/#/guidelines (version 2016.8) has been integrated into the Dovetail test suite. Collaborative working relationship is being developed with Openstack Interoperability Working Group on a continuous basis. In future, we may collaborate to develop NFV extensions to refstack, for instance.

Test cases in the refstack guidelines of the version corresponding to the Openstack verion integrated in OPNFV (for Danube it is Newton) are integrated into Dovetail as is (currently it is version 2016.8). While it is possible that, under enexpected circumstances, some test cases may be found unsuitable for execution in OPNFV environment and may have to be skipped, we do not see such cases today and expect them to be rare exceptions. Should such exceptions occur, they would need to be reviewed by Dovetail for particular reasons, documented clearly and agreed to.

We may consider working with other upstream communities in Euphrates in a similar fashion.

Long Term Road Map

Desired features for Dovetail, if not possible or not accepted in Danube, should be road-mapped using Jira enhancement tickets. Features targeted for E release should start immediately in order to influence E release planning that is ongoing in the community.

EUAG and other user inputs should be incorporated into the road map using the same Jira backlog/to-do ticket mechanism. Such request tickets should be written a JIRA story with sufficient detail for the reader/developers to understand clearly. 

Dovetail's E cycle planning will be based on these Jira backlog/to-do tickets. Please use fixVersion "Future Release" instead of "Dovetail Danube".

Deliverables and JIRA Tasks (fixVersion "Dovetail Danube"):

1. A stand-alone Dovetail client software, to be delivered to and used by testers/vendors as
   1. Dorker container
   2. Git clone python source
      Tracked by JIRA tickets: tbd
2. A User Guide for the client software. Tracked by JIRA ticket: DOVETAIL-73.
3. Backend DB and web server software, and web UI software, for administrators and for reviewers and others to access test results. This item is TBD pending on what falls under LF and what falls under Dovetail.
   1. All backend server software and instructions for someone to create and administer the server
   2. Web UI for test result viewers (tbd)
      Tracked by JIRA tickets: tbd
4. Test strategy document: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-meeting/2017/opnfv-meeting.2017-03-17-13.00.log.html /  http://artifacts.opnfv.org/dovetail/review/30811/testing_user_teststrategy/index.html. Tracked by JIRA tickets:  DOVETAIL-352 - Getting issue details... STATUS
5. Finalized Dovetail test suite: 
   1. test requirements wiki
   2. test area/test case list wiki
   3. compliance_set.yml; 
   4. Test suite Document.
      Tracked by JIRA tickets: DOVETAIL-345.
6. QA: testing Dovetail client and server for meeting feature and quality requirements. Tracked by JIRA tickets;  DOVETAIL-180 - Getting issue details... STATUS
7. Bug fixes.
8. Beta testing
9. Release.

To be updated. It is not accurate nor complete at this time.

Release "Dovetail Danube": https://jira.opnfv.org/projects/DOVETAIL/versions/11001

Kanban board for the overall Dovetail Danube release: https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=149&selectedIssue=DOVETAIL-345&quickFilter=371

Kanban board for tasks related test suite (Testarea & Tesecase): https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=147&quickFilter=357 

1. Timeline:
   1. Project planning complete March 10. TSC review on March 21. Submit to TSC and Board for review/feedback. Test strategy doc by end of April.
   2. Launch announcement in OPNFV Summit (12-15 June). Vendor participation. Tutorial & training. 
   3. Danube release: end of June (~Danube + 3 mon). Launch the CVP program: ~July.
   4. Draft milestones timetable: see below "Project Milestones and Due Dates"
      Action: review and agree on Milestone table below
2. General strategic approach to "Danube Qualification Testing" for CVP:
   1. Scope: Danube qualification testing will qualify compliance to OPNFV Danube release and will test OPNFV as a software platform, i.e. a well-defined set of components with associated interfaces, behaviors and features.
                  Danube qualification testing will include interface compliance testing, capability compliance testing, feature compliance testing of the OPNFV platform.                
                  The OPNFV Danube platform is defined as the sum of all the scenarios participating in OPNFV Danube release. Note that simple interface compliance testing is not seen as sufficient for Danube platform qualification.
   2. OPNFV brand value through Danube Qualification Testing: Danube qualification testing is to ensure that all values provided by the OPNFV Danube release are covered. The Danube qualification testing is to set a "high bar" for passing qualification tests, to ensure OPNFV Danube is understood as high-value.
   3. Controlled process: The Danube qualification testing needs to ensure that test results are objective, repeatable, well-documented and cannot be modified or cheated on. Testing and test results documentation needs to be carried out by an entity independent from the party providing the system under test.
      
      [Wenjing]: Refer to bullet b for scope, and refer to value or high level approach discussions in C&C committee. Here we focus on project planning of dovetail given those guidances.
      Action: None. See OVP document
3.  The SUT for Danube cycle CVP is:
   1. Products from vendors (not OPNFV release artifacts themselves). Development tooling, such as CI/CD, is not part of the SUT. The vendors can bring up the SUT to a pre-Dovetail-test state in any way they choose (and Dovetail will provide documentation to help do so)
   2. The SUT can consist of hardware, NFVI software, and VIM software in one System Under Test. Or, a software only vendor can use third party or white box hardware to be tested as a whole and if the combined whole passes the test suite, the CVP label applies to the software. The Danube cycle does not plan to test hardware-only systems.
   3. To conduct the test (self test or third party test), the tester needs to compose the System Under Test (hardware, NFVI, and VIM) and bring it to the pre-Dovetail-test state. The tester then conducts the Dovetail tests using its automation tool which will report the result directly to a database managed by OPNFV.  (see bullet #4)
      Action: agree on SUT
4.  Test strategy document - blueprint of what to test, how to test
   1. links to Bryan's etherpads: https://etherpad.opnfv.org/p/dovetail-principles,  https://etherpad.opnfv.org/p/dovetail-priorities (already entered into the test case requirement wiki page under brainstorming)
   2. publish by end of April
      1. Defines scope of what should be tested, and what is out of scope
      2.  Refers to how tests are executed, what types of tests are executed
      3.  Defines the test environment, and prerequisites for SUT being tested
      4. Specifies tools to be used for the test process
      5. https://en.wikipedia.org/wiki/Test_strategy  <- :D
      6. How (and why) to create one: http://www.guru99.com/how-to-create-test-strategy-document.html
         
         [Wenjing]: in order to finalize a plan by March 10, we need the main outline of the strategy document (and the main issues) agreed to by March 10 also. Not all details, but main ones that can impact the credibility of the plan.
         Action 1: Summarize the brainstorm information and update criteria in wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Case+Requirements. Publish it. Open call for review.
         Action 2: User facing documentation
5. Scope and precise details of the Danube test suite are to be determined by the following process:
   1. Dovetail scope is a subset of functionalities contained in the OPNFV reference platform(s). When we say Danube based, it further narrows down to the functionalities found in Danube release cycle.  In other words, this sets the upper bound. 
   2. C&C has further reduced the scope for Danube cycle to be NFVI+VIM, and performance testing is currently out of scope.
      1. Plug-in C&C addendum on Danube scope. 2/27?
   3. A set of criteria is to be met by the test cases to be included in the Dovetail test suite. A draft of this set of criteria is in the wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Case+Requirements.  A draft superset of all possible tests is currently maintained in the wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases. However, we are taking a fresh look again with a new work/review process, tracked by Jira, to determine how to organize the test suite, what test areas and what test cases for each test area should be included in Danube CVP. The Jira ticket is https://jira.opnfv.org/browse/DOVETAIL-345. Once this work is concluded, the wiki's will be refreshed to reflect its outcome. Proposal for DOVETAIL-345:
      1. Openstack defcore /refstack integrated in functest
      2. SDN controllers as integrated via Neutron plugins including nosdn, odl, onos, ocl. Test cases driven via Neutron interface. (covers os-nosdn-*-ha, os-odl*-*-ha, os-onos-*-ha and os-ocl-*-ha)
      3. Nfv network basic validation: vPing, vRouter, ...
      4. Nfv use case: vIMS as integrated in functest
      5. HA : as in HA project
      6. IPv6 : as in IPv6 project
      7. others: we can call for proposal, once criteria published
   4. This process is open to all in the community and consensus based. Participation is on Jira (and aided by etherpads as needed) and dovetail calls publicized in the tech-discuss mailing list. In specific cases where consensus can't be reached by this process, requirement/scope issues go to C&C committee/Board, and technical issues go to TSC for determination.
      Action: General agreement on starting point, supporting data needed, and extended review sessions for DOVETAIL-345 until resolved (do we need more than the weekly call time?)
      
      /* next week march 17, we will continue on 6, 7, 8 */
6.  Dovetail will deliver all toolings, documentations and the test suite based on Danube, with the end of June as the final release date. Deliverables include:
   1. standalone test automation tooling
   2. an official test suite for CVP (as the result of 2d process), an extended test suite for feedback or experimental review (ie useful/desirable for other benefits but not accepted by the 2d process)
   3. documentation that includes detailed guides on how to prepare and conduct the Dovetail test, pass criteria, report/review process. Documentation that includes a summary description of test areas and test cases with links to online details.
   4. the above test tooling and test suites and docs must be validated using OPNFV and product references. (ci process, open test/review process, vendor beta process)
   5. hosting the dovetail test result database, and any additional work needed to enable CVP administrative process (eg UI). test the hosting.
   6. need to make quick design decisions on: (i) test result in databases (vs. in files) (ii) use refstack directly is feasible in Danube (vs. later in E). (already in progress)
      Action: agreement on the list and put into JIRA 
      
      
7.  Working with upstream
   1. We are currently evaluating to work with OpenStack Interop WG/Defcore by using refstack tooling for OpenStack testing needs within dovetail. Jose/Howard had a meeting during Atlanta PTG on this topic and OpenStack community seems to be very supportive. Working with upstream helps streamline work and leverage expertise. The current outstanding action item is to have the quick evaluation to help us determine (i) if this is the right direction (ii) if yes, can we deliver this in time for Danube cycle? If time does not permit us doing so in Danube cycle, we will roadmap it to E.
   2. Longer term plan for other upstream projects should follow the roadmap process below.
      Action: pursue Openstack for Euphrates(?)
8.  Long term planning of roadmap
   1. Desired features for dovetail, if not possible/accepted in Danube, should be road mapped using Jira enhancement tickets. Features targeted for E release should start immediately in order to influence E release planning that is ongoing in the community.
   2. EUAG and other user inputs should be incorporated into the roadmap using the same Jira backlog/to-do ticket mechanism. (Write a story for each request)
   3. Dovetail's E cycle planning will be based on these Jira backlog/to-do tickets. Use fixVersion "Future Release" instead of "Dovetail Danube".
      Action: all inputs should be entered in JIRA for dovetail consideration
      

Previous Dovetail Danube planning etherpad:

https://etherpad.opnfv.org/p/Dovetail_Danube_Planning