Versions Compared

Old Version 3

changes.mady.by.user Former user

Saved on

compared with

New Version 4

changes.mady.by.user Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Brainstorming (see below). A summary will then be presented to C&C for sync up, and to TSC, Board for review and approval.
  2. Propose new features or tasks by creating JIRA tickets with fixVersion "Dovetail Danube". For test suite related tasks (as opposed to software code related), also add "Testarea" or "Testcase" label as appropriate. Future JIRA tickets not intended for Danbue Danube cycle should use fixVersion "Future Release".
  3. Review in weekly meetings, revise JIRA tickets as needed.
  4. Identify priority, due date, and assignee.
  5. Describe the main specifications in a rst file in order to have a further technical discussion on Gerrit.

Brainstorming

This is an etherpad a wiki for discussions on Dovetail Danube plan. All inputs must be completed by March 9 ASAP. .We will review on weekly calls on March 3 and 10.

Background:
    The OPNFV board approved the CVP governance document in Dec 2016 with an intent to launching a CVP program and had additional discussions on CVP in the Feb 2017 meeting. They asked the technical community to come back with a plan based on Danube for approval with intended launch mid-this year. The time urgency is part of the Board's ask. 
    This etherpad wiki is a high-level plan, not agreements on all details, but what we need to agree to the core questions to have a clear actionable plan in high level.
    When completed, the content of this etherpad wiki will then be summarized to present to the TSC for technical review and the Board for approval.

...

DRAFT Plan: Please add comments/edits below
        
0)
  1. Timeline:
 

    ...

      1. Project planning complete March 10. Submit to TSC and Board for review/feedback. Test strategy doc by end of April.

    ...

      1. Launch announcement in OPNFV Summit (

    ...

      1. 12-15 June). Vendor participation. Tutorial & training. 

    ...

      1. Danube release: end of June (~Danube + 3 mon).

    ...

      1. Launch the CVP program: ~July.

    ...

      1. Draft milestones timetable: see below 

    ...

      1. "Project Milestones and Due Dates"

    ...

    1. General strategic approach to "Danube Qualification Testing" for CVP:

      ...

        1. Scope: Danube qualification testing will qualify compliance to OPNFV Danube release and will test OPNFV as a software platform, i.e. a well-defined set of components with associated interfaces,

      ...

        1. behaviors and features.

      ...

        1.             Danube qualification testing will include interface compliance testing, capability compliance testing, feature compliance testing of the OPNFV platform.

      ...

        1.                
                      The OPNFV Danube platform is defined as the sum of all the scenarios participating in OPNFV Danube release. Note that simple interface compliance testing is not seen as sufficient for Danube platform qualification.

      ...

        1. OPNFV brand value through Danube Qualification Testing: Danube qualification testing is to ensure that all values provided by the OPNFV Danube release are covered. The Danube qualification testing is to set a "high bar" for passing qualification tests, to ensure OPNFV Danube is understood as high-value.

      ...

        1. Controlled process: The Danube qualification testing needs to ensure that test results are objective, repeatable, well-documented and cannot be modified or cheated on. Testing and test

      ...

        1. results documentation needs to be carried out by an entity independent from the party providing the system under test.

      ...


        1. [Wenjing]: Refer to bullet

      ...

        1. b for scope, and refer to value or high level approach discussions in C&C committee. Here we focus on project planning of dovetail given those guidances.

      ...

      1.  The SUT for Danube cycle CVP is:
                  a)
            1. Products from vendors (not OPNFV release artifacts themselves). Development tooling, such as CI/CD, is not part of the SUT. The vendors can bring up the SUT to a pre-Dovetail-test state in any way they choose (and Dovetail will provide documentation to help do so)
                  b)
            1. The SUT can consist of hardware, NFVI software, and VIM software in one System Under Test. Or, a software only vendor can use third party or white box hardware to be tested as a whole and if the combined whole passes the test suite, the CVP label applies to the software. The Danube cycle does not plan to test hardware-only systems.

        ...

          1. To conduct the test (self test or third party test), the tester needs to compose the System Under Test (hardware, NFVI, and VIM) and bring it to the pre-Dovetail-test state. The tester then

        ...

          1. conducts the Dovetail tests using its automation tool which will report the result directly to a database managed by OPNFV.
                

        ...

        1.  Test strategy document - blueprint of what to test, how to test

          ...

            1. links to Bryan's etherpads: https://etherpad.opnfv.org/p/dovetail-principles,  https://etherpad.opnfv.org/p/dovetail-priorities

          ...

            1. publish by end of April

            ...

                1. Defines scope of what should be tested, and what is out of scope

            ...

                1.  Refers to how tests are executed, what types of tests are executed

            ...

                1.  Defines the test environment, and prerequisites for SUT being tested

            ...

                1. Specifies tools to be used for the test process

            ...

                1. https://en.wikipedia.org/wiki/Test_strategy  <- :D

            ...

                1. How (and why) to create one: http://www.guru99.com/how-to-create-test-strategy-document.html

                  [Wenjing]: in order to finalize a plan

            ...

                1. by March 10, we need the main outline of the strategy document (and the main issues) agreed to by March 10 also. Not all details, but main ones that can impact the credibility of the plan.
            2)
            1. Scope and precise details of the Danube test suite
            is
            1. are to be determined by the following process:
                a)
                  1. Dovetail scope is a subset of functionalities contained in the OPNFV reference platform. When we say Danube based, it further narrows down to the functionalities found in Danube release cycle.  In other words, this sets the upper bound. 
                b)
                  1. C&C has further reduced the scope for Danube cycle to be NFVI+VIM, and performance testing is currently out of scope.
                    *
                        1. Plug-in C&C
                    addentum
                        1. addendum on Danube scope. 2/27?
                    c)
                      1. A set of criteria
                    are
                      1. is to be met by the test cases to be included in the Dovetail test suite. A draft of this set of criteria is in the wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Case+Requirements.  A draft superset of all possible tests
                    are
                      1. is currently maintained in the wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases. However, we are taking a fresh look again with a new work/review process, tracked by Jira, to determine how to organize the test suite, what test areas and what test cases for each test area should be included in Danube
                    cvp
                      1. CVP. The Jira ticket is
                    : 
                      1. https://jira.opnfv.org/browse/DOVETAIL-345. Once this work is concluded, the wiki's will be refreshed to reflect its outcome. 
                    d)
                      1. This process is open to all in the community and consensus based. Participation is on Jira (and aided by etherpads as needed) and dovetail calls publicized in the tech-discuss mailing list. In specific cases where consensus can't be reached by this process, requirement/scope issues go to C&C committee/Board, and technical issues go to TSC for determination.
                    3) Dovetail
                    1.  Dovetail will deliver all toolings, documentations and the test suite based on Danube, with the end of June as the final release date. Deliverables include:
                        a)
                          1. standalone test automation tooling

                  ...

                    1. an official test suite for

                  ...

                    1. CVP (as the result of 2d process), an extended test suite for feedback or experimental review (ie useful/desirable for other benefits but not accepted by the 2d process)
                  c)
                    1. documentation that includes detailed guides on how to prepare and conduct the Dovetail test, pass criteria, report/review process.
                  documentation
                    1. Documentation that includes a summary description of test areas and test cases with links to online details.
                  d)
                    1. the above test tooling and test suites and docs must be validated using
                  opnfv
                    1. OPNFV and product references. (ci process, open test/review process, vendor beta process)

                  ...

                    1. hosting the dovetail test result database, and any additional work needed to enable

                  ...

                    1. CVP administrative process

                  ...

                    1. need to make quick design decisions on: (i) test result in databases (vs. in files) (ii) use refstack directly is feasible in Danube (vs. later in E).
                  4) Working
                  1.  Working with upstream
                      a)
                        1. We are currently evaluating to work with
                      Openstack
                        1. OpenStack Interop WG/Defcore by using refstack tooling for
                      Openstack
                        1. OpenStack testing needs within dovetail. Jose/Howard had a meeting during Atlanta PTG on this topic and
                      Openstack
                        1. OpenStack community seems to be very supportive. Working with upstream helps streamline work and leverage expertise. The current outstanding action item is to have the quick evaluation to help us determine (i) if this is the right direction (ii) if yes, can we deliver this in time for Danube cycle? If time does not permit us doing so in Danube cycle, we will roadmap it to E.
                      b)
                        1. Longer term plan for other upstream projects should follow the roadmap process below.
                      5) Long
                      1.  Long term planning of roadmap
                          a)
                            1. Desired features for dovetail, if not possible/accepted in Danube, should be
                          roadmapped
                            1. road mapped using Jira enhancement tickets. Features targeted for E release should start immediately in order to influence E release planning that is
                          onging
                            1. ongoing in the community.

                      ...

                        1. EUAG and other user inputs should be

                      ...

                        1. incorporated into the roadmap using the same Jira backlog/to-do ticket mechanism. (Write a story for each request)

                      ...

                        1. Dovetail's E cycle planning will be based on these Jira backlog/to-do tickets. Use fixVersion "Future Release" instead of "Dovetail Danube".

                      Project Milestones and Due Dates

                      ...

                      MilestonesFeb 2017March 2017April 2017May 2017June 2017July 2017

                      NOTE: Milestones below does do NOT mean project tasks are sequential. Most will be in progress in parallel.

                      		Preparation     

                      (M1)

                      High level plan of Dovetail completed & reviewed by TSC/Board, C&C addendum completed, reviewed, ready for decision

                      		 March 3&10 - completed in Dovetail
                      March 6 - review with C&C
                      March 14 - TSC review
                      March 20 - Final C&C, to Board                      		    

                      (M2)

                      Plan in Jira with epics defined & owners assigned. Expected completion dates committed.

                      		 March 24    

                      (M3)

                      Dovetail tool software available for Alpha testing by volunteers in Pharos and/or vendor and/or third party tester labs. (Note: this can be using Colorado without dependency with Danube or finalized test plan.)

                      		 March 31    

                      (M4)

                      CVP review/approval

                      		  April 3-6
                      (ONS)                      		   

                      (M5)

                      Test strategy details reviewed, finalized and documented. Test tool migrated to Danube, all design decisions finalized, Tool software completed in a CI for validation. Demo/review in PlugFest. All open decision points closed/finalized.

                      		  

                      April 24-28

                      (PlugFest/HackFest)

                      		   

                      (M6)

                      Test areas and Test cases list ( i.e. the CVP test suite) finalized. Dovetail software linked with the right test suite and frozen. Draft user docs completed. Bug fixes only from this point on.

                      		   May 26  

                      (M7)

                      Beta ready. Test suite/software/doc beta ready. LF/C&C/workflow and tools in place. Test The test suite and tool presentation/demo in OPNFV Summit. CVP announcement.

                      		    June 12-16
                      (OPNFV Summit)                      		 

                      (M8)

                      User trial by vendors and test labs completed. Feedbacks/bugs collected. Final bug JIRA tickets.

                      		    June 30 

                      (M9)

                      Final release of all deliverable

                      		     July 14

                      (M10)

                      CVP launch

                      		     end of July
                             
                             


                      Proposed JIRA Tasks (fixVersion "Dovetail Danube"):

                      To be updated by on March 10. It is not accurate nor complete at this time.

                      ...