The TSC discussed the Dovetail (Danube) Documentation for Review on May 30th. There was no clear conclusion from the meeting but there were several comments. The purpose of this page is to collect the comments for further TSC discussion.
Key point from the discussion: for each test area we need at least some verification of system behavior (beyond pure API verification).
...
- ETSI TST is working on creating test case specifications which can be included in Dovetail. They are not ready yet
- OpenStack Refstack (and Tempest) test cases test the OpenStack API, they could be extended to check the functionality also (example: create a couple of VMs are check that traffic is passing between them)
- Open-O based their testing on scenarios; Dovetail could follow this example
- We agreed on the C&C committee and in DoveTail that we would test per functional area. If we want to propose this change I suggest it be done as a request to the C&C committee with clear argumentation why it serves the industry better than the current approach.
TSC Feedback to Dovetail
Key point from the discussion: for each test area we need at least some verification of system behavior (beyond pure API verification).
Things to do before TSC can approve:
- Have all ongoing Gerrit reviews merged, documents finalized (no editor's notes, open actions etc.)
- Move HA tests to mandatory
- For information: Current HA suite kills OpenStack service process and verifies continuous API availability
- Make Functest's vPing a mandatory test
- Report to TSC that we do not consider additional feature projects ready for compliance verification - show Wenjing's slide
- Ask PTLs to speak up if they disagree
- (Dovetail-internal action): Provide an updated list of documents Dovetail considers as Danube release artefacts
...
...
E-Release work items:
- Include Doctor as optional or mandatory?
- Include Models (the OPNFV project) test cases (launch a sample multi-VM VNF)
- We need to check these against the test case requirements
- incroporate more OPNFV feature projects (e.g. SFC, Doctor, etc.) in Dovetail
List of documents collected at the Paris hackfest:
Currently available documents:
- Test strategy & overview document:
- Test scope and test case specifications draft (each patch has a test area's spec and a patch to test scope file)
- Test case requirements draft
Test scope comments:
- Security: excluded for the time being as no good way to test
- MANO: excluded since not mature yet
- SDN controllers: tested through Neutron APIs, not tested directly
- HA testing could include also SDN controller components [timirnich: this is not available in Danube (for ODL, don't know for ONOS or OpenContrail)]
- For IPv6 networks we create we should validate they are able to carry traffic. (Ping is maybe not the right way to present this)
- There might be "experimental" tests that are not part of CVP
- Are these not simply part of our testing work? Candidate tests and test suites should always be in the pipepline.
Other information:
...
- can be included in Dovetail. They are not ready yet
- OpenStack Refstack (and Tempest) test cases test the OpenStack API, they could be extended to check the functionality also (example: create a couple of VMs are check that traffic is passing between them)
- Open-O based their testing on scenarios; Dovetail could follow this example
- We agreed on the C&C committee and in DoveTail that we would test per functional area. If we want to propose this change I suggest it be done as a request to the C&C committee with clear argumentation why it serves the industry better than the current approach.