Reviewers must follow the below checklist to ensure review consistency for the OPNFV Danube release at a minimum.
- Mandatory Test Areas - all mandatory test areas are present
- Test-Case Count within Test Area - total number of test-cases are present in each test area
- Test-Case Pass Percentage - all tests have passed (100% pass rate)
- Log File Verification - inspect the log file for each test area (osinterop, ha, vping)
- SUT Information Verification - TBA when Dovetail cvp.0.9.0 is available
1. Mandatory Test Areas
Validate that all mandatory test areas are included in the overall test suite. The required mandatory test areas are:
- osinterop
- vping
- ha
Login to the CVP portal at:
Click on the 'My Results' tab in top-level navigation bar.
The CVP administrator will ask for review volunteers using the cvp@opnfv.org email alias. The incoming results for review will be identified by the administrator with a particular 'Test ID', and 'Owner' and will have a status of 'review'.
In the example above, this information will be provided as:
- Test ID: a00c47e8
- Owner: jtaylor
Click on the hyperlink within the 'Test ID' column.
* Note, that the 'Test ID' column in this view condenses the UUID used for 'Test ID' to 8 characters even though the 'Test ID' is a longer UUID in the back-end.
The 'Test ID' hyperlink toggles the view to a top-level listing of the results displayed above. Validate that osinterop, vping and ha test areas are all present within the view.
2. Test-Case Count within Test Area
Validate the test-case count within each test area. For the Danube release, this must break down as follows:
| Mandatory Test Area | Test-Case Count |
|---|---|
| osinterop | 205 |
| ha | 8 |
| vping | 2 |
| Total | 215 |
In the diagram above (from section 1), these counts can be gleaned from the numbers to the right of the test-cases. The total number is given for the osinterop test area at 205. The vping (dovetail.vping.tc001, dovetail.vping.tc002) and ha test-cases are broken down separately with a line for each test-case. Directly above the 'Test Result Overview' listing there's a summary labelled 'Test Run Results' shown below.
The above output can serve as another validation. For Danube, a mandatory total of 215 test-cases must be present (205+8+2).
An example of a listing that should flag a negative review is shown above. This overall output contains only 214 test-cases, which is missing one of the ha test-cases.
3. Test-Case Pass Percentage
All mandatory test-cases must pass. This can be validated in multiple ways. The below diagram of the 'Test Run Results' is one method and shows that 100% of the mandatory test-cases have passed. This value must not be lower than 100%.
Another method to check that all mandatory test-cases have passed is shown in the diagram below. The pass/total is given as a fraction and highlighted here in yellow. For the osinterop test area, the result must display [205/205] and [1/1] for each of the test-cases under the vping and ha test areas.
4. Log File Verification
Three log files must be verified for content within each mandatory test area. The log files for each of the test areas is noted in the table below.
| Mandatory Test Area | Log File | Location |
|---|---|---|
| osinterop | dovetail.osinterop.tc001.log | results/osinterop_logs |
| ha | yardstick.log | results/ |
| vping | functest.log | results/ |
The three log files can be displayed by clicking on the setup icon to the right of the results, as shown in the diagram below.
* Note, while the vping and ha test areas list multiple test-cases in the below diagram, there is a single log file for all test-cases within these test areas.
Within the osinterop log (dovetail.osinterop.tc001.log), scroll down to the area of the log that begins to list the results of each test-case executed. This can be located by looking for lines prefaced with 'tempest.api' and ending with '... ok'.
| osinterop test-case prefix | Count |
|---|---|
| tempest.api.compute | 75 |
| tempest.api.identity | 4 |
| tempest.api.image | 20 |
| tempest.api.network | 46 |
| tempest.api.volume | 60 |
| Total | 205 |
The number of lines within the osinterop log for test-cases must add up according to the table above, where test-cases are broken down according to compute, identity, image, network and volume, with respective counts given in the table.
The ha log (yardstick.log) must contain the 'PASS' result for each of the eight test-cases within this test area. This can be verified by searching the log for the keyword 'PASS'.
The eight lines to validate are listed below:
- 2017-10-16 05:07:49,158 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 INFO The HA test case PASS the SLA
- 2017-10-16 05:08:31,387 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 INFO The HA test case PASS the SLA
- 2017-10-16 05:09:13,669 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 INFO The HA test case PASS the SLA
- 2017-10-16 05:09:55,967 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 INFO The HA test case PASS the SLA
- 2017-10-16 05:10:38,407 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 INFO The HA test case PASS the SLA
- 2017-10-16 05:11:00,030 yardstick.benchmark.scenarios.availability.scenario_general scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m
- 2017-10-16 05:11:22,536 yardstick.benchmark.scenarios.availability.scenario_general scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m
- 2017-10-16 05:12:07,880 yardstick.benchmark.scenarios.availability.scenario_general scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m
The final validation is for the vping test area log file (functest.log). The two entries displayed in the diagrams below must be present in this log file.
- vping_userdata
- vping_ssh
