Status of proposed tests

2018-06-25

Mandatory Tests

Installer

Scenario

vping_userdata

vping_ssh

stress.ping

tempest.osinterop

tempest.compute

tempest.identity_v2

tempest.identity_v3

tempest.image

tempest.network_api

tempest.volume

tempest.trunk_ports

tempest.ipv6_api

security.patrole

ha.nova_api

ha.neutron_server

ha.keystone

ha.glance_api

ha.cinder_api

ha.cpu_load

ha.disk_load

ha.haproxy

ha.rabbitmq

ha.database

Apex

nosdn-nofeature

fraser

PASS

FAIL

DOVETAIL-662 - Getting issue details... STATUS

PASS

not supported

FAIL

DOVETAIL-730 - Getting issue details... STATUS

FAIL

DOVETAIL-730 - Getting issue details... STATUS

FAIL

DOVETAIL-730 - Getting issue details... STATUS

FAIL

DOVETAIL-730 - Getting issue details... STATUS

FAIL

DOVETAIL-730 - Getting issue details... STATUS

FAIL

DOVETAIL-730 - Getting issue details... STATUS

FAIL

DOVETAIL-730 - Getting issue details... STATUS

PASS

FAIL

DOVETAIL-728 - Getting issue details... STATUS

PASS

FAIL

DOVETAIL-728 - Getting issue details... STATUS

PASS

FAIL

HA proxy processes don't recover

FAIL

DOVETAIL-728 - Getting issue details... STATUS

FAIL

DOVETAIL-728 - Getting issue details... STATUS

Compass

nosdn-nofeature

fraser

PASS

FAIL

cannot connect to 192.168.11.xxx

FAIL

DOVETAIL-727 - Getting issue details... STATUS

FAIL

total: 200

DOVETAIL-662 - Getting issue details... STATUS

PASS

not supported

PASS

FAIL

DOVETAIL-728 - Getting issue details... STATUS

PASS

FAIL

DOVETAIL-728 - Getting issue details... STATUS

PASS

FAIL

DOVETAIL-728 - Getting issue details... STATUS

FAIL

DOVETAIL-728 - Getting issue details... STATUS

Fuel

nosdn-nofeature

master

PASS

not supported

PASS

FAIL

SLA failure:
10.725534 > 5.000000

UNSTABLE

most of the time the test detects 11 keystone processes of which only 9 recover

Is the keystone process query correct?

PASS

FAIL

SLA failure:
12.149971 > 5.000000

PASS

FAIL

SLA failure:
12.336285 > 5.000000

rabbitmq-server not found
in the host environment

FAIL

SLA failure: 13.879296 > 5.000000

2018-06-25

Installer	Scenario	stress.ping	tempest.compute	tempest.identity_v2	tempest.identity_v3	tempest.image	tempest.network_api	tempest.volume	tempest.trunk_ports	security.patrole	ha.controller_restart	ha.rabbitmq	ha.database	ha.neutron_l3_agent	tempest.network_scenario	vnf.vEPC	vnf.vIMS	tempest.bgpvpn	snaps.smoke
Apex	nosdn-nofeature stable/fraser	FAIL	PASS	FAIL - only v3 used skip mechanism needed or just part of reviewer instructions (does not need to be automated) DOVETAIL-656 - Getting issue details... STATUS	PASS	PASS	PASS	PASS	PASS	PASS	FAIL IPMI info missing	PASS	PASS	PASS	PASS	FAIL passes on Apex daily loop	PASS	not supported	FAIL test_volume_type_with_encryption test_volume_type_with_qos_and_encryption passes on Apex daily CI loop
	odl stable/fraser	FAIL	FAIL	FAIL - only v3 used skip mechanism needed or just part of reviewer instructions (does not need to be automated)			FAIL	FAIL	FAIL	FAIL	FAIL IPMI info missing	PASS	PASS	not supported: l3 agent not installed	FAIL	FAIL	FAIL	FAIL	FAIL
	nosdn-nofeature master	FAIL	PASS	not supported	PASS	PASS	PASS	PASS	PASS	PASS	FAIL IPMI info missing	PASS	PASS	PASS	PASS	PASS	FAIL VM in error state: "no valid host found" passes on Apex daily CI loop	not supported	FAIL
Compass	nosdn-nofeature stable/fraser	PASS create stack rate 5/5	PASS	SKIP	PASS	PASS	PASS	PASS	PASS skip: 9 pass: 43	PASS skip: 3 (dvr is not enabled) pass: 132 test_create_distributed_router ... skip test_show_distributed_router ... skip test_update_distributed_router ... skip	PASS	PASS	PASS	FAIL SLA failure: 9.023325 > 5.000000	FAIL fail: 5 No IPv4 addresses found	PASS	PASS	not supported	FAIL fail: 1 pass: 120 test_ssh_client_fip_after_reboot ... fail
Compass	odl stable/fraser	PASS create stack rate 5/5	PASS	SKIP	PASS	PASS	PASS	PASS	PASS skip: 9 pass: 43	PASS skip: 5 (l3_agent_scheduler extension not enabled) pass: 130 test_create_router_on_l3_agent ... skip test_delete_router_from_l3_agent ... skip test_list_routers_on_l3_agent ... skip test_create_high_availability_router ... skip test_update_high_availability_router ... skip	PASS	PASS	PASS	not supported: l3 agent not installed	FAIL fail: 2 (False is not true : ip address 192.168.11.114 is reachable) test_update_instance_port_admin_state test_update_router_admin_state	PASS	PASS	not supported	FAIL fail: 2 pass: 119 test_create_network_router_admin_user_to_new_project ... fail test_create_network_router_new_user_to_admin_project ... fail
Fuel (x86)	nosdn-nofeature master	PASS	PASS	FAIL - only v3 used skip mechanism needed or just part of reviewer instructions (does not need to be automated) DOVETAIL-656 - Getting issue details... STATUS	PASS	PASS	PASS	PASS	PASS skip: 9 pass: 43	PASS skip: 3 (dvr is not enabled) pass: 132 test_create_distributed_router ... skip test_show_distributed_router ... skip test_update_distributed_router ... skip	PASS	FAIL rabbitmq located in msg0x VMs https://docs.opnfv.org/en/stable-fraser/submodules/fuel/docs/release/installation/installation.instruction.html#automatic-installation-of-a-baremetal-pod	FAIL database located in dbs0x VMs https://docs.opnfv.org/en/stable-fraser/submodules/fuel/docs/release/installation/installation.instruction.html#automatic-installation-of-a-baremetal-pod	FAIL l3-agent in gtw0x VMs https://gerrit.opnfv.org/gerrit/#/c/58333/	FAIL - Success: 3 - Skipped: 0 - Expected failures: 0 - Unexpected success: 0 - Failures: 2 test_update_instance_port_admin_state ... fail test_update_router_admin_state ... fail	FAIL auth problem passes on FUEL daily CI loop	FAIL auth problem deployment clearwater-opnfv timed out	not supported	PASS

Installer

Scenario

vping_ssh

vping_userdata

tempest.osinterop

tempest.ipv6

tempest.multi_node_scheduling

tempest.security

tempest.vm_lifecycle

ha.nova_api

ha.neutron_server

ha.keystone

ha.glance_api

ha.cinder_api

ha.cpu_load

ha.disk_load

ha.haproxy

sdnvpn.subnet_connectivity

sdnvpn.tenant_separation

sdnvpn.router_assoc

sdnvpn.router_assoc_fip

Apex

nosdn-nofeature

stable/fraser

PASS

FAIL

passes on Apex daily loop

FAIL

passes on Apex daily CI loop

FAIL

PASS

FAIL

PASS

FAIL

PASS

FAIL

IPMI info missing

PASS

not supported

Fuel

nosdn-nofeature

PASS

FAIL

PASS

FAIL

PASS

FAIL

PASS

not supported

Historical results below

2018-05-23 - limited results due to failed CI jobs

Installer	Scenario	vnf.vIMS	snaps.smoke	stress.ping	tempest.compute	tempest.identity_v2	tempest.identity_v3	tempest.image	tempest.network	tempest.volume	tempest.trunk_ports	tempest.bgpvpn	security.patrole	ha.controller_restart	ha.rabbitmq	ha.neutron_l3_agent	ha.database
Apex	nosdn-nofeature	FAIL Exception: Unable to create a number	PASS	FAIL	PASS	FAIL - only v3 used skip mechanism needed DOVETAIL-656 - Getting issue details... STATUS	PASS	PASS	FAIL - Success: 19 - Skipped: 0 - Expected failures: 0 - Unexpected success: 0 - Failures: 1 test_mtu_sized_frames ... fail	PASS	PASS	not supported	PASS	FAIL	FAIL SLA failure: 31.512733 > 30.000000	FAIL stack create failed inner exception: file not exist	PASS
Apex	odl	FAIL	FAIL		PASS	FAIL	PASS	PASS	FAIL	PASS	PASS	FAIL	FAIL	FAIL	FAIL	l3 agent not installed	PASS
Compass	nosdn-nofeature
Compass	odl	FAIL	FAIL fail: 2 pass: 117	FAIL failed to create stacks	PASS	SKIP	PASS	PASS	FAIL fail: 3 pass: 17	PASS	PASS skip: 9 pass: 43	FAIL No .testr.conf file was found	PASS skip: 5 pass: 130	PASS	FAIL No rabbitmq-server process	FAIL No neutron-la-agent process	PASS
Fuel (x86)	nosdn-nofeature	FAIL deployment clearwater-opnfv timed out	PASS	PASS	PASS	SKIP	PASS	PASS	FAIL fail: 1 pass: 19 test_mtu_sized_frames ... fail	PASS	PASS skip: 9 pass: 43	FAIL No .testr.conf file was found	PASS	FAIL IPMI information missing	FAIL No rabbitmq-server process	FAIL failed to create stack with shade	FAIL No mysql process
Fuel (arm)

2018-05-16

Installer	Scenario	smoke.tc001	smoke.tc002	ha.tc009	ha.tc010	ha.tc011	vnf.tc001	stress.tc001	sdnvpn.tc000	security.patrole	tempest.neutron_trunk_ports
Apex	nosdn-nofeature	FAIL 0 failing tests 76 success 19 skipped: skip: Identity api v2 is not enabled skip: AccountQuotasTest skipped as swift is not available => implement mechanism for specifying skippable tests DOVETAIL-656 - Getting issue details... STATUS	FAIL 1 failing test: PortConfigError: Cannot locate network with name - external in project - admin	FAIL ValueError: context not found for server 'node0.LF-8419b4c1' IPMI information missing?	FAIL missing ssh key in Yardstick FIX: https://gerrit.opnfv.org/gerrit/#/c/57401/	FAIL Unable to find network with name or id 'net04_ext' Env variables not passed correctly FIX: https://gerrit.opnfv.org/gerrit/#/c/57265/	FAIL failed to download file (temporary problem?)	FAIL TypeError: cannot concatenate 'str' and 'NoneType' objects => OS_TENANT_NAME not defined in OS credentials on Apex BOTTLENECK-228 - Getting issue details... STATUS	-- not supported by scenario --	PASS	-- not yet run --
Apex	odl	FAIL 0 failing tests 76 success 19 skipped: skip: Identity api v2 is not enabled skip: AccountQuotasTest skipped as swift is not available => implement mechanism for specifying skippable tests DOVETAIL-656 - Getting issue details... STATUS	FAIL 1 failing test: PortConfigError: Cannot locate network with name - external in project - admin	FAIL ValueError: context not found for server 'node0.LF-8419b4c1' IPMI information missing?	FAIL missing ssh key in Yardstick FIX https://gerrit.opnfv.org/gerrit/#/c/57401/	FAIL Unable to find network with name or id 'net04_ext' Env variables not passed correctly FIX https://gerrit.opnfv.org/gerrit/#/c/57265/	FAIL out of floating IPs	FAIL TypeError: cannot concatenate 'str' and 'NoneType' objects => OS_TENANT_NAME not defined in OS credentials on Apex BOTTLENECK-228 - Getting issue details... STATUS	FAIL FIX: https://gerrit.opnfv.org/gerrit/#/c/57263/	PASS	-- not yet run --
Compass	nosdn-nofeature	FAIL 1 failing tests 75 success 19 skipped fail: No IPv4 address found	FAIL 2 failing tests fail: Guest does not have a console available. (HTTP 404) fail: network_id are not the same	PASS	PASS	FAIL SLA failure: 10.010200 > 5.000000	PASS (nosdn-nofeature) FAIL failed to download files	FAIL	---	PASS 0 failing tests 132 success 3 skipped skip: dvr is not enabled	PASS 0 failing tests 43 success 9 skipped skip: vlan, vxlan, gre skip: Advanced image is required to run this test
Compass	odl	FAIL 0 failing tests 76 success 19 skipped	FAIL 1 failing tests fail: <snaps.domain.network.Router instance at 0x7f4f262d6290> is not None	PASS	PASS	---	FAIL failed to download files	FAIL No valid host was found	---	FAIL
Fuel (x86)	nosdn-nofeature	FAIL 0 failing tests 76 success 19 skipped	PASS 119 success	FAIL IPMI information missing	FAIL Ssh is still unavailable: SSHError	FAIL failed to create stack with shade	FAIL failed to download files	PASS	---	PASS 135 success	PASS 0 failing tests 43 success 9 skipped skip: vlan, vxlan, gre skip: Advanced image is required to run this test
Fuel (arm)

2018-05-09

Installer	Scenario	smoke.tc001	smoke.tc002	ha.tc009	ha.tc010	ha.tc011	vnf.tc001	stress.tc001	sdnvpn.tc000	security.patrole	tempest.neutron_trunk_ports
Apex		FAIL 0 failing tests 76 success 19 skipped: skip: Identity api v2 is not enabled skip: AccountQuotasTest skipped as swift is not available	FAIL 1 failing test: PortConfigError: Cannot locate network with name - external in project - admin	FAIL ValueError: context not found for server 'node0.LF-8419b4c1' IPMI information missing	FAIL Ssh is still unavailable: SSHError	FAIL Unable to find network with name or id 'net04_ext' Env variables not passed correctly? Dovetail needs to pass the env file. We (Xudan) have seen this before.	PASS (nofeature) ? (odl)	FAIL TypeError: cannot concatenate 'str' and 'NoneType' objects	FAIL Unknown test case or tier 'bgpvpn', or not supported by the given scenario 'unknown'	PASS
Compass	nosdn-nofeature	FAIL 1 failing tests 75 success 19 skipped fail: No IPv4 address found	FAIL 2 failing tests fail: Guest does not have a console available. (HTTP 404) fail: network_id are not the same	PASS	PASS	FAIL SLA failure: 10.010200 > 5.000000	PASS (nosdn-nofeature) FAIL failed to download files	FAIL	---	PASS 0 failing tests 132 success 3 skipped skip: dvr is not enabled	PASS 0 failing tests 43 success 9 skipped skip: vlan, vxlan, gre skip: Advanced image is required to run this test
Compass	odl	FAIL 0 failing tests 76 success 19 skipped	FAIL 1 failing tests fail: <snaps.domain.network.Router instance at 0x7f4f262d6290> is not None	PASS	PASS	---	FAIL failed to download files	FAIL No valid host was found	---	FAIL
Fuel (x86)	nosdn-nofeature	FAIL 0 failing tests 76 success 19 skipped	PASS 119 success	FAIL IPMI information missing	FAIL Ssh is still unavailable: SSHError	FAIL failed to create stack with shade	FAIL failed to download files	PASS	---	PASS 135 success	PASS 0 failing tests 43 success 9 skipped skip: vlan, vxlan, gre skip: Advanced image is required to run this test
Fuel (arm)