Page Comparison

...

The table below captures the current status of functest integration. Please remove the issue from the table when it is resolved,

Cédric Ollivier

James Gu

Test suite	Test case	Issue	Resolution
Healthcheck
airship-opnfv-functest-healthcheck-latest-connection_check-run
airship-opnfv-functest-healthcheck-latest-

singlevm1

tenantnetwork1-run
airship-opnfv-functest-healthcheck-latest-

singlevm2

tenantnetwork2-run
airship-opnfv-functest-healthcheck-latest-

vping_ssh

vmready1-run
airship-opnfv-functest-healthcheck-latest-vmready2-run
airship-opnfv-functest-healthcheck-latest-singlevm1-run
airship-opnfv-functest-healthcheck-latest-singlevm2-run
airship-opnfv-functest-healthcheck-latest-vping_ssh-run
airship-opnfv-functest-healthcheck-latest-vping_userdata-run
	vping_userdata-run	Upstream bug: AIRSHIP-95. Nova instance console log api returns emptry
airship-opnfv-functest-healthcheck-latest-cinder_test-run
airship-opnfv-functest-healthcheck-latest-odl-run
airship-opnfv-functest-healthcheck-latest-tempest_smoke-run
airship-opnfv-functest-healthcheck-latest-tempest_horizon-run
Smoke
airship-opnfv-

functest-smoke-latest-tempest_neutron-runneutron_tempest_plugin.api.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_dhcp_port_status_activeRequires neutron extensions not supported in Airship DPDKneutron_tempest_plugin.api.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_list_networks_hosted_by_one_dhcpsame neutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_delete_policies_while_tenant_attached_to_netsameneutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_external_update_policy_from_wildcard_to_specific_tenantsameneutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_policy_allows_tenant_to_allocate_floatingipsameneutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_policy_allows_tenant_to_attach_ext_gwsameneutron_tempest_plugin.api.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcp_stateful_fixedips_outrangesameneutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_create_floatingip_with_timestampsameneutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_create_router_with_timestampsameneutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_show_floatingip_attribute_with_timestampsameneutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_show_router_attribute_with_timestampsameneutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_update_floatingip_with_timestampsameneutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_update_router_with_timestampsameQOS test casesUpstream: QOS support in DPDKneutron_tempest_plugin.api.test_security_groups.RbacSharedSecurityGroupTest.test_regular_client_shares_to_another_regular_clientTBDairship-opnfv-functest-smoke-latest-tempest_cinder-runcinder_tempest_plugin.api.volume.test_volume_revert.VolumeRevertTests.test_volume_revert_to_snapshotAIRSHIP-96: upstream bug related to kernel 4.15airship-opnfv-functest-smoke-latest-tempest_keystone-runairship-opnfv-functest-smoke-latest-refstack_platform-runskippedairship-opnfv-functest-smoke-latest-tempest_full-runairship-opnfv-functest-smoke-latest-tempest_heat-run

functest-smoke-latest-tempest_neutron-run
	neutron_tempest_plugin.api.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_dhcp_port_status_active	Requires neutron extensions not supported in Airship DPDK
	neutron_tempest_plugin.api.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_list_networks_hosted_by_one_dhcp	same
	neutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_delete_policies_while_tenant_attached_to_net	same
	neutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_external_update_policy_from_wildcard_to_specific_tenant	same
	neutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_policy_allows_tenant_to_allocate_floatingip	same
	neutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_policy_allows_tenant_to_attach_ext_gw	same
	neutron_tempest_plugin.api.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcp_stateful_fixedips_outrange	same
	neutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_create_floatingip_with_timestamp	same
	neutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_create_router_with_timestamp	same
	neutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_show_floatingip_attribute_with_timestamp	same
	neutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_show_router_attribute_with_timestamp	same
	neutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_update_floatingip_with_timestamp	same
	neutron_tempest_plugin.api.test_timestamp.TestTimeStampWithL3.test_update_router_with_timestamp	same
	QOS test cases	Upstream: QOS support in DPDK
	neutron_tempest_plugin.api.test_security_groups.RbacSharedSecurityGroupTest.test_regular_client_shares_to_another_regular_client	TBD
airship-opnfv-functest-smoke-latest-tempest_cinder-run
	cinder_tempest_plugin.api.volume.test_volume_revert.VolumeRevertTests.test_volume_revert_to_snapshot	AIRSHIP-96: upstream bug related to kernel 4.15
airship-opnfv-functest-smoke-latest-tempest_keystone-run
airship-opnfv-functest-smoke-latest-refstack_platform-run		skipped
airship-opnfv-functest-smoke-latest-tempest_full-run
	tempest.api.compute.admin.test_instance_usage_audit_log_negative.InstanceUsageAuditLogNegativeTestJSON.test_instance_usage_audit_logs_with_nonadmin_user	AIRSHIP-104: Airship rbac policy inconsistent with tempest default	https://review.opendev.org/c/airship/treasuremap/+/767395
	empest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration	1289064: upstream: live migration is broken in numa. Needs Train
	tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration_paused	same
	tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration	same
	tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration_paused	same
	tempest.api.compute.admin.test_migrations.MigrationsAdminTest.test_list_migrations_in_flavor_resize_situation	TBD
	tempest.api.compute.admin.test_migrations.MigrationsAdminTest.test_resize_server_revert_deleted_flavor	TBD
	tempest.api.compute.admin.test_simple_tenant_usage.TenantUsagesTestJSON.test_get_usage_tenant_with_non_admin_user	AIRSHIP-105: Airship rbac policy inconsistent with tempest default	https://review.opendev.org/c/airship/treasuremap/+/767395
	tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_create_list_show_delete_interfaces_by_network_port	fail 36.011
	tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_reassign_port_between_servers	fail 89.588
	tempest.api.compute.servers.test_create_server_multi_nic.ServersTestMultiNic.test_verify_duplicate_network_nics	fail 126.883
	tempest.api.compute.servers.test_create_server_multi_nic.ServersTestMultiNic.test_verify_multiple_nics_order	fail 125.284

	tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_verify_resize_state	TBD
	tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment	TBD
	tempest.api.compute.servers.test_disk_config.ServerDiskConfigTestJSON.test_resize_server_from_auto_to_manual	TBD
	tempest.api.compute.servers.test_disk_config.ServerDiskConfigTestJSON.test_resize_server_from_manual_to_auto	TBD
	tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_get_console_output	Upstream bug: AIRSHIP-95. Nova instance console log api returns emptry
	tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_get_console_output_server_id_in_shutoff_status	same
	tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_get_console_output_with_unlimited_size	same
	tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm	TBD
	tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm_from_stopped	TBD
	tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert	TBD
	tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert_with_volume_attached	TBD
	tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_volume_backed_server_confirm	TBD
	tempest.api.identity.admin.v3.test_application_credentials.ApplicationCredentialsV3AdminTest.test_create_application_credential_with_roles	AIRSHIP-106: Airship rbac policy inconsistent with tempest default
	tempest.api.identity.admin.v3.test_groups.GroupsV3TestJSON.test_list_groups	AIRSHIP-107: possible rbac?
	tempest.api.identity.admin.v3.test_list_users.UsersV3TestJSON.test_list_usersz	AIRSHIP-108: possible rbac
	tempest.api.identity.admin.v3.test_list_users.UsersV3TestJSON.test_list_users_with_name	same
	tempest.api.identity.admin.v3.test_projects_negative.ProjectsNegativeStaticTestJSON.test_list_projects_by_unauthorized_user	AIRSHIP-109: airship rbac policy is different than tempest default
	tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_create_application_credential	AIRSHIP-110: airship rbac policy is different than tempest default	https://review.opendev.org/c/airship/treasuremap/+/767395
	tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_create_application_credential_expires	same
	tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_list_application_credentials	same
	tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_query_application_credentials	same
	tempest.api.identity.v3.test_users.IdentityV3UsersTest.test_password_history_check_self_service_api	TBD: possible rbac
	tempest.api.identity.v3.test_users.IdentityV3UsersTest.test_user_account_lockout	TBD: possible rbac
	tempest.api.identity.v3.test_users.IdentityV3UsersTest.test_user_update_own_password	TBD: possible rbac
	tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_deactivate_reactivate_image	TBD: possible rbac
	tempest.api.network.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_list_networks_hosted_by_one_dhcp	FUNCTEST-1023: L3 test cases need to be skipped based on Neutron extension list or blacklisted
	tempest.api.network.admin.test_external_network_extension.ExternalNetworksTestJSON.test_list_external_networks	same
	tempest.api.network.admin.test_external_network_extension.ExternalNetworksTestJSON.test_show_external_networks_attribute	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcp_stateful	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcp_stateful_fixedips	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcp_stateful_fixedips_duplicate	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcp_stateful_fixedips_outrange	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcp_stateful_router	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_64_subnets	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_invalid_options	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_stateless_eui64	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_stateless_no_ra	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_stateless_no_ra_no_dhcp	same
	tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_two_subnets	same
	tempest.api.network.test_ports.PortsIpV6TestJSON.test_port_list_filter_by_router_id	same
	tempest.api.network.test_ports.PortsTestJSON.test_port_list_filter_by_router_id	same
	tempest.api.volume.admin.test_volume_manage.VolumeManageAdminTest.test_unmanage_manage_volume	TBD
	tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_from_deactivated_image	AIRSHIP-112: possible rbac?



airship-opnfv-functest-smoke-latest-tempest_heat-run
	heat_tempest_plugin.tests.functional.test_nova_server_networks.CreateServerTest.test_create_server_with_port	FUNCTEST_1022: Requires neutron l3 extension not supported in Airship DPDK. Should be dynamically skipped based on neutron extension list.
	heat_tempest_plugin.tests.functional.test_nova_server_networks.CreateServerTest.test_create_server_with_subnet_fixed_ip_sec_group	same
	heat_tempest_plugin.tests.functional.test_nova_server_networks.CreateServerTest.test_create_update_server_with_subnet	same
	heat_tempest_plugin.tests.functional.test_nova_server_networks.UpdateServerNetworksTest.test_create_update_server_add_network	same
	heat_tempest_plugin.tests.functional.test_nova_server_networks.UpdateServerNetworksTest.test_create_update_server_add_same_fixed_ip	same
	heat_tempest_plugin.tests.functional.test_nova_server_networks.

CreateServerTest

UpdateServerNetworksTest.test_create_update_server_

with_portRequires neutron l3 extension not supported in Airship DPDK

add_subnet	same
	heat_tempest_plugin.tests.functional.test_nova_server_networks.

CreateServerTest

UpdateServerNetworksTest.test_create_update_server_

with

multi_

subnet_fixed_ip_sec_group

networks_swaps	same
	heat_tempest_plugin.tests.functional.test_nova_server_networks.

CreateServerTest

UpdateServerNetworksTest.test_create_update_server_swap_

with

network_

subnet

port	same
	heat_tempest_plugin.tests.functional.test_nova_server_networks.UpdateServerNetworksTest.test_create_update_server_

add

swap_network_subnet	same
	heat_tempest_plugin.tests.functional.test_nova_server_networks.UpdateServerNetworksTest.test_create_update_server_swap_

add

subnet_network

same

_fixed_ip


	heat_tempest_plugin.tests.

functional

scenario.test

_nova

_server_

networks

signal.

UpdateServerNetworksTest

ServerSignalIntegrationTest.test_

create

server_

update

signal_

server

userdata_

add

format_

subnet

raw	same
	heat_tempest_plugin.tests.

functional

scenario.test_

nova_

server_

networks

signal.

UpdateServerNetworksTest

ServerSignalIntegrationTest.test_

create

server_

update

signal_

server

userdata_

multi

format_

networks

software_

swaps

config	same
	heat_tempest_plugin.tests.functional.test_

nova

os_

server

wait_

networks

condition.

UpdateServerNetworksTest

OSWaitCondition.test_create_

update

stack_

server

with_

swap

multi_

network

signal_

port

waitcondition	TBD
	heat_tempest_plugin.tests.

functional

scenario.test_

nova

base_

server_networks

resources.

UpdateServerNetworksTest

BasicResourcesTest.test_

create

base_

update_server_swap_network_subnet

resources_integration	TBD
	heat_tempest_plugin.tests.

functional

scenario.test_

nova

remote_

server

deeply_

networks

nested.

UpdateServerNetworksTest

RemoteDeeplyNestedStackTest.test_

create_update_server_swap_subnet_networkheat_tempest_plugin.tests

remote_nested	TBD
airship-opnfv-functest-smoke-latest-tempest_scenario-run
	tempest.scenario.test_server_basic_

signal

ops.

ServerSignalIntegrationTest

TestServerBasicOps.test_server_

signal_userdata_format_rawheat_tempest_plugin.tests.scenario.test_server_signal.ServerSignalIntegrationTest.test_server_signal_userdata_format_software_configheat_tempest_plugin.tests.functional.test_os_wait_condition.OSWaitCondition.test_create_stack_with_multi_signal_waitconditionTBDheat

basic_ops	Upstream bug: AIRSHIP-103. Nova metaservice url not reachable from guest vm
airship-opnfv-functest-smoke-latest-tempest_telemetry-run		skipped
airship-opnfv-functest-smoke-latest-tempest_slow-run
	tempest.scenario.test_volume_backup_restore.TestVolumeBackupRestore.test_volume_backup_restore	FUNCTEST-1019: teste volume backup depends on floating ip
	tempest.api.compute.servers.test_device_tagging.TaggedBootDevicesTest.test_tagged_boot_devices	FUNCTEST-1020: test device tagging still uses tenant network
airship-opnfv-functest-smoke-latest-rally_sanity-run
	NovaServers.boot_and_live_migrate_server	No support from OpenStack Helm. Recommend to blacklist.
airship-opnfv-functest-smoke-latest-patrole-run
	patrole_tempest_plugin.tests.api.

scenario

network.test_

base

floating_ips_

resources

rbac.

BasicResourcesTest

FloatingIpsRbacTest.test_

base

create_

resources_integrationTBDheat

floating_ip	FUNCTEST-1021: Requires neutron l3 extension not supported in Airship DPDK. Should be dynamically skipped based on neutron extension list.
	patrole_tempest_plugin.tests.

scenario

api.network.test_

remote

floating_

deeply

ips_

nested

rbac.

RemoteDeeplyNestedStackTest

FloatingIpsRbacTest.test

_remote_nestedTBDairship-opnfv-functest-smoke-latest-tempest_scenario-runtempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_opsUpstream bug: AIRSHIP-103. Nova metaservice url not reachable from guest vmairship-opnfv-functest-smoke-latest-tempest_telemetry-runskippedairship-opnfv-functest-smoke-latest-tempest_slow-runtempest.scenario.test_volume_backup_restore.TestVolumeBackupRestore.test_volume_backup_restoreFUNCTEST-1019: teste volume backup depends on floating iptempest.api.compute.servers.test_device_tagging.TaggedBootDevicesTest.test_tagged_boot_devicesFUNCTEST-1020: test device tagging still uses tenant networkairship-opnfv-functest-smoke-latest-rally_sanity-runNovaServers.boot_and_live_migrate_serverNo support from OpenStack Helm. Recommend to blacklist.airship-opnfv-functest-smoke-latest-patrole-run

_create_floating_ip_floatingip_address	same
	patrole_tempest_plugin.tests.api.network.test_floating_ips_rbac.FloatingIpsRbacTest.test_delete_floating_ip	same
	patrole_tempest_plugin.tests.api.network.test_floating_ips_rbac.FloatingIpsRbacTest.test_list_floating_ips	same
	patrole_tempest_plugin.tests.api.network.test_floating_ips_rbac.FloatingIpsRbacTest.test_show_floating_ip	same
	patrole_tempest_plugin.tests.api.network.test_floating_ips_rbac.FloatingIpsRbacTest.test_update_floating_ip	same
	patrole_tempest_plugin.tests.api.network.test_networks_rbac.NetworksRbacTest.test_create_network_is_default	same
airship-opnfv-functest-smoke-latest-refstack_compute-run		skipped
airship-opnfv-functest-smoke-latest-refstack_object-run
airship-opnfv-functest-smoke-latest-networking-sfc-run		skipped
airship-opnfv-functest-smoke-latest-networking-bgpvpn-run		skipped
airship-opnfv-functest-smoke-latest-tempest_barbican-run		skipped
airship-opnfv-functest-smoke-latest-tempest_octavia-run		skipped
Benchmark
airship-opnfv-functest-benchmarking-latest-rally_full-run
	NeutronNetworks.create_and_show_ports	needs blacklist. It needs router extension.
	NovaServers.resize_server	SLA. Retest on pod 15. pod17 too small.
	GlanceImages.create_and_deactivate_image	SLA. Retest on pod 15. pod17 too small.
airship-opnfv-functest-benchmarking-latest-rally_jobs-run		skipped
airship-opnfv-functest-benchmarking-latest-vmtp-run		skipped
airship-opnfv-functest-benchmarking-latest-shaker-run		skipped
VNF
airship-opnfv-functest-vnf-latest-cloudify-run		FUNCTEST-1017: cloudify vpn deploy uses floating ip. Shoudl change to use "public" network or skip.
airship-opnfv-functest-vnf-latest-cloudify_ims-run		skipped
airship-opnfv-functest-vnf-latest-heat_ims-run		skipped
airship-opnfv-functest-vnf-latest-vyos_vrouter-run		skipped
airship-opnfv-functest-vnf-latest-juju_epc-run		skipped

Versions Compared

Old Version 4

New Version Current

Key