This page is a temporary location to draft and validate the testing guide documentation. Once it's in a decent draft shape, I will submit it to documentation gerrit and respective section authors can complete the doc through gerrit review.

Reference: CVP and Dovetail Terminology

I will use the terms defined in the above WIKI. Those terms will also be copied to the documentation's term section when they are in decent shape.

The intended audience of this document is the technical staff in vendors who are interested in seeking CVP certifications using Dovetail testing or third party labs who are to assist their clients for the same purpose. Additional audience include anyone who is interested in using Dovetail for testing purposes other than CVP certification. We assume the readers have good technical background in IT (e.g. Linux, Docker containers, security), networking, NFV, in addition to test areas that CVP covers.

1 Creating a Compliant POD

...

/* work in progress */

1.1 High Level Overview

In the top level, the CVP framework requires the tester's lab consist of the System Under Test (SUT), the jumphost that has access both public Internet and the private Management Network (a.k.a. Operation and Management Network, or O&M), necessary networking configuration and security gateways. The tester may realize this by a firewall, as shown below, and putting the jumphost in the DMZ. However, this configuration is just one possibility and a suggestion; it is not an explicit CVP requirement. Other methods of achieving the same goal of allowing the jumphost to have secure access to both the public Internet and private Management Network are equally valid.

Image Removed

The OPNFV CVP server, shown on the right side of the diagram above, is included for reference only. It is not the responsibility of the tester for the purpose of CVP certification. 

In a data center environment, a remote management infrastructure (a.k.a. Light Out Management) is a important part of the running a large scale system efficiently and reliably. Such remote management infrastructure however is not an explicit requirement of CVP at this time. The test suite does not make direct reference to the existence of a remote management system, although it is a recommended component.

1.2 Compute Hardware

The following bare metal hardware is required:

• 1 jumphost
• 5 controller and compute hosts

We require all 5 controller and compute hosts to be bare metal hosts because of high availability and scalability tests require bare metal environment. Other test cases may also be developed with bare metal environment only.

The jumphost does not have to be bare metal for CVP. However, this guide assumes it is a stand alone bare metal machine for simplicity. It is the tester's responsibility to make necessary configurations such that a virtual machine or a container can perform the same duties as a bare metal jumphost.

1.2.1 Jumphost

The jumphost is not part of the SUT in Dovetail, but is required for the Dovetail testing apparatus. Dovetail installs Docker based containers and other testing tools in this host to drive the automated testing procedures. This host must have the necessary network connection such that it has good access to the Internet and the VIM's API interface to drive testing. While Dovetail does not specify any strict minimum hardware requirement, and Docker has very little in its "minimum" requirements, we still recommend at least a workstation level host with 64 bit processor that, for instance, meets or exceeds the following spec, /*please verify these are reasonable recommendations ??? */

Processor: 2-4 cores
Memory: 32 GB RAM
Hard disk space: 60 GB

The jumphost can itself be a virtual machine or container. However, for simplicity, this document always assumes that it is a bare metal machine.

1.2.2 Controller and Compute Hosts

The Controller and Compute hosts are collectively referred to as simply the Hosts. They are part of the SUT in Dovetail. The 5 controller and compute hosts should meet the following minimum hardware requirements.

CPU:

• Intel Xeon E5-2600v2 Series or newer
• AArch64 (64bit ARM architecture) compatible (ARMv8 or newer)

Firmware:

• BIOS/EFI compatible for x86-family blades
• EFI compatible for AArch64 blades

Local Storage:

Below describes the minimum for the Pharos spec, which is designed to provide enough capacity for a reasonably functional environment. We recommend the same as minimum requirement for CVP testing.

• Disks: 2 x 1TB HDD + 1 x 100GB SSD (or greater capacity)
• The first HDD should be used for Operating System and additional software/tool installation
• The second HDD is configured for object storage (e.g. based on CEPH or others)
• The SSD should be used as the object store database journal (e.g. based on CEPH or others)
• Performance testing requires a mix of compute nodes with CEPH (Swift+Cinder) and without CEPH storage /*to confirm that this is Not Applicable for Dovetail in current release because performance is not in scope. */
• Virtual ISO boot capabilities or a separate PXE boot server (DHCP/tftp or Cobbler)

Memory:

• 32GB RAM Minimum

Power Supply

• Single power supply acceptable (redundant power not required/nice to have)

1.3 Networking Hardware

The OPNFV community Pharos specification documents several options for the physical networking configuration of the NFVI POD. In this section, we will use the Pharos as a reference to describe the required networking hardware configuration and point out areas where they are not strictly required for the purpose of CVP testing. The tester should take these as guidelines not strict requirements.

The NFV Infrastructure needs the following networks,

1. Data Network
2. Management Network
3. Storage Network (optional)
4. Administrative Network
5. Lights-Out Management Network

1.3.1 Network Hardware

• 24 or 48 Port Top Of Rack (TOR) Ethernet Switch with either 1GB or 10GB ports (or both): 1 unit is required. Dovetail currently does not test HA over switch failures.
• NICs - Combination of 1GE and 10GE based on network topology options as defined below. These NICs are required per server and can be on-board or PCI-e modules.
• Connectivity for each data or control network is through a separate NIC. This simplifies switch configuration but requires more NICs on the server and more ports on the switch.
• BMC (Baseboard Management Controller) for lights-out management network using IPMI (Intelligent Platform Management Interface) is not required for Dovetail testing at this time.

The options of network topology below have been validated in the OPNFV community and can be recommended. However, you are not required to use these options for the purpose of the CVP. Other network configurations are also valid and can be sufficient for CVP.

1.3.2 Network Option Examples

1. Option I: 4x1G Control, 2x10G Data, 48 Port Switch

   • 1 x 1G for lights-out Management

   • 1 x 1G for Admin/PXE boot

   • 1 x 1G for control-plane connectivity

   • 1 x 1G for storage

   • 2 x 10G for data network (redundancy, NIC bonding, High bandwidth testing)

2. Option II: 1x1G Control, 2x 10G Data, 24 Port Switch

   • Connectivity to networks is through VLANs on the Control NIC

   • Data NIC used for VNF traffic and storage traffic segmented through VLANs

3. Option III: 2x1G Control, 2x10G Data, 2x10G Storage, 24 Port Switch

   • Data NIC used for VNF traffic

   • Storage NIC used for control plane and Storage segmented through VLANs (separate host traffic from VNF)

   • 1 x 1G for lights-out mangement

   • 1 x 1G for Admin/PXE boot

   • 2 x 10G for control-plane connectivity/storage

   • 2 x 10G for data network

1.4 Management

1.5 Preparing the Hosts for CVP Testing

This section provide general guidelines on how to prepare the SUT for getting ready to conduct CVP testing using Dovetail. SInce the SUT is a commercial vendor product that is based on an open source VIM (currently Openstack), we will describe the preparation steps in the context of the open source community distributions only. The tester is advised to use the information here as reference only and prepare the commerdical SUT based on its own system specifics.

1.5.1 Host Operating Systems

The OPNFV CVP does not directly restrict choice of host operating systems. The following operating systems have been used and tested in certain degree in the OPNFV community.

• Ubuntu 16.04.2 LTS (Xenial) for x86_64

• Ubuntu 14.04 LTS (Trusty) for x86_64
• CentOS-7-1611 for x86_64
   

1.5.2 Openstack-based VIM Requirements

The current version of CVP uses Openstack as the only option of a Virtual Infrastructure Manager (VIM). The SUT must therefore use an Openstack based VIM. The following guidelines explain the compliant versions of Openstack and required functional components.

The current CVP test suite includes Openstack DefCore testing through RefStack as a component. Therefore it is advised that testers should also consult documentation of Defcore qualification testing for additional guidance. The version of Refstack in CVP_1.0.0 is 2016.8.

1.5.2.1 Openstack Versions

For now, following versions of open source Openstack have been tested:

• Kilo

• Liberty

• Mitaka

• Newton

It's not guaranteed that those versions not listed here or commercial versions of Openstack are well compatible with CVP_1.0.0. 

The list can be updated if additional versions have been tested.  

1.5.2.2 Openstack POD Configuration

The SUT must be configured such that it consists of 3 controllers nodes in HA mode, and 2 compute nodes.

• controller node( 3 or more)
• compute node( 2 or more)
• storage node(optional)
• network node(optional)

Above is the recommended deployment, but you can still try your own combinations of nodes.

The topologies vary in different deployment, below is a typical deployment:

/*insert diagram */

Additional explainations /*what else needed here?? */

1.5.2.3 Required Openstack Components

The following Openstack components are required,

• Openstack Nova
• Openstack Neutron
• Openstack Glance
• Openstack Swift
• Openstack Keystone
• Openstack Cinder

1.5.3 Configuring Testing Resources in Openstack

/* creating projects, accounts, floating ip, networks, security policies, router,... all that are needed for conducting Dovetail testing */

/* any storage configuration needed? */

/* resource/configuration needed by IPv6 */

/* resource/configuration needed by HA */

/* resource/configuration needed by sdnvpn */

This page is a temporary location to draft and validate the testing guide documentation. Once it's in a decent draft shape, I will submit it to documentation gerrit and respective section authors can complete the doc through gerrit review.

Reference: OVP and Dovetail Terminology

I will use the terms defined in the above WIKI. Those terms will also be copied to the documentation's term section when they are in decent shape.

The intended audience of this document is the technical staff in vendors who are interested in seeking CVP certifications using Dovetail testing or third party labs who are to assist their clients for the same purpose. Additional audience include anyone who is interested in using Dovetail for testing purposes other than CVP certification. We assume the readers have good technical background in IT (e.g. Linux, Docker containers, security), networking, NFV, in addition to test areas that CVP covers.

1 Creating a Compliant POD

Dovetail uses Pharos Specification (Pharos Specification) as a starting point of defining a compliant physical system as a part of the CVP's System Under Test (SUT). The current Pharos Specification is still being developed and updated for Danube as this time. It also has goals and constrains that are outside of the CVP's current scope, for example those required for use in the OPNFV community development. For these reasons, we provide a generalized guide here specifically for testers who need to create and configure a POD for the purpose of conducting CVP testing.

/* work in progress */

1.1 High Level Overview

In the top level, the CVP framework requires the tester's lab consist of the System Under Test (SUT), the jumphost that has access both public Internet and the private Management Network (a.k.a. Operation and Management Network, or O&M), necessary networking configuration and security gateways. The tester may realize this by a firewall, as shown below, and putting the jumphost in the DMZ. However, this configuration is just one possibility and a suggestion; it is not an explicit CVP requirement. Other methods of achieving the same goal of allowing the jumphost to have secure access to both the public Internet and private Management Network are equally valid.

Image Added

The OPNFV CVP server, shown on the right side of the diagram above, is included for reference only. It is not the responsibility of the tester for the purpose of CVP certification. 

In a data center environment, a remote management infrastructure (a.k.a. Light Out Management) is a important part of the running a large scale system efficiently and reliably. Such remote management infrastructure however is not an explicit requirement of CVP at this time. The test suite does not make direct reference to the existence of a remote management system, although it is a recommended component.

1.2 Compute Hardware

The following bare metal hardware is required:

• 1 jumphost
• 5 controller and compute hosts

We require all 5 controller and compute hosts to be bare metal hosts because of high availability and scalability tests require bare metal environment. Other test cases may also be developed with bare metal environment only.

The jumphost does not have to be bare metal for CVP. However, this guide assumes it is a stand alone bare metal machine for simplicity. It is the tester's responsibility to make necessary configurations such that a virtual machine or a container can perform the same duties as a bare metal jumphost.

1.2.1 Jumphost

The jumphost is not part of the SUT in Dovetail, but is required for the Dovetail testing apparatus. Dovetail installs Docker based containers and other testing tools in this host to drive the automated testing procedures. This host must have the necessary network connection such that it has good access to the Internet and the VIM's API interface to drive testing. While Dovetail does not specify any strict minimum hardware requirement, and Docker has very little in its "minimum" requirements, we still recommend at least a workstation level host with 64 bit processor that, for instance, meets or exceeds the following spec, /*please verify these are reasonable recommendations ??? */

Processor: 2-4 cores
Memory: 32 GB RAM
Hard disk space: 60 GB

The jumphost can itself be a virtual machine or container. However, for simplicity, this document always assumes that it is a bare metal machine.

1.2.2 Controller and Compute Hosts

The Controller and Compute hosts are collectively referred to as simply the Hosts. They are part of the SUT in Dovetail. The 5 controller and compute hosts should meet the following minimum hardware requirements.

CPU:

• Intel Xeon E5-2600v2 Series or newer
• AArch64 (64bit ARM architecture) compatible (ARMv8 or newer)

Firmware:

• BIOS/EFI compatible for x86-family blades
• EFI compatible for AArch64 blades

Local Storage:

Below describes the minimum for the Pharos spec, which is designed to provide enough capacity for a reasonably functional environment. We recommend the same as minimum requirement for CVP testing.

• Disks: 2 x 1TB HDD + 1 x 100GB SSD (or greater capacity)
• The first HDD should be used for Operating System and additional software/tool installation
• The second HDD is configured for object storage (e.g. based on CEPH or others)
• The SSD should be used as the object store database journal (e.g. based on CEPH or others)
• Performance testing requires a mix of compute nodes with CEPH (Swift+Cinder) and without CEPH storage /*to confirm that this is Not Applicable for Dovetail in current release because performance is not in scope. */
• Virtual ISO boot capabilities or a separate PXE boot server (DHCP/tftp or Cobbler)

Memory:

• 32GB RAM Minimum

Power Supply

• Single power supply acceptable (redundant power not required/nice to have)

1.3 Networking Hardware

The OPNFV community Pharos specification documents several options for the physical networking configuration of the NFVI POD. In this section, we will use the Pharos as a reference to describe the required networking hardware configuration and point out areas where they are not strictly required for the purpose of CVP testing. The tester should take these as guidelines not strict requirements.

The NFV Infrastructure needs the following networks,

1. Data Network
2. Management Network
3. Storage Network (optional)
4. Administrative Network
5. Lights-Out Management Network

1.3.1 Network Hardware

• 24 or 48 Port Top Of Rack (TOR) Ethernet Switch with either 1GB or 10GB ports (or both): 1 unit is required. Dovetail currently does not test HA over switch failures.
• NICs - Combination of 1GE and 10GE based on network topology options as defined below. These NICs are required per server and can be on-board or PCI-e modules.
• Connectivity for each data or control network is through a separate NIC. This simplifies switch configuration but requires more NICs on the server and more ports on the switch.
• BMC (Baseboard Management Controller) for lights-out management network using IPMI (Intelligent Platform Management Interface) is not required for Dovetail testing at this time.

The options of network topology below have been validated in the OPNFV community and can be recommended. However, you are not required to use these options for the purpose of the CVP. Other network configurations are also valid and can be sufficient for CVP.

1.3.2 Network Option Examples

1. Option I: 4x1G Control, 2x10G Data, 48 Port Switch

   • 1 x 1G for lights-out Management

   • 1 x 1G for Admin/PXE boot

   • 1 x 1G for control-plane connectivity

   • 1 x 1G for storage

   • 2 x 10G for data network (redundancy, NIC bonding, High bandwidth testing)

2. Option II: 1x1G Control, 2x 10G Data, 24 Port Switch

   • Connectivity to networks is through VLANs on the Control NIC

   • Data NIC used for VNF traffic and storage traffic segmented through VLANs

3. Option III: 2x1G Control, 2x10G Data, 2x10G Storage, 24 Port Switch

   • Data NIC used for VNF traffic

   • Storage NIC used for control plane and Storage segmented through VLANs (separate host traffic from VNF)

   • 1 x 1G for lights-out mangement

   • 1 x 1G for Admin/PXE boot

   • 2 x 10G for control-plane connectivity/storage

   • 2 x 10G for data network

1.4 Management

1.5 Preparing the Hosts for CVP Testing

This section provide general guidelines on how to prepare the SUT for getting ready to conduct CVP testing using Dovetail. SInce the SUT is a commercial vendor product that is based on an open source VIM (currently Openstack), we will describe the preparation steps in the context of the open source community distributions only. The tester is advised to use the information here as reference only and prepare the commerdical SUT based on its own system specifics.

1.5.1 Host Operating Systems

The OPNFV CVP does not directly restrict choice of host operating systems. The following operating systems have been used and tested in certain degree in the OPNFV community.

• Ubuntu 16.04.2 LTS (Xenial) for x86_64

• Ubuntu 14.04 LTS (Trusty) for x86_64
• CentOS-7-1611 for x86_64
   

1.5.2 Openstack-based VIM Requirements

The current version of CVP uses Openstack as the only option of a Virtual Infrastructure Manager (VIM). The SUT must therefore use an Openstack based VIM. The following guidelines explain the compliant versions of Openstack and required functional components.

The current CVP test suite includes Openstack DefCore testing through RefStack as a component. Therefore it is advised that testers should also consult documentation of Defcore qualification testing for additional guidance. The version of Refstack in CVP_1.0.0 is 2016.8.

1.5.2.1 Openstack Versions

For now, following versions of open source Openstack have been tested:

• Kilo

• Liberty

• Mitaka

• Newton

It's not guaranteed that those versions not listed here or commercial versions of Openstack are well compatible with CVP_1.0.0. 

The list can be updated if additional versions have been tested.  

1.5.2.2 Openstack POD Configuration

The SUT must be configured such that it consists of 3 controllers nodes in HA mode, and 2 compute nodes.

• controller node( 3 or more)
• compute node( 2 or more)
• storage node(optional)
• network node(optional)

Above is the recommended deployment, but you can still try your own combinations of nodes.

The topologies vary in different deployment, below is a typical deployment:

/*insert diagram */

Additional explainations /*what else needed here?? */

1.5.2.3 Required Openstack Components

The following Openstack components are required,

• Openstack Nova
• Openstack Neutron
• Openstack Glance
• Openstack Swift
• Openstack Keystone
• Openstack Cinder

1.5.3 Configuring Testing Resources in Openstack

/* creating projects, accounts, floating ip, networks, security policies, router,... all that are needed for conducting Dovetail testing */

/* any storage configuration needed? */

/* resource/configuration needed by IPv6 */

/* resource/configuration needed by HA */

/* resource/configuration needed by sdnvpn */

2 Conducting CVP Tests using Dovetail

...

The tester should also validate that the jumphost can reach the public Internet. For example,

%ping 8.8.8.8
%ping%dig https://www.opnfv.org/cvp

...

     Use the following steps to check if the right version of Docker is already installed, and if not, install it.

    % docker version

    As the docker installation process is much complex, you can refer to the scripts.

    It will install the latest version of docker, if you’re not intend to update your docker version, be careful to run the script:

    % wget -qO- https://get.docker.com/ | sh

2.2.4 Installing Dovetail on Jumphost

...

          % cd $DOVETAIL_HOME

          % sudo git clone https://git.opnfv.org/dovetail

          % cd $DOVETAIL_HOME/dovetail

          % sudo pip install -e ./

          /* test dovetail install is successful */

          % dovetail -h

Method 2. Installing Dovetail Docker Container 

       The Dovetail project also maintains a Docker image that has Dovetail test tools preinstalled. 

...

The only <tag> is 'latest'.

% sudo docker run --privileged=true -it -v <openrc_path>:<openrc_path> \

-v $DOVETAIL_HOME/results:$DOVETAIL_HOME/results \

-v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \

-v /home/opnfv/dovetail/userconfig:/home/opnfv/dovetail/userconfig \

-v /var/run/docker.sock:/var/run/docker.sock \

--name <DoveTail_Container_Name> (optional) \

opnfv/dovetail:<Tag> /bin/bash

2.3 Running CVP Test Suite

...

HA test cases need to know the info of all nodes of the OpenStack. It should include every node's name, role, ip, user and key_filename or password. The info should be written in file $DOVETAIL_HOME/dovetail/userconfig/pod.yaml.  There is a sample file $DOVETAIL_HOME/dovetail/userconfig/sample_pod.yaml.

Two methods to login nodes

...

1. The name of each node must be node1, node2,...
2. node1 must be Controller node.
3. If use private key to login, the key_filename must be /root/.ssh/id_rsa in file $DOVETAIL_HOME/dovetail/userconfig/pod.yaml.
4. If use private key to login, you must copy the private key to $DOVETAIL_HOME/dovetail/userconfig/
         cp <private_key_for_login_nodes> $DOVETAIL_HOME/dovetail/userconfig/id_rsa

2.3.3 Making Sense of CVP Test Results

...

% cd $DOVETAIL_HOME/dovetail

% sudo git pull

% sudo pip install -e ./

This step is necessary if dovetail software or the CVP test suite have updates.

...

2.4.1 Running Dovetail Locally 

DoveTail supports uploading results into database. The database can be either a local database or an official one. Before you can use the local database, you need to create the local database and the testapi service. They can be installed on any machine which can talk to the jumphost. Docker 1.12.3 and later should be installed on this machine. There are 3 steps for creating local database and testapi service.

      step1. Set ports for database and testapi service           

            The default ports of database and testapi are 27017 and 8000, respectively. Check whether they are used by other services already.

                    % netstat -nlt

            If 27017 and 8000 are used by other services, you need to set other ports in step2.

        step2. Run a Dovetail container 

            Run a Dovetail container.

                    % sudo docker pull opnfv/dovetail:<Tag>

                    % sudo docker run -itd --privileged=true --name <DoveTail_Container_Name> \

                       -v /var/run/docker.sock:/var/run/docker.sock  opnfv/dovetail:<Tag> /bin/bash

                  % sudo docker exec -it <DoveTail_Container_Name> /bin/bash

            If you need to set ports for database and testapi service,

                    % export mongodb_port=<database_port> 

                  % export testapi_port=<testapi_port> 

      step3. Create local database and testapi service

            % cd /home/opnfv/dovetail/dovetail/utils/local_db/

            % ./launch_db.sh <localhost_ip_address>

            Exit this Dovetail container.

                  % exit

      step4. Check the status of database and testapi service

            Local database and testapi service are actually two containers named mongodb and testapi. You can check whether there are these two containers running.

                  % sudo docker ps -a

            You can try to get data from the database to make sure everything is OK.

                  % wget <localhost_ip_address>:<testapi_port>/api/v1/results

2.4.2 Running Dovetail with an Offline SUT

...

    when "Installing Dovetail Docker Container" method is used,

             % sudo mkdir /home/opnfv/dovetail/userconfig

             % cd /home/opnfv/dovetail/userconfig

            % touch refstack_tempest.conf defcore.txt

            % vim refstack_tempest.conf

            % vim defcore.txt

    the recommend way to set refstack_tempest.conf is shown in https://aptira.com/testing-openstack-tempest-part-1/

    the recommended way to edit defcore.txt is to open https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false and copy all the test cases into defcore.txt.

    Then use “docker run” to create a container,

          % sudo docker run --privileged=true -it -v <openrc_path>:<openrc_path> \

           -v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \

           -v /home/opnfv/dovetail/userconfig:/home/opnfv/dovetail/userconfig \

           -v /var/run/docker.sock:/var/run/docker.sock \

          --name <DoveTail_Container_Name> (optional) \

          opnfv/dovetail:<Tag> /bin/bash

...

3.1 Check dovetail commands

% dovetail -h

Image Modified

Dovetail has three commands: list, run and show.

3.2 List

3.2.1 List help

% dovetail list -h

3.2.2 List a test suite

List command will list all test cases belong to the given test suite.

% dovetail list compliance_set

Image Modified

% dovetail list debug

Image Modified

The ipv6, example and nfvi are test areas. If no <TESTSUITE> is given, it will list all testsuites.

...

Show command will give the detailed info of one certain test case.

3.3.1 Show help

% dovetail show -h

Image Modified

3.3.2 Show test case

3.4 Run

Dovetail supports running a named test suite, or one named test area of a test suite.

3.4.1 Run help

% dovetail run -h

Image ModifiedThere are some options:

...