Observations of Reference Conformance (RC) Participation in the Lakelse Release

This wiki page is based on an email sent to the Anuket TSC on Sept 6, 2021.

Introduction

During the Lakelse release, some issues with RC became apparent.  

An important objective of the release process is to make the development status of deliverables visible to the TSC, and the community.  RC lacks transparency, despite steps in the release process intended to provide transparency.  

RC Mapping to RA

RC is frequently described as being dependent on RA.  However, during the Lakelse release, it became apparent that RC is, at best, loosely coupled to RA.  More accurately, RC seems to be based primarily on OpenStack releases. The apparent assumption is that this will result in sufficient, although unknown, coverage for RA.  There doesn't appear to have been any attempt to understand or even read the RA requirements during the development of RC.

1. M2 requires a description of the coverage of RA.  I was provided a link to RC documentation that does not provide specific mapping to RA requirements.  In fact, there aren't even any references to RA requirement numbers.
   1. This means that we do not have an accurate, or even approximate, measurement of RA coverage.
   2. We also cannot describe how coverage has changed from one release to the next.
   3. This is a contradiction of the RC documentation, itself.  For example:  "The conformance specifications must provide the mapping between tests and requirements to demonstrate traceability and coverage."
2. The RC documentation includes lists of test case names.  However, there are no links to detailed information about the test cases.  
   1. For example, a requirement author, or other stakeholder, might want to understand how a requirement is being tested.
   2. The engineering team conducts a review of RA documentation for each release.  Sometimes they want to see how a requirement is being tested.

RC Validation Testing

We discussed validation testing at the Sept 6 Technical Discussion meeting.  At the meeting I learned that "validation" is performed by running the test suites against a known-good environment and verifying that the test cases pass.

1. M1 requires a validation plan for RC.  Instead of providing a validation plan, I was provided a pointer to the RC documentation.  The result of this is that the community has no documented description of how RC is being validated.
2. Based on feedback from the meeting, the validation testing does not include insertion of faults to ensure that test cases fail when they should.  This may result in false negative tests.
3. The validation testing seems to be conducted in just a single configuration/environment.  This may result in configuration/optimization around a single environment and in turn, inaccurate test results in a valid, but different environment.
4. There is no dashboard or other convenient means to see the results of validation testing.  Also, there are no periodic reports or other updates on the results.

Conclusions

1. If we intend for RC to truly be dependent on RA, then this needs to be expressed through the mapping that is described in RC's own documentation. In addition, we should work toward improving validation testing.
2. If we believe that a loose coupling of RC to RA is sufficient, then we should stop stating that RC has a dependency on RA.  We should also update RC documentation to remove the several references to detailed mapping of RC to RA.