OVAL - Correctness Testing

Correctness Testing — Archive

Important: The OVAL Compatibility Program was moved to "archive" status in December 2009, and replaced with the "OVAL Adoption Program." Under the OVAL Adoption Program product validation is performed by an external organization, allowing the OVAL Team to focus on educating vendors on best practices regarding the use and implementation OVAL and on how OVAL can continue to evolve as needed by the community.

Refer to the OVAL Adoption Program section for addition information and to review all products and services listed.

Schedule | Procedures | Sample Test Suites

Correctness testing represents the final stage in the OVAL Compatibility Program and is used to verify that an implementation has followed the community's recommendations. This aids other organizations in knowing which capabilities can be used to interface with one another. A schedule of upcoming sessions, procedures for correctness testing, the current test suite, and a collection of previous test suites, are included below.

Schedule

Upcoming and recent sessions for Version 5 Correctness Testing are included below:

UPCOMING

Date	Location	Sign-Up
TBD	Bedford, Massachusetts, USA	oval@mitre.org

Date	Location	Sign-Up
April 11, 2007	Bedford, Massachusetts, USA	oval@mitre.org
January 17, 2007	Bedford, Massachusetts, USA	oval@mitre.org
November 16, 2006	Bedford, Massachusetts, USA	oval@mitre.org
October 4, 2006	Bedford, Massachusetts, USA	oval@mitre.org
July 13, 2006	Bedford, Massachusetts, USA	oval@mitre.org

Procedures

Correctness Testing must be passed by a capability in order to achieve official OVAL-Compatible status. A description of the Testing Infrastructure, Testing Procedures, and Review Outcome are included below.

Testing Infrastructure

The final phase of the OVAL Compatibility Program is Correctness Testing, which must be passed by a capability in order to achieve official OVAL-Compatible status. Please see the document "An Introduction to OVAL Compatibility" for more information about the OVAL Compatibility Program and the different phases involved.

The purpose of Correctness Testing is to ensure that companies, and their capabilities, use OVAL as defined by the OVAL Community. This aids other organizations in knowing which capabilities can be used to interface with one another. To be certified by the Review Authority as OVAL Compatible, a capability must run through the procedures defined below.

Testbed

The Review Authority shall provide a number of systems connected by a local network for the review. All necessary access to testbed machines shall be provided, as well as time to install any required software, test and troubleshoot. The testbed will have Internet connectivity. An organization that requires a connection with the Internet should communicate with the Review Authority before the testing session to make sure everything will be setup in a manner that meets the capabilities requirements.

The actual makeup of the testbed will vary, but a typical testbed would consist of 5-10 systems with various OS and applications installed. The testbed will be adjusted for each Correctness Testing session by the Review Authority to appropriately serve the involved capabilities.

Definition Test Suite

The Definition Test Suite is a collection of OVAL definitions that shall be provided by the Review Authority. It will be provided in the form of a single XML file that contains all the definitions. This file may be broken up into individual files if necessary. Should a candidate require advance knowledge of definitions (e.g. to convert to an internal format), they should contact the Review Authority one week prior to the testing session and the Definition Test Suite will be made available. Review previous test suites.

System Characteristics Test Suite

The System Characteristics Test Suite shall be a collection of OVAL System Characteristic data provided by the Review Authority. It will be provided in the form of a single XML file that contains information about each system in the testbed. This file cannot be distributed early. Review previous test suites.

Results Test Suite

The Results Test Suite shall be a collection of OVAL Results provided by the Review Authority. It will be provided in the form of a single XML files that contains the results of an evaluation of the testbed. This file cannot be distributed early. Review previous test suites.

Testing Procedures

OVAL Compatibility consists of six different areas. During Phase I of the OVAL Compatibility Program, each capability declared compliance with one or more of these areas. Correctness Testing for each area is outlined below.

OVAL Definition Consumer Testing

Candidate OVAL Definition Consumers will first be asked by the Review Authority to demonstrate within the capability, validation of the Definition Test Suite against the schema version with which it is stated to comply. Once the test suite has been validated, the candidate's capability shall evaluate the definitions included in the Definition Test Suite using the provided testbed for system information. Each candidate will be expected to produce results (not necessarily OVAL Results) for each definition in the Definition Test Suite. If a capability does not support a particular platform, then this should be presented in the results. Also, all errors during evaluation should be included in the results provided.

The results of a candidate's assessment shall be compared with a predetermined assessment of the target machine(s) by the Review Authority. At a minimum, the capabilities results must be identified by the corresponding OVAL Definition ID and contain the true/false/unknown/error answer determined by the evaluation. Results for individual OVAL tests included in these definitions are highly desirable and will aid in understanding any discrepancies.

Capabilities that do not consume OVAL Definitions at runtime shall educate the Review Authority about the process by which they consume OVAL Definition submissions and then transfer them to their capability.

The candidate capability passes the test and is considered a Compatible OVAL Definition Consumer if the results obtained by evaluating the Definition Test Suite exactly match the official results produced by the Review Authority's predetermined assessment.

OVAL Definition Producer Testing

Candidate OVAL Definition Producers shall make available to the Review Authority all available OVAL Definitions they have produced to date. The Review Authority will choose a sampling of definitions to consider and shall validate the selected definitions against the appropriate schema version to ensure stated compliance. The Review Authority will also confirm that all OVAL IDs are unique with respect to both the candidate and the OVAL Community as a whole.

In addition, the Review Authority will verify that an appropriate reference exists for each submitted definition (CVE for vulnerability class definitions) and that all metadata and tests contained within each OVAL definition are consistent with the Review Authority's guidelines for a complete definition.

OVAL Results Consumer Testing

Candidate OVAL Results Consumers will first be asked by the Review Authority to demonstrate within the capability, validation of the Results Test Suite against the schema version with which it is stated to comply. Once the test suite has been validated, the candidate's capability shall illustrate to the Review Authority how the OVAL Results are incorporated. For each OVAL Definition in the Results Test Suite, the candidate must show successful importation of the included OVAL-ID and the associated OVAL Result. This information must be linked to an appropriate system identifier and displayed to the user.

OVAL Results Producer Testing

Candidate OVAL Results Producers shall produce OVAL Results files corresponding to the evaluation of the testbed using the OVAL Definitions comprising the Definition Test Suite. The Review Authority may require multiple copies of OVAL Results be generated from the same target machine (or machine data set if using OVAL System Characteristics files) to review repeatability of the OVAL Results.

The Review Authority will validate the candidate's OVAL Results against the appropriately specified OVAL Schema version. The OVAL Results will then be compared to an OVAL Results file generated by the Review Authority. The purpose of this comparison is to review the OVAL Results file for accuracy and correctness.

If a candidate's capability performs assessment against some stock models of machine states, (clean installs of OSs, etc) rather than actual systems in the testbed, there are additional requirements. The candidate must provide valid System Characteristics files with the machine states used with the Definition Test Suite to produce the Results. This will allow the Review Authority to verify that the Results are correct. The Review Authority, at its sole discretion, may allow the candidate to generate the OVAL Results using their own models instead of the System Characteristics files provided by the Review Authority.

OVAL System Characteristics Consumer Testing

Candidate OVAL System Characteristic Consumers will first be asked by the Review Authority to demonstrate within the capability, validation of the System Characteristics Test Suite against the schema version with which it is stated to comply. Once the test suite has been validated, the candidate's capability shall run the Review Authority through the process by which they consume OVAL System Characteristics, and how the information is then used within their application. The purpose of this is to make a correctness assessment about the use of the OVAL System Characteristics file.

OVAL System Characteristics Producer Testing

Review Outcome

The Review Authority shall provide the Candidate with a detailed report on the outcome of the OVAL Compatibility Test. The Candidate shall be given the opportunity to explain any discrepancies to the Review Authority. The Review Authority, at its sole discretion, may choose to consider or reject these explanations before making an official ruling on the outcome of the capabilities OVAL compatibility. At that point, all rulings made by the Review Authority are final.