Camden, New Jersey Home · Contact · Links · Search
City Services & Government > Information Technology

GUIDELINES FOR TESTING APPLICATIONS

 Types of Testing

The term Quality Assurance is commonly used to refer to a wide variety of software testing activities, including:
  • Conformance Testing


    • Testing conducted to verify that an implementation conforms to a formal specification (typically one defined by a standards organization).

  • Functional Testing


    • Testing conducted to verify that software meets its functional requirements (which may include, but typically exceed, conformance to formal standards).

  • Interoperability Testing


    • Testing to verify that two or more software products are capable of interacting with each other, perhaps via a communications or messaging protocol, or by exchanging data through some other means. Note that conformance to specifications is a necessary, but probably not sufficient, condition for two systems to be interoperable.

  • Performance Testing


    • Testing conducted to evaluate the compliance of a system or component with specified performance requirements.

  • Stress Testing


    • Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements.

  • Usability Testing


    • Testing conducted to evaluate the ease with which users can learn and use a software product.
These guidelines primarily address conformance testing. However, since conformance testing focuses on verifying conformance to (compliance with) formal specifications, the principles and practices outlined in this document should prove applicable to other forms of testing that can be expressed in these requirements-based terms.

CONFORMANCE REQUIREMENTS:

The vendor must create and publish a test plan for testing the test materials and the test suite as a whole. The results of executing this test plan MUST be published.

If tests are buggy or incorrect this will be noticed eventually, and will lead to their exclusion from the test suite. Similarly, if the test suite as a whole has problems (if it is difficult to install, or has inaccurate documentation) then these problems will be reported by users. It is preferable to discover and correct these problems before the test suite is released rather than to wait for users to discover them while testing their implementations. The test materials must therefore be tested to ensure that they function correctly, and that they correctly test what they claim to test, and the test suite as a whole must be tested to ensure that it is usable and that it performs as designed. Publishing the test plan and test results will help to ensure that users of the test suite have confidence in its quality.

The test materials MUST include a mechanism for users to provide feedback.

Define the mechanisms for reporting and publishing test results

A well defined and consistent mechanism for reporting test results is essential to ensure that the test materials are easy to use and that it is possible to compare the results of executing the tests on different implementations. This will make easier to compare implementation feature-sets to determine interoperability. It also provides data that supports the process of identifying overall and feature-specific levels of support among implementations.

Tests MUST report their execution outcome in a consistent manner. At a minimum, tests MUST report the following statuses:

  • Cannot Determine
  • Fail
  • Not Applicable
  • Passed

The process for executing tests MUST be well defined and documented. Clear and unambiguous instructions on how to run the test suite should be supplied to ensure that no two users running the same test suite following the instructions under the same conditions achieve different results (the test suite should be deterministic).

The test material documentation MUST specify where test results may not be repeatable or reproducible.

If test results may vary depending on the order in which tests are executed, or for any other reason, this must be documented to allow results to be interpreted correctly, and to enable different test runs to be compared. Wherever possible, the documentation should explain what must be done in order to ensure that the results are reproducible and repeatable to the greatest extent possible.

When tests fail, they MUST report on the reason for failure.

All parties (vendor and the City of Camden) must agree on the results and interpretation of the testing.



About Camden | City Services & Government | Doing Business
Economic Development | Events & Attractions | Contact | Links | Disclaimer

For questions or coments about the website, please email the WebMaster.

Copyright 2006 City of Camden
  This site developed by