Integration with Yardstick¶
Problem description¶
For each specified QPI [1], QTIP needs to select a suite of test cases and collect required test results. Based on these results, QTIP calculates the score.
Proposed change¶
QTIP has a flexible architecture [2] to support different mode: standalone and agent. It is recommended to use agent mode to work with existing test runners. Yardstick will act as a runner to generate test result and trigger QTIP agent on the completion of test.
Work Items in Yardstick¶
- Create a customized suite in Yardstick
Yardstick not only has many existing suites but also support customized suites. QTIP could create a suite named QTIP-PoC in Yardstick repo to verify workflow of QTIP agent mode.
- Launch QTIP in Yardstick
Whether to launch QTIP will be determined by checking the existence of OS environment variable QTIP. If it exists, QTIP will be launched by using Yardstick CLI yardstick plugin install [3].
- Yardstick interacts with QTIP
See Yardstick-QTIP+integration for details.
Work Items in QTIP¶
- Provide an API for Yardstick to post test result and environment info
After completing test execution, Yardstick will post test result and enviroment info with JSON format via QTIP API. See Yardstick-QTIP+integration for details.
- Parse yardstick test result
When QTIP agent receive Yarstick test result and enviroment info, QTIP agent will extract metrics which is definded in metric spec configuration file. Based on these metrics, QTIP agent will caculate QPI.
- Provide an API for querying QPI
QTIP will provide an API for querying QPI. See Yardstick-QTIP+integration for details.
Testing¶
The changes will be covered by new unit test.
Documentation¶
TBD