106 lines
4.8 KiB
Plaintext
106 lines
4.8 KiB
Plaintext
|
tdc - Adding test cases for tdc
|
||
|
|
|
||
|
|
Author: Lucas Bates - lucasb@mojatatu.com
|
||
|
|
|
||
|
|
ADDING TEST CASES
|
||
|
|
-----------------
|
||
|
|
|
||
|
|
User-defined tests should be added by defining a separate JSON file. This
|
||
|
|
will help prevent conflicts when updating the repository. Refer to
|
||
|
|
template.json for the required JSON format for test cases.
|
||
|
|
|
||
|
|
Include the 'id' field, but do not assign a value. Running tdc with the -i
|
||
|
|
option will generate a unique ID for that test case.
|
||
|
|
|
||
|
|
tdc will recursively search the 'tc-tests' subdirectory (or the
|
||
|
|
directories named with the -D option) for .json files. Any test case
|
||
|
|
files you create in these directories will automatically be included.
|
||
|
|
If you wish to store your custom test cases elsewhere, be sure to run
|
||
|
|
tdc with the -f argument and the path to your file, or the -D argument
|
||
|
|
and the path to your directory(ies).
|
||
|
|
|
||
|
|
Be aware of required escape characters in the JSON data - particularly
|
||
|
|
when defining the match pattern. Refer to the supplied json test files
|
||
|
|
for examples when in doubt. The match pattern is written in json, and
|
||
|
|
will be used by python. So the match pattern will be a python regular
|
||
|
|
expression, but should be written using json syntax.
|
||
|
|
|
||
|
|
|
||
|
|
TEST CASE STRUCTURE
|
||
|
|
-------------------
|
||
|
|
|
||
|
|
Each test case has required data:
|
||
|
|
|
||
|
|
id: A unique alphanumeric value to identify a particular test case
|
||
|
|
name: Descriptive name that explains the command under test
|
||
|
|
skip: A completely optional key, if the corresponding value is "yes"
|
||
|
|
then tdc will not execute the test case in question. However,
|
||
|
|
this test case will still appear in the results output but
|
||
|
|
marked as skipped. This key can be placed anywhere inside the
|
||
|
|
test case at the top level.
|
||
|
|
category: A list of single-word descriptions covering what the command
|
||
|
|
under test is testing. Example: filter, actions, u32, gact, etc.
|
||
|
|
setup: The list of commands required to ensure the command under test
|
||
|
|
succeeds. For example: if testing a filter, the command to create
|
||
|
|
the qdisc would appear here.
|
||
|
|
This list can be empty.
|
||
|
|
Each command can be a string to be executed, or a list consisting
|
||
|
|
of a string which is a command to be executed, followed by 1 or
|
||
|
|
more acceptable exit codes for this command.
|
||
|
|
If only a string is given for the command, then an exit code of 0
|
||
|
|
will be expected.
|
||
|
|
cmdUnderTest: The tc command being tested itself.
|
||
|
|
expExitCode: The code returned by the command under test upon its termination.
|
||
|
|
tdc will compare this value against the actual returned value.
|
||
|
|
verifyCmd: The tc command to be run to verify successful execution.
|
||
|
|
For example: if the command under test creates a gact action,
|
||
|
|
verifyCmd should be "$TC actions show action gact"
|
||
|
|
matchPattern: A regular expression to be applied against the output of the
|
||
|
|
verifyCmd to prove the command under test succeeded. This pattern
|
||
|
|
should be as specific as possible so that a false positive is not
|
||
|
|
matched.
|
||
|
|
matchCount: How many times the regex in matchPattern should match. A value
|
||
|
|
of 0 is acceptable.
|
||
|
|
teardown: The list of commands to clean up after the test is completed.
|
||
|
|
The environment should be returned to the same state as when
|
||
|
|
this test was started: qdiscs deleted, actions flushed, etc.
|
||
|
|
This list can be empty.
|
||
|
|
Each command can be a string to be executed, or a list consisting
|
||
|
|
of a string which is a command to be executed, followed by 1 or
|
||
|
|
more acceptable exit codes for this command.
|
||
|
|
If only a string is given for the command, then an exit code of 0
|
||
|
|
will be expected.
|
||
|
|
|
||
|
|
|
||
|
|
SETUP/TEARDOWN ERRORS
|
||
|
|
---------------------
|
||
|
|
|
||
|
|
If an error is detected during the setup/teardown process, execution of the
|
||
|
|
tests will immediately stop with an error message and the namespace in which
|
||
|
|
the tests are run will be destroyed. This is to prevent inaccurate results
|
||
|
|
in the test cases. tdc will output a series of TAP results for the skipped
|
||
|
|
tests.
|
||
|
|
|
||
|
|
Repeated failures of the setup/teardown may indicate a problem with the test
|
||
|
|
case, or possibly even a bug in one of the commands that are not being tested.
|
||
|
|
|
||
|
|
It's possible to include acceptable exit codes with the setup/teardown command
|
||
|
|
so that it doesn't halt the script for an error that doesn't matter. Turn the
|
||
|
|
individual command into a list, with the command being first, followed by all
|
||
|
|
acceptable exit codes for the command.
|
||
|
|
|
||
|
|
Example:
|
||
|
|
|
||
|
|
A pair of setup commands. The first can have exit code 0, 1 or 255, the
|
||
|
|
second must have exit code 0.
|
||
|
|
|
||
|
|
"setup": [
|
||
|
|
[
|
||
|
|
"$TC actions flush action gact",
|
||
|
|
0,
|
||
|
|
1,
|
||
|
|
255
|
||
|
|
],
|
||
|
|
"$TC actions add action reclassify index 65536"
|
||
|
|
],
|