Adding tests to autotest¶
Adding a test is probably the easiest development activity to do.
Each test is completely contained in it’s own subdirectory (either in
client/tests for client-side tests or
server/tests for server-side
tests) - the normal components are
- An example control file,e.g.
- A test wrapper, e.g.
- Some source code for the test, if it’s not all done in just the Python script.
Start by taking a look over an existing test, e.g.
note that the name of the subdirectory -
tests/dbench, the test wrapper -
dbench.py and the name of the class inside the test wrapper -
all match. Make sure you do this in your new test too.
The control file is trivial:
That just takes the default arguments to run dbench - mostly, we try to provide sensible default settings to get you up and running easily, then you can override most things later.
There’s a tarball for the source code - dbench-3.04.tar.gz - this will
get extracted under
src/ later. Most of what you’re going to have to do
is in the Python wrapper. Look at
dbench.py - you’ll see it inherits
from the main test class, and defines a version (more on that later).
You’ll see four functions:
initialize()- This is run before everything, every time the test is run.
setup()- This is run when you first use the test, and normally is used to compile the source code.
run_once()- This is called by
job.run_testN times, where N is controlled by the iterations parameter to
run_test(defaulting to one). It also gets called an additional time with profilers turned on, if you have any profilers enabled.
postprocess_iteration()- This processes any results generated by the test iteration, and writes them out into a keyval. It’s generally not called for the profiling iteration, as that may have different performance.
The test will result in a PASS, unless you throw an exception, in which case it will FAIL (error.TestFail?), WARN (error.TestWarn?) or ERROR (anything else). Most things that go wrong in Python will throw an exception for you, so you don’t normally have to worry about this much - you can check extra things and throw an exception if you need. Now let’s look at those functions in some more detail.
This is the one-off setup function for this test. It won’t run again
unless you change the version number (so be sure to bump that if you
change the source code). In this case it’ll extract dbench-3.04.tar.gz
src/, and compile it for us. Look at the first few lines:
# http://samba.org/ftp/tridge/dbench/dbench-3.04.tar.gz def setup(self, tarball='dbench-3.04.tar.gz'): tarball = utils.unmap_url(self.bindir, tarball, self.tmpdir)
A comment saying where we got the source from. The function header -
defines what the default tarball to use for the source code is (you can
override this with a different dbench version from the control file if
you wanted to, but that’s highly unusual). Lastly there’s some magic
unmap_url - that’s just incase you overrode it with a
url - it’ll
download it for you, and return the local path … just copy that bit.
utils.extract_tarball_to_dir(tarball, self.srcdir) os.chdir(self.srcdir) utils.system('./configure') utils.system('make')
OK, so this just extracts the tarball into
self.srcdir (pre-setup for
you to be
src/ under the test), cd’s into that src dir, and runs
./configure; make just as you would for most standard compilations.
We use the local
system() wrapper, not
os.system() - this will
automatically throw an exception if the return code isn’t 0, etc.
Apart from compiling a package from the source,you have an option to
use the client system’s software manager to install a package using
Here is how you do it:
from autotest.client.shared import software_manager backend=software_manager.SoftwareManager() backend.install('<package_name>')
This actually executes the test. The core of what it’s doing is just:
Which says “run dbench and add the output to self.results”. We need to record the output so that we can process it after the test runs in postprocess.
For performance benchmarks, we want to produce a keyval file of
key=value pairs, describing how well the benchmark ran. The key is
just a string, and the value is a floating point or integer number.
dbench, we produce just two performance metrics - “throughput” and
“nprocs”. The function is called once per iteration (except for the
optional profiling iteration), and we end up with a file that looks like
throughput = 217 nprocs = 4 throughput = 220 nprocs = 4 throughput = 215 nprocs = 4
Note that the above was from a run with three iterations - we ran the benchmark 3 times, and thus print three sets of results. Each set is separated by a blank line.
Additional test methods¶
These methods aren’t implemented in the
dbench test, but they can be
implemented if you need to take advantage of them.
For performance tests that need to conduct any pre-test priming to make
the results valid. This is called by
job.run_test before running the
test itself, but after all the setup.
Used for any post-test cleanup. If test may have left the machine in a broken state, or your initialize made a large mess (e.g. used up most of the disk space creating test files) that could cause problems with subsequent tests then it’s probably a good idea to write a cleanup that undoes this. It always gets called, regardless of the success or failure of the test execution.
Used for executing the test, by calling
postprocess. The base test class provides an implementation that already
supports profilers and multiple test iterations, but if you need to
change this behavior you can override the default implementation with
If you want to properly support multi-iteration tests and/or profiling
runs, you must provide that support yourself in your custom execute implementation.
Adding your own test¶
Now just create a new subdirectory under tests, and add your own control
file, source code, and wrapper. It’s probably easiest to just copy
mytest.py, and edit it - remember to change the name of the
class at the top though.
If you have any problems, or questions, drop an email to the Autotest mailing list), and we’ll help you out.