A convenient shortcut for working with test packages
The train to FOSDEM was the perfect setting for creating a dirty little tool for quickly building and running single test packages.
The previous Genode release introduced a new on-target test automation tool called depot autopilot. Martin converted more than 70 former run scripts into packages that are readily deployable using this tool. This dramatically cuts down the number of boot cycles and stresses the endurance of the whole Genode system compared to the execution of only one test per boot cycle.
Those benefits, however, come at the price of a little inconvenience when dealing with an individual test. When instrumenting or extending, let's say, the xml_node test, one first has to create a package (using the depot/create tool) along with the packages required by the depot-autopilot (like init, report_rom, vfs, and fs_rom) before being able to execute the test. Even though the depot-autopilot run script features a few nifty parameters to enable a fairly efficient work flow, one still gets the feeling that the packages stand in the way in such situations.
My little test.run script found at my topic branch attempts to regain the former convenience with dealing with low-level tests. The basic idea is to feed the run script with a single PKG argument with the name of one test package. The run script looks up the matching pkg recipe within the source tree and constructs a static system scenario out of the information found in the pkg's runtime file. This information contains, for example, the needed amount of RAM, the list of ROM modules required by the test (listed in the <contents> node), the test configuration (<config>), and a criterion for the success/failure of the test. Thanks to Martin's herculean work with molding many of our tests into such runtime-description files, those tests follow the same simple formalism now. This allows my little test.run script to extract all information needed for assembling a static system. For example, the list of <content> found in the runtime file corresponds to the list of boot modules specified in the build_boot_image step.
The second piece of the puzzle was the input for the build step. This information is absent from the test's pkg archive. Here, some brutal heuristics come into play: The run script scans the source tree for all target.mk files, in particular it looks for each target's name as defined in via the TARGET variable found in target.mk. For each target name, the run script now knows where in the source tree the corresponding target.mk resides. That is, it knows the build argument needed to build a target with a given name. But what are the names of the targets of the test scenario? Well, the <content> definition of the test's runtime file includes all targets the test relies on. Granted, it may contain also other ROM modules such as libraries the targets depend on, but the targets are definitely in there. So the run script simply tries to find a target.mk file for each <content> element. If there exists such a target.mk, it adds the target's build path to the list of build arguments.
The last missing piece of the puzzle was the detection of the test's failure or success. There are two typical mechanisms at play. First, a timeout indicates that a test has failed. And second, a known-good log output confirms the success of the test. Almost all low-level tests use this simple scheme. The information for both mechanisms is present in the form of the runtime's <events> node. The run script uses this information to feed the run tool's run_genode_until function. It's funny that test.run actually reverses Martin's work when he turned the run scripts into packages. Martin turned the formerly used regexp patterns to plain strings intermixed with wildcards and applied quoting of special characters (like < and >). The test.run script now converts these patterns back to regular expressions.
Test driving the test driver
To give the test.run script a try, you may cherry-pick the commit cited above (at my topic branch) and execute the following command from your build directory:
make run/test PKG=test-xml_node KERNEL=linux
It is quite surprising to me how many of the existing tests can be executed this way out of the box. The tool is certainly not bullet-proof. You may call it a hack. But that doesn't make it any less useful. ;-)
Thanks to Martin for pointing me to the --xpath feature of xmllint and for paving the ground for making tools like test.run possible!