FVT Process

The FVT process has been enabled on the build machine. What follows is a summary of how the FVT works, how to run the FVT tests locally, and what to do in case you are assigned an FVT bug.

Summary

On an regular interval, the build machine will run FVT (Functional Verification Test) across all of the platforms supported by Zero. At first, FVT will be scheduled to run very early Sunday morning, but it may be moved to run daily.

FVT consists of all of the currently run Junit tests -- with the following differences:

• The tests are run on all supported platforms: Windows, Linux and Mac. The platforms might increase, such that multiple types of platforms will be added. (for example, Windows XP and Windows 2003 Server)
• The tests are run from the repository on www.projectzero.org. All dependencies are resolved, and are not co-located in the same tree

Because the Junit tests are run against resolved dependencies in a repo, its possible that some of your tests will fail. This is usually because of assumptions about where files are. Most tests can be easily changed to accommodate running against resolved dependencies instead of co-located dependencies. (98% of all junit tests passed on first attempted FVT.)

The steps taken in the FVT are the following:

• extract a fresh svn tree from trunk
• collect together all of the .tests packages into a zip file
• move zip file to all platforms being tested.
• For each platform:
  • download and install CLI
  • copy repository to local machine
  • resolve test projects
  • run tests
  • collect output files from junit tests
• pull all test output files back to build machine
• Test fails if any test on any platform failed.

Bugs

Because the FVT process runs asynchronously with the regular builds, it is impossible for emails to be sent to developers when one of their changes breaks the build. Instead, team leads will be emailed when the FVT fails, and they will be responsible for assigning someone to fix the failing test.

To run an FVT style build locally, against remotely resolved artifacts, take the following steps:

• extract trunk out into a clean directory / workspace
• cd trunk/BUILD/zero.remote.fvt
• ant -f remotefvt.xml run-tests

The output of the run will be in zero.remote.fvt/export/local -- in the same format as is usually available from the build.