New Build Process (M4)

The intent is to have a single set of build scripts across all Zero Applications/Projects. The build XML files allow you to tweak the various steps as needed.

For a build, there are 4 common steps: build (resolve, compile, package), test (resolve, compile, unit-test), publish, javadoc.

A formal build is a 3 tiered hierarchy which starts in BUILD/zero.build. The config.xml is the cruisecontrol configuration file which kicks off the appropriate build based on SVN changes or dependent components. The cc-build.xml is where all the real work happens.

The formal build will execute all phases of the build with a local only resolver except for publish. Each component build has a repository and test-repository which contains the required external dependencies. It is important to note that the repository of your parent component is copied prior to the component build starting.

When the tests are executed, the emma results are merged together as well as having the individual emma reports for each project.

Component build (zero.tools/ant/component.xml)

Every component (TOOLS, CORE, ASSEMBLE, etc...) has a corresponding build project named zero.<COMPONENT>.build. Its responsibility is to drive the steps of a build across all the projects contained within.

To perform a step, just run

zero <step>, e.g., zero build

If you want to emulate the formal build a bit closer, for the build or test phases, you should run with a local resolver , e.g.,

zero -Dresolver=local build

NOTE: zero.tools.build aka zero.tools is bootstrapped. You will use ANT for its build step and then use zero found in zero.tools/export/zero.tools to execute the rest of the TOOLS build. As an optimization, CORE also uses zero from the same directory.

build

The component repository is merged with the current repository in ${export}/repository. In the formal build, the ${export}/repository is built by copying the repository from your parent component.

The build target for all non-test projects is executed next. One can define projects that need to be included or excluded which do not match the default rules. The patternset must be defined after the import statement. The following shows an example

<import file="${zerohome}/ant/component.xml" />
<patternset id="build.files.additional">
   <exclude name="zero.tools*/build.xml" />
   <exclude name="zero.eclipse*/build.xml" />
</patternset>

test

This follows a similar process as build. There is a test-repository instead of a repository and a ${export}/test-repository. Here is an example to include or exclude projects that are executed when test is invoked.

<import file="${zerohome}/ant/component.xml" />
<patternset id="build.files.tests.additional">
   <exclude name="zero.tools.tests/build.xml" />
</patternset>

publish

This step runs through all the "build" projects and publishes them into the repository.

javadoc

Javadoc is executed on all the "build" projects by default. Most of the current components have specialized behaviors.

generate-repository

To update the component repository or test-repository, one can update the ivy.xml or test-ivy.xml. The appropriate OSSC approval must be obtained first.

zero.build

The BUILD/zero.build project contains the CruiseControl configuration (config.xml) as well as the build script (cc-build.xml) to drive a component build.

To build TOOLS (subset of CORE)

• extract BUILD/zero.build
• cd zero.build
• edit cc.properties
• rm -rf checkout
• ant -Drelease=trunk -f cc-build.xml build-TOOLS
• <zero.tools>/export/zero.tools/zero -Drelease=trunk -Dskip.checkout=true -f cc-build.xml test-TOOLS
  • You can skip the tests and perform a publish by adding a -Dskip.test=true

To build CORE (this assumes you have done the above)

• Extract a command line runtime, creates a zerohome in <tmp dir>/CORE.latest
  • cd zero.build
  • ant -Drelease=trunk -Dcomponent=CORE -f cc-build.xml install-zero
• Execute a CORE build
  • /tmp/CORE.latest/zerohome/zero -Drelease=branches/b_newbuild -Dskip.checkout=true -f cc-build.xml CORE

The results of the build will end up in zero.build/cc/export/<component>.<directory qualifier>, e.g., zero.build/export/CORE.latest.

If you already have a zero command line, you can use the following steps to build any component

• Execute a build
  • zerohome/zero -Drelease=trunk -f cc-build.xml ASSEMBLE

Build knobs

release

the SVN release, e.g., trunk or branches/b_mybranch

skip.checkout

skip the svn checkout

skip.resolve

skip the resolve step

skip.test

skip the test step

skip.native

skip native code compilation

version.generate

use the SVN revision as part of the package version

release.xml

The BUILD/zero.build contains a script, release.xml, to build an entire release.

To build a branch (e.g. 1.0M4) which contains a complete set of all components, do the following

• extract BUILD/zero.build
• cd zero.build
• edit cc.properties
• ant -Drelease=branches/1.0/M4 -f release.xml

Project build

To build a project, its as easy as zero build To test a test project, zero test

If you would like emma support during your test add -Demma=true

zero.tools bootstrapping

zero.tools requires some special help in order to build. Its a project that is a Zero Project, an Eclipse Plugin and the Command Line. Conceptually these three packages are very similar; building one allows you to create the others. There are however some gotchas.

• In order to execute any Zero command, it must be made available to ANT. For all Zero Projects, one would use zero which provides the zero tasks. zero.tools must redefine and load the tasks as needed.
• Since zero.tools is also a Zero Project, it can be re-compiled via a resolve. In doing so, great care must be taken to not destroy anything in the export directory.
• To allow patching of the ivy.jar, the image that will be packaged and publish is created under the export directory.
• The ant JARs are currently stored in a single ZIP which is expanded during the build. This may need to be re-thought. While its easy to either expand the ZIP directly into another ZIP/JAR via zipfileset, it does make zero.tools look incorrect. By having an image under the export directory, we have something that is correct however is not used during a resolve.

Tips and Tricks

zero.properties

• One can pass a -Dzero.properties=<fully qualified filename> to Zero that will define global properties that will override the predefined properties used by any Zero build step.

Optimizations

Trash.Compile & Resolve

The resolve process is required in order to perform a compile. The classpath for the compile step is pulled from the .zeroresolved.properties. All dependents must also be compiled for the "root" project to compile. The recursive compilation occurs through resolve. Trying to determine which of the dependents need to re-compile seems like an obvious optimization, however its fraught with issues.

The original thought was to look at a variety of possibilities:

• Only look at direct dependencies.
• Does a .zeroresolved.properties exist, if not, compile it.
• If there is a source directory,
  • Is there a classes directory, if not, compile it
  • If the source directory last modification time later than the classes last modification time, if yes, compile it.
  • If the .zeroresolved.properties is newer than the source directory last modification time, compile it.
• If the ivy.xml is newer than the .zeroresolved.properties, compile it

A lot of these checks sound promising, however, there are underlying issues that invalidate them all.

Not all filesystems propagate the last modification time up the directory hierarchy. All the checks that compare directory timestamps may not work. For those filesystems, the only answer is to determine the newest directory within and perform that comparison.

The more serious issue is compiling against the latest code. By checking direct dependencies, we are not taking into account the transitive dependencies that exist. For example A -> (depends) B -> C. If something changes in C, and we are compiling A, our checks determine B is not changed. Therefore, when we compile A, we do not see the changes in C.

A logical fix is to deal with all dependents including transitives. For the compile & resolve step to work properly, the dependents must be processed in reverse order. Each dependent would resolve without recursive compilation and then compile itself.

The ultimate question is why is the optimization needed. For most use cases, its actually not needed. When you have a large number of projects in your workspace, and they all have some number of common dependents, you will be compiling and resolving those projects N times.

The simple solution to all this is to make certain that all dependents are compiled and resolved in the appropriate order and as few times as possible. If we go back to our original algorithm, it had some things right and a few things wrong. We can leverage the recursive compile and resolve to guarantee the dependency traversal. For each project, we would do the recursive resolve (see previous statement) and compile since we really cannot tell if we are affect by transitives unless we were to examine each one when compiling ourselves. What can be done is to minimize the number of compile/resolves that are performed for a given project. By keeping track of what we have compiled/resolved, we can prevent duplicate executions. This would limit the number of times to 1. If you are building multiple projects within a single JVM, then the answer is 2 since there is the build for the project itself.

-- fraenkel - 03 Dec 2007

-- fraenkel - 30 Mar 2008