In progress

Dependency Management in Zero

The dependency management in Zero is designed to allow resolving dependencies that the application needs instead of a model where all dependencies are bundled with the environment. The idea is to reduce complexity by allowing the application to declare what behaviors are needed. The typical zero application will depend on zero.core which will include other dependencies as transitives. In order to make this model work there are several required components:

1. A repository to provide packages. This includes zero packages as well as jars available through existing ibiblio servers
2. A means of browsing and searching the repositories to see what is available. This can be as easy as using a browser to look around indexing enabled servers all the way to what zero catalog provides with community features.
3. able declare a dependency on one of these artifacts.
4. ability to download artifacts into a local repository
5. support for transitive dependencies.
6. report capabilities that can be used to find where the dependencies reside

Ivy

The dependency management in Project Zero uses http://ant.apache.org/ivy for the dependency management.

Most of these features are provided by Ivy in Zero dependency management component, also called zero.tools. Ivy provides resolvers that can be used to search a host of different types of repositories including local filesytem, URL and ibiblio servers ( a special case of url resolvers). The repository structure, how artifacts are stored in the repository, is configurable through ivy. The ivy configuration file included with each zero application provides the means of specifying the dependencies.

Zero Projects

A zero project is defined by much of the behavior provided by zero.core in where resources and handlers are defined. But what really makes a project a Zero project with respect to dependency management. The only requirement that dependency management places on a Zero project is the project must have a configy/ivy.xml file.

Resolvers

Resolvers are defined in ${zerohome}/config/ivy

Chains

1. local
2. LocalThenRemote
3. remote ( contains chain, and resolvers for zero and ibiblio
4. RemoteThenLocal

Resolvers

1. private
2. userhome
3. workspace

Local Repository

The local repository is where artifacts are stored to be used by an application. The local repository is a shared repository that can be used by all applications. A corollary to the repository is exanded directory. A simplification to sharing a zero application, is the application is packaged as a simple zip file. The ivy file for a zero application declares a single publication stanza, the zero zip zip file. Ivy can be used resolve the zip, but in order for an application to use the zip it must first be expanded. The local repository also has an expanded directory, where the resolved artifacts are expanded.

What is the local repository used for today?

1. Artifacts need to be local to be used
2. The repository is shared between zero applications, providing a performance enhancement. Since the jars are loaded from the same location for multiple apps they can be shared.
3. The repository can be searched to find dependencies.
4. A stable snapshot of dependencies. The remote repositories can be updated at any time.
5. Local repository can be managed. Dependencies can be pruned or deleted.

Ivy provides a resolver for a local repository, the filesystem resolver. The filesystem resolver resolves local dependencies without copying them to a cache. The resolver is marked with isLocal set to true. The userhome resolver is the resolver for resolving artifacts in the local repository.

The local repository is the repository where dependencies are resolved from. This means that the ivy.xml for a dependency is in the repository, and the artifacts associated with the dependency is also in the local repository. The retrieve step of a resolve is used to build up the information to build the classpath by locating the artifacts using the ivy report. The retrieve step uses a callable for each type of artifact.

1. LibArtifactCallable ( typically used for jar dependencies )
2. ZeroArtifactCallable ( used for zip dependencies, assumes a zero project package )
3. ProjectArtifactCallable ( used for peer zero projects, also builds peer projects in cli environment )

Getting remote artifacts

The dependency management in Zero tries to separate remote resolves from local. Local resolves are fast and do not required an internet connection. The local resolve also does not update to a newer version of an artifact, it will always use the latest in the repository.

One of the issues with an incubation type environment like Zero is newer releases of components may contain breaking changes. So a remote resolve with dependencies specifying a dynamic version can have the side effect of pulling a breaking change. The dependency management feature tries to work around this issue by separating local resolves, from the potentially dangerous remote resolves. A second problem with remote resolves is with license agreements. The user should know that they are going off box to retrieve an artifact and they should be concerned with license agreements.

A remote resolve is not implemented in typical ivy verbs. Ivy has the notion of resolve and publish. The typical ivy resolve against a remote repository will download the artifact into the cache and store meta data that ties that artifact in the cache with the resolver that resolved it. This allows the same artifact to be resolved later on by the same resolver, but the cached version of the artifact is found. The remote artifact is now in a cache and can be found if later resolved by the same resolver. So Zero's dependency management could have defined one set of resolvers and used a local repository that could be used for local publishes and the cache for remote artifacts.

But there were two challenges that forced a different approach.

1. Separate local resolves from remote
2. Provide local repository management

The Zero dependency management instead decided to move the artifacts resolved by the remote resolver into the local repository, eliminating the need for a cache. The resolve of a remote artifact is actually are resolve remote, publish local. This separation of resolver chains allows remote resolves to be a special case to avoid the potential issues involved with resolving remotely.

Ivy cache

The Zero dependency management does not have an ivy cache per say. Instead, the local repository and cache are merged so that any resolver that is marked isLocal=false, will resolve dependencies directly into the local repository. This was an optimization so the artifacts did not have to be copied to the local repository and the cache deleted after each remote resolve.

Fixups that had to occur

Note: even resolvers with isLocal=true are putting artifacts in the local repository

1. ivy file has to be moved ivys directory
2. create checksums for ivy file
3. remove ivydata.properties file
4. clean up root directory of cache/repository

Ivy creates a ivydata properties files for all resolved artifacts, local and remote.

Typically ivydata.properties file.

#ivy cached data file for zero#zero.application.template;1.0.10439.M4
#Tue Feb 05 15:29:50 EST 2008
artifact\:zero.application.template\#zip\#zip\#1424414251.location=http\://www.projectzero.org/repo/zero.repo.latest/zero/zero.application.template/zips/zero.application.template-1.0.10439.M4.zip
artifact\:zero.application.template\#zip\#zip\#1424414251.is-local=false
artifact.resolver=zero1
artifact\:ivy.original\#ivy\#xml\#1307966143.location=http\://www.projectzero.org/repo/zero.repo.latest/zero/zero.application.template/ivys/ivy-1.0.10439.M4.xml
artifact\:ivy.original\#ivy\#xml\#1307966143.is-local=false
resolver=zero1

And here is one for an artifact from a local repository

#ivy cached data file for org.apache.ant#ant;1.7.0
#Tue Feb 05 10:16:01 EST 2008
artifact.resolver=userhome
resolver=userhome
artifact\:ivy.original\#ivy\#xml\#-31980803.location=C\:\\zero\\workspace\\m4.5\\zero.tools.tests\\test_workspace\\zero.tasks.AntTasksTest\\repository\\org.apache.ant\\ant\\ivys\\ivy-1.7.0.xml
artifact\:ivy.original\#ivy\#xml\#-31980803.is-local=true

The problem that we hit was if a resolve found a artifact using the private resolver, then it would fail when another resolve tried to find the same artifact using local.

Zero get is a programmatic means to download artifacts from a remote repository, there is no ivy file, the module descriptor is created programmatically. The zero gets leave properties file in the root directory of the cache.

#zero.tools#resolver.get;1.0.0 resolved revisions
#Tue Feb 05 10:15:59 EST 2008
+branch\:\#@\#\:+@\#\:NULL\:\#@\:\#@\#\:+organisation\:\#@\#\:+zero\:\#@\#\:+module\:\#@\#\:+zero.application.template\:\#@\#\:+revision\:\#@\#\:+latest.revision\:\#@\#\:=1.0.9958.M4 release

<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="1.0">
   <info organisation="zero.tools"
      module="resolver.get"
      revision="1.0.0"
      status="release"
      publication="20080205101559"
   />
   <configurations>
      <conf name="default" visibility="public"/>
   </configurations>
   <publications>
      <artifact name="resolver.get" type="jar" ext="jar" conf="default"/>
   </publications>
   <dependencies>
      <dependency org="zero" name="zero.application.template" rev="latest.revision" conf="*->*"/>
   </dependencies>
</ivy-module>

Note: ivy.original file is showing up in ivys directory, need to figure out why, this is new

Note: useOrigin should be set to false, we don't want to copy artifacts from local repository to cache.

New discoveries

The reason we have to move the ivy file in the cache is because the default ivy cache pattern is different than the one zero uses. New this is configurable

Ivy regardless of the useOrigin setting is moving the ivy from a local repository into the cache. The file ivy.original also needs to be cleaned up.

IvySettings settings = ivy.getSettings();
settings.setCacheArtifactPattern(Manager.ARTIFACT_PATTERN);
settings.setCacheIvyPattern(Manager.IVY_PATTERN);
File _cache = ivy.getSettings().getDefaultCache();
if ( cache == null ) {
   _cache = cache;
}
       
CacheManager manager = new CacheManager(settings, _cache);

Workspace Resolver

The workspace resolver is a zero unique resolver that supports resolving against peer zero projects. The workspace resolver works in the Eclipse and Command Line by abstracting the workspace and project notion. In Eclipse, the eclipse apis are used to get the workspace and the projects in a workspace. For the command line, a cliAppsDir is used as the root directory for the workspace. A project is found by looking for projects that have a config/ivy.xml file.

The local chain specifies the workspace resolver before userhome, so workspace projects are found as dependencies before dependencies located in the local repository. The project projects are indicated in the classpath by a dependency name that ends with ".project". This allows the Zero classpath container in eclipse to correctly reference peer projects. The command line however uses the classpath.cli which walks project references to correctly build the classpath.

Private Repository

The private resolver is a resolver to find artifacts in the private repository of an application. Currently, only standalone packages have the embedded private repository.

Zero Resolved Properties

The .zeroresolved.properties is written out as part of the resolve step. The information in this file is needed in order to run the zero application. The file will list the classpath, which is entries for all of the resolved projects, excluding the current, and can list peer projects as .projects references. The classpath is used by the Zero classpath container in eclipse. The classpath.cli is similar, but includes the entries for the current project, and does not contain project references, instead list the classpath entries for the project and it's transitives. The dependencies entry is all the Zero dependent projects in the virtualized order. Then the path for each of the dependencies is listed, again to support virtualization.

Order of dependencies

Breadth-first, but if a node is found more then once, last one wins.

Remote resolvers

Most of the resolvers are split between chains and resolvers. The local resolver is a chain that references resolvers defined in a separate configuration file. The remote resolvers are different in that a single file specifies both the resolver and the chain. This design was to allow supporting the user to modify the remote resolvers in eclipse. The remote resolvers are specified through a remote.properties file, and the remote.xml is generated.

The resolvers are specified as either zero or maven, and they are mapped to the appropriate Ivy resolver. Only the path can be specified. the ivy pattern is hardcoded in zero ( sort of, anyone that nows ivy can get around that. )

• zero.1=http://www.projectzero.org/repo/zero.repo.latest/
• maven.2=http://repo1.maven.org/maven2/

Note: Each time the ivy instance is created in Manager, the timestamp of the properties file is checked to see if remote.xml needs to be generated.

Ivy Patterns in Zero

• ivy pattern [organisation]/[module]/[type]s/[artifact]-[revision].[ext]
• artifact pattern [organisation]/[module]/[type]s/[artifact]-[revision].[ext]

Maven Repository resolves

Zero dendency management support maven repositories through ibibilo resolvers. Discuss the maven configurations and how to specify when resolving from a zero application

1. transitives = false
2. excludes

Configuration

The ivy configuration can be found in the config/ivy directory under zero.home. The ivyconf.xml file includes all the chains and resolvers that the zero cli uses to resolve and publish an application.

Multiple Local Repositories

Can we support multiple local repositories with how local and cache are merged

One potential trick to support multiple locals. Put the additional locals in the remove.xml and set the attribute local=false. This will treat the additional local repositories as remote and copy the files into the local repository. Using this trick to allow the zero.tools.test to have a seed repository that is accessed through the file resolver as the remote resolver.

-- bonanno - 10 Dec 2007