Configuring a Private Repository

This topic will discuss how to configure a private repository. The zero command line interface contains a zero-repository that provides a repository for each module group. The zero-repository contains a local repository under the modules directory for a module group that contains all resolved modules, including zero and maven artifacts. The zero modules are packaged as zip files and must be unzipped in the expanded directory in order to be used. The central local repository for all artifacts allows the JVM to share the classes to help reduce the memory foot print of a zero application.

The current design supports a local resolver chain, that consists of the following resolvers

1. Private Resolver
2. Workspace Resolver
3. Userhome Resolver

Private Resolver

The private resolver is a filesystem resolver that is set to the private directory in a zero module. The typical use for this resolver is to support the standalone resolver, but any module can contain a private repository. These artifacts are local and are not copied to the userhome repository

Workspace Resolver

The workspace resolver is a custom resolver that defines the repository as peer modules. The workspace concept allows peer modules to be resolvable without requiring that they are packages and published to the userhome repository. This resolver was a convenience for the build and eclipse environment.

Userhome Resolver

The userhome resolver is a zero filesystem resolver, which is an extension to the Ivy filesystem resolver. The zero file system resolver can use the .zero/shared/history.properties file to filter out versions of module. The history is used to lock a module into a certain version of a module when resolved. The userhome is also a known required local repository. When resolving again remote repositories, the modules need to be accessible locally. Ivy uses a caching concept, but zero instead retrieves the artifacts into the local repository. The userhome repository is used to publish remote modules.

Additional Local Repository

Nice to have, but initially focusing on remote repositories

A local repository is typically a filesystem repository. To support private configurable local repositories the command line needs to provide the following features

1. Configure the private local repository
2. Resolve against the private local repository
3. Publish to the private local repository
4. A means to package a local repository

The zero command line has a zero modulegroup addurl command that allows adding zero and maven repositories to the remote resolver chain. The remote repository must be accessible by URL. In order to support a private local repository, the zero modulegroup commands need to be able to add a filesystem resolver to the local chain.

Additional Remote Repository

Remote repositories are the preferred means to share modules/artifacts with other developers. This topic will discuss the 4 aspects of a shareable private repository.

1. The types of repositories
2. The structure of the repository
3. Creating the repository
4. Sharing the repository
5. Enabling others to use the new repository.

Types of repositories

The types of artifacts that can be published to a repository include zero modules and maven artifacts.

1. zero artifacts
2. maven artifacts ( jars with poms )

Typically, maven artifacts are retrieved from ibiblio servers, but they can also be available from a private repository described by an ivy.xml.

The anticipated private shared repository will want to share zero modules. A zero module is defined as a zip file with a corresponding ivy.xml in the config directory. There are other characteristics of a zero module like a zero.config file, but zip and config/ivy.xml will suffice for the discussion about creating the private shared repository. The private repository can be configured to provide both zero modules and maven artifacts, each with an ivy file to specify the artifacts.

Structure of a repository

The resolves use apache ivy, which allows defining the pattern for finding the artifact and ivys. The following is the patterns used for zero modules in the repository.

ZERO_ARTIFACT_PATTERN = "[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"
ZERO_IVY_PATTERN = "[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"

When a resolve occurs, the [arg] values are substituted with the values from the dependency line. So for example, when resolving the following artifact ivy will use the dependency line to first try to find the ivy file using the ZERO_IVY_PATTERN.

      <dependency org="zero" name="zero.core" rev="1.0"/>         

The substituted line will be zero/zero.core/ivys/ivy-1.0.xml, the type value is ivy and the artifact is also ivy. If the ivy.xml file is found, then the information about the artifact can be found, like the type is zip. The two patterns define how the repository has to be structured so the zero modules can be found.

An example of the zero repository on projectzero.org shows how the repository is configured.

https://www.projectzero.org/zero/silverstone/latest/repo/ is the base URL for the repository. The configured private repository will also have a base URL that will be provided to users that will be used when adding the new repository with zero modulegroup addurl.

Note: That checksum files are needed for the ivy and artifacts.

Creating the repository

Now that we have seen the structure of the repository, how should a private shared repository be created. A private repository can be created using manual steps to place the files in the appropriate paths, but this can be difficult and a utility needs to be written to create the checksums.

1. Do we need to provide a utility to create the checksums?

The easiest way is to develop the zero modules locally and then use zero package, and zero publish to publish the artifacts to a local repository. The local repository can then be copied to the appropriate directory under the apache server. This approach does require removing the modules from the copy of the repository that should not be included in the private repository, like the modules obtained from projectzero/smash repositories.

1. Should we provide instructions to use zero to host the repository?

Publishing to repository using fileserver resolver

This option requires that the private shared repository is on the same machine as where the modules are created.

Add fileserver resolver to site.xml, don't need to add to chain. Use same url resolver if you need to also resolve from the repository.

<filesystem name="my_private_repository" allownomd="false">
   <ivy pattern="/usr/local/apache2/htdocs/repo/${ivy.local.default.ivy.pattern}" />
   <artifact pattern="/usr/local/apache2/htdocs/repo/${ivy.local.default.artifact.pattern}" />
</filesystem>

1. Extend zero modulegroup addurl to support file resolvers.
2. Extend zero publish so resolver can be specified.

Publishing to repository using sftp resolver

This option is in the case the private shared repository is on another machine from where the modules are created.

Most likely use a unix server to host repository. Repository is still made available to users through apache, but you can use the sftp resolver to publish the modules.

Need to configure a ssh client, [[jsch][http://www.jcraft.com/jsch/] is one option.

Add sftp resolver to site.xml, don't need to add to chain. Use same url resolver if you need to also resolve from the repository.

http://ant.apache.org/ivy/history/latest-milestone/resolver/sftp.html?

    <sftp name="places2see" user="jim" userPassword="passw0rd" host="places2see.raleigh.ibm.com">
       <ivy pattern="/home/jim/repo_test/${ivy.local.default.ivy.pattern}"/>
       <artifact pattern="/home/jim/repo_test/${ivy.local.default.artifact.pattern}"/>
    </sftp> 

1. Extend zero publish so resolver can be specified. ( actually just need to add -resolver to options, code is there ).
2. Add a way to specify user name and password so they don't have to hard code in ivy.xml

Process:

1. Need to bootstrap cli, cli does not have ibiblio resolvers.
2. Add jsch to ivy file,
3. Resolve the cli.
4. Edit the site.xml and add the sftp resolver, snippet above.
5. Use this cli for publishing.
6. Create a new cli, zero modulegroup create devel, zero modulegroup addurl http://www.projectzero, zero switch
7. Create the apps using the new cli.
8. To publish apps, switch back to cli with sftp support, and zero publish -f myapp/export/myapp-1.0.0.zip

Issues There is a real issue with updating the ivy files with the sftp resolver, and making jsch jars available. We can describe the process, but then we have the issue all over again for each module, since once a module is resolved it has it's own classpath. If the site.xml file has been updated then every module has to add the dependency on jsch.

Sharing the repository

Set up apache, indexing must be enabled

Open the apache config in an editor, which by default is located at /usr/local/apache2/conf/httpd.conf, to define the repository as an accessible directory with the Indexes option turned on, and restart apache when you are finished to load the new config. See below, where $zerohome and $module.group.name should be replaced appropriately with your information as defined in the above instructions, and $repo_uri is the URI of your new repository location:

    Alias /$repo_uri /$zerohome/zero-repository/$module.group.name/modules
    <Directory "$zerohome/zero-repository/$module.group.name/modules">
        Options Indexes
    </Directory>

Note: The configuration above is a minimum setup, you will probably want to have additional options on the Directory directive to make it more secure.

Note: If you're testing from Windows, you may have a permission problem unless you use the following <Directory> setting, which replaces the one from above:

<Directory "$zerohome/zero-repository/$module.group.name/modules">
    Options Indexes
    Order allow,deny
    Allow from all
</Directory>

Enable others to use the repository

Once the apache server is running, make sure that indexing is turned on and you can browse the published modules. The next step is to provide the base url to the new users of this repository.

They will type the following command to add the new private shared repository to the chain of remote repositories. The private repository, will be the last in the chain.

zero modulegroup add url 

Authenticate Remote Repository

Future but if repository is protected the authentication credentials can be added to resolver in the ivy.

-- bonanno - 24 Jun 2008