Retrieving the native libraries for a specific architecture

Currently a module is packaged as a single zip. If the module includes natives, then the natives for all platforms are included in that zip. As sMash adds support for more platforms this solution won't scale. If a developer is running locally and has no intentions of packaging the module for other platforms then downloading natives for other architectures is increases download times and also wastes disk space.

The redesign for natives will affect packaging, publishing and resolving ( resolve, update, mg update, mg get).

Plan B for natives

Using configurations

Ivy offers configurations where the artifacts of a module can be split up and targeted to a configuration. The configuration support can be used to split the module zip from the natives. A configuration will be added to the ivy and the configuration name will match the architecture and operating system. The package step will look to see if a configuration exists for the natives and then exclude the corresponding directoreies under lib from the module zip. In this case there will also be a native archive for each of the supported native libraries.

Here is an example of the natives that should be added to any module that will package the natives as separate artifacts.

    <configurations>
        <conf name="x86_linux" description="artifacts required on x86 linux platforms"/>
        <conf name="x86_windows" description="artifacts required on x86 windows platforms"/>
        <conf name="x86_mac" description="artifacts required on x86 windows platforms"/>
        <conf name="all_natives" extends="x86_linux, x86_windows, x86_mac" description="all platform support"/>
    </configurations>

Other configurations can be supported, but we are creating the convention that the configuration name should not exist as directory under lib.

Note These configurations are added to all ivy files programmatically during a resolve. Ivy requires that configurations are defined to resolve that configuration.

The ivy file also needs to list the natives publications and the corresponding configurations.

    <publications>
      <artifact type="zip" />
      <artifact name="x86_windows" type="zip" conf="x86_windows" />;
      <artifact name="x86_linux" type="zip" conf="x86_linux" />
    </publications>

The publish step for the typical module creates an ivys and zips directory. If the module defines natives then there will also be a natives directory on the publish step.

Note If these modifications are not made to the ivy file, then the package step will build a single zip file that includes all the natives.

Package

The packaging step will be changed to determine the supported natives by reading the ivy file to find the specified configurations. The naming convention needs to mirror the directories under lib with the _ (underscore) changed to a file separator. If the directory under lib that corresponds to the configuration, then the directory is excluded from the module zip and a native archive is created.

Standalone

Standalone will only allow resolving the natives that are currently in the local repository.

The zero package standalone will support a new option, -natives. This option can be set to any of the supported natives configurations. The default package will use the value set in local.properties, which is resolve all natives. However, an interesting us case is building a standalone for a specific platform, reducing the size of the shared zip. In this case the -natives can be set to all_natives or x86_linux for example.

The following example shows creating a standalone package for linux from a windows machine

C:\cli\sebring\5534\zero\myapp10>..\zero package standalone -natives=x86_linux -overwrite
CWPZT0628I: Packaging C:\cli\sebring\5534\zero\myapp10
CWPZT0503I: Build standalone package myapp10
CWPZT0502I: Created standalone package C:\cli\sebring\5534\zero\myapp10\export\myapp10-all-1.0.0.zip
CWPZT0600I: Command package was successful

If a known native is specified, but the native has not been downloaded then an error is displayed.

C:\cli\sebring\5534\zero\myapp10>..\zero package standalone -overwrite -natives=all_natives
CWPZT0628I: Packaging C:\cli\sebring\5534\zero\myapp10
CWPZT0503I: Build standalone package myapp10
CWPZT0538E: Download Failed for the following dependencies
        zero:native_process!x86_mac:1.1.0.0.25292
CWPZT0601E: Error: Command package failed

Unfortunately, the only fix for this is to update the module

If an unsupported native is specified then an error is displayed.

C:\cli\sebring\5534\zero\myapp10>..\zero package standalone -overwrite -natives=x86_aix
The configuration x86_aix is not a supported configuration,
        Supported configurations are [x86_mac, x86_windows, x86_linux, all_natives, default]
CWPZT0601E: Error: Command package failed

Publish

The publish step still publishes the module zip. The -f option is used to specify the module zip. If the ivy file has publications specified for the native archive, then the publish step will publish those archives as well.

Resolve

The GetModules needs to determine the conf and programmatically set it in the module descriptor. Typically it can determine the os and arch using the tools utils.

The expand step needs to find the native zips and expand them in the lib directory.

P8

One of the suggestions was to only split up p8 into natives. The current module zip is 18 meg.

Before

Size	Archive
18,206,021	p8-2.0.0.0.26178.zip

When the ivy file for p8 is modified the archives are

After

Size	Archive
2,646,397	p8-1.1.0.0.26474.zip
7,051,669	x86_linux-1.1.0.0.26474.native
1,489,084	x86_mac-1.1.0.0.26474.native
7,020,090	x86_windows-1.1.0.0.26474.native

So a linux resolve for p8 after the split will download

Size	Archive
2,646,397	p8-1.1.0.0.26474.zip
7,051,669	x86_linux-1.1.0.0.26474.native

or 9698066 instead of 18206021, roughly half. The mac is much less for some reason.

Issues

Module Parser

The ivy XmlModuleParser has a default configuration that can not be updated. So there is no way to change the default configuration for dependencies when one is not specified in the ivy. Ivy also declares the XmlModuleParser final so extending the class will not work.

The configurations need to be defined in all ivy's. The fix is to programmatically add the supported configurations to the module descriptor.

Native providers need to declare the artifact and the supported configuration. There is a naming convention that needs to be adhered to. Also may need a different artifact type of native libraries, otherwise the expand will have to distinguish between a module zip and a native zip. The lack of an ivy can be used to tell.

Publish

publish -f doesn't make sense anymore. The a zip is specified with the -f, but this is only one of the potential artifacts for publish. The -f option will specify the zip file that contains the ivy.xml file to use. The parent directory of the -f zip will be used to obtain the other artifacts.

GetModules

The behavior we want to achieve is if the module has been resolved with a subset of the natives, and later additional natives are required they are retreived. A problem was encountered with this behavior with GetModules. The issue is when a module is resolved by ivy it is expected that all artifacts have been published with the module. The scenario where the ivy is downloaded along with the zip and a subset of the natives and then moved to the local repository, some of the artifacts specified in the ivy are missing. So the subsequent resolve with a different configuration fails because the download for the needed artifact fails.

The problem is caused by how we move artifacts from the cache to the local repository. Ivy expects that if the ivy is found in a repository, then all of the artifacts are also available. We broke the rules with natives, because we only retrieve the natives we need. The second part of the problem is we use a localThenRemote resolver. So when the ivy is found in the local repository ivy stops looking. We need to use the localThenRemote, otherwise we will re-download modules already retrieved.

Workaround: 1. Detected the failed download, and issue another GetModules for just the failed module and transitives set to false. This will retrieve only the failed module.

-- bonanno - 06 Nov 2008