December 30, 2009

Empower Hudson with Artifactory - Track and Replay Your Build Artifacts

Overview
In this blog, I will demonstrate how to integrate Hudson with JFrog's Artifactory repository manager to have full build-to-artifacts traceability. We will use Artifactory plug-in to deploy the Hudson build artifacts and track them back to their original build.

Keeping the history and reproducibility of code is a must-have for any modern project.
Using one of the different flavors of version control applications, you can easily reproduce the state of any point in the past using the different methods of SCM tagging.

But what happens when you want to reproduce binary products from a certain phase?
Are dependencies considered? Does anyone really remember what version of dependency X was used in version 1.0 or in version 3.1 of your application? What if you used version ranges or dynamic properties? Was the application compiled using JDK 5 or 6?

All this information can be recorded during the publication of your binaries, which is usually done by a CI server of your choice.
Your CI server has all the knowledge required in order to reproduce a build:
  • Information on the builds themselves
  • The published items
  • Version information
  • Dependencies
  • Build environment details
But how can you capture all this data? This is where Artifactory kicks in!

Artifactory (v2.1.3+ OSS) is open for communication from any build process to receive information needed for tracing/reproducing a build - the sender of this information is typically your build server!
The information is transferred
via REST in the form of a BuildInfo JSON object and contains details about the modules, artifacts, dependencies, environments, properties, and more.
All builds and binaries are provided with bi-directional links that enable you to reproduce and analyze the impact of any action.

Presently, JFrog provides a first integration with Hudson and Maven 2. Other technology stacks are coming, but for the purpose of this blog I will use a setup of Hudson with a Maven 2 build.

So let's get "crackin'"!

We assume that your instance of Hudson is already configured to request all it's dependencies from Artifactory. This of course ensures that all your build's dependencies are cached in Artifactory and can be used for build reproducibility.

Installing Hudson's Artifactory Plug-in
To install the Artifactory plug-in, simply browse from the main menu to "Manage Hudson" -> "Manage Plugins" -> "Available" Tab, and check to enable "Artifactory Plug-in". Once the plug-in has been downloaded and installed, restart Hudson for the changes to take effect.

Now we'll configure the plug-in on a system-wide level and point it to the Artifactory to which we would like to publish the information (please note that Artifactory should be running and available at this point).
To do this, enter the "Configure System" menu via "Manage Hudson" -> "Configure System", and then configure the URL (up to the application context name, like "http://localhost:8081/artifactory"), and optional credentials (if anonymous access is enabled, you don't need to provide them). The need for credentials of an authenticated user comes from the fact that Hudson requests a list of deployable repositories from Artifactory, so you can choose the destination of your binaries at a later stage.
Notice that you can add multiple Artifactory configurations to suit your needs.

Next, we'll configure the plug-in at the "Job" level.
Enter the Job configuration by selecting your Job and clicking the "Configuration" link. Scroll down to the Post-build Actions option group, and select "Deploy artifacts to Artifactory". Once selected, the menu will expand and will let you choose the "Artifactory server" and "Target repository" to which to deploy. As implied by the field names and the "Deployer username" and "Deployer password" credentials you provide, you must have "Deploy" permission on the target repository you select.
Being able to select your deployment target from a ready-made list, which is received directly from Artifactory, helps you avoid the pitfall of configuring your "Distribution Management" with typos.

The Artifactory plug-in deploys via REST API, which optimizes the process of unique/non-unique snapshots and doesn't require credential and distribution management configuration in your settings.xml and POM files.
Unlike Maven, which deploys each module as its build is completed (which may result in a partial deployment of your project's artifacts if your build fails at some point), the Artifactory plug-in deploys only when the entire build completes successfully (much like the built-in Hudson deployer). Each deployed artifact is tagged with buildName and buildNumber properties, and finally the Build Info is published.

At this point, you can run your Job, and then view the "Console output" to see the deployment and build info publication log messages.


Artifactory's Build Management

Now that the Job is complete, the artifacts are deployed, and the build info is published, we can view the build info in Artifactory by clicking the "Artifacts" tab under the new "Browse:Builds" sub-menu.
Here, we can see a list of all the published build names and the time each was last built.
Drilling down through the build number list, we can view the general info, the published modules, and the XML representation of the selected build.
Notice that the top of the build browser displays navigable breadcrumbs that are also synchronized with a RESTful URL that provides easy access to every part of every build.
The general info tab displays the main details about the build (name, number, type, etcetera) properties that were attached, and even the option to save the published module's artifacts and dependencies as saved search results (requires the "Smart Searches" add-on).

Clicking on the name of a module displays a list of the artifacts and dependencies that are part of the selected module.





When deleting an artifact that's associated with a build, either as a product or a dependency, Artifactory will notify you of the association prior to the removal.

Promotion of published modules is also made possible by the "Save search results" actions that are available through the General Build Info tab (requires the "Smart Searches" add-on).
Moreover, buildName and buildNumber properties, allows us to manually search build artifacts through the Property Searcher (requires the "Properties" add-on).

Conclusion
Using Hudson (and others to be supported soon) and Artifactory we've:
  • Supplied Hudson with all the needed dependencies from Artifactory—helping us keep the exact dependencies that were used in each build
  • Deployed all produced binaries to Artifactory—helping us keep and promote all the products of the build
  • Published build information to Artifactory—helping us manage and keep track of every build, environment, product, and dependency
With the assistance of these tools and methods, you will be able to reproduce and execute a build from any point of recorded time or compare information between different builds.
You may want to visit our build integration wiki page for a more in-depth explanation of the process.

Have fun, and be careful not to break the build. ;)




9 comments:

  1. Excellent review!

    Within every organization using Maven as its main build tool, CI build server must have a good integration facility with the local repository.

    The best way to examine night builds it to actually use them on a regular base.

    Thanks for a good work.

    ReplyDelete
  2. It seems as though the plugin fails when a multi-module maven job is using hudson's incremental build option and not all modules are built. Further, I'm guessing that if the previous build's artifacts are still present, this failure does not occur, since it seems to do with the plugin's attempt to find the artifact.

    ReplyDelete
  3. Will the TeamCity support be comming soon?

    ReplyDelete
  4. @John: Yes :) We are already working on it.

    ReplyDelete
  5. We have ant scripts (in Hudson) that builds a bunch of jars / dlls (ant on msbuild on VS .sln). What's the recommended way to put the jars/dlls produced by our ant scripts into artifactory?

    The jars will be used by other maven/non-maven projects.

    We would like to use maven coordinates (groupid, artifactId, version) to identify the jars/dlls. We would like to avoid writing POMs, is that possible?

    Thanks...

    ReplyDelete
  6. Once i had configure more then one Artifact path to hudson project, is it possible to configure with respect to user so if i login and run the built the build output will get copy to aaa path and if U login and run the build it will get copy to bbb path

    ReplyDelete
  7. Hi,

    Anyone else faced a problem while hudson performing deploy, I am getting the following exception. It seems to be related to atlassian ant task dependencies.

    [ArtifactDeployer] - Starting deployment from the post-action ...
    FATAL: Could not initialize class com.atlassian.ant.tasks.PermissionsUtils
    java.lang.NoClassDefFoundError: Could not initialize class com.atlassian.ant.tasks.PermissionsUtils
    at com.atlassian.ant.tasks.CopyWithPerms.doFileOperations(CopyWithPerms.java:75)
    at org.jenkinsci.plugins.artifactdeployer.service.LocalCopy$1CopyImpl.doFileOperations(LocalCopy.java:38)
    at org.apache.tools.ant.taskdefs.Copy.execute(Copy.java:549)

    ReplyDelete
    Replies
    1. I'm getting the same error. Any fix for it?

      Delete
    2. Not sure the blog comments is the best place asking for help :) Can you please post it on forums.jfrog.org or stackoverflow?

      Thanks!

      Delete