December 30, 2009

Empower Hudson with Artifactory - Track and Replay Your Build Artifacts

Overview
In this blog, I will demonstrate how to integrate Hudson with JFrog's Artifactory repository manager to have full build-to-artifacts traceability. We will use Artifactory plug-in to deploy the Hudson build artifacts and track them back to their original build.

Keeping the history and reproducibility of code is a must-have for any modern project.
Using one of the different flavors of version control applications, you can easily reproduce the state of any point in the past using the different methods of SCM tagging.

But what happens when you want to reproduce binary products from a certain phase?
Are dependencies considered? Does anyone really remember what version of dependency X was used in version 1.0 or in version 3.1 of your application? What if you used version ranges or dynamic properties? Was the application compiled using JDK 5 or 6?

All this information can be recorded during the publication of your binaries, which is usually done by a CI server of your choice.
Your CI server has all the knowledge required in order to reproduce a build:
  • Information on the builds themselves
  • The published items
  • Version information
  • Dependencies
  • Build environment details
But how can you capture all this data? This is where Artifactory kicks in!

Artifactory (v2.1.3+ OSS) is open for communication from any build process to receive information needed for tracing/reproducing a build - the sender of this information is typically your build server!
The information is transferred
via REST in the form of a BuildInfo JSON object and contains details about the modules, artifacts, dependencies, environments, properties, and more.
All builds and binaries are provided with bi-directional links that enable you to reproduce and analyze the impact of any action.

Presently, JFrog provides a first integration with Hudson and Maven 2. Other technology stacks are coming, but for the purpose of this blog I will use a setup of Hudson with a Maven 2 build.

So let's get "crackin'"!

We assume that your instance of Hudson is already configured to request all it's dependencies from Artifactory. This of course ensures that all your build's dependencies are cached in Artifactory and can be used for build reproducibility.

Installing Hudson's Artifactory Plug-in
To install the Artifactory plug-in, simply browse from the main menu to "Manage Hudson" -> "Manage Plugins" -> "Available" Tab, and check to enable "Artifactory Plug-in". Once the plug-in has been downloaded and installed, restart Hudson for the changes to take effect.

Now we'll configure the plug-in on a system-wide level and point it to the Artifactory to which we would like to publish the information (please note that Artifactory should be running and available at this point).
To do this, enter the "Configure System" menu via "Manage Hudson" -> "Configure System", and then configure the URL (up to the application context name, like "http://localhost:8081/artifactory"), and optional credentials (if anonymous access is enabled, you don't need to provide them). The need for credentials of an authenticated user comes from the fact that Hudson requests a list of deployable repositories from Artifactory, so you can choose the destination of your binaries at a later stage.
Notice that you can add multiple Artifactory configurations to suit your needs.

Next, we'll configure the plug-in at the "Job" level.
Enter the Job configuration by selecting your Job and clicking the "Configuration" link. Scroll down to the Post-build Actions option group, and select "Deploy artifacts to Artifactory". Once selected, the menu will expand and will let you choose the "Artifactory server" and "Target repository" to which to deploy. As implied by the field names and the "Deployer username" and "Deployer password" credentials you provide, you must have "Deploy" permission on the target repository you select.
Being able to select your deployment target from a ready-made list, which is received directly from Artifactory, helps you avoid the pitfall of configuring your "Distribution Management" with typos.

The Artifactory plug-in deploys via REST API, which optimizes the process of unique/non-unique snapshots and doesn't require credential and distribution management configuration in your settings.xml and POM files.
Unlike Maven, which deploys each module as its build is completed (which may result in a partial deployment of your project's artifacts if your build fails at some point), the Artifactory plug-in deploys only when the entire build completes successfully (much like the built-in Hudson deployer). Each deployed artifact is tagged with buildName and buildNumber properties, and finally the Build Info is published.

At this point, you can run your Job, and then view the "Console output" to see the deployment and build info publication log messages.


Artifactory's Build Management

Now that the Job is complete, the artifacts are deployed, and the build info is published, we can view the build info in Artifactory by clicking the "Artifacts" tab under the new "Browse:Builds" sub-menu.
Here, we can see a list of all the published build names and the time each was last built.
Drilling down through the build number list, we can view the general info, the published modules, and the XML representation of the selected build.
Notice that the top of the build browser displays navigable breadcrumbs that are also synchronized with a RESTful URL that provides easy access to every part of every build.
The general info tab displays the main details about the build (name, number, type, etcetera) properties that were attached, and even the option to save the published module's artifacts and dependencies as saved search results (requires the "Smart Searches" add-on).

Clicking on the name of a module displays a list of the artifacts and dependencies that are part of the selected module.





When deleting an artifact that's associated with a build, either as a product or a dependency, Artifactory will notify you of the association prior to the removal.

Promotion of published modules is also made possible by the "Save search results" actions that are available through the General Build Info tab (requires the "Smart Searches" add-on).
Moreover, buildName and buildNumber properties, allows us to manually search build artifacts through the Property Searcher (requires the "Properties" add-on).

Conclusion
Using Hudson (and others to be supported soon) and Artifactory we've:
  • Supplied Hudson with all the needed dependencies from Artifactory—helping us keep the exact dependencies that were used in each build
  • Deployed all produced binaries to Artifactory—helping us keep and promote all the products of the build
  • Published build information to Artifactory—helping us manage and keep track of every build, environment, product, and dependency
With the assistance of these tools and methods, you will be able to reproduce and execute a build from any point of recorded time or compare information between different builds.
You may want to visit our build integration wiki page for a more in-depth explanation of the process.

Have fun, and be careful not to break the build. ;)




December 24, 2009

The one that talks, the one that does!

In a blog "Why Putting Repositories in your POMs is a Bad Idea", Sonatype "asked" the open source community to manage their Maven2 POM file correctly.

This is a good and important request, since Maven will not work correctly:
  • Over time (due to URL changes)
  • In a closed environment (no direct access to the Internet from a developer machine)
  • Because it will shortcut the repository manager of your choice (Nexus, Archiva, or Artifactory) for resolving dependencies—this is most important.


We are facing this problem for almost every customer that uses Maven, and most of them are using the lazy and dirty solution of "mirrorOf". It is argued in the blog that mirroring all Maven requests to a single URL is a good idea: We know it is a bad idea as it completely takes away control of isolating the source for releases, snapshots, and plug-ins!

There is however, a dramatic sentence in the blog about POM files coming from Open Source projects (it actually applies to everyone):
"The entries you have defined will be burned forever into your released POMs."
It sounds like Maven is broken by design and forever, because of all the bad POM files that already exist out there.

Since we support our customers, and they are suffering from actions that are not under their control, we decided to fix it.

In the latest version of Artifactory (2.1.3), there is now a new feature: Automatic cleanup of remote repositories declared in POMs.
You can now configure any virtual repository to automatically clean up rogue remote repositories declared in POM files.

By default, Artifactory will do it for repositories and plug-in repositories directly declared under the project POM entity, or declared inside an active-by-default profile. You can enforce a deeper cleanup that removes all repository and plug-in repository declarations in all profiles.

Using Artifactory as your repository manager means that you will never get "burnt forever" by innocent mistakes done in POM files of nice, popular Open Source projects.

"The one that talks, the one that does!"