vCloud Director has a nice feature allowing to organize vApps into catalogs. If like me you need to replicate these catalog across organization or even different vCloud hosts you may find your solution here.

I have written a solution to do it a long time ago but never had the time to release it publicly. It is now done and here are the features.

Features

This package comes with a rich set of features.

Catalog comparison and replication

Replicate a catalog from one source catalog to a destination organization. If a catalog with the same name exists in the destination it will be modified to have the same content as in the source catalog. If not a new catalog with the same name will be created. The replication is one way (source to destination).
The actions taken are:

• Delete catalog items in destination not present in source
• Update name and description of previously copied catalog items that would have been changed in the destination
• Copy new source catalog item to destination (download once to vCO, upload to one or more vCD)

Each of these are optional. Destination VDC is also optional. If provided all the elements to be copied will have for destination the provided vDC. If not the workflow will find destination VDCs with matching names. If not the workflow will raise an exception.

Connected replication

When using the "Replicate catalog" workflow a single vCO server is used to download the vApp Templates from the vCloud Director hosting the source vApp Templates and upload these to the other vCloud Director. This requires to have the two vCloud hosts configured in vCO.

vCloud Director source -----------> vCO -----------> vCloud Director Destination

While the vCO server could be located on any network having access to both vCloud servers the vCO server should be located to optimize the network speed for example if the upload speed between the source and destination is a lot slower then the download speed the vCO server should be on the same network as the destination.

Several instances of the same workflow can run on the same vCO server with different destinations (there is a locking system to avoid uploading a file being downloaded). Below an example of a vCO server replicating a catalog from one cloud source to two cloud destinations.

Replication from filesystem

When using the "Export catalog to filesystem" a first vCO server is used to download the vApp Templates from the vCloud Director hosting the source vApp Templates. XML files describing the catalog and the catalog items are also downloaded.

You can then use the "Replicate catalog from filesystem" workflow to compare the file based catalog to the one on the destination server.

The main idea behind this two steps replication is to optimize the replication process by using a third party replication using deduplication, compression, safe transfers.

vCloud Director source -> vCO 1 -> Third party replication system source -----------> Third party replication system destination -> vCO 2 -> vCloud Director Destination

A first vCO server is located on the same network as the source vCloud Director to download the vAppTemplates as fast as possible to a filesystem hosted on a replication system (this requires having this filesystem accessible from vCO) . The replication system copies the changed blocks to the destination replication system. Once the replication finished the destination vCO upload the new files to the vCloud DIrector on the same network.

Parallel operations

The workflows use and abuse asynchronous sub workflows to make several operations running at the same time. and also copies the VMDK files within a vApp Templates in parallel. The level of parallelism is configurable.

The "Replicate catalog" workflow ran.

It started one to many "Copy catalog item" workflows in parallel

The "Copy catalog item" ran a single "Copy vApp Template multithreade" workflow

Which in turn ran one to many "Copy file across vAppTemplates" workflows in parallel

Which in turn runs:

• "Download single file from vApp Template" workflow (not shown on the picture since in this case it was already downloaded previously and the workflow optimize the process with not downloading the vApp Template again)
• Upload single file to vApp Template which will run several "Upload File chink to vAppTemplate" in serial (parallel upload of the chunks is not supported, they need to arrive in the right order).

Safer uploads

Sending over a multi GB file over http may cause reliability issues. If the upload fails you have to restart again. To avoid this the workflows uploads the files in chunks of a configurable size. If a chunk fails to be uploaded the workflow retries to upload it hence avoiding the need to restart completely. The chunk size is configurable.

Here I have run the same replication as before. At 14:35 I have unpluged my vCO server network connection to make the file chunk being uploaded failed. I have then plugged it again to simulate a temporary network issue. As you can see below the upload was resumed. The file chunk that was uploaded starting at 14:36:32 seconds was the same as the previous one (this can be checked by clicking on the workflow run and check the variables tab) . This is why we had 11 chunks uploaded in the previous replication and 12 in this one.

The other advantage is since vCO is an orchestration engine supporting checkpointing if the replication workflow would be stopped because of vCO server maintenance (i.e Windows update rebooting) the workflow would resume from the point where it uploaded the last chunk (as long as the maintenance time is not longer then the timeout vCloud Director has for waiting on the vApp Template update which is one hour according to my observation).

vApp Template Metadata

The vApp Templates metadatas are copied as part of the catalog replication (connected or from filesystem). If the metadatas are copied from a vCloud Host using a System admin organization and credentials to a host connection using a specific organization then the hidden and read only metadata will be created as read / write (this is the only option when connecting as org admin).

Not included

There are a lot of things that are doable and other that were not because of lack of API support in vCLoud Director or lack of time to add more features:

• Does not replicate medias : There is no API yet to download a media so no replication possible.
• Does not reconnect the vApp Template to the networks of the target cloud. This requires deploying the template, delete it from the catalog match the network to the ones in th e new environment, capture the modified vApp as a vApp Template. You can do this manually (one time operation) or create your workflow to do it (there is something similar already implemented in the mass VM import workflow in communities). The updated vAppTemplate will not be considered as a new one and will not be deleted / overwritten by the replication mechanism.
• Does not clean up the downloaded files. A separate workflow (scheduled or run by policy) could do this.
• The update of the catalog item does not update metadatas. I will consider adding it if this is something people ask for.

Documentation

Import the replication package using the vCO client.

The "Catalog Replication Settings" saves all the configuration in a configuration element. Open either the vSphere Web client or the vCO client and search for the "Catalog Replication Settings" workflow.

Catalog Replication Settings

Click on it, then right click / "Run a workflow".

General configuration

The first step is to define a path where vCO will download the vApp Templates. The pre-requesites are:

• There is enough disk space in this path (if you use the appliance you will have to mount an NFS share or create an additional drive)
• You have set proper share / file permissions
• The user running the vCO service has access to this path (by default this is not the case if this is a network share)
• You have changed the vCO access rights as documented here.

Replicate a catalog settings

The next step is for configuring the vApp Template transfer. This are the settings used when running the "Replicate a Catalog" workflow. You can set the number of parallel copies (one copy = 1 download + 1 upload) and a time out. When timing out the copy will be canceled. This is to avoid a catalog replication to ait forever on a copy that would be stalled.

The next setting is to set the number of parallel copies within a single vApp Template. For example if a vApp Template has several VMs with several disks these can be copied concurently.

Export a catalog to a filesystem

These settings apply for the "Export a catalog to a filesystem" workflow. These are similar settings as above except it has an option to keep some vApp Template settings during the (lossless) download. If this option is selected it is not possible to download files concurently within a vApp Template.

If the lossless option was not selected it is possible as before to provide the number of concurent downloads.

Replicate a catalog from the filesystem

The same settings as above but this time when uploading the vApp Templates.

And the same settings for the the vApp template file. The last one is for specifying the size of each chunk when uploading. This allows to have safer uploads that can be resumed in case of a network issue or vCO host maintenance.

Operations

Now that the replication configuration is done it is possible to start the replication workflows.

Replicating a catalog using a single vCO server

To replicate a catalog using a single vCO orchestrating a source and destination cloud run the "Replicate a catalog" workflow.

The first parameter is the source catalog.

The next step is to provide a destination organization and a destination VDC. If the catalog does not exist in the destination organization it is created. If it exists it will be updated. A one time full copy will be required before the workflow can snychronize the changes.

The next step defines what are the changes to apply. Beware that if you already have catalog items in the destination catalog the workflow will delete these on the first run and then copying the ones from the source catalog if the copy and delete options are selected.

You have the option to run the synchronization now or to Schedule it. Scheduling a recurrent task is a good way to automate the synchronization. Another way is to trigger the workflow using the vCloud Director notification on catalog changes.

The workflow will synchronize the catalog.

Replicating a catalog to and from the filesystem

To optimize the replication speed / bandwith usage you may want to use a third party system leveraging deduplication, compression and other mechanisms. To do this you need to run the "Export a catalog to a filesystem" workflow on the vCO server local to your source vCloud Director. You need to provide the catalog that you want to replicate.

Then you need to provide the export path. As a default it will use the one you have configured previously.

As before you can run the workflow immediately or schedule it.

Once the export completed and your third party replication is completed.

Run "Replicate a catalog from the filesystem" on the vCO server local to your destination cloud. This vCO server must have access to a copy of the files that were exported. Provide the full path to the catalog file (The catalog file name is the catalog name + ".xml").

Select the destination organization and VDC.

Select the synchronization options.

And as always you can run it right away or schedule it.

Download

You can find the package on the vCenter Orchestrator Communities