Extend vCloud Director with service builder

Thursday, 28 February 2013 04:14 Last Updated on Monday, 04 March 2013 12:04 Written by Christophe Decanini

In a previous article I explained how we could extend vCloud Director with vCenter Orchestrator and then provided a full implementation. This was using the vCloud Director blocking tasks and notifications feature which allow to extend existing vCoud Director operations.

vCloud Director 5.1 introduced a new feature called API extensions which basically allows a cloud provider to extend the vCloud Director API with custom services with leveraging VMware or third party applications. In this article I discuss why someone would want to leverage this feature and explain the "service builder" solution I have created around it.

The first question you may ask is "Why would you want to extend vCloud Director API ?"

As the vCloud Director Administrator I need to provide a feature not available in the vCloud API to cloud API consumers (i.e. the portal development team)

It is a combination of several vCloud API calls, and or API calls with predetermined parameters I can define
It is included in another VMware or third party API
It is a mix of both

OK, but then why not use the existing APIs instead of extending vCloud Director's one ?

Lets see some of the common portal / cloud implementations.

The first one is the old fashioned portal calling the vCloud and other APIs. Here are two variations exposing or not the vCloud API. Exposing the vCloud API allows to leverage all the applications consuming the vCloud API. For example a service provider will want his tenants to use vCloud Connector to copy back & forth vApps from / to vSphere / vCloud. This design is IMHO flawed in several aspects:

The portal must include all the automation you would typically put in an orchestration solution. This means re-inventing the wheel to re-create a lot of facilities offered by an orchestration engine, making these operation only usable by the portal or creating a custom web service when thousands of customers already used standard orchestration APIs. This will be a solution difficult to support that will be tied with the scripting language used to consume the underlying cloud and third parties APIs. These APIs will support different types of web services, different type of authentications.

- Please note - I have used a vCloud Automation Center picture to represent the portal. This could be any portal. vCloud Automation Center is not only a portal but also a workflow engine.

alt

The second approach is that orchestration rules them all ! The portal calls in the orchestrator calling the underlying systems. The two designs below are using this approach with keeping the orchestration API private and either exposing the vCloud REST API or not. The Orchestrator APi could ahve been made public to expose the additional services but there are some serious workflows / resources access control considerations required.

This approach was a good one when there was no cloud API and that the target system was vSphere / vCenter. The Orchestrator would expose a subset of the functionality with using several API calls chained together to perform the desired automation. Using orchestration has several benefits including supportability, modularity, use of hundreds of supported workflows, API abstraction, caching, high availability and many other things I detail in this article.

However this has a few flaws as well. An orchestration engine is meant to automate complex operations. A lot of the API calls started from the portal will be about almost real time operations such as refreshing a list of vApps, presenting information in a panel and so on. I have written about this in details in this article : Building your custom cloud portal - Knowing when to use the vCloud API and the vCenter Orchestrator web service

alt

Several successful cloud implementation involving using custom services I have seen have implemented the Cloud & orchestration design below in the left. While this resolve some of the issues mentioned previously it still make the custom services only available to the portal. With vCloud Director 1.5 blocking tasks and notifications allowed to customize existing vCloud Director operations, hence diminishing the number of calls to be made from the portal to the orchestration engine but direct calls to orchestration are most often required for operations required that cannot be started by vCloud blocking tasks and notifications. With vCloud Director 5.1 you can use the bottom right design.

alt

The reasons for prefering this design on the right are the following: As a cloud administrator I do not want to give direct access to my cloud backend applications:

Access: I provide only one single point of access : the vCloud Director API and I want to publish as a service only a given operation, not give access to all the API of any of my cloud back-end components.
Security: The only authentication, session management is the vCloud one.
Consistency : My customers only use the vCloud API which reduces their development effort (avoid using different web services and different authentications)

Giving API access for the functionality exposed in the portal is also an important differentiator between different cloud providers. This provides ways to develop client applications for specific uses, build automation. At VMware API parity with functionality exposed in the UI is an important goal for engineering for these reasons.

OK but what for ?

There are several use cases for providing functionality that is not out of the box for example:

vApp custom provisioning or decommissioning
vApp backup
vApp load balancing
vApp fault tolerance
vApp updates
vApp VMs security hardening
vApp VMs affinity / anti-affinity rules
vShield security groups
Install agents, inject parameters in vApp VMs
Database as a service
Update user password in Active Directory
iSCSI target for MSCS Clusters

A lot of these are achievable with the components of the vCloud Suite. There are a lot more use cases with integrating third party systems such as management systems. You can "cloudify" the whole datacenter if needed.

How ?

My colleagues at VMware have done an excellent job exploring the API documentation and explaining vCloud API extensions. Here are some references I have used to create "service builder". If you are not affraid of going deep technical I definitely recommend going through these:

Blog articles written by Christopher Knowles:

Blog articles written by Thomas Kraus:vCloud Director API Extensions (implementation example)

VMworld 2013 session OPS-CSM1379

I do not have the time to invest a lot of time in these articles and create my services manually. How can I get started quickly ?

With using service builder. Service Builder is:

A wizard based workflow to create a custom service and service operations (called service links in the vCloud extensibility API)
Expose workflow as service operations (/api/myservice/myOperation)
Expose this workflow as well in the "Workflow run service" so they can be managed (/api/workflow/workflowID/executionId)

In addition service builder creates a "workflow run service" that allows vCloud Director API to manage vCenter Orchestrator workflows.

And I forgot to mention it is free.

How do I get started with service builder ?

I have put together an online presentation explaining in details :

What is "vCloud service builder" and "vCloud workflow run service"
Pre-requisites
How it works animations
Installation
Usage
Examples for calling the newly created APIs
Custom operations monitoring with vCloud Director UI
Troubleshooting
Download link

Note : The presentation requires your Email and organization. This is solely for me to better understand who has interest in this tool and eventually to contact you in case I need further information on your submitted comments. This information will not be used for anything else and not be transmitted.