Plug-in Generator v2 advanced techniques

In previous tutorials we created inventory object types, their children objects, workflows to operate these that we packaged for our end users.
In this tutorial we are going to create an orchestrator plug-in to list vCO / vRO workflows ! You may wonder why doing so ?
This is a good use case for leveraging some of the more advanced features of the plug-in generator v2 to handle complex REST APIs and improving performance when needed.

Prerequisites


Optionally you can download the multi-node plug-in to have some nice icons.
If you do so download the dar file, rename it .zip, extract it. Once extracted from the vCO client get in the design view / resources, browse to /Library/Dynamic Types/images. Select images, right click / import resources, browse your plug-in extracted folder to the resources/images folder and select the icons you want to import (vcoServer, workflow, workflow category, execution).


Create a plug-in


Right click on the inventory "Type hierarchy" / run Plugin gen -1- Create new plug-in

DTVCO-001.png


Name your plug-in and select an icon (any of your choice if you did not bother importing the multi node plug-in icons).
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-002.png


Submit / Run in background
Hit the F5 key to refresh the inventory and see your vCO namespace.


Add a host

We need a vCO host to interact with.

Right click on the vCO namespace / run the workflow "Plugin gen -2- Add new host:

DTVCO-003.png


Let the selected namespace.

b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-004.png

Enter a name for the vCO server you are going to use and its URL (note that I used the API URL)
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-005.png

Select basic authentication
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-006.png

Enter a vCO admin credential
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-007.png

Submit.

Create a workflow type


Right click on the vCOHost type / run the workflow "Plugin gen -3- Create a type


DTVCO-008.png

Enter the parameters:
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-009.png

  • Type name : workflow (no upper case)
  • Folder name : Workflows
Click next. It is going to show the host you have defined.
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-010b.png

Click next. Choose not to provide a findById and findAll URL

b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-011.png

Click next. Since we will need to know the URL to get the workflows let's check the built in API docs. Enter https://yourvCOServer:8281/vco/api/docs and browse to the workflow service.

b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-011a.png

The URL is /workflows. Enter it in the findRelation URL input
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-012.png

Hit tab - After a few seconds the response should be filled. Copy and paste it to a JSON viewer (I.E http://jsonviewer.stack.hu)

DTVCO-013.png
The array of workflow objects is under link.
Click next. Set the objects selector to .link, hit tab
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-014.png

Copy the objects preview. Click next.

b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-015.png

There is an error as it seems the properties that were collected are missing the required id and name properties. Click on the properties selectors input.
DTVCO-016.png

Unfortunately the properties there does not seem to match workflow properties such as id, name. description, version.
Paste the JSON you just copied to the JSON viewer.


b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-017.png

With unfolding the attributes property you can see that each of them has a name and a value. The issue with having such objects attributes instead of regular object properties is that you cannot just define id as being .id since there is no id property. You may think a few seconds that you can always use .attributes.1.value with assuming .1 will always contain the id but this may not always be the case, at least the API documentation does not guarantee it. What you need is to get the attribute value for the attribute having a name equal to "id".
With the plug-in generator V1 there was no way to write an expression for doing so but the V2 now include JSONSelect.
To understand how to write these queries you can read JSONSelect documentation. To test your expressions called selectors you can use online JSONSelect expression testers such as http://jsonselect.curiousconcept.com
Open it in a web browser, paste the JSON you copied and test your expressions to get the name, id and other values.


b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-018.png

If you did not find on your own to get the id the expression is .name:val("id") ~ .value
Basically it means that we look for the .value property for a .name sibling set with a value of "id".

b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-019.png

And it returns 5 workflow ids which is the sample I was working on (Preview size was set to 5 to avoid working with the full set of workflows which in my case are 1036).

You can now remove the existing properties that are not useful and add the id property.
DTVCO-020.png

Repeat the last operation for name, description, version
DTVCO-021.png


If you got your expression right the properties preview should now be populated

b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-022.png

Click next.
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-023.png

The properties are listing description and version (id and name being mandatory, they are not listed here). Submit.

Unfold the inventory. You should see something similar as below.

DTVCO-024.png

You may have noticed that during the creation of the object and unfolding the inventory is taking a long time.
We will now see what we can do to make the plug-in be faster. Using JSON Select is a code free way of implementing gathering objects properties of complex objects but while powerful making a JSON Select query per object properties * total objects number can be too long. Also in some cases getting some properties may require transforming the properties values with doing some parsing or even making new REST calls to get linked objects.

Testing and improving performance

Let's test getting all the Workflows objects.

Right click on the workflow type / Run the "Find all objects for type" workflow.

DTVCO-028.png

Submit. Click "Show workflow run". Select the General tab and wait for the workflow to finish.


DTVCO-029.png
In my case it took 1 minute 39 seconds to get all workflows. While this is not a disaster it is not offering a very good end user experience when unfolding the inventory or listing all the elements in a workflow input.


Right click again on the workflow type and run the "Plugin gen - Define a get properties action"

DTVCO-030.png


Fill in the parameters
b_600_0_16777215_01_images_phocagallery_tutorials_DTVCO-031.png

  • method : findAll
  • operationAction : getWorkflowsProperties (and do not set getWorkflowProperties - singular workflow)

Submit.
Right click again on the workflow type and run the "Plugin gen - Define a get properties action" - Show workflow run / General tab.

DTVCO-032.png
We are now down to 22 seconds which is a lot better for end users.

The getWorkflowsProperties is included in the plug-in generator package as an example on how to get object properties from the JSON returned by the HTTP query

DTVCO-033.png

It takes a jsonString as input, use JSON.parse to transform it in a JavaScript object, get the link property as we did previously and add each attribute name and value to a Properties() object and add it into an array of object properties it returns.
Here it was pretty straight forward. In other cases you may have to add more logic to get the properties, for example do additional HTTP GET operations to get linked properties. If you need so you can add a second input to the action that will be set with the REST Host this object is from.

If needed / wanted you can use actions for all three methods (findRelation, findAll, findById - in this case return a single Properties).
If the action is defined for an object and a method it will always be used before the configured selectors.


You may wonder how the cache timeout setting available during the type creation is affecting the performances. By default it was set to 30 seconds.

Change the cache setting by editing the cache configuration element. Right click / edit.
DTVCO-033b.png


Set a very high number here to make sure it exceed the time in second since the last time you ran the workflow to get all the workflow objects.


DTVCO-034.png


Right click again on the workflow type and run the "Plugin gen - Define a get properties action" - Show workflow run / General tab.


DTVCO-035.png
We are now down to 3 seconds to get hundreds to thousands of objects.
While making the cache timeout longer may sound like a good idea to increase a lot the performance it has a drawback : if you set a 1 hour timeout your inventory may be different from the remote system if change happened in the last hour. Using very small timeout will give you a very accurate inventory but then vCO will send a LOT of queries. For example vCO does a findById between each workflow step involving a given object. If you did not define a findById URL it will call the findRelation one that will pull a lot of objects and get their properties. If you do so in a workflow loop it is going to slow down vCO, the remote system and eventually cause the remote system to reject your queries.
This is why the timeout value should be adjusted to a value that will likely avoid repeating the queries returning the same results. The workflows operating the inventory objects should handle the case where there is a discrepancy between the inventory and the remote system (For example deleting an object that has been deleted by another process a few seconds before).

Conclusion

In this tutorial we could get the (complex) workflow object properties leveraging JSONSelect and tested the performance improvements using an action.
JSONSelect gives more flexibility to gather objects properties and the action allows for even more flexibility and for much better performance.






Comments  

0 # dmitry 2018-01-17 15:45
Can you run a workflow from the list?

Maybe I did something wrong, my workflows are not runnable.

NUC Lab Kit

Below are my thoughts for a vSAN nuc lab. Since I already have cables, not including them here. I ordered (and received by Nov 30, 2016)
3 x nuc, 3 x 32GB Crucial mem, 3 x Toshiba NVMe drive, 3 x Startech USB to GB NIC, and 3 x Crucial 1TB SSD. I've also been very happy with my Cisco SG300-10 so I bought one more since my existing one only has one port available. Each of the items listed here are linked below - all were purchased using the provided links below.
single NIC (See this post for details on using the USB -> GB NIC item listed below

I stayed with the i5 for the power consumption and form factor vs. the i7 Skull Canyon ;)

 

Search

Experts Exchange