Monitoring the vCenter Orchestrator Java Virtual Machine

alt

Ever wondered how your vCenter Orchestrator is doing on memory usage ? vCenter Orchestrator Java Virtual Machine allocates a default of 2 GB Heap space.

It will consume more heap space as processes are running and will ultimately call the garbage collector to keep in memory only the used items.

If you are running a lot of workflows on your server or if you are experimenting custom workflows and plug-ins you may want to know how much of this memory is consumed. The easiest way is to download this utility.

You can copy it in $INSTALL_DIR/app-server/server/vmo/deploy/ then restart your server and check the server.log file. Every ten seconds the memory usage will be written in the log as follow:

2011-09-19 11:59:22.149+0200 INFO  [Server] JBoss (MX MicroKernel) [4.2.2.GA (build: SVNTag=JBoss_4_2_2_GA date=200710221139)] Started in 28s:819ms
2011-09-19 11:59:22.233+0200 INFO  [STDOUT] ============================ VMO server ready ==============================
2011-09-19 11:59:32.010+0200 INFO  [MEMORY_DUMP] 281.561 MB / 1.926 GB
2011-09-19 11:59:42.010+0200 INFO  [MEMORY_DUMP] 282.606 MB / 1.926 GB

If you want to follow this real time instead of checking the logs after the facts you can download VisualVM, install it on your vCO server and run it to monitor the ch.dunes.jboss.Main. Below is a capture of vCO running hundreds or workflows in parallel.

alt

Now if you are getting serious about this because you need to include vCenter Orchestrator in your enterprise monitoring solution you should use the Java Management Extensions (JMX). To enable JMX for a vCenter Orchestrator server started as a service:

  • Create a jmxremote.password file in install_directory\app- server\server\vmo\conf. The file takes the form of a username and a password separated by a space on each line. An example is: monitor password
  • Create a jmxremote.access file in install_directory\app- server\server\vmo\conf. The file takes the form of a username and a permission separated by a space on each line. An example is: monitor readonly
  • Secure each of the jmxremote files on Windows as described here: http://download.oracle.com/javase/6/docs/technotes/guides/management/security-windows.html, and for non-Windows systems simply use the following command:
    chmod 600 jmxremote.password jmxremote.password

Note: Essentially, each of the jmxremote files must only be accessible to the owner. Final security properties on a Windows server should show Full Access to the owner (default is Administrators group) and no other users, groups, or SYSTEM listed for access. Additionally, windows explorer will add a lock icon next to the filename.

Edit install_directory\app- server\bin\wrapper.conf by adding the lines below after line "wrapper.java.additional.9 ...":

  wrapper.java.additional.10=-Dcom.sun.management.jmxremote.authenticate=true wrapper.java.additional.11=-Dcom.sun.management.jmxremote.password.file= ../server/vmo/conf/jmxremote.password wrapper.java.additional.12=-Dcom.sun.management.jmxremote.access.file=../server/vmo/conf/jmxremote.access wrapper.java.additional.13=-Dcom.sun.management.jmxremote.ssl=false wrapper.java.additional.14=-Dcom.sun.management.jmxremote.port=1099 wrapper.java.additional.15=-Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl wrapper.java.additional.16=-Djboss.platform.mbeanserver

Note: The numbers in wrapper.java.additional.## must be in order without gap between them, otherwise wrapper ignores them. If the last property is wrapper.java.additional.10 then all above properties should be shift by 1 starting at wrapper.java.additional.11
JMX monitoring will be available after restarting the vCenter Orchestrator server.


Testing of the monitoring
JConsole is a GUI application from the Java Development Kit designed for monitoring Java applications.
The jconsole executable is in JDK_HOME/bin, where JDK_HOME is the directory where the JDK is installed. If this directory is in system path, the tool can be started by typing jconsole in a command (shell) prompt. Otherwise, it is necessary to change the current directory.
JConsole will list local processes and give the option to enter a remote one with the hostname:port syntax.
Once connected it is possible to monitor in details the memory, threads and managed beans.
The following table represents a subset of MBeans that can be used for monitoring the performance of a vCenter Orchestrator instance.

Workflow Execution

Mbean

ch.dunes.workflow.engine.mbean.WorkflowEngine

Description

Statistics about active workflows

Attribute

Description

ExecutorsActiveCount

Number of currently active workflows

ExecutorsQueueSize

Number of workflows queued

Web views

Mbean

jboss.web:type=Cache,host=[hostname],path=/vmo

Description

webview statistics

Attribute

Description

accessCount

Number of accesses to the cache

cacheMaxSize

Max size of resources which will have their content cached

cacheSize

Max size of resources which will have their content cached

desiredEntryAccessRatio

Entry hit ratio at which an entry will never be removed from the cache.

hitsCount

Number of cache hits

Apache Tomcat Global Request Processor Metrics

Mbean

jboss.web:type=GlobalRequestProcessor,name=http-[hostname]-[port]

Description

Apache Tomcat Global Request Processor Metrics

Attribute

Description

bytesSent

Bytes sent by all the request processors running on the Apache Tomcat container

bytesReceived

Bytes received by all the request processors running on the Apache Tomcat container

processingTime

Total processing time (in milliseconds) since startup

errorCount

Error count on all the request processors running on the Apache Tomcat container

maxTime

Maximum time it took to process a request

requestCount

Request count on all the request processors running on the Apache Tomcat container

vCO web service

Mbean

jboss.web:type=Manager,path=/vmware-vmo-webcontrol,host=[hostname]

Description

Attribute

Description

activeSessions

Number of currently active sessions

expiredSessions

Number of sessions that have expired

maxActive

Maximum number of sessions that have been active at the same time

processingTime

Total processing time (in milliseconds) since startup

sessionAverageAliveTime

Average time (in seconds) that expired sessions had been alive

sessionCounter

Total number of sessions created by this manager

sessionMaxAliveTime

Longest time (in seconds) that an expired session had been alive

WebViewEngine

Mbean

jboss.web:j2eeType=Servlet,name=VSO-WebViewEngine,WebModule=//localhost/vmo,J2EEApplication=none,J2EEServer=none

Description

Attribute

Description

maxTime

Maximum time taken (in milliseconds) for processing a request

processingTime

Total processing time (in milliseconds) since startup

sessionMaxAliveTime

The longest time (in seconds) that an expired session had been alive

requestCount

Total number of requests served since startup

Web Tread pool

Mbean

jboss.web:type=ThreadPool,name=http-[hostname]-[port]

Description

Attribute

Description

currentThreadCount

Number of threads created on the Apache Tomcat container

currentThreadsBusy

Number of busy threads on the Apache Tomcat container


Possible implementation through JMX includes VMware vFabric Hyperic. It can be used to collect, store metrics from JMX and custom MBeans, and produce graphs over a long period of time.

vCenter Orchestrator has a throttling mechanism limiting the number of running workflows to 300 (default value). if additional workflows are started they will be stored in an execution queue. Even under heavy load I have never seen memory usage after garbage collecting over 1500 MB. Now this depends a lot in how the plug-ins and associated workflows were designed. In a next article I will write guidelines on how to minimize vCO resource footprint.