JBoss BPM Suite 6.1 Design Time / Runtime Clustering

JBoss BPM Suite can be clustered both in Design time and Runtime.

For Design time, clustering leverages Apache Zookeeper and Apache Helix, to replicate the git repo.
For Runtime, the container manages most of the clustering & HA, however, if you choose to use timers in your process or the execution server, you will need to configure one extra component, Quartz.

Disclaimer: These configurations are for getting a test BPM cluster environment up and running. It may not work on your environment without modifications. This is not a production environment setup, modifications, updates, tweaks, multiple hosts and security hardening may be necessary.

Acquire your binaries from the Red Hat Customer Portal. The postgresql driver is available from postgresql.org.

As of this writing you need the following binaries:
jboss-eap-6.4.0.zip
jboss-bpmsuite-6.1.0.GA-deployable-eap6.x.zip
jboss-bpmsuite-6.1.0.GA-supplementary-tools.zip
postgresql-9.2-1002.jdbc4.jar

After following these instructions you should end up with a 2 node design & runtime cluster. The configurations and instructions are for installing onto single host (localhost). Configurations and some instructions will need to be tweaked if you desire to install on multiple hosts.

Setup NODE 1
Unzip the binaries


mkdir -p /tmp/bpmcluster
cd /tmp/bpmcluster/
unzip jboss-bpmsuite-6.1.0.GA-supplementary-tools.zip

mkdir -p /tmp/bpmcluster/node1
cd /tmp/bpmcluster/node1
unzip jboss-eap-6.4.0.zip
unzip -u jboss-bpmsuite-6.1.0.GA-deployable-eap6.x.zip

Setup Postgres Database

Login to postgres
The quartz database script needs to be executed externally. Creation of the bpms database assumes ownership as noted below in the command, this is not typical in most customer environments.


sudo su - postgres
psql

CREATE USER bpms WITH PASSWORD 'bpms';
CREATE DATABASE bpms OWNER bpms;
\q

psql -U bpms bpms
\i /tmp/bpmcluster/jboss-brms-bpmsuite-6.1.0.GA-redhat-2-supplementary-tools/ddl-scripts/postgresql/quartz_tables_postgres.sql
\q

Optional depending on database privileges. The applications can automatically create the tables via hibernate, if allowed.

 
\i /tmp/bpmcluster/jboss-brms-bpmsuite-6.1.0.GA-redhat-2-supplementary-tools/ddl-scripts/postgresql/postgresql-jbpm-schema.sql
\i /tmp/bpmcluster/jboss-brms-bpmsuite-6.1.0.GA-redhat-2-supplementary-tools/ddl-scripts/postgresql/postgres-dashbuilder-schema.sql

Return to your regular user account

Setup Zookeeper


cd /tmp/bpmcluster/jboss-brms-bpmsuite-6.1.0.GA-redhat-2-supplementary-tools/zookeeper

Copy zookeeper/conf/zoo_sample.cfg to zoo.cfg


cp zoo_sample.cfg zoo.cfg

Modify zoo.cfg
Update the the following to your choosing


dataDir=/tmp/zookeeper # make sure the folder exists prior to starting zookeeper

clientPort=2181

Start Zookeeper


cd /tmp/bpmcluster/jboss-brms-bpmsuite-6.1.0.GA-redhat-2-supplementary-tools/zookeeper/bin

./zkServer.sh start

Logs will be in zookeeper.out

Setup Helix


cd /tmp/bpmcluster/jboss-brms-bpmsuite-6.1.0.GA-redhat-2-supplementary-tools/helix-core/bin

Create the BPM Cluster


./helix-admin.sh --zkSvr localhost:2181 --addCluster bpmCluster

Add node1 to bpmCluster


./helix-admin.sh --zkSvr localhost:2181 --addNode bpmCluster localhost_bpmClusterNode1

Add node2 to bpmCluster


./helix-admin.sh --zkSvr localhost:2181 --addNode bpmCluster localhost_bpmClusterNode2

Add the resource vfs-repo to bpmCluster, configure the cluster for LeaderStandby mode

This means that one node will be primary and the other nodes will be a slave, replication only goes from the primary to the slaves


./helix-admin.sh --zkSvr localhost:2181 --addResource bpmCluster vfs-repo 1 LeaderStandby AUTO_REBALANCE

Tell the cluster to rebalance and initialize


./helix-admin.sh --zkSvr localhost:2181 --rebalance bpmCluster vfs-repo 2

Start Helix on Node 1
You only need 1 helix-controller for each OS you want managed. We are installing both bpm nodes on the same OS, so we only need 1 helix-controller. If you have mulitple hosts (OS), each host will need to run the helix controller as well.


nohup ./run-helix-controller.sh --zkSvr localhost:2181 --cluster bpmCluster 2>&1 > ./controller.log &

Logs will be in controller.log

Now to modify your EAP NODE 1 instance


cd /tmp/bpmcluster/node1/jboss-eap-6.4

EAP Configuration
When you unzipped the bpm binaries, it overwrote the standalone configuration files.

Some of these configurations will be slightly different for each node. For instance paths, hosts, and port numbers.

standalone/configuration/standalone-full-ha.xml

Add the following to the <system-properties> section.


<property name="jboss.node.name" value="node1"/>
<property name="org.uberfire.nio.git.dir" value="/tmp/bpmcluster/node1/jboss-eap-6.4/bin/repo"/>
<property name="org.quartz.properties" value="/tmp/bpmcluster/node1/jboss-eap-6.4/standalone/configuration/quartz-definition.properties"/>
<property name="org.uberfire.nio.git.daemon.enabled" value="true" />
<property name="org.uberfire.nio.git.daemon.host" value=“localhost" />
<property name="org.uberfire.nio.git.daemon.port" value="9418" />
<property name="org.uberfire.nio.git.ssh.enabled" value="true" />
<property name="org.uberfire.nio.git.ssh.host" value="localhost" />
<property name="org.uberfire.nio.git.ssh.port" value="8001" />
<property name="org.uberfire.cluster.id" value="bpmCluster"/>
<property name="org.uberfire.cluster.zk" value="localhost:2181"/>
<property name="org.uberfire.cluster.local.id" value="localhost_bpmClusterNode1"/>
<property name="org.uberfire.cluster.vfs.lock" value="vfs-repo"/>
<property name="org.uberfire.metadata.index.dir" value="/tmp/bpmcluster/node1/jboss-eap-6.4/bin/index"/>
<property name="org.uberfire.cluster.autostart" value="false"/>

Add the following datasource / driver definitions for postgres, <datasources> section:


                <datasource jta="false" jndi-name="java:jboss/datasources/bpmsNOXADS" pool-name="java:jboss/datasources/bpmsNOXADS_Pool" enabled="true" use-java-context="true" use-ccm="true">
                    <connection-url>jdbc:postgresql://localhost:5432/bpms</connection-url>
                    <driver>postgresql</driver>
                    <security>
                        <user-name>bpms</user-name>
                        <password>bpms</password>
                    </security>
                </datasource>
                <datasource jta="true" jndi-name="java:jboss/datasources/bpmsDS" pool-name="java:jboss/datasources/bpmsDS_Pool" enabled="true" use-java-context="true" use-ccm="true">
                    <connection-url>jdbc:postgresql://localhost:5432/bpms</connection-url>
                    <driver>postgresqlXA</driver>
                    <security>
                        <user-name>bpms</user-name>
                        <password>bpms</password>
                    </security>
                </datasource>
                <drivers>

                    <driver name="postgresql" module="org.postgresql">
                        <datasource-class>org.postgresql.Driver</datasource-class>
                    </driver>
                    <driver name="postgresqlXA" module="org.postgresql">                         <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>                     </driver>                 </drivers>

Add the postgres db driver to EAP


mkdir -p /tmp/bpmcluster/node1/jboss-eap-6.4/modules/system/layers/base/org/postgresql/main
cd /tmp/bpmcluster/node1/jboss-eap-6.4/modules/system/layers/base/org/postgresql/main

Create a file called module.xml with the following contents


<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.1" name="org.postgresql">
    <resources>
        <resource-root path="postgresql-9.2-1002.jdbc4.jar"/>
    </resources>
    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>

Copy your postgres db driver to this location

Setup the quartz property file. You've already set up the datasources above.


cd /tmp/bpmcluster/node1/jboss-eap-6.4/standalone/configuration

Create a file called quartz-definition.properties with the following contents


#============================================================================
# Configure Main Scheduler Properties
#============================================================================
org.quartz.scheduler.instanceName = jBPMClusteredScheduler
org.quartz.scheduler.instanceId = AUTO
#============================================================================
# Configure ThreadPool
#============================================================================
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 5
org.quartz.threadPool.threadPriority = 5
#============================================================================
# Configure JobStore
#============================================================================
org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreCMT
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
org.quartz.jobStore.useProperties=false
org.quartz.jobStore.dataSource=managedDS
org.quartz.jobStore.nonManagedTXDataSource=notManagedDS
org.quartz.jobStore.tablePrefix=QRTZ_
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval = 20000
#============================================================================
# Configure Datasources
#============================================================================
org.quartz.dataSource.managedDS.jndiURL=jboss/datasources/bpmsDS
org.quartz.dataSource.notManagedDS.jndiURL=jboss/datasources/bpmsNOXADS

Modify the application property files to associate the appropriate datasource and database configurations

business-central.war

Modify the jta-data-source and hibernate-dialect properties


vi /tmp/bpmcluster/node1/jboss-eap-6.4/standalone/deployments/business-central.war/WEB-INF/classes/META-INF/persistence.xml


<jta-data-source>java:jboss/datasources/bpmsDS</jta-data-source>
<property name="hibernate.dialect" value="org.hibernate.dialect.PostgreSQLDialect" />

dashbuilder.war

Modify the jndi-name and add the postgres module


vi /tmp/bpmcluster/node1/jboss-eap-6.4/standalone/deployments/dashbuilder.war/WEB-INF/jboss-web.xml


<jndi-name>java:jboss/datasources/bpmsDS</jndi-name>


vi /tmp/bpmcluster/node1/jboss-eap-6.4/standalone/deployments/dashbuilder.war/WEB-INF/jboss-deployment-structure.xml


<module name="org.postgresql" />

Force the apps to redeploy on the start of the EAP instance


touch /tmp/bpmcluster/node1/jboss-eap-6.4/standalone/deployments/business-central.war.dodeploy
touch /tmp/bpmcluster/node1/jboss-eap-6.4/standalone/deployments/dashbuilder.war.dodeploy

Add an application admin user with the group admin


cd /tmp/bpmcluster/node1/jboss-eap-6.4/bin
./adduser.sh

Setup EAP NODE 2
You can zip up your EAP NODE 1 configuration and unzip it to a node2 folder or refollow the steps from EAP NODE 1


cd /tmp/bpmcluster/node1
zip -r jbosseap.zip jboss-eap-6.4
mkdir -p /tmp/bpmcluster/node2
cd /tmp/bpmcluster/node2
unzip ../node1/jbosseap.zip

Modify the standalone-full-ha.xml system-properties


<property name="jboss.node.name" value="node2"/>
<property name="org.uberfire.nio.git.dir" value="/tmp/bpmcluster/node2/jboss-eap-6.4/bin/repo"/>
<property name="org.quartz.properties" value="/tmp/bpmcluster/node2/jboss-eap-6.4/standalone/configuration/quartz-definition.properties"/>
<property name="org.uberfire.nio.git.daemon.enabled" value="true" />
<property name="org.uberfire.nio.git.daemon.host" value=“localhost" />
<property name="org.uberfire.nio.git.daemon.port" value="9419" />
<property name="org.uberfire.nio.git.ssh.enabled" value="true" />
<property name="org.uberfire.nio.git.ssh.host" value="localhost" />
<property name="org.uberfire.nio.git.ssh.port" value="8002" />
<property name="org.uberfire.cluster.id" value="bpmCluster"/>
<property name="org.uberfire.cluster.zk" value="localhost:2181"/>
<property name="org.uberfire.cluster.local.id" value="localhost_bpmClusterNode2"/>
<property name="org.uberfire.cluster.vfs.lock" value="vfs-repo"/>
<property name="org.uberfire.metadata.index.dir" value="/tmp/bpmcluster/node2/jboss-eap-6.4/bin/index"/>
<property name="org.uberfire.cluster.autostart" value="false"/>

Start your EAP instances
The messaging cluster password is necessary. This property can also be modified in the standalone-full-ha.xml so that it doesn't need to be specified as a command line argument

EAP NODE 1


cd /tmp/bpmcluster/node1/jboss-eap-6.4/bin
./standalone.sh -c standalone-full-ha.xml -Djboss.messaging.cluster.password=password

EAP NODE 2


cd /tmp/bpmcluster/node2/jboss-eap-6.4/bin
./standalone.sh -c standalone-full-ha.xml -Djboss.messaging.cluster.password=password -Djboss.socket.binding.port-offset=100

*** NOTES ***

Console log printing - by default, the console log is not configured for standalone-full-ha.xml

For each EAP Node, you can run the following jboss cli command from the bin folder.

EAP NODE 1


./jboss-cli.sh 
connect localhost:9999
/subsystem=logging/console-handler=CONSOLE:add(level=INFO,formatter="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n")
/subsystem=logging/root-logger=ROOT:add-handler(name="CONSOLE")

EAP NODE 2


./jboss-cli.sh 
connect localhost:10099
/subsystem=logging/console-handler=CONSOLE:add(level=INFO,formatter="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n")
/subsystem=logging/root-logger=ROOT:add-handler(name="CONSOLE")

If running on OS X, hornetq may have an issue with multicast. Enable the appropriate route.

sudo route add 224.0.0.0 127.0.0.1 -netmask 240.0.0.0


16:08:14,988 ERROR [org.hornetq.core.server] (Thread-0 (HornetQ-scheduled-threads-2044127527)) HQ224033: Failed to broadcast connector configs: java.io.IOException: Can't assign requested address
    at java.net.PlainDatagramSocketImpl.send(Native Method) [rt.jar:1.7.0_76]
    at java.net.DatagramSocket.send(DatagramSocket.java:697) [rt.jar:1.7.0_76]
    at org.hornetq.api.core.UDPBroadcastGroupConfiguration$UDPBroadcastEndpoint.broadcast(UDPBroadcastGroupConfiguration.java:136) [hornetq-core-client-2.3.25.Final-redhat-1.jar:2.3.25.Final-redhat-1]

You can list the helix zookeeper configuration information by running the following command

./helix-admin.sh --zkSvr localhost:2181 --listResourceInfo bpmCluster vfs-repo

This is the ideal configuration based on the commands above


{
  "id" : "vfs-repo",
  "mapFields" : {
    "vfs-repo_0" : {
      "localhost_bpmClusterNode1" : "LEADER",
      "localhost_bpmClusterNode2" : "STANDBY"
    }
  },
  "listFields" : {
    "vfs-repo_0" : [ "localhost_bpmClusterNode1", "localhost_bpmClusterNode2" ]
  },
  "simpleFields" : {
    "IDEAL_STATE_MODE" : "AUTO",
    "NUM_PARTITIONS" : "1",
    "REBALANCE_MODE" : "SEMI_AUTO",
    "REPLICAS" : "2",
    "STATE_MODEL_DEF_REF" : "LeaderStandby",
    "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
  }
}
ExternalView for vfs-repo:
{
  "id" : "vfs-repo",
  "mapFields" : {
    "vfs-repo_0" : {
      "localhost_bpmClusterNode1" : "OFFLINE",
      "localhost_bpmClusterNode2" : "OFFLINE"
    }
  },
  "listFields" : {
  },
  "simpleFields" : {
    "BUCKET_SIZE" : "0"
  }
}

For single sign on (SSO), assuming security is setup correctly. Adding an application admin user on both systems should be sufficient.

Modify standalone-full-ha.xml on both nodes, <sso/> property to


<sso cache-container="web" cache-name="sso"/>

For your clustered runtime instances you will usually have a maven central repository, like nexus to be your maven central. A central artifact repository is ideal, but you still have to create the appropriate scripts to pull the artifacts. However, replication of maven artifacts via rsync is also an option as noted in the documentation.

In the quartz-definiton.properties, if you specify a jndiURL for managedDS that is different from what is specified as the jta-data-source in business-central.war you may encounter an exception where quartz is unable to locate the appropriate jndi url reference.

13:41:14,467 WARN  [org.drools.persistence.SingleSessionCommandService] (http-/127.0.0.1:8080-6) Could not commit session: org.jbpm.workflow.instance.WorkflowRuntimeException: [TestProj7.Timer-Delay:6 - Delay:3] -- org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'managedDS': java.sql.SQLException: Could not retrieve datasource via JNDI url 'jboss/datasources/quartzXADS' java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction [See nested exception: java.sql.SQLException: Could not retrieve datasource via JNDI url 'jboss/datasources/quartzXADS' java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction]

References:
https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_BPM_Suite/6.1/html/Installation_Guide/chap-Clustering.html
http://planet.jboss.org/post/clustering_in_jbpm_v6
https://opensourcebpm.wordpress.com/2014/04/27/redhat-jboss-bpms-design-time-clustering/