An increasing number of customers are using SOA Suite in a cluster configuration, I might hazard to say that the majority of production deployments are now using SOA clusters.  So I thought it may be useful to detail the steps in building an 11g cluster and explain a little about why things are done the way they are.

In this series of posts I will explain how to build a SOA/BPM cluster using the Enterprise Deployment Guide.

This post will explain the setting required to prepare the cluster for installation and configuration.

Software Required

The following software is required for an 11.1.1.3 SOA/BPM install.

Software	Version	Notes
Oracle Database	Certified databases are listed here	SOA & BPM Suites require a working database installation.
Repository Creation Utility (RCU)	11.1.1.3	If upgrading an 11.1.1.2 repository then a separate script is available.
Web Tier Utilities	11.1.1.3	Provides Web Server, 11.1.1.3 is an upgrade to 11.1.1.2, so 11.1.1.2 must be installed first.
Web Tier Utilities	11.1.1.3	Web Server, 11.1.1.3 Patch.  You can use the 11.1.1.2 version without problems.
Oracle WebLogic Server 11gR1	10.3.3	This is the host platform for 11.1.1.3 SOA/BPM Suites.
SOA Suite	11.1.1.2	SOA Suite 11.1.1.3 is an upgrade to 11.1.1.2, so 11.1.1.2 must be installed first.
SOA Suite	11.1.1.3	SOA Suite 11.1.1.3 patch, requires 11.1.12 to have been installed.

My installation was performed on Oracle Enterprise Linux 5.4 64-bit.

Database

I will not cover setting up the database in this series other than to identify the database requirements.  If setting up a SOA cluster then ideally we would also be using a RAC database.  I assume that this is running on separate machines to the SOA cluster.  Section 2.1, “Database”, of the EDG covers the database configuration in detail.

Settings

The database should have processes set to at least 400 if running SOA/BPM and BAM.

• alter system set processes=400 scope=spfile

Run RCU

The Repository Creation Utility creates the necessary database tables for the SOA Suite.  The RCU can be run from any machine that can access the target database.  In 11g the RCU creates a number of pre-defined users and schema with a user defiend prefix.  This allows you to have multiple 11g installations in the same database.

After running the RCU you need to grant some additional privileges to the soainfra user.  The soainfra user should have privileges on the transaction tables.

• grant select on sys.dba_pending_transactions to prefix_soainfra
• Grant force any transaction to prefix_soainfra

Machines

The cluster will be built on the following machines.

• EDG Name is the name used for this machine in the EDG.
• Notes are a description of the purpose of the machine.

EDG Name	Notes
LB	External load balancer to distribute load across and failover between web servers.
WEBHOST1	Hosts a web server.
WEBHOST2	Hosts a web server.
SOAHOST1	Hosts SOA components.
SOAHOST2	Hosts SOA components.
BAMHOST1	Hosts BAM components.
BAMHOST2	Hosts BAM components.

Note that it is possible to collapse the BAM servers so that they run on the same machines as the SOA servers. In this case BAMHOST1 and SOAHOST1 would be the same, as would BAMHOST2 and SOAHOST2. The cluster may include more than 2 servers and in this case we add SOAHOST3, SOAHOST4 etc as needed.

My cluster has WEBHOST1, SOAHOST1 and BAMHOST1 all running on a single machine.

Software Components

The cluster will use the following software components.

• EDG Name is the name used for this machine in the EDG.
• Type is the type of component, generally a WebLogic component.
• Notes are a description of the purpose of the component.

EDG Name	Type	Notes
AdminServer	Admin Server	Domain Admin Server
WLS_WSM1	Managed Server	Web Services Manager Policy Manager Server
WLS_WSM2	Managed Server	Web Services Manager Policy Manager Server
WLS_SOA1	Managed Server	SOA/BPM Managed Server
WLS_SOA2	Managed Server	SOA/BPM Managed Server
WLS_BAM1	Managed Server	BAM Managed Server running Active Data Cache
WLS_BAM2	Managed Server	BAM Manager Server without Active Data Cache
	Node Manager	Will run on all hosts with WLS servers
OHS1	Web Server	Oracle HTTP Server
OHS2	Web Server	Oracle HTTP Server
LB	Load Balancer	Load Balancer, not part of SOA Suite

The above assumes a 2 node cluster.

Network Configuration

The SOA cluster requires an extensive amount of network configuration.  I would recommend assigning a private sub-net (internal IP addresses such as 10.x.x.x, 192.168.x.x or 172.168.x.x) to the cluster for use by addresses that only need to be accessible to the Load Balancer or other cluster members.  Section 2.2, "Network", of the EDG covers the network configuration in detail.

• EDG Name is the hostname used in the EDG.
• IP Name is the IP address name used in the EDG.
• Type is the type of IP address:
  • Fixed is fixed to a single machine.
  • Floating is assigned to one of several machines to allow for server migration.
  • Virtual is assigned to a load balancer and used to distribute load across several machines.
• Host is the host where this IP address is active.  Note for floating IP addresses a range of hosts is given.
• Bound By identifies which software component will use this IP address.
• Scope shows where this IP address needs to be resolved.
  • Cluster scope addresses only have to be resolvable by machines in the cluster, i.e. the machines listed in the previous section.  These addresses are only used for inter-cluster communication or for access by the load balancer.
  • Internal scope addresses
• Notes are comments on why that type of IP is used.

EDG Name	IP Name	Type	Host	Bound By	Scope	Notes
ADMINVHN	VIP1	Floating	SOAHOST1-SOAHOSTn	AdminServer	Cluster	Admin server, must be able to migrate between SOA server machines.
SOAHOST1	IP1	Fixed	SOAHOST1	NodeManager, WLS_WSM1	Cluster	WSM Server 1 does not require server migration.
SOAHOST2	IP2	Fixed	SOAHOST1	NodeManager, WLS_WSM2	Cluster	WSM Server 2 does not require server migration
SOAHOST1VHN	VIP2	Floating	SOAHOST1-SOAHOSTn	WLS_SOA1	Cluster	SOA server 1, must be able to migrate between SOA server machines
SOAHOST2VHN	VIP3	Floating	SOAHOST1-SOAHOSTn	WLS_SOA2	Cluster	SOA server 2, must be able to migrate between SOA server machines
BAMHOST1	IP4	Fixed	BAMHOST1	NodeManager	Cluster
BAMHOST1VHN	VIP4	Floating	BAMHOST1-BAMHOSTn	WLS_BAM1	Cluster	BAM server 1, must be able to migrate between BAM server machines
BAMHOST2	IP3	Fixed	BAMHOST2	NodeManager, WLS_BAM2	Cluster	BAM server 2 does not require server migration
WEBHOST1	IP5	Fixed	WEBHOST1	OHS1	Cluster
WEBHOST2	IP6	Fixed	WEBHOST2	OHS2	Cluster
soa.mycompany.com	VIP5	Virtual	LB	LB	Public	External access point to SOA cluster.
admin.mycompany.com	VIP6	Virtual	LB	LB	Internal	Internal access to WLS console and EM
soainternal.mycompany.com	VIP7	Virtual	LB	LB	Internal	Internal access point to SOA cluster

Floating IP addresses are IP addresses that may be re-assigned between machines in the cluster.  For example in the event of failure of SOAHOST1 then WLS_SOA1 will need to be migrated to another server.  In this case VIP2 (SOAHOST1VHN) will need to be activated on the new target machine.  Once set up the node manager will manage registration and removal of the floating IP addresses with the exception of the AdminServer floating IP address.

Note that if the BAMHOSTs and SOAHOSTs are the same machine then you can obviously share the hostname and fixed IP addresses, but you still need separate floating IP addresses for the different managed servers.  The hostnames don’t have to be the ones given in the EDG, but they must be distinct in the same way as the ETC names are distinct.  If the type is a fixed IP then if the addresses are the same you can use the same hostname, for example if you collapse the soahost1, bamhost1 and webhost1 onto a single machine then you could refer to them all as HOST1 and give them the same IP address, however SOAHOST1VHN can never be the same as BAMHOST1VHN because these are floating IP addresses.

Notes on DNS

IP addresses that are of scope “Cluster” just need to be in the hosts file (/etc/hosts on Linux, C:\Windows\System32\drivers\etc\hosts on Windows) of all the machines in the cluster and the load balancer.  IP addresses that are of scope “Internal” need to be available on the internal DNS servers, whilst IP addresses of scope “Public” need to be available on external and internal DNS servers.

Shared File System

At a minimum the cluster needs shared storage for the domain configuration, XA transaction logs and JMS file stores.  It is also possible to place the software itself on a shared server.  I strongly recommend that all machines have the same file structure for their SOA installation otherwise you will experience pain!  Section 2.3, "Shared Storage and Recommended Directory Structure", of the EDG covers the shared storage recommendations in detail.

The following shorthand is used for locations:

• ORACLE_BASE is the root of the file system used for software and configuration files.
• MW_HOME is the location used by the installed SOA/BPM Suite installation.  This is also used by the web server installation.  In my installation it is set to <ORACLE_BASE>/SOA11gPS2.
• ORACLE_HOME is the location of the Oracle SOA components or the Oracle Web components.  This directory is installed under the the MW_HOME but the name is decided by the user at installation, default values are Oracle_SOA1 and Oracle_Web1.  In my installation they are set to <MW_HOME>/Oracle_SOA and <MW_HOME>/Oracle _WEB.
• ORACLE_COMMON_HOME is the location of the common components and is located under the MW_HOME directory.  This is always <MW_HOME>/oracle_common.
• ORACLE_INSTANCE is used by the Oracle HTTP Server and/or Oracle Web Cache.  It is recommended to create it under <ORACLE_BASE>/admin.  In my installation they are set to <ORACLE_BASE>/admin/Web1, <ORACLE_BASE>/admin/Web2 and <ORACLE_BASE>/admin/WC1.
• WL_HOME is the WebLogic server home and is always found at <MW_HOME>/wlserver_10.3.

Key file locations are shown below.

Directory	Notes
<ORACLE_BASE>/admin/domain_name/aserver/domain_name	Shared location for domain.  Used to allow admin server to manually fail over between machines.  When creating domain_name provide the aserver directory as the location for the domain. In my install this is <ORACLE_BASE>/admin/aserver/soa_domain as I only have one domain on the box.
<ORACLE_BASE>/admin/domain_name/aserver/applications	Shared location for deployed applications.  Needs to be provided when creating the domain. In my install this is <ORACLE_BASE>/admin/aserver/applications as I only have one domain on the box.
<ORACLE_BASE>/admin/domain_name/mserver/domain_name	Either unique location for each machine or can be shared between machines to simplify task of packing and unpacking domain.  This acts as the managed server configuration location.  Keeping it separate from Admin server helps to avoid problems with the managed servers messing up the Admin Server. In my install this is <ORACLE_BASE>/admin/mserver/soa_domain as I only have one domain on the box.
<ORACLE_BASE>/admin/domain_name/mserver/applications	Either unique location for each machine or can be shared between machines.  Holds deployed applications. In my install this is <ORACLE_BASE>/admin/mserver/applications as I only have one domain on the box.
<ORACLE_BASE>/admin/domain_name/soa_cluster_name	Shared directory to hold the following   dd – deployment descriptors   jms – shared JMS file stores   fadapter – shared file adapter co-ordination files   tlogs – shared transaction log files In my install this is <ORACLE_BASE>/admin/soa_cluster.
<ORACLE_BASE>/admin/instance_name	Local folder for web server (OHS) instance. In my install this is <ORACLE_BASE>/admin/web1 and <ORACLE_BASE>/admin/web2. I also have <ORACLE_BASE>/admin/wc1 for the Web Cache I use as a load balancer.
<ORACLE_BASE>/product/fmw	This can be a shared or local folder for the SOA/BPM Suite software.  I used a shared location so I only ran the installer once. In my install this is <ORACLE_BASE>/SOA11gPS2

All the shared files need to be put onto a shared storage media.  I am using NFS, but recommendation for production would be a SAN, with mirrored disks for resilience.

Collapsing Environments

To reduce the hardware requirements it is possible to collapse the BAMHOST, SOAHOST and WEBHOST machines onto a single physical machine.  This will require more memory but memory is a lot cheaper than additional machines.  For environments that require higher security then stay with a separate WEBHOST tier as per the EDG.  Similarly for high volume environments then keep a separate set of machines for BAM and/or Web tier as per the EDG.

Notes on Dev Environments

In a dev environment it is acceptable to use a a single node (non-RAC) database, but be aware that the config of the data sources is different (no need to use multi-data source in WLS).  Typically in a dev environment we will collapse the BAMHOST, SOAHOST and WEBHOST onto a single machine and use a software load balancer.  To test a cluster properly we will need at least 2 machines.

For my test environment I used Oracle Web Cache as a load balancer.  I ran it on one of the SOA Suite machines and it load balanced across the Web Servers on both machines.  This was easy for me to set up and I could administer it from a web based console.