Building a SOA/BPM/BAM Cluster Part I – Preparing the Environment
- by antony.reynolds
An increasing number of customers are using SOA Suite in a cluster configuration, I might hazard to say that the majority of production deployments are now using SOA clusters.  So I thought it may be useful to detail the steps in building an 11g cluster and explain a little about why things are done the way they are.  In this series of posts I will explain how to build a SOA/BPM cluster using the Enterprise Deployment Guide.  This post will explain the setting required to prepare the cluster for installation and configuration.  Software Required  The following software is required for an 11.1.1.3 SOA/BPM install.              Software        Version        Notes                  Oracle Database        Certified databases are listed here        SOA & BPM Suites require a working database installation.                           Repository Creation Utility (RCU)               11.1.1.3        If upgrading an 11.1.1.2 repository then a separate script is available.                           Web Tier Utilities               11.1.1.3        Provides Web Server, 11.1.1.3 is an upgrade to 11.1.1.2, so 11.1.1.2 must be installed first.                           Web Tier Utilities               11.1.1.3        Web Server, 11.1.1.3 Patch.  You can use the 11.1.1.2 version without problems.                  Oracle WebLogic Server 11gR1        10.3.3        This is the host platform for 11.1.1.3 SOA/BPM Suites.                  SOA Suite        11.1.1.2        SOA Suite 11.1.1.3 is an upgrade to 11.1.1.2, so 11.1.1.2 must be installed first.                  SOA Suite        11.1.1.3        SOA Suite 11.1.1.3 patch, requires 11.1.12 to have been installed.          My installation was performed on Oracle Enterprise Linux 5.4 64-bit.  Database  I will not cover setting up the database in this series other than to identify the database requirements.  If setting up a SOA cluster then ideally we would also be using a RAC database.  I assume that this is running on separate machines to the SOA cluster.  Section 2.1, “Database”, of the EDG covers the database configuration in detail.  Settings  The database should have processes set to at least 400 if running SOA/BPM and BAM.     alter system set processes=400 scope=spfile    Run RCU  The Repository Creation Utility creates the necessary database tables for the SOA Suite.  The RCU can be run from any machine that can access the target database.  In 11g the RCU creates a number of pre-defined users and schema with a user defiend prefix.  This allows you to have multiple 11g installations in the same database.  After running the RCU you need to grant some additional privileges to the soainfra user.  The soainfra user should have privileges on the transaction tables.     grant select on sys.dba_pending_transactions to prefix_soainfra     Grant force any transaction to prefix_soainfra    Machines  The cluster will be built on the following machines.     EDG Name is the name used for this machine in the EDG.     Notes are a description of the purpose of the machine.                EDG Name        Notes                  LB        External load balancer to distribute load across and failover between web servers.                  WEBHOST1        Hosts a web server.                  WEBHOST2        Hosts a web server.                  SOAHOST1        Hosts SOA components.                  SOAHOST2        Hosts SOA components.                  BAMHOST1        Hosts BAM components.                  BAMHOST2        Hosts BAM components.          Note that it is possible to collapse the BAM servers so that they run on the same machines as the SOA servers. In this case BAMHOST1 and SOAHOST1 would be the same, as would BAMHOST2 and SOAHOST2. The cluster may include more than 2 servers and in this case we add SOAHOST3, SOAHOST4 etc as needed.  My cluster has WEBHOST1, SOAHOST1 and BAMHOST1 all running on a single machine.  Software Components  The cluster will use the following software components.     EDG Name is the name used for this machine in the EDG.     Type is the type of component, generally a WebLogic component.     Notes are a description of the purpose of the component.                EDG Name        Type        Notes                  AdminServer        Admin Server        Domain Admin Server                  WLS_WSM1        Managed Server        Web Services Manager Policy Manager Server                  WLS_WSM2        Managed Server        Web Services Manager Policy Manager Server                  WLS_SOA1        Managed Server        SOA/BPM Managed Server                  WLS_SOA2        Managed Server        SOA/BPM Managed Server                  WLS_BAM1        Managed Server        BAM Managed Server running Active Data Cache                  WLS_BAM2        Managed Server        BAM Manager Server without Active Data Cache                           Node Manager        Will run on all hosts with WLS servers                  OHS1        Web Server        Oracle HTTP Server                  OHS2        Web Server        Oracle HTTP Server                  LB        Load Balancer        Load Balancer, not part of SOA Suite          The above assumes a 2 node cluster.  Network Configuration  The SOA cluster requires an extensive amount of network configuration.  I would recommend assigning a private sub-net (internal IP addresses such as 10.x.x.x, 192.168.x.x or 172.168.x.x) to the cluster for use by addresses that only need to be accessible to the Load Balancer or other cluster members.  Section 2.2, "Network", of the EDG covers the network configuration in detail.     EDG Name is the hostname used in the EDG.     IP Name is the IP address name used in the EDG.     Type is the type of IP address:              Fixed is fixed to a single machine.         Floating is assigned to one of several machines to allow for server migration.         Virtual is assigned to a load balancer and used to distribute load across several machines.             Host is the host where this IP address is active.  Note for floating IP addresses a range of hosts is given.     Bound By identifies which software component will use this IP address.     Scope shows where this IP address needs to be resolved.              Cluster scope addresses only have to be resolvable by machines in the cluster, i.e. the machines listed in the previous section.  These addresses are only used for inter-cluster communication or for access by the load balancer.         Internal scope addresses             Notes are comments on why that type of IP is used.                EDG Name        IP Name        Type        Host        Bound By        Scope        Notes                  ADMINVHN        VIP1        Floating        SOAHOST1-SOAHOSTn        AdminServer        Cluster        Admin server, must be able to migrate between SOA server machines.                  SOAHOST1        IP1        Fixed        SOAHOST1        NodeManager, WLS_WSM1        Cluster        WSM Server 1 does not require server migration.                  SOAHOST2        IP2        Fixed        SOAHOST1        NodeManager, WLS_WSM2        Cluster        WSM Server 2 does not require server migration                  SOAHOST1VHN        VIP2        Floating        SOAHOST1-SOAHOSTn        WLS_SOA1        Cluster        SOA server 1, must be able to migrate between SOA server machines                  SOAHOST2VHN        VIP3        Floating        SOAHOST1-SOAHOSTn        WLS_SOA2        Cluster        SOA server 2, must be able to migrate between SOA server machines                  BAMHOST1        IP4        Fixed        BAMHOST1        NodeManager        Cluster                           BAMHOST1VHN        VIP4        Floating        BAMHOST1-BAMHOSTn        WLS_BAM1        Cluster        BAM server 1, must be able to migrate between BAM server machines                  BAMHOST2        IP3        Fixed        BAMHOST2        NodeManager, WLS_BAM2        Cluster        BAM server 2 does not require server migration                  WEBHOST1        IP5        Fixed        WEBHOST1        OHS1        Cluster                           WEBHOST2        IP6        Fixed        WEBHOST2        OHS2        Cluster                           soa.mycompany.com        VIP5        Virtual        LB        LB        Public        External access point to SOA cluster.                  admin.mycompany.com        VIP6        Virtual        LB        LB        Internal        Internal access to WLS console and EM                  soainternal.mycompany.com        VIP7        Virtual        LB        LB        Internal        Internal access point to SOA cluster          Floating IP addresses are IP addresses that may be re-assigned between machines in the cluster.  For example in the event of failure of SOAHOST1 then WLS_SOA1 will need to be migrated to another server.  In this case VIP2 (SOAHOST1VHN) will need to be activated on the new target machine.  Once set up the node manager will manage registration and removal of the floating IP addresses with the exception of the AdminServer floating IP address.  Note that if the BAMHOSTs and SOAHOSTs are the same machine then you can obviously share the hostname and fixed IP addresses, but you still need separate floating IP addresses for the different managed servers.  The hostnames don’t have to be the ones given in the EDG, but they must be distinct in the same way as the ETC names are distinct.  If the type is a fixed IP then if the addresses are the same you can use the same hostname, for example if you collapse the soahost1, bamhost1 and webhost1 onto a single machine then you could refer to them all as HOST1 and give them the same IP address, however SOAHOST1VHN can never be the same as BAMHOST1VHN because these are floating IP addresses.  Notes on DNS  IP addresses that are of scope “Cluster” just need to be in the hosts file (/etc/hosts on Linux, C:\Windows\System32\drivers\etc\hosts on Windows) of all the machines in the cluster and the load balancer.  IP addresses that are of scope “Internal” need to be available on the internal DNS servers, whilst IP addresses of scope “Public” need to be available on external and internal DNS servers.  Shared File System  At a minimum the cluster needs shared storage for the domain configuration, XA transaction logs and JMS file stores.  It is also possible to place the software itself on a shared server.  I strongly recommend that all machines have the same file structure for their SOA installation otherwise you will experience pain!  Section 2.3, "Shared Storage and Recommended Directory Structure", of the EDG covers the shared storage recommendations in detail.  The following shorthand is used for locations:     ORACLE_BASE is the root of the file system used for software and configuration files.     MW_HOME is the location used by the installed SOA/BPM Suite installation.  This is also used by the web server installation.  In my installation it is set to <ORACLE_BASE>/SOA11gPS2.     ORACLE_HOME is the location of the Oracle SOA components or the Oracle Web components.  This directory is installed under the the MW_HOME but the name is decided by the user at installation, default values are Oracle_SOA1 and Oracle_Web1.  In my installation they are set to <MW_HOME>/Oracle_SOA and <MW_HOME>/Oracle _WEB.     ORACLE_COMMON_HOME is the location of the common components and is located under the MW_HOME directory.  This is always <MW_HOME>/oracle_common.     ORACLE_INSTANCE is used by the Oracle HTTP Server and/or Oracle Web Cache.  It is recommended to create it under <ORACLE_BASE>/admin.  In my installation they are set to <ORACLE_BASE>/admin/Web1, <ORACLE_BASE>/admin/Web2 and <ORACLE_BASE>/admin/WC1.     WL_HOME is the WebLogic server home and is always found at <MW_HOME>/wlserver_10.3.    Key file locations are shown below.              Directory        Notes                  <ORACLE_BASE>/admin/domain_name/aserver/domain_name        Shared location for domain.  Used to allow admin server to manually fail over between machines.  When creating domain_name provide the aserver directory as the location for the domain.         In my install this is <ORACLE_BASE>/admin/aserver/soa_domain as I only have one domain on the box.                  <ORACLE_BASE>/admin/domain_name/aserver/applications        Shared location for deployed applications.  Needs to be provided when creating the domain.         In my install this is <ORACLE_BASE>/admin/aserver/applications as I only have one domain on the box.                  <ORACLE_BASE>/admin/domain_name/mserver/domain_name        Either unique location for each machine or can be shared between machines to simplify task of packing and unpacking domain.  This acts as the managed server configuration location.  Keeping it separate from Admin server helps to avoid problems with the managed servers messing up the Admin Server.         In my install this is <ORACLE_BASE>/admin/mserver/soa_domain as I only have one domain on the box.                  <ORACLE_BASE>/admin/domain_name/mserver/applications        Either unique location for each machine or can be shared between machines.  Holds deployed applications.         In my install this is <ORACLE_BASE>/admin/mserver/applications as I only have one domain on the box.                  <ORACLE_BASE>/admin/domain_name/soa_cluster_name        Shared directory to hold the following           dd – deployment descriptors            jms – shared JMS file stores            fadapter – shared file adapter co-ordination files            tlogs – shared transaction log files          In my install this is <ORACLE_BASE>/admin/soa_cluster.                  <ORACLE_BASE>/admin/instance_name        Local folder for web server (OHS) instance.         In my install this is <ORACLE_BASE>/admin/web1 and <ORACLE_BASE>/admin/web2.          I also have <ORACLE_BASE>/admin/wc1 for the Web Cache I use as a load balancer.                  <ORACLE_BASE>/product/fmw        This can be a shared or local folder for the SOA/BPM Suite software.  I used a shared location so I only ran the installer once.         In my install this is <ORACLE_BASE>/SOA11gPS2           All the shared files need to be put onto a shared storage media.  I am using NFS, but recommendation for production would be a SAN, with mirrored disks for resilience.  Collapsing Environments  To reduce the hardware requirements it is possible to collapse the BAMHOST, SOAHOST and WEBHOST machines onto a single physical machine.  This will require more memory but memory is a lot cheaper than additional machines.  For environments that require higher security then stay with a separate WEBHOST tier as per the EDG.  Similarly for high volume environments then keep a separate set of machines for BAM and/or Web tier as per the EDG.  Notes on Dev Environments  In a dev environment it is acceptable to use a a single node (non-RAC) database, but be aware that the config of the data sources is different (no need to use multi-data source in WLS).  Typically in a dev environment we will collapse the BAMHOST, SOAHOST and WEBHOST onto a single machine and use a software load balancer.  To test a cluster properly we will need at least 2 machines.  For my test environment I used Oracle Web Cache as a load balancer.  I ran it on one of the SOA Suite machines and it load balanced across the Web Servers on both machines.  This was easy for me to set up and I could administer it from a web based console.