I wanted to use two Oracle SPARC T5-4 servers to simultaneously host both Oracle RAC and a WebLogic Server Cluster. I chose to use Oracle VM Server for SPARC to create a cluster like this:
There are plenty of trade offs and decisions that need to be made, for example:
Rather than configuring the system by hand, you might want to use an Oracle SuperCluster T5-8
My configuration is similar to jsavit's: Availability Best Practices - Example configuring a T5-8
but I chose to ignore some of the advice. Maybe I should have included an alternate service domain, but I decided that I already had enough
redundancy
Both Oracle SPARC T5-4 servers were to be configured like this:
Cntl
0.25 4 64GB
App LDom 2.75 CPU's 44 cores 704 GB
DB LDom One CPU 16 cores 256 GB
The systems started with everything in the primary domain: # ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
primary active -n-c-- UART 512 1023G 0.0% 0.0% 11m
# ldm list-spconfig
factory-default [current]
primary
# ldm list -o core,memory,physio
NAME
primary
CORE
CID CPUSET
0 (0, 1, 2, 3, 4, 5, 6, 7)
1 (8, 9, 10, 11, 12, 13, 14, 15)
2 (16, 17, 18, 19, 20, 21, 22, 23)
-- SNIP
62 (496, 497, 498, 499, 500, 501, 502, 503)
63 (504, 505, 506, 507, 508, 509, 510, 511)
MEMORY
RA PA SIZE
0x30000000 0x30000000 255G
0x80000000000 0x80000000000 256G
0x100000000000 0x100000000000 256G
0x180000000000 0x180000000000 256G # Give this memory block to the DB LDom
IO
DEVICE PSEUDONYM OPTIONS
[email protected] pci_0
[email protected] pci_1
[email protected] pci_2
[email protected] pci_3
[email protected] pci_4
[email protected] pci_5
[email protected] pci_6
[email protected] pci_7
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE1
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE2
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/MB/SASHBA0
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RIO/NET0
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE3
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE4
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE9
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE10
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE11
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE12
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE5
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE6
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE7
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE8
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE13
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE14
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE15
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE16
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/MB/SASHBA1
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RIO/NET2
Added an additional service processor configuration: # ldm add-spconfig split # ldm list-spconfig factory-default primary split [current]
And removed many of the resources from the primary domain: # ldm start-reconf primary
# ldm set-core 4 primary
# ldm set-memory 32G primary
# ldm rm-io
[email protected] primary
# ldm rm-io
[email protected] primary
# ldm rm-io
[email protected] primary
# ldm rm-io
[email protected] primary
# ldm rm-io
[email protected] primary
# ldm rm-io
[email protected] primary
# ldm rm-io
[email protected] primary
# init 6
Needed to add resources to the guest domains: # ldm add-domain db
# ldm set-core cid=`seq -s"," 48 63` db # ldm add-memory mblock=0x180000000000:256G db
# ldm add-io
[email protected] db
# ldm add-io
[email protected] db # ldm add-domain app
# ldm set-core 44 app
# ldm set-memory 704G app
# ldm add-io
[email protected] app
# ldm add-io
[email protected] app
# ldm add-io
[email protected] app
# ldm add-io
[email protected] app
# ldm add-io
[email protected] app
Needed to set up services: # ldm add-vds primary-vds0 primary # ldm add-vcc port-range=5000-5100 primary-vcc0 primary
Needed to add a virtual network port for the WebLogic application domain: # ipadm NAME CLASS/TYPE STATE UNDER ADDR lo0 loopback ok -- -- lo0/v4 static ok -- ... lo0/v6 static ok -- ... net0 ip ok -- ... net0/v4 static ok -- xxx.xxx.xxx.xxx/24 net0/v6 addrconf ok -- .... net0/v6 addrconf ok -- ... net8 ip ok -- -- net8/v4 static ok -- ... # dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net1 Ethernet unknown 0 unknown ixgbe1 net0 Ethernet up 1000 full ixgbe0 net8 Ethernet up 10 full usbecm2 # ldm add-vsw net-dev=net0 primary-vsw0 primary # ldm add-vnet vnet1 primary-vsw0 app
Needed to add a virtual disk to the WebLogic application domain: # format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t5000CCA02505F874d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB> /scsi_vhci/
[email protected] /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD0/disk 1. c0t5000CCA02506C468d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB> /scsi_vhci/
[email protected] /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD1/disk 2. c0t5000CCA025067E5Cd0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB> /scsi_vhci/
[email protected] /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD2/disk 3. c0t5000CCA02506C258d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB> /scsi_vhci/
[email protected] /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD3/disk Specify disk (enter its number): ^C # ldm add-vdsdev /dev/dsk/c0t5000CCA02506C468d0s2
[email protected] # ldm add-vdisk HDD1
[email protected] app
Add some additional spice to the pot: # ldm set-variable auto-boot\\?=false db # ldm set-variable auto-boot\\?=false app # ldm set-var boot-device=HDD1 app
Bind the logical domains: # ldm bind db # ldm bind app
At the end of the process, the system is set up like this: # ldm list -o core,memory,physio NAME primary CORE CID CPUSET 0 (0, 1, 2, 3, 4, 5, 6, 7) 1 (8, 9, 10, 11, 12, 13, 14, 15) 2 (16, 17, 18, 19, 20, 21, 22, 23) 3 (24, 25, 26, 27, 28, 29, 30, 31) MEMORY RA PA SIZE 0x30000000 0x30000000 32G IO DEVICE PSEUDONYM OPTIONS
[email protected] pci_0
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE1
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE2
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/MB/SASHBA0
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RIO/NET0 ------------------------------------------------------------------------------ NAME app CORE CID CPUSET 4 (32, 33, 34, 35, 36, 37, 38, 39) 5 (40, 41, 42, 43, 44, 45, 46, 47) 6 (48, 49, 50, 51, 52, 53, 54, 55) 7 (56, 57, 58, 59, 60, 61, 62, 63) 8 (64, 65, 66, 67, 68, 69, 70, 71) 9 (72, 73, 74, 75, 76, 77, 78, 79) 10 (80, 81, 82, 83, 84, 85, 86, 87) 11 (88, 89, 90, 91, 92, 93, 94, 95) 12 (96, 97, 98, 99, 100, 101, 102, 103) 13 (104, 105, 106, 107, 108, 109, 110, 111) 14 (112, 113, 114, 115, 116, 117, 118, 119) 15 (120, 121, 122, 123, 124, 125, 126, 127) 16 (128, 129, 130, 131, 132, 133, 134, 135) 17 (136, 137, 138, 139, 140, 141, 142, 143) 18 (144, 145, 146, 147, 148, 149, 150, 151) 19 (152, 153, 154, 155, 156, 157, 158, 159) 20 (160, 161, 162, 163, 164, 165, 166, 167) 21 (168, 169, 170, 171, 172, 173, 174, 175) 22 (176, 177, 178, 179, 180, 181, 182, 183) 23 (184, 185, 186, 187, 188, 189, 190, 191) 24 (192, 193, 194, 195, 196, 197, 198, 199) 25 (200, 201, 202, 203, 204, 205, 206, 207) 26 (208, 209, 210, 211, 212, 213, 214, 215) 27 (216, 217, 218, 219, 220, 221, 222, 223) 28 (224, 225, 226, 227, 228, 229, 230, 231) 29 (232, 233, 234, 235, 236, 237, 238, 239) 30 (240, 241, 242, 243, 244, 245, 246, 247) 31 (248, 249, 250, 251, 252, 253, 254, 255) 32 (256, 257, 258, 259, 260, 261, 262, 263) 33 (264, 265, 266, 267, 268, 269, 270, 271) 34 (272, 273, 274, 275, 276, 277, 278, 279) 35 (280, 281, 282, 283, 284, 285, 286, 287) 36 (288, 289, 290, 291, 292, 293, 294, 295) 37 (296, 297, 298, 299, 300, 301, 302, 303) 38 (304, 305, 306, 307, 308, 309, 310, 311) 39 (312, 313, 314, 315, 316, 317, 318, 319) 40 (320, 321, 322, 323, 324, 325, 326, 327) 41 (328, 329, 330, 331, 332, 333, 334, 335) 42 (336, 337, 338, 339, 340, 341, 342, 343) 43 (344, 345, 346, 347, 348, 349, 350, 351) 44 (352, 353, 354, 355, 356, 357, 358, 359) 45 (360, 361, 362, 363, 364, 365, 366, 367) 46 (368, 369, 370, 371, 372, 373, 374, 375) 47 (376, 377, 378, 379, 380, 381, 382, 383) MEMORY RA PA SIZE 0x30000000 0x830000000 192G 0x4000000000 0x80000000000 256G 0x8080000000 0x100000000000 256G IO DEVICE PSEUDONYM OPTIONS
[email protected] pci_1
[email protected] pci_2
[email protected] pci_3
[email protected] pci_4
[email protected] pci_5
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE3
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE4
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE9
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE10
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE11
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE12
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE5
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE6
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE7
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE8 ------------------------------------------------------------------------------ NAME db CORE CID CPUSET 48 (384, 385, 386, 387, 388, 389, 390, 391) 49 (392, 393, 394, 395, 396, 397, 398, 399) 50 (400, 401, 402, 403, 404, 405, 406, 407) 51 (408, 409, 410, 411, 412, 413, 414, 415) 52 (416, 417, 418, 419, 420, 421, 422, 423) 53 (424, 425, 426, 427, 428, 429, 430, 431) 54 (432, 433, 434, 435, 436, 437, 438, 439) 55 (440, 441, 442, 443, 444, 445, 446, 447) 56 (448, 449, 450, 451, 452, 453, 454, 455) 57 (456, 457, 458, 459, 460, 461, 462, 463) 58 (464, 465, 466, 467, 468, 469, 470, 471) 59 (472, 473, 474, 475, 476, 477, 478, 479) 60 (480, 481, 482, 483, 484, 485, 486, 487) 61 (488, 489, 490, 491, 492, 493, 494, 495) 62 (496, 497, 498, 499, 500, 501, 502, 503) 63 (504, 505, 506, 507, 508, 509, 510, 511) MEMORY RA PA SIZE 0x80000000 0x180000000000 256G IO DEVICE PSEUDONYM OPTIONS
[email protected] pci_6
[email protected] pci_7
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE13
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE14
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE15
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RCSA/PCIE16
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/MB/SASHBA1
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected] /SYS/RIO/NET2
Start the domains: # ldm start app LDom app started # ldm start db LDom db started
Make sure to start the vntsd service that was created, above. # svcs -a | grep ldo disabled 8:38:38 svc:/ldoms/vntsd:default online 8:38:58 svc:/ldoms/agents:default online 8:39:25 svc:/ldoms/ldmd:default # svcadm enable vntsd
Now use the MAC address to configure the Solaris 11 Automated Installation.
Database Logical Domain
# telnet localhost
5000
{0} ok devalias
screen
/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]
disk7
/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]
disk6
/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]
disk5
/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]
disk4
/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]
scsi1
/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]
net3
/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected],1
net2
/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]
virtual-console /virtual-devices/
[email protected]
name aliases
{0} ok boot net2
Boot device:
/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected]/
[email protected] File
and args:
1000 Mbps full duplex Link up Requesting Internet Address for xx:xx:xx:xx:xx:xx Requesting Internet Address for xx:xx:xx:xx:xx:xx
WLS Logical Domain
# telnet localhost
5001
{0} ok devalias hdd1
/
[email protected]/
[email protected]/
[email protected] vnet1
/
[email protected]/
[email protected]/
[email protected] net
/
[email protected]/
[email protected]/
[email protected] disk
/
[email protected]/
[email protected]/
[email protected] virtual-console
/virtual-devices/
[email protected] name aliases
{0} ok boot net Boot device:
/
[email protected]/
[email protected]/
[email protected] File
and args: Requesting Internet Address for xx:xx:xx:xx:xx:xx Requesting Internet Address for xx:xx:xx:xx:xx:xx
Repeat the process for the second SPARC T5-4, install Solaris, RAC
and WebLogic Cluster, and you are ready to go. Maybe buying a
SuperCluster would have been easier.