Using Oracle Enterprise Manager Ops Center to Update Solaris via Live Upgrade
- by LeonShaner
Introduction:
This Oracle Enterprise Manager Ops Center blog entry provides tips for using Ops Center to update Solaris using Live Upgrade on Solaris 10 and Boot Environments on Solaris 11.
Why use Live Upgrade?
Live Upgrade (LU) can significantly reduce downtime associated with patching
Live Upgrade avoids dropping to single-user mode for long periods of time during patching
Live Upgrade relies on an Alternate Boot Environment (ABE)/(BE), which is patched while in multi-user mode; thereby allowing normal system operations to continue with the active BE, while the alternate BE is being patched
Activating an newly patched (A)BE is essentially a reboot; therefore the downtime is ~= reboot
Admins can easily revert to the prior Boot Environment (BE) as a safeguard / fallback.
Why use Ops Center to patch via Live Upgrade, Alternate Boot Environments, and Solaris 11 equivalents?
All the benefits of Ops Center's extensive patch and package knowledge base can be leveraged on top of Live Upgrade
Ops Center can orchestrate patching based on Live Upgrade and Solaris 11 features, which all works together to minimize downtime
Ops Centers advanced inventory and reporting features assurance that each OS is updated to a verifiable, consistent standard, rather than relying on ad-hoc (error prone) procedures and scripts
Ops Center gives admins control over the boot environment specifications or they can let Ops Center decide when a BE is necessary, thereby reducing complexity and lowering the opportunity for user error
Preparing to use Live Upgrade-like features in Solaris 11
Requirements and information you should know:
Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems
Solaris 11 has features which are similar in concept to Live Upgrade on Solaris 10, but differ greatly in implementationImportant distinctions:
Solaris 11 assumes ZFS root
Solaris 11 adds Boot Environments (BE's) as an integrated feature (see beadm)
Solaris 11 BE's avoid single-user patching (vs. Solaris 10 w/ ZFS snapshot=ABE).
Solaris 11 Image Packaging System (IPS) has hooks for BE creation, as needed
Solaris 11 allows pkgs to be installed + upgraded in alternate BE (e.g. instead of the live system) but it is controlled on a per-pkg basis
Boot Environments are activated across a reboot; instead of spending long periods installing + upgrading packages in single user mode.
Fallback to a prior BE is a function of the BE infrastructure (a la beadm).
(Generally) Reboot + BE activation can be much much faster on Solaris 11
Preparing to use Live Upgrade on Solaris 10
Requirements and information you should know:
Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems
Live Upgrade Pre-requisite patches must be applied before the first Live Upgrade Alternate Boot Environments are created (see "Pre-requisite Patches" section, below...)
Solaris 10 Update 6 or newer on ZFS root is the practical starting point for Live Upgrade
Live Upgrade with ZFS root is far more straight-forward than any scheme based on Alternative Boot Environments in slices or temporarily breaking mirrors
Use Solaris best practices to upgrade the OS to at least Solaris 10 Update 4 (outside of Ops Center)
UFS root can (technically) be used, but it is significantly more involved (e.g. discouraged) -- there are many reasons to move to ZFS while going through the process to update to Solaris 10 Update 6 or newer (out side of Ops Center)
Recommendation: Start with Solaris 10 Update 6 or newer on ZFS root
Recommendation: Start with Ops Center 12c or newer
Ops Center 12c can automatically create your ABE's for you, without the need for custom scripts
Ops Center 12c Update 2 avoids kernel panic on unpatched Solaris 10 update 9 (and older) -- unrelated to Live Upgrade, but more on the issue, below.
NOTE: There is no magic! If you have systems running Solaris 10 Update 5 or older on UFS root, and you don't know how to get them updated to Solaris 10 on ZFS root, then there are services available from Oracle Advanced Customer Support (ACS), which specialize in this area.
Live Upgrade Pre-requisite Patches (Solaris 10)
Certain Live Upgrade related patches must be present before the first Live Upgrade ABE's are created on Solaris 10.Use the following MOS Search String to find the “living document” that outlines the required patch minimums, which are necessary before using any Live Upgrade features:
Solaris Live Upgrade Software Patch Requirements(Click above – the link is valid as of this writing, but search in MOS for the same "Solaris Live Upgrade Software Patch Requirements" string if necessary)
It is a very good idea to check the document periodically and adapt to its contents, accordingly.IMPORTANT: In case it wasn't clear in the above document, some direct patching of the active OS, including a reboot, may be required before Live Upgrade can be successfully used the first time.HINT: You can use Ops Center to determine what to expect for a given system, and to schedule the “pre-patching” during a maintenance window if necessary.
Preparing to use Ops Center
Discover + Manage (Install + Configure the Ops Center agent in) each Global Zone
Recommendation: Begin by using OCDoctor --agent-prereq to determine whether OS meets OC prerequisites (resolve any issues)
See prior requirements and recommendations w.r.t. starting with Solaris 10 Update 6 or newer on ZFS (or at least Solaris 10 Update 4 on UFS, with caveats)
WARNING: Systems running unpatched Solaris 10 update 9 (or older) should run the Ops Center 12c Update 2 agent to avoid a potential kernel panic
The 12c Update 2 agent will check patch minimums and disable certain process accounting features if the kernel is not sufficiently patched to avoid the panic
SPARC: 142900-05 Obsoleted by: 142900-06 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit)
X64: 142901-05 Obsoleted by: 142901-06 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit)
OR
SPARC: 142909-17 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit)
X64: 142910-17 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit)
Ops Center 12c (initial release) and 12c Update 1 agent can also be safely used with a workaround (to be performed BEFORE installing the agent):
# mkdir -p /etc/opt/sun/oc
# echo "zstat_exacct_allowed=false" > /etc/opt/sun/oc/zstat.conf
# chmod 755 /etc/opt/sun /etc/opt/sun/oc
# chmod 644 /etc/opt/sun/oc/zstat.conf
# chown -Rh root:sys /etc/opt/sun/oc
NOTE: Remove the above after patching the OS sufficiently, or after upgrading to the 12c Update 2 agent
Using Ops Center to apply Live Upgrade-related Pre-Patches (Solaris 10)Overview:
Create an OS Update Profile containing the minimum LU-related pre-patches, based on the Solaris Live Upgrade Software Patch Requirements, previously mentioned.
SIMULATE the deployment of the LU-related pre-patches
Observe whether any of the LU-related pre-patches will require a reboot
The job details for each Global Zone will advise whether a reboot step will be required
ACTUALLY deploy the LU-related pre-patches, according to your change control process (e.g. if no reboot, maybe okay to do now; vs. must do later because of the reboot).
You can schedule the job to occur later, during a maintenance window
Check the job status for each node, resolving any issues found
Once the LU-related pre-patches are applied, you can Ops Center to patch using Live Upgrade on Solaris 10
Using Ops Center to patch Solaris 10 with LU/ABE's -- the GOODS!(this is the heart of the tip):
Create an OS Update Profile containing the patches that make up your standard build
Use Solaris Baselines when possible
Add other individual patches as needed
ACTUALLY deploy the OS Update Profile
Specify the appropriate Live Upgrade options, e.g.
Synchronize the active BE to the alternate BE before patching
Do not activate the BE after patching
Check the job status for each node, resolving any issues found
Activate the newly patched BE according to your change control process
Activate = Reboot to the ABE, making the ABE the new active BE
Ops Center does not separate LU activate from reboot, so expect a reboot!
Check the job status for each node, resolving any issues found
Examples (w/Screenshots)
Solaris 10 and Live Upgrade: Auto-Create the Alternate Boot Environment (ZFS root only)
ABE to be created on ZFS with name S10_12_07REC (Example)
Uses built in feature to call “lucreate -n S10_12_07REC” behind scenes if not already present
NOTE: Leave “lucreate” params blank (if you do specify options, the will be appended after -n $ABEName)
Solaris 10 and Live Upgrade: Alternate Boot Environment Creation via Operational Profile (script)
The Alternate Boot Environment is to be created via custom, user-supplied script, which does whatever is needed for the system where Live Upgrade will be used.
Operational Profile, which provides the script to create an ABE:
Very similar to the automatic case, but with a Script (Operational Profile), which is used to create the ABE
Relies on user-supplied script in the form of an Operational Profile
Could
be used to prepare an ABE based on a UFS root in a slice, or on a
separate device (e.g. by breaking a mirror first) – it is up to the script author to do the right thing!
EXAMPLE: Same result as the ZFS case, but illustrating the Operational Profile (e.g. script) approach to call:
# lucreate -n S10_1207REC
NOTE: OC special variable is $ABEName
Boot Environment Profile, which references the Operational Profile
Script = Operational Profile on this screen
Refers to Operational Profile shown in the previous section
The user-supplied S10_Create_BE Operational Profile will be run
The Operational Profile must send a non-zero exit code if there is a problem (so that the OS Update job will not proceed)
Solaris 10 OS Update Profile (to provide the actual patch specifications)
Solaris 10 Baseline “Recommended” chosen for “Install”
Solaris 10 OS Update Plan (two-steps in this case)
“Create a Boot Environment” + “Update OS” are chosen.
Using Ops Center to patch Solaris 11 with Boot Environments (as needed)
Create a Solaris 11 OS Update Profile containing the packages that make up your standard build
ACTUALLY deploy the Solaris 11 OS Update Profile
BE will be created if needed (or you can stipulate no BE)
BE name will be auto-generated (if needed), or you may specify a BE name
Check the job status for each node, resolving any issues found
Check if a BE was created; if so, activate the new BE
Activate = Reboot to the BE, making the new BE the active BE
Ops Center does not separate BE activate from reboot
NOTE: Not every Solaris 11 OS Update will require a new BE, so a reboot may not be necessary.
Solaris 11: Auto BE Create (as Needed -- let Ops Center decide)
BE to be created as needed
BE to be named automatically
Reboot (if necessary) deferred to separate step
Solaris 11: OS Profile
Solaris 11 “entire” chosen for a particular SRU
Solaris 11: OS Update Plan (w/BE)
“Create a Boot Environment” + “Update OS” are chosen.
Summary:
Solaris 10 Live Upgrade, Alternate Boot Environments, and their equivalents on Solaris 11 can be very powerful tools to help minimize the downtime associated with updating your servers. For very old Solaris, there are some important prerequisites to adhere to, but once the initial preparation is complete, Live Upgrade can be used going forward. For Solaris 11, the built-in Boot Environment handling is leveraged directly by the Image Packaging System, and the result is a much more straight forward way to patch, and far fewer prerequisites to satisfy in getting there. Ops Center simplifies using either approach, and helps you improve consistency from system to system, which ultimately helps you improve the overall up-time across all the Solaris systems in your environment.
Please let us know what you think? Until next time...\Leon--
Leon Shaner | Senior IT/Product ArchitectSystems Management | Ops Center Engineering @ Oracle
The
views expressed on this [blog; Web site] are my own and do not
necessarily reflect the views of Oracle.
For more information, please go to Oracle Enterprise Manager web page or follow us at :
Twitter | Facebook | YouTube | Linkedin | Newsletter
