Introduction:  
  This Oracle Enterprise Manager Ops Center blog entry provides tips for using Ops Center to update Solaris using Live Upgrade on Solaris 10 and Boot Environments on Solaris 11. 
  Why use Live Upgrade? 
   
    Live Upgrade (LU) can significantly reduce downtime associated with patching 
    Live Upgrade avoids dropping to single-user mode for long periods of time during patching 
    Live Upgrade relies on an Alternate Boot Environment (ABE)/(BE), which is patched while in multi-user mode; thereby allowing normal system operations to continue with the active BE, while the alternate BE is being patched 
    Activating an newly patched (A)BE is essentially a reboot; therefore the downtime is ~= reboot 
    Admins can easily revert to the prior Boot Environment (BE) as a safeguard / fallback. 
   
  Why use Ops Center to patch via Live Upgrade, Alternate Boot Environments, and Solaris 11 equivalents? 
   
    All the benefits of Ops Center's extensive patch and package knowledge base can be leveraged on top of Live Upgrade 
    Ops Center can orchestrate patching based on Live Upgrade and Solaris 11 features, which all works together to minimize downtime 
    Ops Centers advanced inventory and reporting features assurance that each OS is updated to a verifiable, consistent standard, rather than relying on ad-hoc (error prone) procedures and scripts 
    Ops Center gives admins control over the boot environment specifications or they can let Ops Center decide when a BE is necessary, thereby reducing complexity and lowering the opportunity for user error 
   
  Preparing to use Live Upgrade-like features in Solaris 11 
  Requirements and information you should know:  
   
    Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems 
   
   
    Solaris 11 has features which are similar in concept to Live Upgrade on Solaris 10, but differ greatly in implementationImportant distinctions:
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
       
        Solaris 11 assumes ZFS root 
        Solaris 11 adds Boot Environments (BE's) as an integrated feature (see beadm) 
        Solaris 11 BE's avoid single-user patching (vs. Solaris 10 w/ ZFS snapshot=ABE). 
        Solaris 11 Image Packaging System (IPS) has hooks for BE creation, as needed 
         
          Solaris 11 allows pkgs to be installed + upgraded in alternate BE (e.g. instead of the live system) but it is controlled on a per-pkg basis 
          Boot Environments are activated across a reboot; instead of spending long periods installing + upgrading packages in single user mode. 
         
        Fallback to a prior BE is a function of the BE infrastructure (a la beadm). 
        (Generally) Reboot + BE activation can be much much faster on Solaris 11 
          
       
     
   
  Preparing to use Live Upgrade on Solaris 10 
  Requirements and information you should know: 
   
     Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems 
   
   
    Live Upgrade Pre-requisite patches must be applied before the first Live Upgrade Alternate Boot Environments are created (see "Pre-requisite Patches" section, below...) 
    Solaris 10 Update 6 or newer on ZFS root is the practical starting point for Live Upgrade 
     
      Live Upgrade with ZFS root is far more straight-forward than any scheme based on Alternative Boot Environments in slices or temporarily breaking mirrors 
      Use Solaris best practices to upgrade the OS to at least Solaris 10 Update 4 (outside of Ops Center) 
      UFS root can (technically) be used, but it is significantly more involved (e.g. discouraged) -- there are many reasons to move to ZFS while going through the process to update to Solaris 10 Update 6 or newer (out side of Ops Center) 
     
   
   
    Recommendation: Start with Solaris 10 Update 6 or newer on ZFS root 
    Recommendation: Start with Ops Center 12c or newer 
     
      Ops Center 12c can automatically create your ABE's for you, without the need for custom scripts 
      Ops Center 12c Update 2 avoids kernel panic on unpatched Solaris 10 update 9 (and older) -- unrelated to Live Upgrade, but more on the issue, below. 
     
  NOTE: There is no magic!  If you have systems running Solaris 10 Update 5 or older on UFS root, and you don't know how to get them updated to Solaris 10 on ZFS root, then there are services available from Oracle Advanced Customer Support (ACS), which specialize in this area. 
   
     
        
     
   
  Live Upgrade Pre-requisite Patches (Solaris 10) 
  Certain Live Upgrade related patches must be present before the first Live Upgrade ABE's are created on Solaris 10.Use the following MOS Search String to find the “living document” that outlines the required patch minimums, which are necessary before using any Live Upgrade features: 
  Solaris Live Upgrade Software Patch Requirements(Click above – the link is valid as of this writing, but search in MOS for the same "Solaris Live Upgrade Software Patch Requirements" string if necessary) 
  It is a very good idea to check the document periodically and adapt to its contents, accordingly.IMPORTANT:  In case it wasn't clear in the above document, some direct patching of the active OS, including a reboot, may be required before Live Upgrade can be successfully used the first time.HINT: You can use Ops Center to determine what to expect for a given system, and to schedule the “pre-patching” during a maintenance window if necessary. 
    
  Preparing to use Ops Center 
   
    Discover + Manage (Install + Configure the Ops Center agent in) each Global Zone 
    Recommendation:  Begin by using OCDoctor --agent-prereq to determine whether OS meets OC prerequisites (resolve any issues) 
    See prior requirements and recommendations w.r.t. starting with Solaris 10 Update 6 or newer on ZFS (or at least Solaris 10 Update 4 on UFS, with caveats) 
    WARNING: Systems running unpatched Solaris 10 update 9 (or older) should run the Ops Center 12c Update 2 agent to avoid a potential kernel panic 
     
      The 12c Update 2 agent will check patch minimums and disable certain process accounting features if the kernel is not sufficiently patched to avoid the panic 
       
        SPARC: 142900-05 Obsoleted by: 142900-06 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit) 
        X64: 142901-05 Obsoleted by: 142901-06 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit) 
       
      OR 
       
        SPARC: 142909-17 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit) 
        X64: 142910-17 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit) 
       
      Ops Center 12c (initial release) and 12c Update 1 agent can also be safely used with a workaround (to be performed BEFORE installing the agent): 
     
   
   
     
       
        # mkdir -p /etc/opt/sun/oc 
        # echo "zstat_exacct_allowed=false" > /etc/opt/sun/oc/zstat.conf 
        # chmod 755 /etc/opt/sun /etc/opt/sun/oc 
        # chmod 644 /etc/opt/sun/oc/zstat.conf 
        # chown -Rh root:sys /etc/opt/sun/oc 
       
     
     
       
        NOTE: Remove the above after patching the OS sufficiently, or after upgrading to the 12c Update 2 agent  
       
     
   
    
  Using Ops Center to apply Live Upgrade-related Pre-Patches (Solaris 10)Overview: 
   
    Create an OS Update Profile containing the minimum LU-related pre-patches, based on the Solaris Live Upgrade Software Patch Requirements, previously mentioned. 
    SIMULATE the deployment of the LU-related pre-patches 
    Observe whether any of the LU-related pre-patches will require a reboot 
     
      The job details for each Global Zone will advise whether a reboot step will be required 
     
    ACTUALLY deploy the LU-related pre-patches, according to your change control process (e.g. if no reboot, maybe okay to do now; vs. must do later because of the reboot). 
     
      You can schedule the job to occur later, during a maintenance window 
     
    Check the job status for each node, resolving any issues found 
    Once the LU-related pre-patches are applied, you can Ops Center to patch using Live Upgrade on Solaris 10 
   
    
  Using Ops Center to patch Solaris 10 with LU/ABE's -- the GOODS!(this is the heart of the tip): 
   
    Create an OS Update Profile containing the patches that make up your standard build 
     
      Use Solaris Baselines when possible 
      Add other individual patches as needed 
     
    ACTUALLY deploy the OS Update Profile 
     
      Specify the appropriate Live Upgrade options, e.g. 
       
        Synchronize the active BE to the alternate BE before patching 
        Do not activate the BE after patching 
       
     
    Check the job status for each node, resolving any issues found 
    Activate the newly patched BE according to your change control process 
     
      Activate = Reboot to the ABE, making the ABE the new active BE 
       
        Ops Center does not separate LU activate from reboot, so expect a reboot! 
       
     
    Check the job status for each node, resolving any issues found 
   
    
    
  Examples (w/Screenshots) 
  Solaris 10 and Live Upgrade: Auto-Create the Alternate Boot Environment (ZFS root only) 
  ABE to be created on ZFS with name S10_12_07REC (Example) 
  Uses built in feature to call “lucreate -n S10_12_07REC” behind scenes if not already present 
  NOTE: Leave “lucreate” params blank (if you do specify options, the will be appended after -n $ABEName)  
    
    
    
    
    
    
  Solaris 10 and Live Upgrade: Alternate Boot Environment Creation via Operational Profile (script) 
  The Alternate Boot Environment is to be created via custom, user-supplied script, which does whatever is needed for the system where Live Upgrade will be used. 
  Operational Profile, which provides the script to create an ABE: 
   
    Very similar to the automatic case, but with a Script (Operational Profile), which is used to create the ABE 
    Relies on user-supplied script in the form of an Operational Profile 
    Could
 be used to prepare an ABE based on a UFS root in a slice, or on a 
separate device (e.g. by breaking a mirror first) – it is up to the script author to do the right thing! 
    EXAMPLE: Same result as the ZFS case, but illustrating the Operational Profile (e.g. script) approach to call: 
   
  # lucreate -n S10_1207REC 
    
   
    
  NOTE: OC special variable is $ABEName 
   
    
    
    
  Boot Environment Profile, which references the Operational Profile 
   
    Script = Operational Profile on this screen 
    Refers to Operational Profile shown in the previous section 
    The user-supplied S10_Create_BE Operational Profile will be run 
    The Operational Profile must send a non-zero exit code if there is a problem (so that the OS Update job will not proceed) 
   
    
   
   
      
   
    
  Solaris 10 OS Update Profile (to provide the actual patch specifications) 
  Solaris 10 Baseline “Recommended” chosen for “Install” 
    
    
   
      
   
    
  Solaris 10 OS Update Plan (two-steps in this case) 
  “Create a Boot Environment” + “Update OS” are chosen. 
    
    
   
      
   
    
  Using Ops Center to patch Solaris 11 with Boot Environments (as needed) 
   
     Create a Solaris 11 OS Update Profile containing the packages that make up your standard build 
    ACTUALLY deploy the Solaris 11 OS Update Profile 
     
      BE will be created if needed (or you can stipulate no BE) 
      BE name will be auto-generated (if needed), or you may specify a BE name 
     
    Check the job status for each node, resolving any issues found 
    Check if a BE was created; if so, activate the new BE 
     
      Activate = Reboot to the BE, making the new BE the active BE 
       
        Ops Center does not separate BE activate from reboot 
       
     
    NOTE: Not every Solaris 11 OS Update will require a new BE, so a reboot may not be necessary. 
   
   
  Solaris 11: Auto BE Create (as Needed -- let Ops Center decide) 
   
    BE to be created as needed 
    BE to be named automatically 
    Reboot (if necessary) deferred to separate step 
   
    
   
   
      
   
    
  Solaris 11: OS Profile 
  Solaris 11 “entire” chosen for a particular SRU 
    
    
    
    
  Solaris 11: OS Update Plan (w/BE) 
   “Create a Boot Environment” + “Update OS” are chosen. 
    
    
    
    
    
    
    
    
    
    
    
    
    
  Summary: 
  Solaris 10 Live Upgrade, Alternate Boot Environments, and their equivalents on Solaris 11 can be very powerful tools to help minimize the downtime associated with updating your servers.  For very old Solaris, there are some important prerequisites to adhere to, but once the initial preparation is complete, Live Upgrade can be used going forward.  For Solaris 11, the built-in Boot Environment handling is leveraged directly by the Image Packaging System, and the result is a much more straight forward way to patch, and far fewer prerequisites to satisfy in getting there.  Ops Center simplifies using either approach, and helps you improve consistency from system to system, which ultimately helps you improve the overall up-time across all the Solaris systems in your environment. 
    
  Please let us know what you think?  Until next time...\Leon--   
  Leon Shaner | Senior IT/Product ArchitectSystems Management | Ops Center Engineering @ Oracle 
  The
views expressed on this [blog; Web site] are my own and do not
necessarily reflect the views of Oracle. 
    
   
  For more information, please go to Oracle Enterprise Manager  web page or  follow us at :  
  
  
  
  
    
    Twitter |  Facebook |  YouTube |  Linkedin |  Newsletter