Upgrade Best Practices Fault-Tolerant Deployments

Upgrades in a fault-tolerant environment are supported. However, take care to follow the recommendations in this topic and in the related procedure. All your Primary and Secondary sss must be upgraded within a small window of a few hours.
casp1032
Upgrade Best Practices Fault-Tolerant Deployments
Upgrades in a fault-tolerant environment are supported. However, follow the recommendations in this topic. All your Primary and Secondary 
SpectroSERVER
s must be upgraded within a small window of a few hours.
Before conducting any upgrades, verify that all 
DX NetOps Spectrum
 components are up-to-date with current service packs. Also, consult the list of the communication ports and protocols that 
DX NetOps Spectrum
 uses. Your security parameters may require you to temporarily disable these ports and protocols during the upgrade. For more information about firewall ports and protocols, see .
First, upgrade the OneClick Web Servers, after that upgrade the primary 
SpectroSERVER
s, and finally upgrade the secondary 
SpectroSERVER
s.
FAQs
 
Q: In a distributed SS (DSS) environment, what is the supported and best practice for upgrading servers?
 
 
A:
 Upgrade the Primary MLS (main location server) SpectroSERVER first, followed by the Primary OneClick server, followed by the remaining Primary SpectroSERVERs. It is important that you start the primary MLS first, before starting the other primary SpectroSERVERs. This allows all of the cross landscape and "MOM" (manager of managers) functions to process and complete. Once the primary MLS is running and active, you can start the rest of the primaries.  Upgrade the Secondary SpectroSERVERs last (including all SpectroSERVER and non-primary OneClick servers). Once all SpectroSERVERs have been upgraded, be sure to stop and restart all OneClick tomcat processes 
again
 to update the tomcat cache with the updated SS info. Not doing this will cause OneClick issues, such as landscapes not showing in the client.
In a DSS setup, if you stop a primary SS to test your failover scenario without performing a successful database synchronization (via online backup) first, you may see the landscapes disappear in OneClick. To make sure this does not happen, manually start the online backup and allow the database synchronization to occur. Once complete, you can then test your failover scenario.
Upgrade in a Fault-Tolerant Environment
The following procedure describes an upgrade in a fault-tolerant environment. Use the same procedure for a single-server or DSS upgrade. Upgrade the OneClick Web Servers first, and then upgrade primary 
SpectroSERVER
s. Upgrade Secondary 
SpectroSERVER
s at last.
Alarms may display incorrectly in OneClick until the upgrade process is complete, including any post-upgrade steps that may be required. Avoid using 
DX NetOps Spectrum
 for management until you have completed all the steps.
 
Follow these steps:
 
  1. Disable automatic online backups on the Primary 
    SpectroSERVER
    s by taking the following steps:
    1. Highlight the VNM model in the Universe Topology view.
    2. In the Component Detail panel, select the Information tab.
    3. Locate and expand the Online Database Backup subview.
    4. Set Automatic Backups to “Disabled”.
  2. Perform a manual online backup on every Primary 
    SpectroSERVER
     to preserve the current database. Take the following steps:
    1. In the Online Database Backup subview, click Begin Backup Now.
    2. Verify that the online backup has succeeded.
    We recommend differentiating the databases by the version number. Set the “Prefix for Backup File Name” parameter from the default of “db_” to something like “db_
    version
    _” or another value that identifies the version.
  3. Edit the .vnmrc file to increase the 'maximum event records' parameter on all Secondary 
    SpectroSERVER
    s. For example, change the following parameter:
    max_event_records=20000
    to the following value:
    max_event_records=200000
    The new value prevents event loss during the upgrade.
  4. Restart all Secondary 
    SpectroSERVER
    s so that the change takes effect:
    1. Launch a Spectrum Control Panel.
    2. Click Stop 
      SpectroSERVER
      .
    3. Once the Status changes to “INACTIVE,” click Start 
      SpectroSERVER
      .
    4. Verify that the Status changes to “RUNNING”.
  5. Instruct all OneClick users who are logged in to the OneClick Web Servers to close their clients.
  6. Stop each of the Primary 
    SpectroSERVER
    s:
    1. Launch a Spectrum Control Panel
    2. Click Stop 
      SpectroSERVER
      .
    3. You are prompted to verify the action to stop the 
      SpectroSERVER
       and Archive Manager.
    4. Verify that the Status changes to “INACTIVE”.
    5. Exit the Spectrum Control Panel.
  7. On the OneClick clients, verify that the 
    SpectroSERVER
    s have failed over to the Secondary 
    SpectroSERVER
    s.
    The landscape icons in the Explorer hierarchy panel change from green to yellow
    Yellow borders appear around the Contents and Component Detail panels.
  8. Stop all OneClick Web Servers using the following command:
    $SPECROOT/tomcat/bin/stopTomcat.sh
  9. Upgrade all OneClick web servers.
  10. Upgrade all the Primary 
    SpectroSERVER
    s, and verify that the installations complete successfully.
     Wait for every installation to complete before continuing to the next step.
  11. Manually start the Archive Manager on every Primary 
    SpectroSERVER
    :
    1. Launch a Spectrum Control Panel.
    2. Select Control, and click Start Archive Manager.
    Starting the Archive Manager ensures that the events that are being stored locally on the Secondary 
    SpectroSERVER
     are sent over to the Primary Archive Manager.
  12. Wait for all events on the Secondary 
    SpectroSERVER
    s to be sent over to the Primary Archive Manager. To verify their status, take the following steps:
    1. In the OneClick client, highlight the VNM model for each 
      SpectroSERVER
       in the Universe Topology view.
    2. In the Component Detail panel, select the Information tab.
    3. Locate and expand the 
      SpectroSERVER
       Control subview.
    4. Locate and expand the Event Log Information subview.
    5. Wait for the total of Locally Stored Events to reach zero (0).
  13. Start the 
    SpectroSERVER
     on the Primary MLS:
    1. Launch a Spectrum Control Panel on the Primary MLS.
    2. Click Start 
      SpectroSERVER
      .
    3. Verify that the Status changes to “RUNNING”.
  14. Start the remaining Primary 
    SpectroSERVER
    s.
  15. Stop each of the secondary 
    SpectroSERVER
    s.
  16. Upgrade all the Secondary 
    SpectroSERVER
    s, and verify that the installations complete successfully.
    Wait for every installation to complete before continuing to the next step.
  17. Start the 
    SpectroSERVER
     on every Secondary 
    SpectroSERVER
    :
    1. Launch a Spectrum Control Panel on the 
      SpectroSERVER
      .
    2. Click Start 
      SpectroSERVER
      .
    3. Verify that the Status changes to “RUNNING”.
  18. Verify that the clients open successfully and that the Connection Status all the Primary 
    SpectroSERVER
    s are up and green.
  19. Perform a manual online backup on every Primary 
    SpectroSERVER
     to preserve the current database. Take the following steps:
    1. In the Online Database Backup subview, click Begin Backup Now.
    2. Verify that the online backup has succeeded.
    3. Verify that the re-synchronizing the Primary 
      SpectroSERVER
       database with the Secondary 
      SpectroSERVER
       is succeeded.
    We recommend differentiating the databases by the version number. Set the “Prefix for Backup File Name” parameter from the default of “db_” to something like “db_
    version
    _” or another value that identifies the version.
    To continue management activities during the upgrade process, do not upgrade all the OneClick web servers at one time. Retain few OneClick web servers, so that you can monitor your infrastructure using the secondary 
    SpectroSERVER
    s until all the primary 
    SpectroSERVER
    s are upgraded. When all primary 
    SpectroSERVER
    s are upgraded, you can upgrade the retained OneClick web servers.