Upgrade an Enterprise Manager Cluster in APM Command Center

You run the automated cluster upgrade utility through a standalone CA APM Command Center. APM Command Center manages the upgrade of Enterprise Manager clusters from any version so that you can obtain the latest capabilities of CA APM. When you use Command Center to upgrade CA APM, you significantly reduce performance management down time and you retain your customizations.
apmdevops106
You run the automated cluster upgrade utility through a standalone CA APM Command Center. APM Command Center manages the upgrade of Enterprise Manager clusters from any version so that you can obtain the latest capabilities of CA APM. When you use Command Center to upgrade CA APM, you significantly reduce performance management down time and you retain your customizations.
You can only use this capability to upgrade from earlier versions of APM to APM 10.7 SP3. You cannot use this capability to upgrade from APM 10.7 to 10.7 SP3.
Discovery (Manual or Automatic) Cluster upgrade runs in seven stages:
  • Backup Creation
  • Pre-upgrade Check
  • Upgrade
  • Verification
  • Rollback
  • Cleanup
You can discover more than one cluster simultaneously, but you can only upgrade one cluster at a time. If you upgrade multiple clusters in Enterprise Team Center (ETC), ETC requires that the Master cluster and Provider clusters are all on the latest CA APM version to function.
  • You cannot run the Cluster Upgrade through the Command Center component in the Installer. Use a Command Center bundle with the Cluster Upgrade functionality.
  • Cluster Upgrade upgrades APM components only, such as Enterprise Managers, Collectors, and the PostgreSQL Database. Cluster Upgrade does not upgrade Command Center. For more information about upgrading Command Center, see Upgrade CA APM Command Center.
4
Prerequisites
Before you begin the cluster upgrade, complete the following steps:
  • PostgreSQL Databases not installed by APM Installer are unsupported for Cluster Upgrade.
  • Cluster Upgrade is unsupported for MOMs that run in the high-availability mode. For example, when two MOMs are present for the same cluster.
  • The use of HTTPS secure connections between and Command Center Controller and Command Center Server prevents Automatic Discovery.
  • Print and complete the Cluster Checklist to ensure that you have all necessary information. For more information, see Checklist: Enterprise Manager Cluster Upgrade.
  • Configure the firewall so that the Controllers that are deployed on the cluster component servers communicate with the Command Center Server that is used for the Cluster Upgrade. For more information, see Configure CA APM Command Center
  • (Automatic Discovery only) Write down the WebView Hostname or Manager of Managers (MOM) Hostname.
  • (Automatic and Manual Discovery) Write down the APM Database credentials for PostgreSQL only.
    You do not need database credentials for Oracle for either Automatic or Manual Discovery.
  • (Automatic Discovery only) Write down the SSH credentials or the SSH private keys and passphrase of all servers running CA APM components. These components include the WebView, MOM, and Collectors.
    For Manual Discovery, you do not need the SSH credentials or SSH private keys and passphrase.
  • (Automatic and Manual Discovery) Have permissions to the Enterprise Manager installation and processes, preferably as the root user. Permissions are granted on the servers.
Run Discovery
Before you start the cluster upgrade, Command Center runs a check to discover all the components in your environment. You cannot start the cluster upgrade in Command Center until Command Center identifies all cluster components. The components are WebView, the MOM, the APM Database, and the Enterprise Manager Collectors. If you do not have WebView, you can run the Automatic Discovery by starting with the MOM instead. You must use 
one
 of three discovery options to complete this step:
  • Automatic Discovery for clusters with components on the Linux platform
    Use this option to discover all Linux cluster components when you have identical SSH credentials for all components.
  • Manual Discovery for clusters with components on both Linux and Windows platforms
    Use this option to discover all cluster components when the components are on both the Linux and Windows platforms. This option involves both Manual and Automatic Discovery steps.
  • Manual Discovery for clusters with components on a single platform
    Use this option when your environment meets one of the following conditions: 
    • (Windows only) All cluster components are running on Windows
    • (Windows or Linux) Your SSH credentials differ for various components
    • (Windows or Linux) You do not use SSH for the security protocol
    • If you set up SSL communication between the Configuration Servers and the Controllers
Enter all WebView or MOM Hostnames into Command Center. Command Center produces a list of these clusters showing versions and locations. Instruct Command Center to upgrade one or more clusters simultaneously.
If you upgrade an environment with multiple clusters, you must discover all clusters before initiating the upgrade either manually or automatically. Command Center does not display whether a cluster is the ETC Master cluster or Provider cluster. You must know the cluster environment setup.
Command Center lets you refresh the discovery data for all clusters and their components. After you refresh the data, the previously discovered data is erased. Use this option before every new upgrade so that Command Center obtains a current understanding of the environment. This capability is useful when you upgrade one cluster after another or when you upgrade your cluster over a longer period.
Run Automated Discovery for Clusters on Linux 
Command Center runs the automated Discovery to verify all cluster components. 
The use of HTTPS and SSL between Command Center Server and Command Center Controller prevents Automatic Discovery. If you set up SSL communication between the Configuration Server and the Controllers you must run Manual Discovery. To configure the Configuration Server and the Agent Controller, see Enable ActiveMQ Secure Communication.
Follow these steps:
  1. Open Command Center and select 
    Servers
    .
    The Servers tab opens. Information about the clusters that are connected to the Command Center instance appears.
  2. (Optional) Select the drop-down arrow next to the cluster name.
    A list of every discovered cluster and every cluster component appears.
    • Click each cluster component in the drop-down list.
      Detailed information appears for that component, such as the Version, Port, and Hostname. 
  3. Select the drop-down arrow on the 
    Discover
     button, and select 
    Automatically Discover APM Server Components
    The 
    Discover APM Infrastructure Component
     dialog opens.
  4. Enter the 
    WebView Hostname
    Discover more than one cluster simultaneously. Add other WebView Hostnames or MOM Hostnames
    separated by a comma.
  5. Select one 
    Authentication Method
    :
    • SSH with username and password
      • Enter the 
        Username
         and 
        Password
         for the server where the WebView or MOM is located.
    • SSH with private key and passphrase
      Select a 
      Private Key
       from the drop-down list.
      Cluster Upgrade uses the SSH private keys under which Command Center runs. The SSH private keys are managed by SSH tools, such as
      ssh-keygen
      or
      ssh-copy-id
      ,
      and not by Command Center.
  6. (Optional) Select 
    Show Advanced Options
    .
    The SSH port appears. 
    • If necessary, change the SSH port.
      Default Port:
       22
  7. Click 
    Discover
     to run Discovery.
    Command Center detects all components and determines that an upgrade is possible.
  8. (Optional) To obtain a copy of the Discovery log file, select 
    Download
    .
  9. If Discovery is successful, the 
    Upgrade
     options appear when you select the cluster.
    If Discovery is unsuccessful, see Automatic Discovery Failed.
You ran the automated Discovery process. For next steps, go to Create a Backup and Perform a Pre-upgrade Check.
Run Manual Discovery
To run Manual Discovery, install the Controller on each server where an APM component is installed. You must know where every component of your cluster is located to complete Manual Discovery successfully.
  • The Controller package on the 
    Home
     screen of Command Center cannot be used for the upgrade. Use the Controller package on the
    Servers
    tab.
  • If you upgrade an environment with a ETC Master and Provider clusters, you must discover all clusters before starting the upgrade. Command Center discovers the clusters simultaneously. Command Center does not display whether a cluster is the ETC Master or Provider cluster. You must know the cluster environment setup.
Run Manual Discovery on a Cluster with all Components on either Linux or Windows
Run Manual Discovery when all components are either on Linux or Windows. Apply these steps when you do not use SSH protocol or your SSH credentials differ among your components.
Follow these steps:
  1. Open Command Center and select 
    Servers
    .
    The Servers tab opens. 
  2. Select the drop-down arrow on the 
    Discover
     button and select 
    Download the Controller to Add Components Manually
    .
    A window opens.
  3. Select your environment platform. 
  4. To download the Controller to the target server for every component, select the 
    Package Download URL 
    and follow the install instructions.
    If your environment is on one server, download the package only to that server. If your components are on different servers, install the Controller on each server that contains an APM component or Discovery fails. Do not install the Controller on the Oracle Database. Install the Controller on the PostgreSQL Database.
  5. To start the Controller, open Command Line and use the 
    apmccctrl.sh start
     command.
    The Controller connects to Command Center automatically.
  6. Go to the 
    Servers
     tab of Command Center, and select
    Automatically Discover APM Server Components
    . For complete steps about Automatic Discovery, see Run Automated Discovery for Clusters on Linux.
  7. View the Discovery progress in the 
    Servers
     tab of Command Center. 
    If Discovery is successful, the 
    Upgrade
     button appears.
    If undiscovered components remain, the component appears with a caution icon. 
  8. Download the controller package and install it on the correct server for that component.
  9. Rerun Automated Discovery to ensure that all components are detected.
  10. (Optional) If Discovery is unsuccessful, select 
    Download
     to obtain the cluster logs and determine the reason. For more information, see Manual Discovery Failed.
Run Manual Discovery on a Cluster with both Linux and Windows Components
Run a Manual Discovery for components on Windows servers because Windows does not support SSH protocol. You can then proceed with Automatic Discovery for components on Linux servers.
If you upgrade an environment with ETC Master and Provider clusters, you must discover all clusters first before starting the upgrade. Command Center discovers the clusters simultaneously when you use either automatic or manual discovery. Command Center does not display if a cluster is the ETC Master cluster or Provider cluster. You must know the cluster environment setup.
Follow these steps:
  1. Download one of the Cluster Upgrade Repositories for the missing platform:
    • (Windows) Download the Windows repository.
    • (Linux) Download the Linux repository.
  2. Save the file to the 
    ACC_HOME\import
    directory.
  3. Restart Command Center.
  4. Log into the Command Center UI.
  5. Go to the
    Servers
    tab. 
  6. Select
    Discover
    , and select
    Download the Controller to Add Components Manually
    .
    The
    Add APM Infrastructure Component
     dialog opens.
  7. Select the
    Package Download URL
    and follow the installation instructions.
  8. (Windows only) For each server that has an APM component:
    Unzip the Controller package onto every server that has an APM component except the Oracle Database. If you miss any other component, Discovery fails. If you use the PostgreSQL Database, install the Controller on that server.
    1. Unzip the Windows Controller package that you previously downloaded from the
      Servers
      tab to the folder where you want to install it.
    2. Open Command Line and start the Controller using the
      apmccctrl.cmd start
       command.
  9. (Optional) (Linux only) If you want to deploy the Command Center Controller manually on your Linux components, complete the following steps. If you want to run Discovery automatically for Linux components, go to Step 10.
    • Run Manual Discovery on APM components on Linux if your SSH credentials differ for various components. Run Manual Discovery if you not use SSH for the security protocol.
    • Unzip the Controller package onto every server that has an APM component. If you miss a component, Discovery fails.
    1. In the Command Center UI, download Linux ACC Controller package from the 
      Servers
       tab.
    2. Unzip Linux Controller package that you downloaded to the folder where you want to install it.
    3. Open Command Line, and start the Controller using the 
      apmccctrl.cmd start
       command.
  10. (Linux only) Return to the Command Center UI, and start Automatic Discovery. For more information about running Automatic Discovery, see Run Automated Discovery.
You completed Discovery. To proceed with the upgrade, see Create a Backup and Perform a Pre-upgrade Check.