No data is being ingested during the upgrade.
No data is being sent to DX App Experience Analytics and DX APM.
All the DX APM agent integrations are stopped or all the agent traffic is blocked. To stop the integrations, log in to the DX APM agent systems, and stop the Tomcat process.
No third-party data is being ingested into DX OI.
All the RESTmon Agents are stopped and all the agent traffic is blocked.

SSH to the same system that was used earlier for the installation.
Create a directory name
<latest_install_dir>
.
From the Support site, download the latest installer distribution file to the directory you created.
Log in to the
Support
site using your credentials.
Click
Enterprise Software
and then click
Product Downloads
.
In the
DOWNLOAD MANAGEMENT
page, search for
Application Performance Management
.
In the
Product Download
section, select the
DX Application Performance Management Multi-Platform
version as
20.2.1.
Click the
DX Application Performance Management Multi-Platform
link to open and view the files.
Download the required file:
DX Common Platform Installer r20.2.1-online
DX Common Platform Installer r20.2.1-offline
Extract the downloaded zip file.

Backup the Elasticsearch Cluster Snapshot
Backup the PostgreSQL Database
Backup the APM OI Metric Publisher Configurations
Backup and Restore the YAML Files
Backup All of the Management Modules
Backup the JavaScript Calculator Scripts

In the web console, navigate to

Application > Deployments > jarvis-esutils

.

Open the

Environment

tab and check for the

EXCLUDE_INDICES

variable.

Remove

ao_.*

.

Check for the

SNAPSHOT_CRON

variable and ensure that you update the expression according to your requirement. For example, to take a snapshot every 23 hours daily, update the expression as

0 0 23 **?

.

Click

Save

.

Ensure that the

jarvis-esutils

pod is restarted and the snapshots are created successfully.

Run the following Elasticsearch query to check the snapshots:

http://ES_ROUTE/_snapshot/Repository/_all

Sample output:
          
        {
  "snapshots" : [
    {
      "snapshot" : "ao-snapshot_2020-09-29_23:00:00",
      "uuid" : "YwhqvlnSTgSJuKzBg1J4Sw",
      "version_id" : 7050199,
      "version" : "7.5.1",
      "indices" : [
        "jarvis_jmetrics_1.0_1",
        "jarvis_kron",
        "jarvis_jmessages_1.0_1",
        "jarvis_healthcheck_2.0_1",
        "jarvis_metadata",
        ".kibana_tadmin-userstore",
        "jarvis_config",
        "audit",
        ".ca_es_acl",
        ".kibana_d1b9a308-b53b-4e89-ab1a-f490d11d1193"
      ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2020-09-29T22:59:59.489Z",
      "start_time_in_millis" : 1601420399489,
      "end_time" : "2020-09-29T23:00:00.490Z",
      "end_time_in_millis" : 1601420400490,
      "duration_in_millis" : 1001,
      "failures" : [ ],
      "shards" : {
        "total" : 10,
        "failed" : 0,
        "successful" : 10
      }
    },
    .....
        ..............

Navigate to the

tools

directory of the 20.2 downloaded zip file.

cd <latest_install_dir>/tools

Create a directory named

db-backup

.

mkdir db-backup

Copy the

db-backup.tar

package from the

tools

directory to the

db-backup

directory.

For example, cp /root/<latest_install_dir>/tools/db-backup.tar /root/<latest_install_dir>/tools/db-backup

Extract the

db-backup.tar

package.

Run the

run.sh

script from the

db-backup

directory.

For Kubernetes:
OS=kubernetes ./run.sh

For OpenShift:
./run.sh

For Kubernetes:
OS=kubernetes ./cleanup.sh

For OpenShift:
./cleanup.sh

kubectl get cm apmservices-oimetricpublisher -o yaml > oim_configmap.yaml -n <namespace>

oc get cm apmservices-oimetricpublisher -o yaml > oim_configmap.yaml

Navigate to the

tools

directory of the 20.2.1 downloaded zip file.

cd <latest_install_dir>/tools

Run the following command to backup the YAML files:

./backup-yamls.sh [-n| <namespace>] [-f <filename>]
Where: 
-n : proceed in the namespace <namespace> ("dxi" is default)
-f : save backup as <filename> instead of yaml-backup-YYYY-MM-DD-HH-MM-SS.tgz in the current directory. <filename> can be /path/to/mybackup.tgz or backup.tgz in the current directory.

We recommend that you save the backup process logs as shown in the following example:

./backup_yamls.sh -n dx -f backup-2020-Jun-28.tgz 2>&1 | tee backup-2020-Jun-28.log

Where

2>&1

redirects STDERR to STDOUT, and then

tee

sends output into the display and into the specified file. To append to the filename instead of overwriting, use

tee -a <filename>

.

You can also run this script without any parameters from the

tools

directory:

./backup-yamls.go

Run the following command to list all the scheduled jobs:

crontab -e

The cron jobs editor is displayed.

Define the date and time to run the job in the editor. For example, to run the job at 4:05 am every day, add the following entry:

5 4 * * * /bin/bash -c "/path/to/tools/backup-yamls.sh -f /path/to/backups/filename-$(date '+\%Y-\%m-\%d-\%H-\%M-\%S').tgz”
          
Note: You must escape % in the command because % is a special sign for the cron daemon.

Navigate to the
tools
directory of the 20.2.1 downloaded zip file.
cd <latest_install_dir>/tools
Run the following command:
./restore-dxi.sh [-d|--backup-dir <DIRECTORY> | -f|--backup-file <TARBALL>]

Where:
backup-dir: Unpack the backed up tgz to view the files and specify this directory. Takes backups from the DIRECTORY (under the installation directory).
backup-file: Takes backups from the TARBALL instead of the 
backup directory.
You can use
-d
or
-f
to provide the location of the backed up data.

Navigate to the host where the NFS server is running.

Create a folder under the

configs

directory for every tenant to be backed up. For example, create a folder named

backup_production_tenant

.

For example, mkdir <base-dir>/configs/backup_production_tenant

Where,

<base-dir>

is the NFS folder that you chose during the installation.

For every tenant, copy the folder named

customize

into the

backup_production_tenant

folder that you created earlier.

For example,
cp -a <base-dir>/em/<tenantid>-random_int/001/customize <base-dir>/configs/backup_production_tenant

Folder to backup a Standalone Tenant:

<base-dir>/em/<tenantid>-<random_int>/001/customize

Folder to backup a Small, Regular, Large and Max Tenant:

<base-dir>/em/<tenantid>-<random_int>/000/customize

Navigate to the JavaScripts Extensions UI (
http://apmservices-gateway.<defaultSubDomain>/<tenantID>/apm/atc/?#/extensions
).
From the JavaScript Calculator list, locate the required JavaScript file.
Click
Download
in the
Action
column for the required file.

Run the following command to get the list of all the APM OIMP deployments:

kubectl get deployment -n <namespace> | grep oimetricpublisher

If the number of deployments is more than one, run the following command to scale down the

additional APM OIMP

deployments to zero:

kubectl get deployment -n <namespace> | grep oimetricpublisher | awk '{print $1}' | xargs kubectl -n <namespace> scale deployment --replicas=0

Ensure that only one APM OIMP pod is running.

Run the following command to delete the scaled down deployments:

kubectl delete deployment <deployment name> -n <namespace>

Run the following command to delete the config map references that point to the deleted deployments:

kubectl get configmaps | grep oimetricpublisher | awk '{print $1}' | xargs kubectl delete configmap -n <namespace>

Login to the Kubernetes web console.
Navigate to the project.
Search for the APM OIMP deployment and check if the number of APM OIMP deployments is more than one.
If yes, scale down those deployments to zero and ensure that only one deployment is scaled up.
Delete the scaled down deployments.
Delete the config map references that point to the deleted deployments.

Login to OpenShift.

oc login -u <username> -p <password>

Open the project:

oc project <dxi project>

Run the following command to get the list of all the APM OIMP deployments:

oc get deployment | grep oimetricpublisher

If the number of deployments is more than one, run the following command to scale down the

additional APM OIMP

deployments to zero:

oc get deployment | grep oimetricpublisher | awk '{print $1}' | xargs oc scale deployment --replicas=0

Run the following command to delete the scaled down deployments:

oc delete deployment <deployment name>

Ensure that only one APM OIMP pod is running.

Run the following command to delete the config map references that point to the deleted deployments:

oc get configmaps | grep oimetricpublisher | awk '{print $1}' | xargs oc delete configmap

Login to the OpenShift console.
Navigate to the project.
Navigate to
Applications, Deployments
page.
Ensure that only one APM OIMP deployment is scaled up. If there are additional deployments, scale down those deployments to zero.
Delete the APM OIMP deployments that you scaled down.
Navigate to the
Resources, Config Maps
page.
Delete the config map references that point to the deleted deployments.

Access the Jarvis API UI (
http://apis.<defaultSubDomain>
). For more information, see the
DX Platform Routes and Ingresses
section.
Ensure to run on the browser that brings up the Swagger APIs.
Go to the
Mapping
section and select
PATCH
.
Patch the following mappings for the
axa_crashes
index:
{
"product_id": "ao",
"doc_type_id": "axa_crashes",
"doc_type_version": "1",
"mappings": {
"data": {
"dynamic": "strict",
"_all": {
"enabled": false
},
"properties": {
"txn_start": {
"type": "text",
"analyzer": "keyword_lowercase"
},
"service_name": {
"type": "text",
"analyzer": "keyword_lowercase"
}
}
}
}
}
Verify if the mappings (
txn_start
and
service_name
) are successfully updated using the
GET Mapping API
. To verify, provide the following information in the API:
product_id:
ao
doc_type_id:
axa_crashes
doc_type_version:
1
Similarly, patch the mappings for the
axa_sessions
index.
{
"product_id": "ao",
"doc_type_id": "axa_sessions",
"doc_type_version": "1",
"mappings": {
"data": {
"dynamic": "strict",
"_all": {
"enabled": false
},
"properties": {
"network_events": {
"type": "nested",
"properties": {
"urln": {
"type": "keyword"
},
"parent_urln": {
"type": "keyword"
}
}
}
}
}
}
}
Verify if the mappings (
urln
and
parent_urln
) are successfully updated using the
GET Mapping API
. To verify, provide the following information in the API:
product_id
: ao
doc_type_id
: axa_sessions
doc_type_version
: 1

Run the following command to get the template:
curl -XGET $ES_HOST:$ES_REST_PORT/_template/ao_axa_session_events_1?pretty >ao_axa_session_events_1_template.json
Remove the following lines in the
ao_axa_session_events_1
json file:
"ao_axa_session_events_1" : 
          { 
          
          
          and the corresponding }
Update the
session_events
mapping with the highlighted properties as shown:
"session" : {
"properties" : {
"duration" : {
"type" : "long"
},
"custom" : {
"type" : "object",
"dynamic" : true,

"properties": {
"session_field1": {
"type": "keyword"
},
"session_field2": {
"type": "keyword"
},
"session_field3": {
"type": "keyword"
},

"session_field4": {
"type": "keyword"
},

"session_field5": {
"type": "keyword"
},

"session_field6": {
"type": "keyword"
}
}
},
"start" : {
"format" : "epoch_millis",
"type" : "date"
},
"end" : {
"format" : "epoch_millis",
"type" : "date"
},
"id" : {
"type" : "keyword"
}
}
}
 
Run the following command to update the template:
curl -XPUT $ES_HOST:$ES_REST_PORT/_template/ao_axa_session_events_1 -H 'Content-Type: application/json' -d @ao_axa_session_events_1_template.json
Verify if the template was updated using the following URL:
http(s)://$ES_HOST:$ES_REST_PORT/_template/ao_axa_session_events_1?pretty
The template should display the
session_field1
to
session_field6
properties added to the mapping.

Open the
jarvis-esutils
pod.
In the
Environment
tab, set the
INDEX_LIMIT
environment variable as
ao_axa_session_events_1=1mb
.
Where,
ao
is the product ID.
axa_session_events
is the doc_type_id.
1
is the doc_version.
Restart the
jarvis-esutils
pod.
After the rollover is completed, delete the value for the
INDEX_LIMIT
variable.