Install and Configure Symbolicator

This section describes the steps to install and configure symbolicator.
The symbolication server is needed only if the iOS crash symbolication is required in the UI. The iOS crash logs are built out of symbols. Desymbolication changes the symbols into method names and function names.   
 
Download the AXA Symbolicator Installer
The DX AXA Symbolicator Installer is bundled with the DX APM distribution zip file.
Follow these steps:
  1. Create a directory on MAC as the base location for the Symbolicator files. This directory is referenced as 
    $CA_EMM_HOME
    .
  2. Log in to the Support site using your credentials.  
  3. Navigate to the 
    DOWNLOAD MANAGEMENT
     section. 
  4. Search for Application Performance Management.
  5. In the 
    Product Downloads
     section, select the version and click the link to display all the bundled files.
  6. Download the 
    DX AXA Symbolicator Installer
     file.
Prerequisites
Before you install the Symbolicator, perform the following steps:
  • Update the /etc/hosts File
     
    Update the 
    /etc/hosts
     file with the MAC OSX server FQDN to enable Crash Handler to access the MAC OSX. The Symbolicator hostname is the FQDN of the MAC OSX. Update the FQDN in the following format.
     To edit the value, you must be a root user.
    <IP Address> <Host Name with domain> <Alias> Example:10.131.58.124 user01-I193384.ca.com user01-I193384
  • Crash Handler
     
    • Ensure that the following properties are set in the 
      $CA_EMM_HOME/conf/axa-crash-handler_instance.properties
       file:
      • symbolicator.available=true
         
      • symbolicator.username=
         
      • symbolicator.password=
         
      • symbolicator.url =http://<symbolicator_hostname>:<symbolicator_port> 
         
        The Symbolicator hostname is the FQDN of the MAC OSX hostname and must be accessible from your system. The default Symbolicator port is 8070 and is available in the 
        $CA_EMM_HOME/conf/axa-crash-symbolicator_instance.properties
         file.
  • Frontend (EMM Proxy) Changes
    • Ensure that the entry for the Crash Handler proxy forward is correct on the server where the Frontend is installed.
      cd $CA_EMM_HOME/apache-tomee-plus-1.7.1/webapps/emm-proxy/WEB-INFvi web.xml
      Ensure that the entry for 
      <servlet-name>crashhandler</servlet-name>
       is 
      http://<crashhandler_hostname>
      :<crashhandler_port> without any additional paths. In case of a load balancer availability for Crash Handler, ensure that this entry points to the load balancer URL.
  • Crash Symbolicator
    • Set the maximum open files limit to 400000 (ulimit -n 400000). 
      $ sudo launchctl limit maxfiles 400000 unlimited
    • Ensure that the following Hardware and Software Requirements are met:
      • MAC with a minimum of 16GB RAM, 500GB SSD/Flash, 8 Core CPU
      • Latest version of Xcode is installed.
      • Xcode Command Line Tools are installed. If not, trigger the installation:
        $xcode-select --install
      • The latest MAC OS symbol files and caches for all the versions that your application supports are present for complete symbolication of crashes. 
        Ensure to extract the developer symbol images to the following directory of the user:
        cd ~/Library/Developer/Xcode/iOS DeviceSupport ( ~ = HOME of the user)
        For a specific version of iOS, download the corresponding symbol files as follows:
        cd ~/Library/Developer/Xcode/iOS DeviceSupport/11.2.2 (15C202)
        You can get the example images from the 
    • Turn on the spotlight preferences for the 
      ~/Library/Developer 
      directory.
    • Perform the following tasks on the MAC where Symbolicator is being installed:
      • Turn off Auto Updates.
      • Turn off the Sleep Settings so that the hard disk does not sleep.
Install Symbolicator Manually
To install the Symbolicator manually, complete the following tasks.  
Perform the following steps if Homebrew and Java are not installed on the MAC OS. 
Follow these steps:
 
  1. Run the following command to install.
    ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  2. Run the following command:
    brew install caskroom/cask/brew-cask
  3. Run the following command:
    brew tap caskroom/versions
  4. Run the following command to display the available Java version:
    brew cask search java
  5. Run the following command to install Java JDK 1.8.
    brew cask install java8
  6. Run the following command to display the Java version:
    java -version
    The java version should be 1.8 or above.
Set the JAVA_HOME and CA_EMM_HOME Variables
 
Follow these steps:
 
  1.  
    Navigate to the installation directory.
    cd <CA_EMM_HOME>
  2. Open the bash_profile and set the JAVA_HOME and CA_EMM_HOME variables.
    export CA_EMM_HOME=/opt/ca/axa export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer/
Install Kafka
Perform the following steps to install Kafka. 
Follow these steps:
 
  1. Navigate to the installation directory.
    cd <CA_EMM_HOME>
  2. Download Kafka.
    curl -OL https://archive.apache.org/dist/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
  3. Extract the downloaded file.
    tar -zxvf kafka_2.11-0.10.1.0.tgz
  4. Navigate to the conf directory.
    cd kafka_2.11-0.10.1.0/conf
  5. Open the server.properties file and add the following settings to start Kafka.
    listeners=PLAINTEXT://localhost:9092 zookeeper.connect=localhost:2181
  6. Navigate to the bin directory.
    cd kafka_2.11-0.10.1.0/bin
  7. Run the following commands to start the Kafka server.
    nohup ./zookeeper-server-start.sh ../config/zookeeper.properties & nohup ./kafka-server-start.sh ../config/server.properties &
  8. Create the following topics:
    ./bin/kafka-topics.sh --create --zookeeper <ZK_HOST>:<ZK_PORT> --replication-factor 1 --partitions 1 --config retention.ms=3600000 --topic mdoSymbolicatorCrashQueueName Where ZK_HOST: localhostZK_PORT: 2181
    ./bin/kafka-topics.sh --create --zookeeper <ZK_HOST>:<ZK_PORT> --replication-factor 1 --partitions 1 --config retention.ms=3600000 --topic mdoSymbolicatorProcessedQueueCrashName Where ZK_HOST: localhost ZK_PORT: 2181
     Run the following command to check if the topics are created: 
     
    ./kafka-topics.sh --list --zookeeper <ZK_HOST>:<ZK_PORT>
     
     
Create the Directory Structure
Perform the following steps to create the directory structure  in $CA_EMM_HOME. 
Follow these steps:
 
  1. Create a directory named symbol-files.
    mkdir <CA_EMM_HOME>/symbol-files
  2. Set the execute permissions to this folder.
    chmod 755 symbol-files
  3. Create a directory named logs.
    mkdir <CA_EMM_HOME>/logs
  4. Set the permissions to this folder.
    chmod 755 logs
  5. Create a directory named data.
    mkdir <CA_EMM_HOME>/data
  6. Set the permissions to this folder.
    chmod 755 data
  7. Create a directory named bin.
    mkdir <CA_EMM_HOME>/bin
  8. Set the permissions to this folder.
    chmod 755 bin
  9. Create a directory named conf.
    mkdir <CA_EMM_HOME>/conf
  10. Set the permissions to this folder.
    chmod 755 conf
  11. Download the 
    AXA_Sybolicator_MacOS_installer.zip
     file from the Support site to a directory. For example, a directory named 
    Symbolicator
  12. Extract this zip file.
  13. Copy the 
    axa-crash-symbolicator
     folder from the 
    <Symbolicator>
     directory to the 
    $CA_EMM_HOME
     folder on the MAC. 
  14. Copy the following files from the 
    <Symbolicator>/server/conf/*
     folder to the 
    $CA_EMM_HOME/conf
     folder on the MAC.
    • axa-crash-symbolicator_instance.properties 
    • log4j-axa-crash-symbolicator.xml
  15. Move the following files from 
    $CA_EMM_HOME/axa-crash-symbolicator/bin
     to 
    $CA_EMM_HOME/bin
    .
    • axasymbolicator.sh   symbolicator   jq
  16. Update the following properties in the 
    $CA_EMM_HOME/
     
    conf/axa-crash-symbolicator_instance.properties
     file. Update the 
    axa.crash.handler.username
     and 
    axa.crash.handler.password
     with the Global Admin credentials.
    server.port=8070 axa.crash.handler.username= Username of the Global Administrator provided during the installation. axa.crash.handler.password= Password for the Global Administrator provided during the installation. axa.crash.handler.ess.url=<ESSSERVER:PORT>/ess/security/v1/token (Admin UI Host and Port) axa.crash.handler.base.url=<ESSSERVER_SERVER:PORT>/crashhandler (Admin UI Host and Port) axa.symbolicator.base.dsym.dir=$CA_EMM_HOME/symbol-files/ (/ at the end is important) Notes: Always provide the Read Server hostname and port for these properties. If you have multiple Crash Handlers, then the Crash Handler should be load balanced for ESS PROXY FORWARD of Crash Handler API to work.
Start the Symbolicator Server
 
Follow these steps:
 
  1. Navigate to the bin folder.
    cd $CA_EMM_HOME/axa-crash-symbolicator/bin
  2. Run the following script:
    ./axa-crash-symbolicator.sh start
Verify the Connectivity Between Crash Handler and Symbolicator
Use the following commands to verify that the Crash Handler and Symbolicator can connect to each other at the designated ports:
  • From the Crash Handler: 
    telnet <symbolicator_hostname> <symbolicator_port>
     
     
  • From Symbolicator:
    telnet <crash_handler_hostname> <crash_handler_port>
     
     
    If telnet does not work, restart the Crash Handler (./startServices.sh -CH).
    Curl from Crash Handler to Symbolicator.
Verify the Symbolicator Installation
Perform the following steps to verify if the Symbolicator was installed successfully. 
Prerequisite:
 
  • dsym file is available. Ensure that the dsym file is zipped.
 
Follow these steps:
 
  1. Register an app with CA App Experience Analytics.
  2. Start some sessions to ensure that the analytics data is being recorded by the iOS device.
  3. Upload the dsym file to the CA App Experience Analytics Admin UI.
    1. Go to Manage Apps > Select your app. 
    2. Select the iOS version and upload the dsym file.
    3. Crash the app on the mobile device and push the metrics to the CA App Experience Analytics Admin Console.
      If the Symbolicator is installed successfully, the desymbolicated crashes are displayed in the App Crashes section.