Troubleshooting Greenbook

Contents
camm27
Contents
Basic Troubleshooting
  • Connect to the DX NetOps MM Web Manager (see Using the DX NetOps MM Web Manager) to gather information about the DX NetOps MM Cluster and collect the following info:
    • Are all the components running?
    • Is it a Single Server install? If not, how many LC’s are there?
    • Which Device Pack is the customer having problems with?
  • Review the logs of the device pack Engine to see if the issue is in the Engine and check if restarting Engine solves the problem.
  • Review the logs of the device pack Presenter to see if the issue is in the Presenter and check if restarting Presenter solves the problem.
  • If the “DX NetOps MM for DX NetOps MM” Device Pack is installed on the server, DX NetOps MM Engine polls itself to indicating the current Run state of each component giving you an overall performance of the entire DX NetOps MM environment.
  • By default CAMM is installed under $CAMM_HOME\COMPONENTS\ENGINE_CAMM and $CAMM_HOME\COMPONENTS\PRESENTER_CAMM directory.
How to Stop and Start DX NetOps MM Components and Sub Components
Visit the following pages to learn about the scripts to start, stop, and check the status of the components:
Using the DX NetOps MM Web Manager
CAMM Web Manager reflects the high level operational status and health of DX NetOps MM components such as MultiController, LocalController and Sub Components such as Device Pack’s Engine and Presenter. However, it does not indicate if Sub Components are actually collecting data. The utilization of resources on the server by each Component is also indicated. If one or more of the components are not running, they can be restarted from the web interface.
Log into DX NetOps MM Web Manager from the URL http://server:port (8880 is default port). The default username is ‘admin’ and the password is ‘camm’.
Using the Log Files
The CAMM directory structure is created in a logical manner. Every Component (MultiController, LocalController, Generic Executor, Delivery Service) and Sub Component (Engine & Presenter) has a log directory.
The logs are automatically archived as per settings specified in the LocalConfig-ge.xml: $CAMM_HOME/GE_<user>/LocalConfig-ge.xml
Logs for DX NetOps MM main Components are located in the following directories:
  • MultiController
    :
    $CAMM_HOME/MC/logs/CAMM-Multi-Controller-<timestamp>.log
  • LocalController
    :
    $CAMM_HOME/LC/logs/CAMM-Local-Controller-<timestamp>.log
  • GenericController
    :
    $CAMM_HOME/GE_<username>/logs/*.log
  • Device Pack:
    $CAMM_HOME/COMPONENTS/<device-pack-name>/logs/*.*
  • Web:
    $CAMM_HOME/WEBCAMM/logs/*.log
The “STD-ERROR-<timestamp>.log” file located in the same directories contains the error messages. This is often the most important file to look at when troubleshooting an issue.
Logs for DX NetOps MM Sub Components (Engine & Presenter):
The Sub Components of each device pack installed on the server has its own sub directory under the COMPONENTS directory. The Device Pack names are included in the name of the directory. Each Sub Component has a logs directory under the specific device pack ENGINE or PRESENTER sub directories. Log files for each component would indicate if there is a problem with the Device Pack. For example, if it failed to collect/parse data from the EMS.
  • Device pack Engine:
    $CAMM_HOME/COMPONENTS/ENGINE_<DEVICE_PACK>/logs directory
    Example:
    /opt/CA/CAMM/COMPONENTS/ENGINE_CAMM/logs/CAMM-Application-<YYYY-MM-DD>.log
  • Device Pack Presenter:
    $CAMM_HOME/COMPONENTS/PRESENTER_<DEVICE_PACK>/logs directory.
    Example:
    /opt/CA/CAMM/COMPONENTS/PRESENTER_CAMM/logs/CAMM-Application-<YYYY-MM-DD>.log
The “STD-ERROR-<timestamp>.log” file located in the same directories contains the error messages. This is often the most important file to look at when troubleshooting an issue.
Increasing log verbosity
All the above mentioned Components and Sub Components log files by default contain informational level messages (INFO). The granularity of logging can be increased temporarily for troubleshooting purposes by modifying the corresponding “logging.properties” file located in the $CAMM_HOME/<sub-component>/logs directory of each Component and Sub-Component.
Follow these steps:
  1. Edit the “logging.properties” file in the same logs directory; replace the two instances of INFO with FINEST and save the file.
    For example, change it from the default:
    com.torokina.common.logging.apache.FileHandler.directory=E\:\\CA\\CAMM/DS/logs
    handlers=com.torokina.common.logging.apache.FileHandler com.torokina.common.logging.apache.FileHandler.level=INFO
    .level=INFO
    com.torokina.common.logging.apache.FileHandler.prefix=CAMM-Delivery-System-
    To this:
    com.torokina.common.logging.apache.FileHandler.directory=E\:\\CA\\CAMM/DS/logs
    handlers=com.torokina.common.logging.apache.FileHandler
    com.torokina.common.logging.apache.FileHandler.level=FINEST
    .level=FINEST
    com.torokina.common.logging.apache.FileHandler.prefix=CAMM-Delivery-System-
  2. Enable the new logging level by either restarting the component or sub-component (stopall/startall or cammCtrl) or by running the following command from within the relevant log directory to enable without restart: $CAMM_HOME/tools/cammCtrl a 127.0.0.1 p <port> -l ./logging.properties
    Increasing the log level will result in significantly more detail being logged and consequently more disk space usage. Make sure to monitor the disk space and revert back to default INFO settings as soon as reproduction of the problem has been completed.