How to Identify Performance Problems in CA SDM

As a system administrator, you gather information about your installation environment, resource usage, configuration details, and other available system resources. For example, understand the computer configuration to determine the Operating System type, version, and available system resources. This data helps you to identify and diagnose a performance or memory-related issue in CA SDM. The CA Diagnostic Report tool and the Interval Logging utility help you collect diagnostic information about your CA SDM environment.
casm171
HID_How_to_Identify_Performance_Problems_in_CA _SDM
As a system administrator, you gather information about your installation environment, resource usage, configuration details, and other available system resources. For example, understand the computer configuration to determine the Operating System type, version, and available system resources. This data helps you to identify and diagnose a performance or memory-related issue in CA SDM. The CA Diagnostic Report tool and the Interval Logging utility help you collect diagnostic information about your CA SDM environment.
The following diagram describes how to gather diagnostic information and identify performance problems in CA SDM:
Identifying Performance Problems in CA SDM Servers
Identifying Performance Problems in CA SDM Servers
Follow these steps:
Define the Performance Problem
To define performance problems, begin by collecting the system information. Then consider the following example questions for relevance:
  • What did users experience to disrupt their tasks?
  • When did users first discover the problem?
  • Has your environment changed recently, such as hardware upgrades?
  • What functionality of the product experiences issues?
  • How many users does the issue impact, and what type of users?
  • What types of users are not impacted?
  • What is the geographic location of the users that see the problem?
  • What is the Access Level of the impacted users?
  • Do you host CA SDM on a VMware ESX server or other virtualized environment?
  • What other software are you running on the ESX server or host machine?
  • What are the specifications of this environment?
  • How many CPUs do you have on each computer?
  • How much memory did you configure for each computer?
Verify the Prerequisites
  1. For Windows operating system, install pslist.exe tools and add its directory path variable on each CA SDM server.
  2. Install and configure the CA SDM servers before running the diagnostic tool.
    We recommend that you contact CA Support Online before you use the diagnostic tool.
Install PS Tools for Windows
For a Windows installation, install the pslist.exe tool and add its directory path variable to the system path variable.
Follow these steps:
  1. Download PS Tools Suite from Microsoft.
  2. Extract pstools.zip to any directory of your choice and add the directory to the Windows %PATH% environment variable.
  3. Execute pslist.exe once manually from the command prompt as the Local System user and accept the license agreement. To execute pslist.exe as the Local System user, run the following command:
    psexec.exe -s -i pslist.exe
    You can start the CA SDM Windows service under a different user account other than the Local System account. For that service, execute pslist.exe under that account and accept the license agreement. To add pslist.exe after installing and configuring CA SDM, accept the PsList license agreement and restart the CA SDM services.
Execute the SDM Diagnostic Report Tool
The CA Diagnostic tool creates a .CAZ file on the Windows OS and .tar.gz on UNIX in the $NX_ROOT\diag\rpt directory. You upload the file that the CA Diagnostic tool creates with your issue at http://support.ca.com.
Follow these steps:
  1. Execute supp_diag.cmd on Windows, or execute supp_diag.sh on UNIX.
    The diagnostic tool can take five to 10 minutes to complete.
  2. If the data collection process does not complete, navigate to the $NX_ROOT\diag\<host_name>_supp_diag.log log file. To determine the errors that occurred, examine the log file.
    If you want to cancel the background batch job, use CTRL-C to cancel the batch file. Some processes still run in the background, such as MSINFO32.exe. If you have any questions about using this diagnostic tool, contact the CA Support Online.
  3. The script directory structure displays the location of the script files, diagnostic zip files, and log files:
    • The $NX_ROOT\diag\bin directory contains script files.
    • The $NX_ROOT\diag\rpt directory contains the diagnostics zip file (in .caz format on Windows systems and in .tar.gz format on UNIX systems).
    • The $NX_ROOT\diag\misc_logs directory contains the log files that can be automatically included in the zip file.
  4. To unzip the gathered files, complete the steps that are appropriate for the operating system:
    • Windows
    • Open a command prompt.
    • Cd to $NX_ROOT\diag\rpt or any directory where the .CAZ file is located.
    • Execute the following command:
      $NX_ROOT\diag\bin\cazipxp -u <package_name>.CAZ
    • UNIX
    • Open a command prompt.
    • Cd to $NX_ROOT/diag/rpt or any directory where the .tar or .tar.gz file is located.
    • Uncompress and untar the file:
      gunzip -d <package_name>.tar.gz tar -xvf <package_name>.tar
Collect Information from CA Diagnostic Report Tool
The CA SDM installation media includes the Diagnostic Report Tool to help collecting information for diagnosing performance problems. You can use the diagnostic tool to collect information relevant to the operating system. You can use the diagnostic tool to determine the commands for collecting data from CA SDM.
Configure the CA SDM server before running the diagnostic tool.
Follow these steps:
  • Windows
  • Set $NX_ROOT as your CA SDM installation root directory.
    The default for $NX_ROOT is the C:\Program Files\CA\Service Desk\ on Windows. You can change the default when you want to change the default directory during the installation process.
  • Verify that the system path variable includes $NX_ROOT\bin.
  • UNIX/Linux
  • Set $NX_ROOT as your CA SDM installation root directory.
    The default for $NX_ROOT is -- /opt/CAisd/ on a Unix or Linux operating system. You can change the default when you want to change the default directory during the installation.
  • Verify that your $PATH includes $NX_ROOT/bin.
Verify Windows Report
The following list describes the Windows report files that are created and included in the CAZ\tar file package.
  • Ca.olf
    The Ca.olf report file specifies the CA licensing information from ca_lic directory.
  • Lic98.log
    The Lic98.log report file specifies the log file that is related to CA licensing from ca_lic directory.
  • Lic98version.log
    The Lic98version.log report file specifies the log file that is related to CA licensing from ca_lic directory.
  • Licdebug.log 
    The Licdebug.log report file specifies the file that is related to CA licensing from ca_lic directory.
  • Drwatsoninfo.txt
    The Drwatsoninfo.text report file specifies the Dr. Watson configuration of the computer.
  • <host name>_env.txt
    The <host name>_env.txt report file specifies the environment variables that are set on the computer.
  • <host name>_slstat.txt
    The <host name>_sistat.txt report file specifies the output of slstat command.
  • <host name>_pdm_status.txt
    The <host name>_pdm_status.txt file specifies the output of slstat command.
  • <host name>_dir_listing.txt
    The <host name>_dir_listing.txt report file specifies the Service Desk install directory listing.
  • <host name>_pslist.txt
    The <host name>_splist.txt report file specifies the process listing when the PsList Microsoft tool is installed.
  • <host name>_MSINFO32.NFO
    The <host name>_MSINFO32.NFO
     
    report file specifies the MSINFO output gathering system information.
  • <host name>_SYSTEMINFO.TXT
    The <host name>_SYSTEMINFO.TXT
     
    report file specifies the system information.
  • <host name>_appevents.csv
    The <host name>_appevents.csv report file specifies the application event logs created in the past seven days.
  • <host name>_sysevents.csv
    The <host name>_sysevents.csv report file specifies the application event logs created in the past seven days.
  • <host name>_hostinfo.txt
    The <host name>_hostinfo.txt report file specifies the computer information.
  • <host name>_prodinstallinfo.txt
    The <host name>_prodinstallinfo.txt report file specifies installation information for CA products.
  • <host name>_caprod_registry.txt
    The <host name>_caprod_registry.txt report file specifies the Registry information of installed CA products.
  • <host name>_softfeatures.txt
    The <host name>_softfeatures.txt report file specifies the list of software features that are installed for Service Desk.
  • <host name>_ipconfig.txt
    The <host name>_ipconfig.txt report file specifies the IP configuration information.
  • <host name>_supp_diag.log
    The <host name>_supp_diag.log
     
    report file specifies the log created for running the supp_diag tool.
Verify UNIX Report
The following list describes the UNIX report files that are created and included in the CAZ\tar file.
  • Ca.olf
    The Ca.olf
     
    report file specifies the CA licensing information from ca_lic directory.
  • Lic98.log
    The Lic98.log
     
    report file specifies the log file that is related to the CA licensing from ca_lic directory.
  • <host name>_env.txt
    The <host name>_env.txt
     
    report file specifies the environment variables that are set on the computer.
  • <host name>_slstat.txt
    The <host name>_slstat.txt
     
    report file specifies the output of slstat command.
  • <host name>_pdm_status.txt
    The <host name>_pdm_status.txt
     
    report file specifies the output of the pdm status command.
  • <host name>_dir_listing.txt
    The <host name>_dir_listing.txt
     
    report file specifies the Service Desk install directory listing.
  • <host name>_pslist.txt
    The <host name>_pslist.txt
     
    report file specifies the process listing when the pslist Microsoft tool is installed.
  • <host name>_uname.txt
    The <host name>_uname.txt
     
    report file specifies the output of the uname operating system command.
  • <host name>_diskinfo.txt
    The <host name>_diskinfo.txt
     
    report file specifies the output of df operating system command.
  • <host name>_freemem.txt
    The <host name>_freemem.txt
     
    report file specifies the output of memory information.
  • <host name>_supp_diag.log
    The <host name>_supp_diag.log
     
    report file specifies the log created for running the supp_diag tool.
  • <host name>_prtconf.txt
    The <host name>_prtconf.txt
     
    report file specifies the output of the prtconf operating system command on Solaris and the AIX computers.
  • <host name>_solrev.txt
    The <host name>_solrev.txt report file specifies the OS version and patches Information on Solaris computers.
  • <host name>_netconf.txt
    The <host name>_netconf.txt report file specifies the IP configuration Information on AIX computers.
Verify Collected Diagnostic Report
The default CA SDM documentation file\directory list includes the following reports in the CAZ\tar file. Add files or directories that you include in the CAZ\tar file by placing it in the $NX_ROOT\diag\misc_logs directory.
  • $NX_ROOT/GENLEVEL or $NX_ROOT/.GENLEVEL
  • $NX_ROOT/<COMPUTERNAME>.his
  • $NX_ROOT/NX.env
  • $NX_ROOT/NX.env.last
  • $NX_ROOT/log\
  • $NX_ROOT/pdmconf\
  • $NX_ROOT/pdmconf\version
  • $NX_ROOT/bopcfg\www\*.cfg
  • $NX_ROOT/site\mods\
  • $NX_ROOT/site\ddict.sch
  • $NX_ROOT/site\eh\
  • $NX_ROOT/bopcfg\www\CATALINA_BASE\logs\
  • $NX_ROOT/bopcfg\www\CATALINA_BASE\webapps\CAisd\WEB-INF\web.xml
  • $NX_ROOT/bopcfg\www\CATALINA_BASE\webapps\*.xml
Gather Database Server Environment Details
You can gather details about your database server to help identify performance problems in CA SDM.
Follow these steps:
  1. Determine the location of your database server, such as local or remote.
  2. Determine the DBMS version, Operating System version, and patch level.
  3. Complete the appropriate steps for your database type:
    • Execute the following queries for SQL Server and note the results:
      select @@version SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel')
    • Execute the following query for Oracle:
      select * from v$version where banner like 'Oracle%';
  4. Confirm the version of the database client that you installed on the application server.
  5. If available, gather information about Environmental Data, such as the Operating System and other databases.
  6. Gather network topology and topology information or other products that you integrate with CA SDM.
    For example, locate information about the products from available PDF files or diagrams.
Collect Performance Data from the CA SDM Servers
You can collect resource usage data on each SDM server in a multiple server configuration to determine the problems. The collected data sets are used to analyze and troubleshoot any performance or memory-related issues in the CA SDM servers. The interval logging utility allows you to start or stop the diagnostic data collection using the CA SDM web interface. Share the collected data set with CA Support Online. CA Support can help identify and resolve the CA SDM server problems.
Interval logging utility runs on pdm_intrvlog_nxd daemon that collects log data from a server. The daemon automatically starts when a Service Desk server start.
  • To stop the interval logging daemon manually, remove the entry of interval logging utility from pdm_startup.
  • To start the interval logging daemon manually, double-click the pdm_intrvlog_nxd.bat file in the $NX_ROOT/bin folder.
Collect Usage Data using Interval Logging
To collect the resource usage data on the CA SDM servers, configure the server for interval logging. You can select the type of data you want to collect from each server. You can change the logging options for the servers at any time.
For example, if you select only the CPU usage option, the utility collects only CPU usage data on the server.
Follow these steps:
  1. Log in to CA SDM.
  2. Select Systems, Interval Logging under the Administration tab.
    The Interval Logging Configurations List opens.
  3. Click Create New to add an interval log configuration.
  4. Complete the following fields:
    • Server Name
      The Server Name field specifies the server for which you want to collect the log data. Clicking Search displays the list of servers you can configure for the interval logging.
    • Recurrence Interval
      The Recurrence Interval field specifies the time interval during which the log data is collected for the server. For example, if the recurrence interval is set to 3 minutes, the log is collected for every 3 minutes.
      • Default: 3 minutes
      • Minimum: 2 minutes
    • Enabled
      The Enabled field indicates whether the interval logging is enabled or disabled for the server. If enabled, interval log is collected for the server.
      To stop the interval logging before the scheduled end date, change the Enabled status to NO.
    • Record Status
      The Record Status field indicates whether the interval log configuration is active or inactive.
    • Scheduled Start Date
      The Scheduled Start Date field specifies the start date and time for collecting the log data. If a start date is not provided, interval logging starts immediately or whenever you activate and enable the configuration.
    • Scheduled End Date
      The Scheduled End Data field specifies the end date and time for collecting the log data. If you do not enter an end date, interval logging runs until you inactivate or disable the configuration.
  5. Select the appropriate log option that you want to collect from the server.
    For example, select CPU Usage if you want to capture the log for CPU usage data.
    • CPU Usage
      The CPU Usage log option collects CPU usage statistics for the server. This log option executes "pslist - x" on Windows and “ps” on UNIX. Depending on your configuration type, you can collect the diagnostic data on the following servers:
      • Conventional: Primary server.
      • Advanced Availability: All servers.
    • Memory Usage
      The Memory Usage log option collects memory usage data for the server. This log option executes the "pslist - m" command on Windows and the “ps” command on UNIX. Depending on your configuration type, you can collect the diagnostic data on the following servers:
      • Conventional: Primary server.
      • Advanced Availability: All servers.
    • Network Status
      The Network Status log option collects information on all active connections. This log option collects network statistics by executing the "netstat /b" command or the "netstat /a" command. Depending on your configuration type, you can collect the diagnostic data on the following servers:
      • Conventional: Primary server.
      • Advanced Availability: All servers.
    • Task List
      The Task List log option collects application and services information for all tasks running on the server. Depending on your configuration type, you can collect the diagnostic data on the following servers:
      • Conventional: Primary server.
      • Advanced Availability: All servers.
    • Web Session Counts
      The Web Session Counts log option collects CA SDM sessions and user statistics for the web engine processes. This log option executes the "pdm_webstat" command. You can collect the data for any CA SDM servers.
    • Server Status
      The Server Status log option collects information about all the CA SDM daemons or processes on the server. This log option executes the "pdm_status" command. You can collect the data for any CA SDM servers.
    • DB Report
      The DB Report log option collects information of database record types by executing the db_report command. You can collect the data for any CA SDM servers.
    • Virtual DB Info
      The Virtual DB Info log option collects information that is related to the queued database requests. This log option executes the "pdm_vdbinfo" command. You can collect the data for any CA SDM servers.
    • List Server Connections
      The List Server Connections log option collects information on active connections for the server by executing the pdm_listconn command. You can collect the data for any CA SDM servers.
    • List Slump Processes
      The List Slump Processes log option collects information about slump connections and processes. This log option executes the "slstat" command. You can collect the data for any CA SDM servers.
  6. Click Save.
    The server is configured for the interval logging.
  7. (Optional) Repeat Steps 1-5 to create more interval log configurations.
  8. Go to the $NX_ROOT\log directory and view the generated log files. The output files that are generated depend on the Interval Logging options you select, where files include:
    • cpu_usage_hostname
    • db_report_hostname
    • memory_usage_hostname
    • netstat_hostname
    • pdm_listconn_hostname
    • pdm_vdbinfo_hostname
    • server_status_hostname
    • session_counts_hostname
    • tasklist_hostname
    The maximum limit of the output file size defaults to 30 KB. You can modify the file size by changing the @NX_LOGFILE_LIMIT value in NX_env file. If the generated output file exceeds the maximum file size limit, a new file is created. The new file name is appended with the suffix of 1. Files that are generated later have the suffix of the number incrementally.
    You can share the collected log data with CA Support to help identify performance problems in your SDM installation.
Review General Tuning Recommendations
As a best practice, we recommend you to monitor the key performance indicators and the resource consumption regularly. Also, ensure that routine maintenance is applied to identify small problems before they become serious.
If you suspect an issue or users complain about slow performance, open a CA Support Online issue. Include information about the problem. For example, send the Interval Logging data, CA SDM Diagnostic Tool Report, and CA SDM standard logs. Send the information to CA Support from each CA SDM server.
The following list describes common signs of performance problems:
  • The long messages in the CA SDM standard logs (stdlog.x files)
  • Search for the messages in the stdlogs that state "The following query took #### milliseconds...".
  • Queuing on database agents in pdm_vdbinfo output
  • Search for
    Queued Requests(#)
    in the pdm_vdbinfo output. You can execute this command at the OS prompt manually. The Interval Logging script also executes this command.
  • A large number of connected users to each web engine. For example, more than 200 users.
     To display the number of concurrent users per web engine, execute pdm_webstat. You can execute this command manually from the OS command prompt. The Interval Logging script also executes this command.
  • User complaints
  • When CA SDM is up and running, it is recommended to check the process memory. For optimal performance, we recommend the following method:
    1. Set a notification when the process memory exceeds 1.25 GB and begin a check on the processes that are running.
    2. Set a warning notification when the process memory exceeds 1.5 GB and take corrective actions to check the memory usage.