Troubleshooting Operator Console

2
2
UIM 20.3.3 has removed dependency on CA Business Intelligence (CABI) for rendering the native OC screens: Home page, Group view page, Device view page, and Monitoring Technologies (probes) view page. Custom and Out-of-the-Box dashboards and reports are still rendered by using CABI; that is, they have a dependency on CABI. However, the native OC screens are no longer dependent on CABI (Jaspersoft) and are rendered by using HTML5. For more information about the native OC screens using HTML5, see the Configuring and Viewing Monitoring Data article or the "Removing CABI Dependency (Native Operator Console)" section in the UIM 20.3.3 article.
Operator Console (OC) Installing on Primary Hub Instead of UMP Server
Symptom:
I am upgrading my 8.51, 9.02 , 9.2, or 20.1 environment to 20.3, and the Operator Console (OC) installer is trying to install the OC on the primary hub instead of the UMP server. How can I resolve this issue?
Solution:
The Operator Console searches for the wasp probe and then looks for a file. If it finds the file on the primary hub, the OC is installed on the primary hub. And, you cannot change the installation destination to the UMP server.
If the OC installer tries to deploy the OC on your primary hub, cancel the upgrade and do the following.
This process also works if you have already installed the OC on the primary hub.
  1. Take a backup of the wasp.cfg file that is available on the primary hub.
  2. Run the wasp uninstaller.
  3. In the installed packages list of the controller, check if any UMP/OC-related packages exist. (See step 4 for the list of packages that should be removed.)
    1. If packages exist, go to Step 4 and uninstall the packages using the probe utility.
    2. Otherwise, go to Step 11 and delete the wasp folder.
  4. From IM, click the primary robot controller probe to highlight it (do not open the UI).
    primary robot controller probe
  5. Press Ctrl-p to open the probe utility.
  6. Use the inst_pkg_remove callback of the controller probe to remove the UMP/OC packages. That is, inst_pkg_remove wasp. Find the inst_pkg_remove package from the drop-down list.
  7. In the package section, enter the package you want to delete.
    primary robot controller probe
  8. Use the probe utility to remove the following probes:
    • wasp_service_wrapper
    • nisapi_wasp
    • ump
    • ump_operatorconsole
    • wasp_alarmviewer_api
    • policy-management-ws
    • mcs-ui-app-portlet
    • ump_cabi
    • ump_accountadmin
    • ump_dashboard
    • adminconsoleapp
    • uimhome
    • uimesdplatelemetry
    • mps
    • wasp
  9. If you get an error when you click the run button (play icon), deploy the probe again on the primary hub and then run the inst_pkg_remove package.
  10. Verify that you have removed the apps by opening the controller UI and accessing the Status tab, Installed packages.
    primary robot controller probe
  11. Remove the wasp folder from the filesystem.
  12. Deploy the wasp, adminconsoleapp, uimhome, and uimesdplatelemetray packages on the primary hub again.
  13. If the previous wasp.cfg file has to be reused, then clear the UMP/OC webapps from the webapps section of the .cfg file and replace it.
  14. Activate the wasp probe.
  15. Run the OC installer again. It should find your UMP server. In the following screenshot, the setup has two UMP servers and the installer selected the secondary UMP server. Allow the installer to run there.
    primary robot controller probe
  16. After the OC installation on the secondary UMP completes, deactivate the robot on the now secondary OC.
  17. Run the OC installer again to upgrade the primary UMP.
    primary robot controller probe
  18. Activate the secondary OC robot.
  19. Verify that the deployment is working on both the servers.
Browsers Not Working with CABI
Symptom:
After upgrading Chrome to  91.0.4472.77 and Microsoft Edge to 91.0.864.37, I am unable to access CABI-related pages in Operator Console. I am getting the data access error.
Also, when clicking Reports in the OC and running a report, the CABI login screen opens instead of the respective report.
Solution:
This issue is specific to a same-site change in both the browsers. It should affect only those users who are currently using HTTP to connect to Operator Console and CABI.
You can resolve this issue as follows:
Alternatively, you can use the following configuration, if required:
  • For Chrome, add the following to the computer where Chrome is installed:
    • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome
      • Add dword value
        • type: dword
        • name: LegacySameSiteCookieBehaviorEnabled
        • value: 00000001
    • Add Key
      • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome\LegacySameSiteCookieBehaviorEnabledForDomainList
        • Add string value:
          • type: string
          • name: 1
          • value: *.domain.com
After applying the registry changes for the respective browser, clear the cache and reopen the browser.
Operator Console CABI System Component Missing - Unable to Contact the cabi Probe
Symptom:
I have deployed CABI and reviewed all the configurations. Everything is correctly configured; however, the OC wasp is still not able to locate CABI. The OC CABI pages display the CABI System Component Missing error.
Solution:
This issue can occur because of the incorrect CABI address in the OC wasp. If the OC wasp is unable to find CABI, it generates this error.
Verify whether the OC wasp contains the correct CABI key. For example, cabi = /hubname_domain/hubname_hub/robotname/cabi.
This address is case sensitive. To ensure that the correct address is set in the cabi probe:
  1. Right-click on the cabi probe in IM and select
    Edit, Copy to Clipboard
    .
  2. In the copied text, the exact address will be visible. Copy that and paste it into the OC wasp.
  3. Restart the cabi robot and check again.
JasperReport Server 7.1.1 for UIM external_cabi Becoming Unresponsive
Symptom:
My JasperSoft Server 7.1.1 for UIM external_cabi is becoming unresponsive very frequently.
Solution:
This issue can occur because of the RAM memory space availability. You can try the following observations:
  • Verify whether you have the RAM space available (based on the Xms and XMX values).
  • Verify whether you have the "setenv.sh" file in the ../CA/SC/CA Business Intelligence/apache-tomcat/bin folder.
  • Add "-Djava.awt.headless=true" to the setenv.sh file as follows:
    • -Djava.awt.headless=true parameter should be added to the same file in the JAVA_OPTS parameter value after the license directory.
    • So, it should be something like this:
      • JAVA_OPTS="$JAVA_OPTS -Djs.license.directory=../webapps/jasperserver-pro -Djava.awt.headless=true"
CABI Dashboard Not Displaying in OC
Symptom:
CABI dashboards are not displaying in OC for all users with all browser types. The
System component missing
error message is displaying in the OC UI. The wasp.log from OC is as follows:
DEBUG [https-jsse-nio-443-exec-7, com.firehunter.ump.utils.ProbeAddress] Unable to get port_list for robot : null ERROR [https-jsse-nio-443-exec-7, com.ca.cabi4uim.controllers.CABIController] Unable to initialize cabi probe client: Unable to communicate with any of the possible cabi probes: cabi,cabi_external
The environment details are as follows:
  • UIM 20.3 with ump_operatorconsole v2.06 and CABI v4.30 are on separate robots.
  • HTTPS is enabled on both OC and CABI.
  • UIM CABI dashboards are working fine with a direct logon to CABIJS.
Solution:
These steps might help you resolve the issue:
  • Download robot_update_9.32.zip from CA Unified Infrastructure Management Hotfix Index and import to the UIM archive.
  • On the CABI robot, deploy robot_update_9.32.
  • On the OC robot, follow these steps:
    • Deactivate the wasp probe.
    • Rename..\Nimsoft\probes\service\wasp\webapps\cabi.
    • Rename..\Nimsoft\probes\service\wasp\work.
    • Deploy robot_update_9.32.
    • Deploy ump_cabi 4.22 and validate that the date modified for ...\Nimsoft\probes\service\wasp\webapps\cabi.war is updated.
    • Activate the wasp probe.
OC Home page Not Loading
Symptom:
When I log in to OC 20.3, the OC Home page does not load.
Solution:
Do not close the page or click any other option. Wait for the OC Home page to load completely; it will load after some time.
OC Redirecting to CABI Login Page
Symptom:
The OC Home page was not loading when I logged in to OC 20.3. And, when I clicked the Reports option in the left pane, I was redirected to the CABI login page.
Solution:
Allow the OC Home page to load completely before selecting any other option. OC and CABI are different/separate web applications. For JasperServer to create a session for CABI, the process of loading the Home page allows that to happen. If you select the Reports option before the OC Home page loads, you are
unexpectedly
redirected to the CABI login page. After the Home page is loaded, the session is cached and the redirect to the CABI login page no longer occurs. If the OC session is idle for 15-20 minutes, the CABI session may timeout. In that case, select the Home page and let it load once again.
OC Trying to Install on CABI Server
Symptom:
In my environment, while upgrading to 20.3.1, the OC installation was trying to install OC on the CABI server. It would not try to install on UMP (not even on primary hub as documented above). During installation, the drop-down was not listing any server where I could install OC. It was not giving me any choice; for example, it was not allowing to install on the UMP server. It was only trying to install on the CABI server.
Solution:
As a workaround, you can follow these steps:
  • Before starting the OC installation, shut down the CABI robot.
  • Start the OC installation. The drop-down list now lets you select the UMP server where you can install OC.
SLM Groups, Accounts, SLAs Not Visible in OC
Symptom:
After upgrading to 20.3.x, I do not see SLM groups, Accounts, or SLAs in OC. I have multiple SLAs and multiple accounts, and they are in the database.
Solution:
If there is an SLA alarm with warning_severity = null, this issue may occur. This would generate an error that prevents the browser to retrieve the SLM list (in OC) with the accounts and groups. As a workaround, follow these steps:
  1. Run the following query on the database:
    select sla_id from S_SLA_ALARM where warning_severity is null
  2. Copy the sla_id and run the following update query:
    update S_SLA_ALARM set warning_severity =0 where sla_id =<from the first query>
    Repeat the operation for all the warning_severity having null value and replace with a 0.
  3. Log off and log in again.
    The issue should be resolved.
Data Access Error in Operator Console
Symptom:
When I access Operator Console in UIM 20.3.2, I see Data Access Error. UIM 20.3.2 and CABI are configured correctly. Also, when loading the Home page, I see no CABI content (Data Access Error) and noticed 404 errors in the network tab.
Solution:
This is a content issue. The data access error is occurring because CABi is missing dashboards. Deploy or redeploy the following packages on the CABI robot:
  • uim_core_dashboard
  • uim_unified_reporter
Auditing Operator Console Logins
Symptom:
I upgraded to UIM 20.3.2. I want to monitor external user sign-ins to the portal. Is there a way to collect audit logs from Operator Console that shows sign-ins with timestamps of Account Users and Bus users? Also is there a quick way to see last logon date of account users?
Solution:
In 20.3.x, the User_ table is now the CM_User_ table. It works the same way. The entries in this table are created each time any user (nimbus user or account user) logs into OC for the first time. However, the loginDate and lastLoginDate fields are not attributes in the new table. The alternative is monitoring the wasp.log with logmon for the OC logins as described in the KB Article. Although the KB still references UMP, the wasp.log logs the logging attempts.
Examples:
  • Administrator logs into OC
    wasp.log
    Dec 23 12:36:13:261 DEBUG [http-nio-80-exec-17, com.fr.ump.auth.NmsAuth] Login from request user {userId=10159, screenName=
    administrator
    , [email protected], locale=en_US, firstName=administrator, middleName=null, lastName=} Dec 23 12:36:13:403 DEBUG [http-nio-80-exec-17, com.fr.ump.auth.NmsAuth] User prin [email protected]cf24917(administrator) found for 10159
  • Account user logs in:  (
    ipxxxxx
    )
    wasp.log
    Dec 23 12:45:47:211 DEBUG [http-nio-80-exec-12, com.nimsoft.nimbus.probe.service.wasp.db.DbPreparedStatement] Query pNJt took: 0.001s Dec 23 12:45:47:211 DEBUG [http-nio-80-exec-12, com.nimsoft.nimbus.probe.service.wasp.auth.LoginModule] ippma03 logged in. Dec 23 12:45:47:211 DEBUG [http-nio-80-exec-12, com.firehunter.ump.auth.NmsAuth] User:
    ipxxxxx
    , NimBUS login milliseconds: 129 Dec 23 12:45:47:215 DEBUG [http-nio-80-exec-12, com.firehunter.ump.auth.NmsAuth] Login from request user {userId=10161, screenName=ipxxxxx, [email protected], locale=en_US, firstName=dicjiod, middleName=null, lastName=dsmopc}