Troubleshooting Secure Hub and Robot

This article includes troubleshooting topics for secure hub and robot.
uim902
This article includes troubleshooting topics for secure hub and robot.
Unable to Deploy Secure Hub
 
Symptom:
 
In my environment, I have a primary hub, secondary hubs, and configured tunnels. Now, when I try to deploy a secure hub, I am unable to do so.
 
Solution:
 
You must ensure that you first upgrade your robot to 7.97 or later (non-secure) and then deploy the secure hub.
Infrastructure Manager Not Working After Upgrading to Secure State
 
Symptom:
 
After upgrading from CA UIM 9.0.2 to 9.2.0 secure state, Infrastructure Manager (IM) stops working. In this case, when I try to log in to the primary hub using 127.0.0.1, no hubs or domains are listed.
 
Solution:
 
If you have a saved user profile, delete it from the 
..\Nimsoft\hub\profiles
 location.
No UI to Validate Probes in All Linux Environment
 
Symptom:
 
In an all Linux implementation, once I am on a secure hub, I do not have any GUI method to validate probes. 
 
Solution:
 
To validate probes in an all Linux environment, you can use the 
probe_verify
 callback to validate probes. Specify the name of the probe that you want to validate and run the callback.
Unable to Secure a Hub Due to Communication Issues Between Tunnel Server and Tunnel Client
 
Symptom:
 After running the 9.2.0 upgrade installer in the secure mode, when I try to convert my hub into a secure hub, I am facing a communication issue between the tunnel server and tunnel client. I am unable to make my hub a secure one.
 
Solution:
 If you face any communication issue between a tunnel server and tunnel client and unable to make your hub a secure hub, you can follow these steps:
  1. Explicitly log in to the hub.
  2. Download the vs2017-vcredist_x64 1.01 and vs2017-vcredist_x86 1.01 packages into the hub archive.
  3. Download robot 7.97 into the hub archive.
  4. Deploy robot 7.97 on the hub archive.
  5. Manually copy hub_adapter and hub_secure into the hub.
  6. Deploy hub_adapter and then hub_secure on the hub.
    When the hub becomes secure, it should be able to communicate with the other secure hubs in the domain. You can then proceed with remaining steps to deploy the certificate package and robot_update_secure to this hub. 
Non-Secure Robots Not Getting Disconnected After Disabling hub_adapter
 
Symptom:
 
After I deactivate hub_adapter on a secure hub, the non-secure robot pointing to that hub is not getting disconnected immediately.
 
Solution:
 
When you deactivate hub_adapter on a secure hub, the non-secure robots pointing to that hub do not get disconnected immediately. It takes more than 30 minutes for the non-secure robots to show as disconnected in the hub IM. During this time, the non-secure robots cannot process any alarms or QoS messages. The non-secure robot configuration is possible during this time.
Discovery Wizard Not Displaying in UMP After Upgrading to 9.2.0
 
Symptom:
 After upgrading to 9.2.0, when I log in to UMP, the Discovery wizard is not displaying by default.
 
Solution:
 The display of the Discovery wizard depends on the key needDiscovery, which is available in the umpsettings table. If the value of the key is true, the Discovery wizard is displayed; otherwise, it is not displayed. By default, the value is set to true. Therefore, when you log in for the first time in UMP, you see the Discovery wizard. However, if you click the close button and select the 'do not show again' option, the value of the needDiscovery key is set to false, which disables the automatic display of the wizard.
CABI Dashboards Not Appearing in UMP After Upgrade
 
Symptom:
 I used the secure robot option in the 9.2.0 upgrade installer while upgrading CA UIM 9.0.2 server. After the upgrade, I upgraded CABI to 4.10. Now, I do not see the CABI dashboard in UMP. How can I resolve this issue?
 
Solution:
 Sometimes, it is possible that you do not see the CABI dashboard (by default) in UMP after upgrading CABI. If you encounter this issue, we recommend that you add the 
cabi
 key to the UMP raw configuration for the CABI dashboard to appear in UMP. The key is 
cabi
 and the value is 
/
domainname
/hub/cabirobot/cabi
. Use this key-value pair only when you do not see the CABI dashboard in UMP after the upgrade.
Admin Console Inaccessible After Primary Hub Upgrade
 
Symptom:
 
After I upgrade the primary hub, the CA UIM Server Home Page and Admin Console (AC) are inaccessible. 
 
Solution:
 
To gain access, add the primary hub IP address to the service_host configuration file, 
UIMHOME
/probe/service/service_host/service_host.cfg.
  • Add the following entry to the 
    <http_connector>
     section:
    •  
      address = 
      primary_hub_IP_address
       
       
  • If Admin Console is configured for https, add the entry to the 
    <https_connector>
     section instead of the 
    <http_connector>
     section.
  • Restart the service_host probe for the changes to take effect.
Communication with Infrastructure Manager and Admin Console Disrupting After Upgrade in Mixed Mode
 
Symptom:
 
After I upgrade a non-secure hub to a secure hub, communication with Infrastructure Manager (IM) and Admin Console (AC) is disrupting. 
 
Solution:
 
The hubs.sds file contains static route information that is used by non-secure hubs. Secure hubs do not use static routes. When a non-secure hub, which is connected to other non-secure hubs, is upgraded to secure, the static routes are invalidated. All hub-to-hub communication occurs over tunnels that you create. If you have a domain with non-secure and secure hubs, take the following actions on each non-secure hub:
  • Edit 
    UIMHOME
    /hubs/hub.cfg. In the 
    <hubs>
     section, add the key 
    broadcast_on = no
    .
  • Delete the 
    UIMHOME
    /hub/hubs.sds
     
    file from non-secure hubs to remove the old static routes.
  • Restart the hub.
If the hubs.sds file is not deleted from non-secure hubs, you can experience the following problems:
  • A non-secure hub can appear to be disconnected in IM or AC.
  • Robots that are controlled by a non-secure hub can disappear from IM or AC.
  • Robots that are controlled by a non-secure hub are listed, but AC displays the message, 
    No probes to display.
     
Enable CABI and UMP to Communicate Securely from Separate Secure Robots
To enable CABI and UMP to communicate securely from separate secure robots, follow these steps:
  1. Whitelist the IP subnets where UMP and cabi are running by using the existing probe administration UI in IM.
    This configuration ensures that the secure bus understands that wasp and CABI on these subnets can be trusted by the bus.
    UMP CABI Communication 01.png  
  2. Using pu, use the hubsec_setup_put callback on the primary hub to add the IPs of the UMP robot and CABI robot as “trusted_ips” in the security.cfg file of the hub. Use a comma-separated list.
    This configuration ensures that the secure bus understands that these are special robots, and user logins from these robots must be allowed without downgrading access.
    UMP CABI Communication 02.png  
  3. Restart the primary hub, restart UMP, wait until UMP is fully up, and then restart cabi.
Tunnel Communication Not Occurring Correctly
When secure hub and robot are configured and tunnel communication is not happening properly, go to the tunnel client hub and check the tunnel status. If it is showing following errors, follow the related steps to correct the problem:
  •  
    Invalid or missing certificate password. Reason: Failed to decrypt the certificate.
    Verify and correct the certificate paths in hub.cfg and restart the tunnel client robot.
  •  
    An application specific error.
    Add the tunnel server and the tunnel client computer details in the 
    /etc/hosts
     file in all the secure hubs and then restart the tunnel client robots.
  •  
    Could not connect to hub. Please check that the server is running.
    Follow these steps:
    1. Verify whether certificate details are provided in the hub.cfg file.
    2. Change the names for the 
      cert
       and 
      password
       keys.
    3. Provide the valid certificate path in hub.cfg, save the file, and restart the robot.
Unable to Access a Robot Connected to a Secondary Hub
 
Symptom:
 
After I remove hub_adapter from the secondary hub, I cannot access the robot that is connected to the secondary hub.
 
Solution:
 
Secure hub does not accept communication from non-secure robots without the hub_adapter. The hub_adapter should not be removed until all of the robots under a secure hub have been upgraded. Otherwise, communication with the robots is lost unless the hub_adapter is restored. 
Robot Upgrade Failing
 
Symptom:
 
My robot upgrade installation failed. What happened?
 
Solution:
 
The secure bus robot installer runs a pre-installation check for safety before the installation is allowed to proceed. This checks your robot configuration and runs some communication checks to ensure that if the new robot were to be installed that it would come up successfully. If this check fails, more information can be found in robot/proxy_check.log on the robot. Typos or incorrect formatting of the new configuration parameters is the most common cause.
Tunnel Server Not Starting Up
 
Symptom:
 
After you set up the secure hub tunnel, the tunnel server does not start up as it reports an inconsistency with the host name (short host name) and the host name extracted from the certificate (FQDN).
 
Solution:
 
Change the order of the entries in the host file to IP FQDN host name (instead of IP host name FQDN).
Receiving SID Expired Errors
 
Symptom:
 
The IM message window shows "SID expired" errors.
I have the following architecture: Primary hub is tunnel client to tunnel server; third-tier hub is tunnel client to the same tunnel server. From the primary hub, I am unable to expand the hub on the third-tier/client hub. If I restart that hub, it seems to start working again after a few minutes, then stops again. If I open an IM client on the tunnel server, I can successfully communicate with both its clients. Also, from the primary hub, a "transfer check" and "response check" are successful. It is only IM that is failing.
 
Solution:
 
To address this issue, verify that no time difference exists between the clocks on the primary and remote client.
ACLs Not Working Correctly
 
Symptom:
 
On a Linux primary hub, ACLs are behaving strangely, downgraded to the Operator from Administrator when the hub is being replaced with secure hub.
 
Solution:
 
In a secure CA UIM Linux environment, IM does not work. While accessing the Linux primary hub from the IM running on a Windows computer, the ACLs are downgraded from Administrator to Operator and probe deployment from Archive also fails. To resolve this issue, add the IP address of the Windows computer where IM is installed to the list of trusted IPs on the Linux primary hub.
Follow these steps:
  1. Run the command: 
    /opt/nimsoft/bin ./pu hub -u <username> -p<password> hubsec_setup_put
     
  2. Create a key 
    trusted_ips 
    and specify IP address of the Windows computer as the 
    value
    .
  3. Restart the IM.
UMP AccountAdmin Portlet Not Listing ACLs
 
Symptom:
 
UMP AccountAdmin portlet cannot list ACLs with secure bus in place. When accessing UMP after applying all the patches, in the AccountAdmin portlet, I cannot load the ACLs. I get a "500 internal server error."
 
Solution:
 
If the UMP is installed on a secure secondary robot and you are accessing the UMP using browser from a different computer, you might see the "500 internal serve error" while accessing the AccountAdmin portlet. To resolve this issue, add the IP address of the computer that you are using to access UMP to the list of trusted IPs in the primary hub.
Follow these steps:
  1. Run the command from a primary hub computer:
    (Windows)
    C:/Program Files (x86)/Nimsoft/bin pu -u <username> -p <password> hubsec_setup_put
    (Linux)
     /opt/nimsoft/bin ./pu hub -u <username> -p<password> hubsec_setup_put
     
  2. Create a key 
    trusted_ips 
    and specify IP address of the computer that you are using to access UMP as the 
    value
    .
  3. Restart UMP.
wasp Not Working
 
Symptom:
 
The wasp probe does not work when the UMP is installed on a robot (non-secure) pointing to a secure hub.
 
Solution:
 
For the wasp probe to work, the non-secure robot on which UMP is installed must be converted to secure robot because secure hub can communicate with non-secure robots only if hub_adapter is deployed on the secure hub.
Removing hub_adapter
 
Symptom:
 
When should I remove the hub_adapter probe and what things should I verify before removing?
 
Solution:
 
You should remove the hub_adapter probe only when your complete environment is converted to a secure environment. However, if you have a mixed environment where you have a combination of secure and non-secure robots communicating with secure hub, then the hub_adapter probe should not be removed because secure hub communicates with the non-secure robots using hub_adapter.