Notification cache is empty
Follow these steps when notifications are not appearing in the Notification cache.
Review this before you start the troubleshooting steps.
- If alerts show up in the Alerts EDAA but not in M&R, the issue is in M&R.
- If you see a backup of messages in RabbitMQ, the SAM publisher is working but the ingestion service or elasticsearch service may be failing.
- If no messages are passing through into RabbitMQ, the problem is on the SAM publisher side.
- Verify all services are running, as described in Status checks on services.
- Check that SAM has started to publish to RabbitMQ and started with--bootstrap=bootstrap-amqp.conf.
- Verify SAM is exposed via the EDAA and started with--edaa=sam-presentation/2.0.
- Under theConnectionstab in the RabbitMQ web-based management interface, do the following:
- Verify that there are two connections: one from RabbitMQ / Java and the other from rabbitmq-c.
- If RabbitMQ / Java is not present, check if Tomcat is running and view the<.SAM_base_directory>/SAM/smarts/local/logs/tomcat/alert-ingestion
- If rabbitmq-c is not present and if SAM is running, check the presentation SAM log to see if it is publishing to RabbitMQ.
- From the host running the presentation SAM server, check if you can access the elasticsearch database and verify the count from a browser or using a tool like curl on Linux.Browser example:http://localhost:9200/smarts/alert/_countLinux example:curl http://localhost:9200/smarts/alert/_countA result similar to this appears:{"count":50002,"_shards":{"total":5,"successful":5,"failed":0}}
- If the count is zero, check if there are any notifications in SAM.
- From a browser, check if you can access Tomcat in the SAM server host.For example:http://hostname:8080
- Verify elasticsearch is running by checking<SAM_base_directory>/SAM/smarts/local/logs/tomcat/alert-ingestion.logIf the service was running and is now shut down or unable to process notifications, a message similar to this may appear in thealerts-ingestion.logfile:{date} {time},{pid} INFO 'ALERTS-IN' EventConsumerService EventConsumerPool-1 watchdog attemps to reconnect {date} {time},{pid} INFO 'ALERTS-IN' plugins EventConsumerPool-1 [La Lunatica] loaded [], sites [] {date} {time} {date} {time},{pid} ERROR 'ALERTS-IN' ElasticsearchNodeConfigurationService EventConsumerPool-1 ElasticSearchNodeConfigurationService::verifyIndexing: ElasticSearch : SERVICE_UNAVAILABLE :org.elasticsearch.client.transport.NoNodeAvailableException: No node available {date} {time},{pid} INFO 'ALERTS-IN' transport elasticsearch[Ent][generic][T#8] [Ent] failed to get node info for [#transport#-1][inet [localhost/127.0.0.1:9300]], disconnecting... org.elasticsearch.transport.NodeDisconnectedException: [][inet[localhost/127.0.0.1:9300]][cluster/nodes/info] disconnected