Viewing and Filtering Logs

This page describes how to view and filter logs for the Portal, which can be useful when troubleshooting.
apip43
This page describes how to view and filter logs for the Portal, which can be useful when troubleshooting. It applies to logs collected with
journald
, which is the default logging method in the Portal Hardened Image. 
Viewing Container Names in Logs
Older system packages (like the ones that are included in RHEL/Centos 7.x) do not show container names in the default 
journalctl
 output and do not allow you to customize what information is shown when you use 
journalctl
. However, that information is still included in the internal
systemd-journald
database. To see the available fields, you can view the output in human readable json format using the
json-pretty
switch to
journalctl
:
journalctl -u docker -o json-pretty
The following sample shows a single log entry from
journalctl
:
{
        "__CURSOR" : "s=36befbf1657e47dfaba6bb5f233825ce;i=781f;b=114693c3f58c409ca894c039e1b78349;m=3c64365b;t=572f42703deb8;x=109bea720daca391",
        "__REALTIME_TIMESTAMP" : "1533767835705016",
        "__MONOTONIC_TIMESTAMP" : "1013200475",
        "_BOOT_ID" : "114693c3f58c409ca894c039e1b78349",
        "PRIORITY" : "6",
        "_UID" : "0",
        "_GID" : "0",
        "_SYSTEMD_SLICE" : "system.slice",
        "_MACHINE_ID" : "21a1482b50b84b2bb774c7c0a9153d7e",
        "_HOSTNAME" : "localhost.localdomain",
        "_TRANSPORT" : "journal",
        "_CAP_EFFECTIVE" : "1fffffffff",
        "_COMM" : "dockerd",
        "_EXE" : "/usr/bin/dockerd",
        "_CMDLINE" : "/usr/bin/dockerd",
        "_SYSTEMD_CGROUP" : "/system.slice/docker.service",
        "_SYSTEMD_UNIT" : "docker.service",
        "_PID" : "2153",
        "_SELINUX_CONTEXT" : "system_u:system_r:container_runtime_t:s0",
        "MESSAGE" : "Pinging zookeeper1:2181",
        "CONTAINER_NAME" : "portal_kafka1.1.xtasrlkqe9rbd6hkawnzju92j",
        "CONTAINER_TAG" : "1589b2f261af",
        "SYSLOG_IDENTIFIER" : "1589b2f261af",
        "CONTAINER_ID" : "1589b2f261af",
        "CONTAINER_ID_FULL" : "1589b2f261af27748a12ac4fac9d58ae4d9102c9038b1d6b3c66a04539bdcdcf",
        "_SOURCE_REALTIME_TIMESTAMP" : "1533767835704755"
}
Filtering Log Fields
As the previous
journalctl
sample output illustrates, there are quite a few fields in the full json output. To make troubleshooting easier, you can show only the container name, timestamp, and log message when the output of the 
journalctl
 command is piped to it by using the
logfilter
utility that API Portal includes.
The 
logfilter
 utility requires that each line is a self-contained json object, so you must invoke 
journalctl
using 
-o json
 when piping to it (not 
json-pretty
).
To view the log output for Docker with container names in real time, use the following command:
journalctl -u docker -o json -f | <portal installation directory>/util/logfilter
If you want to view logs for a specific container, you can filter the output using 
grep
. The following example uses
grep
to filter the output to display only the logs from the pssg container:
journalctl -u docker -o json -f | ./logfilter | grep "portal_pssg"
portal_pssg.hjuawe8742zm83znhw18gpbl4.ld91netddjsqlpj4ze3yalli2 2018-08-13 19:15:03 UTC [2018/08/13-19:15:03,085]-[INFO   ]-[152]-[com.l7tech.server.policy.assertion.ServerAuditDetailAssertion]--4: [pssg, message-completed] request.url = https://pssg:8443/portalTenantInfo/Apis?$filter=Name+eq+'Portal+Authorization+API+(apim)'+and+TenantId+eq+'apim', request.http.method = GET, remoteHost = 10.0.0.219, request.time.millis = 1534187703079, elapsedTimeMs = 5, request.size = 0, response.size = 2454, response.http.status = 200, routingStatus = None
portal_pssg.hjuawe8742zm83znhw18gpbl4.ld91netddjsqlpj4ze3yalli2 2018-08-13 19:15:03 UTC [2018/08/13-19:15:03,117]-[INFO   ]-[171]-[com.l7tech.external.assertions.odata.server.producer.jdbc.GenerateSqlQuery]- SELECT *  FROM api  WHERE uuid = ?