Troubleshooting the Thin Agent on Linux

The Guest Introspection thin agent is installed with
VMware Tools™
on each guest virtual machine.

Troubleshooting the Thin Agent on Linux

If a virtual machine is slow in reading and writing operations, and unzipping or saving files then there might be problems with the thin agent.
  1. Check the compatibility of all the components involved. You need the build numbers for ESXi, vCenter Server, NSX Manager, and the Security solution you have selected (for example, Trend Micro, McAfee, Kaspersky, or Symantec). After this data has been collected, compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.
  2. Ensure that File Introspection is installed on the system.
  3. Verify that the thin agent is running by with the
    systemctl status vsepd.service
    command.
  4. If you suspect that the thin agent is causing a performance problem with the system, stop the service by running the
    service vsepd stop
    command.
  5. Then perform a test to get a baseline. You can then start the vsep service and perform another test by running the
    service vsepd start
    command.
  6. For deployments consuming network events,
    vmw_conn_notify
    also needs to be checked by running the
    systemctcl status vmw_conn_notifyd.service
    .
  7. Enable debugging for the Linux thin agent:
    1. Run
      /etc/vsep/vsep
      refresh-logging
      .
    2. Usage:
      /etc/vsep/vsepd refresh-logging <dest> <<level> sub-component-name>
      Where,
      <dest>
      : [1-2] 1 - log to the VM and 2 - log to the ESX host.
      <level>
      :
      [1-7]
      where
      4
      is for logging level
      INFO
      ,
      7
      is for logging level
      DEBUG
      .
      <sub-component-name>
      : one or more of transport, timer, file, network, process, system
      When logging to host is enabled, logs are stored in
      vmware.log
      of respective
      vmfs
      directory of VMs on
      ESXi
      hosts.
      Enabling full logging might result in heavy log activity flooding the
      vmware.log
      file. Disable full logging as soon as possible.

Enable debugs based on context (file, process, network or system)

Enhanced logging support allowed thin agent to log module debug level information of specific feature/functionality to vmware.log on host or syslog in the VM.
One needs to restart Thin Agent service in case debug logs are not generated in respective files. Note that logging to
vmware.log
on host may get throttled if logging is heavy. The
refresh-logging
input parameter has been added to
/etc/vsep/vsepd
. Its usage can be displayed by running:
Debugging:
# /etc/vsep/vsepd refresh-logging
Usage:
/etc/vsep/vsepd refresh-logging <dest> <<level> sub-component-name>
where,
<dest>
:
[1-2]
:
1
is log to the VM and
2
is log to the ESX host. When logging to VM is enabled, the logs will be stored at the following location based on the Linux distribution software.
On Ubuntu VMs :
/var/log/syslog
On CentOS, RHEL and SLES:
/var/log/messages
When logging to host is enabled, the logs will be stored in
vmware.log
of respective
vmfs
directory of VMs on ESXi host.
<level>
:
[1-7]
, where
4
is for logging level INFO,
7
is for logging level DEBUG.
<sub-component-name>
: one or more of transport, timer, file, network, process, system
Example:
Enabling the following commands only print logs from that context.
Debug logging for network introspection can be enabled using the following command.
/etc/vsep/vsepd refresh-logging 1 7 network
Debug logging for process introspection:
/etc/vsep/vsepd refresh-logging 1 7 process
Debug logging for anti-virus use case:
/etc/vsep/vsepd refresh-logging 1 7 file
For command processing in timer context (all use cases):
/etc/vsep/vsepd refresh-logging 1 7 timer
For user monitoring:
/etc/vsep/vsepd refresh-logging 1 7 system
For framework communication between SVM and Context Mux (all use cases):
/etc/vsep/vsepd refresh-logging 1 7 transport

Troubleshooting Thin Agent Crashes on Linux

Thin agent dumps core when it crashes. However, it is dependent on the operating system configuration for core dumps. Each Linux distro has different ways and configurations of generating core dump when a system crashes.
For example, you can use
apport
for applications to dump core on a crash, where as in Red Hat you use
abrtd
. However, thin agent dumps backtrace in
/var/log/syslog
(Ubuntu) or
/var/log/messages
(CentOS, RHEL and SLES) in VM or
vmware.log
if logging is enabled on host, depending on the logging destination.
A sample backtrace:
localhost systemd: Started Session 4 of user root. localhost vsep: EMERG: 0: sig_handler(): Received signal: 11 localhost vsep: EMERG: 0: sig_handler(): backtrace returned 7 pointers localhost vsep: EMERG: 0: sig_handler(): /usr/sbin/vsep(+0x1d35e) [0x7fa2e4c9135e] localhost vsep: EMERG: 0: sig_handler(): /lib64/libc.so.6(+0x35a00) [0x7fa2e3d76a00] localhost vsep: EMERG: 0: sig_handler(): /usr/sbin/vsep(+0x3f789) [0x7fa2e4cb3789] localhost vsep: EMERG: 0: sig_handler(): /lib64/libglib-2.0.so.0(+0x6e0fc) [0x7fa2e47960fc] localhost vsep: EMERG: 0: sig_handler(): /lib64/libglib-2.0.so.0(+0x6d745) [0x7fa2e4795745] localhost vsep: EMERG: 0: sig_handler(): /lib64/libpthread.so.0(+0x7df3) [0x7fa2e4109df3] localhost vsep: EMERG: 0: sig_handler(): /lib64/libc.so.6(clone+0x6d) [0x7fa2e3e373dd] localhost vsep: EMERG: 0: sig_handler(): Unmarking all fanotify marked mount points