Troubleshooting the Thin Agent on
Linux
The Guest Introspection thin agent is
installed with
VMware Tools™
on each
guest virtual machine. Troubleshooting the Thin Agent on
Linux
If a virtual machine is slow in reading and
writing operations, and unzipping or saving files then there might be problems with
the thin agent.
- Check the compatibility of all the components involved. You need the build numbers for ESXi, vCenter Server, NSX Manager, and the Security solution you have selected (for example, Trend Micro, McAfee, Kaspersky, or Symantec). After this data has been collected, compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.
- Ensure that File Introspection is installed on the system.
- Verify that the thin agent is running by with thesystemctl status vsepd.servicecommand.
- If you suspect that the thin agent is causing a performance problem with the system, stop the service by running theservice vsepd stopcommand.
- Then perform a test to get a baseline. You can then start the vsep service and perform another test by running theservice vsepd startcommand.
- For deployments consuming network events,vmw_conn_notifyalso needs to be checked by running thesystemctcl status vmw_conn_notifyd.service.
- Enable debugging for the Linux thin agent:
- Run/etc/vsep/vseprefresh-logging.
- Usage:/etc/vsep/vsepd refresh-logging <dest> <<level> sub-component-name>Where,<dest>: [1-2] 1 - log to the VM and 2 - log to the ESX host.<level>:[1-7]where4is for logging levelINFO,7is for logging levelDEBUG.<sub-component-name>: one or more of transport, timer, file, network, process, systemWhen logging to host is enabled, logs are stored invmware.logof respectivevmfsdirectory of VMs onESXihosts.Enabling full logging might result in heavy log activity flooding thevmware.logfile. Disable full logging as soon as possible.
Enable debugs based on context (file,
process, network or system)
Enhanced logging support allowed thin agent to log
module debug level information of specific feature/functionality to vmware.log on
host or syslog in the VM.
One needs to restart Thin Agent service
in case debug logs are not generated in respective files. Note that logging to
vmware.log
on host may get throttled if logging is heavy.
The refresh-logging
input parameter has been added to
/etc/vsep/vsepd
. Its usage can be displayed by running:Debugging:
# /etc/vsep/vsepd refresh-logging
Usage:
/etc/vsep/vsepd
refresh-logging <dest> <<level>
sub-component-name>
where,
<dest>
:
[1-2]
: 1
is log to the VM and
2
is log to the ESX host. When logging to VM is enabled,
the logs will be stored at the following location based on the Linux distribution
software. On Ubuntu VMs :
/var/log/syslog
On CentOS, RHEL and SLES:
/var/log/messages
When logging to host is enabled, the logs
will be stored in
vmware.log
of respective
vmfs
directory of VMs on ESXi host.<level>
:
[1-7]
, where 4
is for logging
level INFO, 7
is for logging level DEBUG.<sub-component-name>
: one or more of transport, timer,
file, network, process, systemExample:
Enabling the following commands only
print logs from that context.
Debug logging for network introspection
can be enabled using the following command.
/etc/vsep/vsepd refresh-logging 1 7 network
Debug logging for process
introspection:
/etc/vsep/vsepd refresh-logging
1 7 process
Debug logging for anti-virus use
case:
/etc/vsep/vsepd refresh-logging 1 7 file
For command processing in timer context
(all use cases):
/etc/vsep/vsepd refresh-logging
1 7 timer
For user monitoring:
/etc/vsep/vsepd refresh-logging
1 7 system
For framework communication between SVM
and Context Mux (all use cases):
/etc/vsep/vsepd refresh-logging 1 7 transport
Troubleshooting Thin Agent
Crashes on Linux
Thin agent dumps core when it crashes.
However, it is dependent on the operating system configuration for core dumps. Each
Linux distro has different ways and configurations of generating core dump when a
system crashes.
For example, you can use
apport
for applications to dump core on a crash, where as in Red Hat you use
abrtd
. However, thin agent dumps backtrace in
/var/log/syslog
(Ubuntu) or
/var/log/messages
(CentOS, RHEL and SLES) in VM or
vmware.log
if logging is enabled on host, depending on the
logging destination. A sample backtrace:
localhost systemd: Started Session 4 of user root. localhost vsep: EMERG: 0: sig_handler(): Received signal: 11 localhost vsep: EMERG: 0: sig_handler(): backtrace returned 7 pointers localhost vsep: EMERG: 0: sig_handler(): /usr/sbin/vsep(+0x1d35e) [0x7fa2e4c9135e] localhost vsep: EMERG: 0: sig_handler(): /lib64/libc.so.6(+0x35a00) [0x7fa2e3d76a00] localhost vsep: EMERG: 0: sig_handler(): /usr/sbin/vsep(+0x3f789) [0x7fa2e4cb3789] localhost vsep: EMERG: 0: sig_handler(): /lib64/libglib-2.0.so.0(+0x6e0fc) [0x7fa2e47960fc] localhost vsep: EMERG: 0: sig_handler(): /lib64/libglib-2.0.so.0(+0x6d745) [0x7fa2e4795745] localhost vsep: EMERG: 0: sig_handler(): /lib64/libpthread.so.0(+0x7df3) [0x7fa2e4109df3] localhost vsep: EMERG: 0: sig_handler(): /lib64/libc.so.6(clone+0x6d) [0x7fa2e3e373dd] localhost vsep: EMERG: 0: sig_handler(): Unmarking all fanotify marked mount points