NSX Guest Introspection Platform
Thin Agent Logs on Windows

The thin agent is installed on the VM Guest OS and intercepts various types of IO activity that include file, network, processes and so on.

Log Path and Sample Message

The thin agent consists of NSX Guest Introspection drivers –
vsepflt.sys
,
vnetwfp.sys
.
The thin agent logs are pushed on the ESXi host, as part of the vCenter Log Bundle. The log path is
/vmfs/volumes/<datastore>/<vmname>/vmware.log
For example:
/vmfs/volumes/5978d759-56c31014-53b6-1866abaace386/Windows10-(64-bit)/vmware.log
Thin agent messages follow the format of <timestamp> <VM Name><Process Name><[PID]>: <message>.
In the log example below
Guest: vnet or Guest:vsep
, indicate log messages related to the respective GI drivers, followed by debug messages.
For example:
2017-10-17T14:25:19.877Z| vcpu-0| I125: Guest: vnet: AUDIT: DriverEntry : vnetFilter build-4325502 loaded 2017-10-17T14:25:20.282Z| vcpu-0| I125: Guest: vsep: AUDIT: VFileSocketMgrConnectHelper : Mux is connected 2017-10-17T14:25:20.375Z| vcpu-0| I125: Guest: vsep: AUDIT: DriverEntry : vfileFilter build-4286645 loaded 2017-10-17T18:22:35.924Z| vcpu-0| I125: Guest: vsep: AUDIT: VFileSocketMgrConnectHelper : Mux is connected 2017-10-17T18:24:05.258Z| vcpu-0| I125: Guest: vsep: AUDIT: VFileFltPostOpCreate : File (\Windows\System32\Tasks\Microsoft\Windows\ SoftwareProtectionPlatform\SvcRestartTask) in a transaction, ignore

Enabling
NSX
File Introspection driver logs

Because the debug setting can flood the vmware.log file to the point that it throttles, we recommend you disable the debug mode as soon as you have collected all the required information.
This procedure requires you to modify the Windows registry. Before you modify the registry, ensure to take a backup of the registry. For more information on backing up and restoring the registry, see the Microsoft Knowledge Base article 136393.
  1. Click
    Start > Run
    . Enter regedit, and click
    OK
    . The Registry Editor window opens. For more information seen the Microsoft Knowledge Base article 256986.
  2. Create this key using the registry editor:
    HKEY_LOCAL_Machine\SYSTEM\CurrentControlSet\services\vsepflt\parameters
    .
  3. Under the newly created parameters key, create these DWORDs. Ensure that hexadecimal is selected when putting in these values:
    Name: log_dest Type: DWORD Value: 0x2 Name: log_level Type: DWORD Value: 0x10
    Other values for
    log level
    parameter key:
    Audit 0x1 Error 0x2 Warn 0x4 Info 0x8 Debug 0x10
  4. If you need to restart the File Introspection driver, open a command prompt as an administrator. Run these commands to unload and reload the
    NSX
    Endpoint filesystem mini driver:
    • fltmc unload vsepflt
    • fltmc load vsepflt
    You can find the log entries in the vmware.log file located in the virtual machine.

Enabling NSX Network Introspection Driver Logs

Because the debug setting can flood the
vmware.log
file to the point that it can make it to throttle, we recommend you disable the debug mode as soon as you have collected all the required information.
This procedure requires you to modify the Windows registry. Before you modify the registry, ensure to take a backup of the registry. For more information on backing up and restoring the registry, see the Microsoft Knowledge Base article 136393.
  1. Click
    Start > Run
    . Enter regedit, and click
    OK
    . The Registry Editor window opens. For more information seen the Microsoft Knowledge Base article 256986.
  2. Edit the registry:
    Windows Registry Editor Version 5.0 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vnetwfp\Parameters] "log_level" = DWORD: 0x0000001F "log_dest" = DWORD: 0x00000001

With the
log_dest
registry settings
DWORD: 0x00000001
, the endpoint thin agent driver logs that forwards the logs to the debugger. Run the debugger (DbgView from SysInternals or windbg) to capture the debug output.
Alternatively, you can set the
log_dest
registry setting to
DWORD:0x000000002
, in which case the driver logs will be printed to
vmware.log
file, which is located in the corresponding virtual machine folder on the ESXi Host.

Enabling UMC logging

The Endpoint Protection user-mode component (UMC) runs within the VMware Tools service in the protected virtual machine.
  1. On a Windows VM, create a
    tools config
    file if it doesn’t exist in the following path:
    C:\ProgramData\VMWare\VMware Tools\tools.conf
  2. Add these lines in the
    tools.conf
    file to enable UMC component logging.
    [logging] log = true vsep.level = debug vsep.handler = vmx
    With the
    vsep.handler = vmx
    setting, the UMC component logs into the
    vmware.log
    file, which is located in the corresponding virtual machine folder on the ESXi host.
    With the following setting logs, the UMC component logs will be printed in the specified log file.
    vsep.handler = file vsep.data = c:/path/to/vsep.log

Troubleshooting the Thin Agent on Windows

  1. Check the compatibility of all the components involved. You need the build numbers for ESXi, vCenter Server, NSX Manager, and the Security solution you have selected (for example, Trend Micro, McAfee, Kaspersky, or Symantec). After this data is collected, you can compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.
  2. Ensure that
    VMware Tools™
    is up-to-date. If you see that only a particular virtual machine is affected, see Installing and upgrading VMware Tools in vSphere (2004754).
  3. Verify that the thin agent is loaded by running the PowerShell command
    fltmc
    .
    Verify that vsepflt is included in the list of drivers. If the driver is not loaded, try loading the driver with the
    fltmc load vsepflt
    command.
  4. If the thin agent is causing a performance problem with the system, unload the driver with this command:
    fltmc unload vsepflt
    .
  5. If you are not using Network Introspection, remove or disable this driver.
    Network Introspection can also be removed through the Modify VMware Tools installer:
    1. Mount the VMware Tools installer.
    2. Navigate to
      Control Panel > Programs and Features
      .
    3. Right-click
      VMware Tools > Modify
      .
    4. Select
      Complete install
      .
    5. Find NSX File Introspection. This contains a subfolder for Network Introspection.
    6. Disable
      Network Introspection
      .
    7. Reboot the VM to finish the uninstallation of the driver.
  6. Enable debug logging for the thin agent. All debugging information is configured to log to the vmware.log file for that virtual machine.
  7. Review the file scans of the thin agent by reviewing the procmon logs. For more information, see Troubleshooting vShield Endpoint performance issues with anti-virus software (2094239).

Troubleshooting Thin Agent Crashes on Windows

If the thin agent kernel mode components crashes, the the memory dump is generated in the
/%systemroot%\MEMORY.DM
.