Scalable Masking with Docker

From 4.8, masking functionality is available for Docker-based deployments of TDM Portal. For more information on masking in Portal, see .
tdm48
From 
Test Data Management
 4.8, masking functionality is available for Docker-based deployments of TDM Portal. For more information on masking in Portal, see Mask Data with CA TDM Portal.
To mask data with the TDM Portal Masking Engine container. This functionality allows you to distribute your masking jobs across multiple hosts, to run concurrently.
 Use the files 
docker-compose-messaging.yml
 and 
docker-compose-masking.yml
, to add these containers to your Docker network. For more information, see Docker-compose files.
You can also use these Docker containers with TDM Portal in Windows, to distribute masking jobs.
 
 
Overview
Scalable masking is possible with TDM Portal and the following two Docker containers:
  •  
    This container contains a Java Messaging Service (JMS) queue of the tasks that constitute your masking job, and distributes them to the Masking Engine container(s). This container uses RabbitMQ messaging software.
     The TDM Portal Masking Service splits your masking job into tasks, to send to the Message Bus Server container.
  •  
    This container starts up to four instances of the Fast Data Masker masking engine to perform each masking task. You can start multiple instances of this container.
Scale the masking service
You can run multiple instances of the Masking Engine container with the 
--scale
 flag in your 
docker-compose up
 command. The following example starts a Docker network with the 
tdmweb
orientdb
 and 
messaging
 containers active, and 3 instances of the 
masking
 container:
docker-compose -f docker-compose.yml -f docker-compose-messaging.yml -f docker-compose-masking.yml up -d
--scale masking=3
You can use this feature to optimize the performance of your masking jobs.
For more information, see Optimize performance with Scalable Masking.
Use Cases
The main use cases for scalable masking with the TDM Portal masking container are as follows:
  •  
    With TDM Portal in Docker - local masking
    Your TDM environment runs within a Docker network. This network includes the Message Bus Server container, and the Masking Engine container(s).
  •  
    With TDM Portal in Docker - remote masking
    Your TDM environment runs within a Docker network. This network includes the Message Bus Server container, but the Masking Engine container(s) to which it sends tasks, are on remote hosts.
  •  
    With TDM Portal in Windows
    Your TDM environment runs in Windows. The Message Bus Server container and Masking Engine container(s) must be on Docker networks available to the Masking Service, but they can be local or remote.
Diagram of scalable masking operation
The diagram below illustrates the following steps:
  1. User initiates masking job via REST request
  2. The Masking Manager in the TDMWeb Portal server (in Windows or in Docker) resolves job information and splits the job into tasks (maximum of one per schema), which it adds to the Message Bus Server's queue.
  3. Masking engine(s) pull tasks from Message Bus Server (JMS) queue.
  4. Masking engine begins masking operations on database table.
  5. Masking engine provides status updates and final audit via JMS queue.
  6. TDM Portal Server pulls status and audit messages from JMS queue.
  7. After completion, TDM Portal Server writes final audit information to masking store.
  8. TDM Portal Server writes job status to repository.
scalable masking
scalable masking
Security password
For data security, a password is necessary, which must match on all services. Supply it as the following parameters or environment variables:
  • In TDM Portal in Windows:
    tdmweb.TDMMaskingService.messaging.password
  • In TDM Portal in Docker:
    MESSAGING_PASS
  • In the Messaging container:
    DEFAULT_PASS
  • In the Masking Engine container:
    MESSAGING_PASS
This password can be either plain text or encrypted.
How to Implement the independent masking engine
To use the masking engine, it is necessary to perform 3 steps:
1a. Configure connection from TDM Portal in Windows to the Messaging container
To send masking jobs from TDM Portal in a Windows environment, to the Messaging container, it is necessary to add the following lines to the 
application.properties
 file (by default, this is at 
C:\Program Files\CA\CA Test Data Manager Portal\conf
) :
tdmweb.TDMMaskingService.messaging.host=
messaging
tdmweb.TDMMaskingService.messaging.port=
5671
tdmweb.TDMMaskingService.messaging.username=
Admin
tdmweb.TDMMaskingService.messaging.password=
{cry}1hY5pZrm87PWjgPdmypDbVZnL4a108lxy8YLuUVRMCr8
# The tableTaskRowThreshold parameter is not specific to scalable masking
# tdmweb.TDMMaskingService.tableTaskRowThreshold = 1000000
Where
  •  
    tdmweb.TDMMaskingService.messaging.host
    Specifies the 
    hostname
     of the Message Bus Server container. Default: 
    messaging
    .
     
  •  
    tdmweb.TDMMaskingService.messaging.port
    Specifies the 
    port number
     of the Message Bus Server container. Default: 
    5671
    .
  •  
    tdmweb.TDMMaskingService.messaging.username
    Defines the username with which you access the Message Bus Server.
  •  
    tdmweb.TDMMaskingService.messaging.password
    Defines the password. Can be encrypted or unencrypted.
  •  
    (Optional) tdmweb.TDMMaskingService.tableTaskRowThreshold
    Sets the maximum number of rows to assign per instance of FDM (i.e. per container). Default: 1000000.
     This parameter is not specific to scalable masking. It also applies to masking jobs you perform without Docker containers.
1b. Configure connection from TDM Portal in Docker to the Messaging container
The Messaging container, with reference to the following entries under 
environment
 in 
docker-compose.yml
 (the file that you use to start the TDM Portal container):
services:
tdmweb
:
...
hostname:
tdmweb
environment:
- 'MESSAGING_SERVER=
messaging'
- '
MESSAGING_PORT=
5671'
- '
MESSAGING_USER=
Admin'
- '
MESSAGING_PASS=
{cry}1hY5pZrm87PWjgPdmypDbVZnL4a108lxy8YLuUVRMCr8'
#
-
'
APPLICATION_PROP="tdmweb.TDMMaskingService.tableTaskRowThreshold=
1000000
"
'
Where
  •  
    MESSAGING_SERVER
    Specifies the 
    hostname
     of the Message Bus Server. Default: 
    messaging
    .
  •  
    MESSAGING_PORT
    Specifies the port number of the Message Bus Server container. Default: 
    5671
    .
  •  
    MESSAGING_USER
    Defines the username with which you access the Messaging container.
    Must match Messaging container environment variable 
    DEFAULT_USER
    .
  •  
    MESSAGING_PASS
    Defines the password for the user with which you access the Messaging container. Can be encrypted or unencrypted.
    Must match Messaging container environment variable 
    DEFAULT_PASS
    .
     For encrypted passwords, begin this value with 
    {cry}
     
  • (Optional) 
    APPLICATION_PROP="tdmweb.TDMMaskingService.tableTaskRowThreshold
    "
    Sets 
    application.properties
     parameter 
    tableTaskRowThreshold
    , to set the maximum number of rows to assign per instance of FDM (i.e. per container). Default: 1000000.
2. Configure the Messaging Container's connection to TDM Portal and the Masking Engine(s)
For the Masking Engine container, it refers to the following values in the file 
docker-compose-messaging.yml
:
services:
messaging:
hostname: messaging
ports:
- '5671:5671'
environment:
- '
DEFAULT_USER=
Admin'
- '
DEFAULT_PASS=
{cry}1hY5pZrm87PWjgPdmypDbVZnL4a108lxy8YLuUVRMCr8
'
Where
  •  
    hostname:messaging
    Defines the hostname of the service. Default: 
    messaging
    .
  •  
    - '5671:5671'
    Defines the port through which this container communicates. Default: 
    5671
    .
  •  
    - '
    DEFAULT_USER'
    Specifies the username with which to send and receive masking tasks.
    Must match the value of 
    messaging.username
     (Portal in Windows) or 
    MESSAGING_USER
     (Portal in Docker), and 
    MESSAGING_USER
     in the Masking Engine container.
  •  
    - '
    DEFAULT_PASS'
    Specifies the password for the user with which to send and receive masking tasks.
    Must match the value of 
    messaging.password
     (Portal in Windows) or 
    MESSAGING_PASS
     (Portal in Docker), and 
    MESSAGING_PASS
     in the Masking Engine container.
3. Configure a Masking Engine container's connection to the Messaging container
For Messaging container, they refer to the following values in the 
docker-compose-masking.yml
 file that you use to start your Masking Engine container(s):
services:
masking
:
...
hostname:
masking
environment:
- 'FDM_LICENSE=
FDM_License_key
'
- 'MESSAGING_SERVER=
messaging
'
- 'MESSAGING_PORT=
5671
'
- 'MESSAGING_USER=
Admin'
- 'MESSAGING_PASS=
{cry}1hY5pZrm87PWjgPdmypDbVZnL4a108lxy8YLuUVRMCr8
'
Where
  •  
    MESSAGING_SERVER
    Specifies the 
    hostname
     of the Messaging container. Default: 
    messaging
    .
  •  
    MESSAGING_PORT
    Specifies the port number of the Messaging container. Default: 
    5671
    .
  •  
    MESSAGING_USER
    Specifies the username with which the service accepts tasks from the Messaging container.
    Must match the value of 
    messaging.username
     (Portal in Windows) or 
    MESSAGING_USER
     (Portal in Docker), and 
    DEFAULT_USER
     in the Messaging container.
  •  
    MESSAGING_PASS
    Specifies the password for the user with which the service accepts tasks from the Messaging container.
    Must match the value of 
    messaging.password
     (Portal in Windows) or 
    MESSAGING_PASS
     (Portal in Docker), and 
    DEFAULT_PASS
     in the Messaging container.
Additional Considerations
Configure a masking engine container's License key
The masking engine behaves the same way as an instance of Fast Data Masker - it does not take the license key from TDM Portal. For this reason, it requires you to add your License key as the environment variable 
FDM_LICENSE
. This license key is the same as your TDM License key.
Expose logs for masking and messaging
You can expose the logs from the Message Bus Server container and the Masking Engine container, so that they are available for review independently of the container.
For more information, see Masking Engine container data volumes.
Use custom seedtables
You can use custom seedtables to mask data with the Masking Engine container.
Optimize masking jobs across multiple masking containers
You can use multiple Masking Engine containers to perform the masking tasks that make up your masking job concurrently.
For more information, see Optimize performance with Scalable Masking.