Scalable Masking with Docker

From 4.8, masking functionality is available for Docker-based deployments of TDM Portal. For more information on masking in Portal, see .
tdm49
From
Test Data Manager
4.8, masking functionality is available for Docker-based deployments of TDM Portal. For more information on masking in Portal, see Mask Data with CA TDM Portal.
To mask data with the TDM Portal Masking Engine container. This functionality allows you to distribute your masking jobs across multiple hosts, to run concurrently.
Use the files
docker-compose-messaging.yml
and
docker-compose-masking.yml
, to add these containers to your Docker network. For more information, see Docker-compose files.
You can also use these Docker containers with TDM Portal in Windows, to distribute masking jobs.
Overview
Scalable masking is possible with TDM Portal and the following two Docker containers:
  • This container contains a Java Messaging Service (JMS) queue of the tasks that constitute your masking job, and distributes them to the Masking Engine container(s). This container uses RabbitMQ messaging software.
    The TDM Portal Masking Service splits your masking job into tasks, to send to the Message Bus Server container.
  • This container starts up to four instances of the Fast Data Masker masking engine to perform each masking task. You can start multiple instances of this container.
Scale the masking service
You can run multiple instances of the Masking Engine container with the
--scale
flag in your
docker-compose up
command. The following example starts a Docker network with the
tdmweb
,
orientdb
and
messaging
containers active, and 3 instances of the
masking
container:
docker-compose -f docker-compose.yml -f docker-compose-messaging.yml -f docker-compose-masking.yml up -d
--scale masking=3
You can use this feature to optimize the performance of your masking jobs.
Use Cases
The main use cases for scalable masking with the TDM Portal masking container are as follows:
  • With TDM Portal in Docker - local masking
    Your TDM environment runs within a Docker network. This network includes the Message Bus Server container, and the Masking Engine container(s).
  • With TDM Portal in Docker - remote masking
    Your TDM environment runs within a Docker network. This network includes the Message Bus Server container, but the Masking Engine container(s) to which it sends tasks, are on remote hosts.
  • With TDM Portal in Windows
    Your TDM environment runs in Windows. The Message Bus Server container and Masking Engine container(s) must be on Docker networks available to the Masking Service, but they can be local or remote.
Diagram of scalable masking operation
The diagram below illustrates the following steps:
  1. User initiates masking job via REST request
  2. The Masking Manager in the TDMWeb Portal server (in Windows or in Docker) resolves job information and splits the job into tasks (maximum of one per schema), which it adds to the Message Bus Server's queue.
  3. Masking engine(s) pull tasks from Message Bus Server (JMS) queue.
  4. Masking engine begins masking operations on database table.
  5. Masking engine provides status updates and final audit via JMS queue.
  6. TDM Portal Server pulls status and audit messages from JMS queue.
  7. After completion, TDM Portal Server writes final audit information to masking store.
  8. TDM Portal Server writes job status to repository.
scalable masking
scalable masking
Security password
For data security, a password is necessary, which must match on all services. Supply it as the following parameters or environment variables:
  • In TDM Portal in Windows:
    tdmweb.TDMMaskingService.messaging.password
  • In TDM Portal in Docker:
    MESSAGING_PASS
  • In the Messaging container:
    DEFAULT_PASS
  • In the Masking Engine container:
    MESSAGING_PASS
This password can be either plain text or encrypted.
How to Implement the independent masking engine
To use the masking engine, it is necessary to perform 3 steps:
1a. Configure connection from TDM Portal in Windows to the Messaging container
To send masking jobs from TDM Portal in a Windows environment, to the Messaging container, it is necessary to add the following lines to the
application.properties
file (by default, this is at
C:\Program Files\CA\CA Test Data Manager Portal\conf
):
tdmweb.TDMMaskingService.messaging.host=
messaging
tdmweb.TDMMaskingService.messaging.port=
5671
tdmweb.TDMMaskingService.messaging.username=
Admin
tdmweb.TDMMaskingService.messaging.password=
{cry}1hY5pZrm87PWjgPdmypDbVZnL4a108lxy8YLuUVRMCr8
# The tableTaskRowThreshold parameter is not specific to scalable masking # tdmweb.TDMMaskingService.tableTaskRowThreshold = 1000000
Where
  • tdmweb.TDMMaskingService.messaging.host
    Specifies the
    hostname
    of the Message Bus Server container. Default:
    messaging
    .
  • tdmweb.TDMMaskingService.messaging.port
    Specifies the
    port number
    of the Message Bus Server container. Default:
    5671
    .
  • tdmweb.TDMMaskingService.messaging.username
    Defines the username with which you access the Message Bus Server.
  • tdmweb.TDMMaskingService.messaging.password
    Defines the password. Can be encrypted or unencrypted.
  • (Optional) tdmweb.TDMMaskingService.tableTaskRowThreshold
    Sets the maximum number of rows to assign per instance of FDM (i.e. per container). Default: 1000000.
    This parameter is not specific to scalable masking. It also applies to masking jobs you perform without Docker containers.
1b. Configure connection from TDM Portal in Docker to the Messaging container
The Messaging container, with reference to the following entries under
environment
in
docker-compose.yml
(the file that you use to start the TDM Portal container):
services:
tdmweb
: ... hostname:
tdmweb
environment: - 'MESSAGING_SERVER=
messaging'
- 'MESSAGING_PORT=
5671'
- 'MESSAGING_USER=
Admin'
- 'MESSAGING_PASS=
{cry}1hY5pZrm87PWjgPdmypDbVZnL4a108lxy8YLuUVRMCr8'
# - 'APPLICATION_PROP="tdmweb.TDMMaskingService.tableTaskRowThreshold=
1000000
"'
Where
  • MESSAGING_SERVER
    Specifies the
    hostname
    of the Message Bus Server. Default:
    messaging
    .
  • MESSAGING_PORT
    Specifies the port number of the Message Bus Server container. Default:
    5671
    .
  • MESSAGING_USER
    Defines the username with which you access the Messaging container.
    Must match Messaging container environment variable
    DEFAULT_USER
    .
  • MESSAGING_PASS
    Defines the password for the user with which you access the Messaging container. Can be encrypted or unencrypted.
    Must match Messaging container environment variable
    DEFAULT_PASS
    .
    For encrypted passwords, begin this value with
    {cry}
  • (Optional)
    APPLICATION_PROP="tdmweb.TDMMaskingService.tableTaskRowThreshold
    "
    Sets
    application.properties
    parameter
    tableTaskRowThreshold
    , to set the maximum number of rows to assign per instance of FDM (i.e. per container). Default: 1000000.
2. Configure the Messaging Container's connection to TDM Portal and the Masking Engine(s)
For the Masking Engine container, it refers to the following values in the file
docker-compose-messaging.yml
:
services: messaging: hostname: messaging ports: - '5671:5671' environment: - 'DEFAULT_USER=
Admin'
- 'DEFAULT_PASS=
{cry}1hY5pZrm87PWjgPdmypDbVZnL4a108lxy8YLuUVRMCr8'
Where
  • hostname:messaging
    Defines the hostname of the service. Default:
    messaging
    .
  • - '5671:5671'
    Defines the port through which this container communicates. Default:
    5671
    .
  • - 'DEFAULT_USER'
    Specifies the username with which to send and receive masking tasks.
    Must match the value of
    messaging.username
    (Portal in Windows) or
    MESSAGING_USER
    (Portal in Docker), and
    MESSAGING_USER
    in the Masking Engine container.
  • - 'DEFAULT_PASS'
    Specifies the password for the user with which to send and receive masking tasks.
    Must match the value of
    messaging.password
    (Portal in Windows) or
    MESSAGING_PASS
    (Portal in Docker), and
    MESSAGING_PASS
    in the Masking Engine container.
3. Configure a Masking Engine container's connection to the Messaging container
For Messaging container, they refer to the following values in the
docker-compose-masking.yml
file that you use to start your Masking Engine container(s):
services:
masking
: ... hostname:
masking
environment: - 'FDM_LICENSE=
FDM_License_key
' - 'MESSAGING_SERVER=
messaging
' - 'MESSAGING_PORT=
5671'
- 'MESSAGING_USER=
Admin'
- 'MESSAGING_PASS=
{cry}1hY5pZrm87PWjgPdmypDbVZnL4a108lxy8YLuUVRMCr8'
Where
  • MESSAGING_SERVER
    Specifies the
    hostname
    of the Messaging container. Default:
    messaging
    .
  • MESSAGING_PORT
    Specifies the port number of the Messaging container. Default:
    5671
    .
  • MESSAGING_USER
    Specifies the username with which the service accepts tasks from the Messaging container.
    Must match the value of
    messaging.username
    (Portal in Windows) or
    MESSAGING_USER
    (Portal in Docker), and
    DEFAULT_USER
    in the Messaging container.
  • MESSAGING_PASS
    Specifies the password for the user with which the service accepts tasks from the Messaging container.
    Must match the value of
    messaging.password
    (Portal in Windows) or
    MESSAGING_PASS
    (Portal in Docker), and
    DEFAULT_PASS
    in the Messaging container.
Additional Considerations
Expose logs for masking and messaging
You can expose the logs from the Message Bus Server container and the Masking Engine container, so that they are available for review independently of the container.
For more information, see Masking Engine container data volumes.
Use custom seedtables
You can use custom seedtables to mask data with the Masking Engine container.
Optimize masking jobs across multiple masking containers
You can use multiple Masking Engine containers to perform the masking tasks that make up your masking job concurrently.