How to Parse IMS Database Copybooks and Mask Data

The purpose of the CA TDM FM for IMS Integrator is to extract and mask data from an IMS database without having to work on the mainframe yourself. CA TDM FM for IMS Integrator uses CA File Master Plus for IMS to access IMS databases on the mainframe, and it uses CA Mainframe Datamaker to mask the extracted IMS data. The Data Collector service for z/OS with the TDM FM Integration plugin drives the interactions between the components of this solution.
tdm10
The purpose of the CA TDM FM for IMS Integrator is to extract and mask data from an IMS database without having to work on the mainframe yourself. CA TDM FM for IMS Integrator uses CA File Master Plus for IMS to access IMS databases on the mainframe, and it uses CA Mainframe Datamaker to mask the extracted IMS data. The Data Collector service for z/OS with the TDM FM Integration plugin drives the interactions between the components of this solution.
Work together with your System Programmer and Database Admin to initialize the setup once for each user.
Masking an IMS database involves 3 steps:
  1. Extracting data from the source IMS database into a flat file.
  2. Masking the data in the flat file.
  3. Reloading the masked data into the target IMS database.
To process flat files in CA TDM Datamaker, you have to register the record layouts for the files. These definition files are suffixed *DM.txt. You can create the DM.txt definition files manually. The z/OS files might come with custom COBOL or PL1 declaration copybooks that provide the definitions.
If you are masking more than one segment type in the database, the flat file that contains the extracted data is a multi-format file, in which one record represents an instance of a segment. For such a multi-format file, an Advanced File Layout file contains record definitions for all record types in one file. A layout file is suffixed *AFL.DM.txt.
For your convenience, your
Test Data Manager
installation includes the required utilities as Windows executables and as generic .jar files. The Integrator is packaged with the File Definition Manager component. The parser is written in Java and requires a JVM (Java Virtual Machine) to run. The executables do not require any installation steps.
Follow these steps:
2
Verify Prerequisites
Work with your System Programmer to fulfill the following prerequisites.
  1. Verify that you have a recent JVM (Java Virtual Machine) installed to run the utilities.
  2. Install and deploy the Data Collector. For more information, see Deploy and Configure the Data Collector for z/OS.
  3. Find the JOB card template in the
    templates
    directory of the Data Collector for IMS home directory. CA provides this JOB card template to generate valid JCLs.
  4. Review the JOB card template and adapt it to your system and security environment, if needed. This template applies to all users.
    Tip: ISPF option 3.17 lists the z/OS UNIX directory in a way that is easy to navigate.
  5. Provide a TDM FM User Space for each user of the integration service to store per-user configuration, including created JCLs, AFL files, MAP files. The TDM FM User Space instance ID is the high-level qualifier (HLQ) of these user-specific data sets.
    Allocate the following data sets by hand for each user:
    • HLQ
      .RUNJCL – Stores all JCLs – library (fixed block, record length 80)
    • HLQ
      .FM.REPT – Stores reports of CA File Master extract or reload operations – library (fixed block ANSI (FBA), record length 133)
    • HLQ
      .TDM.AFL – is the data set for Advanced File Layout files – library (fixed block, record length 120)
    • HLQ
      .TDM.MAP – is the data set for Transformation Map files – library (fixed block, record length 255)
    • HLQ
      .DB.
      dbdName
      – the data set for the extracted database data – sequential
  6. Work with your Database Admin to locate and prepare your copybooks. For more information, see Create an Advanced File Layout (AFL) with File Definition Manager.
Note: The generated jobs are submitted in the security context of the user who runs the utilities.
Create Connection Profile
You create connection profiles to store commonly used connections to remote IMS Databases. You can create source and target connection profiles, so that you can extract from, and import data into, the same or different databases.
Follow these steps:
  1. Launch the CA TDM FM for IMS Integrator from your local computer.
  2. Open the
    Connection Profile
    tab and specify the following information:
    • Connection Profile List
      Select either "No Connection Profile" or "Select Connection Profile" and fill in the fields to create a connection profile.
    • Connection Profile Name
      Defines the name under which to save this connection profile.
    • Data Set Prefix
      Defines the high level qualifier of your TDM FM User Space.
    • Server Name
      Defines the server name where Data Collector for z/OS is installed.
    • Port Number
      Defines the port number of the server where Data Collector for z/OS is installed.
    • User Name
      Defines the user name for your TDM FM user space.
    • Password
      Defines your password.
    • Sysplex Name
      Defines the name of the sysplex where the Data Collector for z/OS is running.
    • z/OS Name
      Defines the LPAR name where the Data Collector for z/OS is running.
    • File Master Clist
      Defines the CA File Master executable that you want to run.
      Click
      GET IMS ENV List
      to retrieve the list of IMS Environments to populate the following two drop-downs.
    • IMS ENV DSN
      Defines the partitioned data set name with IMS environments definitions in its members.
    • IMS ENV List
      Select the IMS environment that contains the database definition.
    • Access Database
      Defines whether to access the database using Program Specification Blocks (PSB) or Database Descriptors (DBD). Ask your DB Admin for details.
      • Using PSB
        — Click
        Get PSB List
        to select a database.
        PSB List
        defines the name of the Program Specification Block which contains the Program Communication Block (PCB) for the selected database. Click
        Get PCB List
        to select a PCB.
      • Using DBD
        — Click
        Get Database List
        to select a database.
    • Account Code
      (Optional) Defines the account code, if your JOB card needs one.
    • Programmer Name
      (Optional) Defines the programmer name, if your JOB card needs it.
  3. Click
    New
    .
    The connection profile for the IMS database server is stored.
Edit a Connection Profile
For details on the fields, see Create a Connection Profile.
  1. Launch the CA TDM FM for IMS Integrator from your local computer.
  2. Open the
    Connection Profile
    tab.
  3. Select an existing profile from the
    Connection Profile List
    . The list is empty if you have not created any connection profiles yet.
  4. Edit the fields and click
    Save
    .
Delete a Connection Profile
  1. Launch the CA TDM FM for IMS Integrator from your local computer.
  2. Open the
    Connection Profile
    tab.
  3. Select the profile from the
    Connection Profile List
    . The list is empty if you have not created any connection profiles yet.
  4. Click
    Delete
    .
Get Remote Copybook Extract
The CA TDM FM for IMS Integrator uses connection profiles to store connection details for your IMS database server. This remote connection lets you get all unparsed copybook extract files under a given IMS layout data set name. Contact your Database Admin for details.
Follow these steps:
  1. Launch the CA TDM FM for IMS Integrator from your local computer.
  2. Specify the following information under the
    Get Remote Copybook Extract
    tab:
    • IMS Layout DSN
      Defines the IMS Layout Data Set Name where your copybooks are stored on the mainframe.
    • Select Connection Profile
      Defines the connection profile that connects to the remote IMS database server.
  3. Click
    Get Copybook Extract
    Wait for the extract to complete, and then click
    OK
    in the success message dialog.
  4. Click
    Parse Copybooks
    .
    The File Definition Manager utility opens. The copybook location that you defined in the IMS Integrator is passed on to the File Definition Manager configuration.
  5. Use the File Definition Manager to generate AFL files. Before you can use them, use File Definition Manager to review and prepare them.
    For more information, see Create an Advanced File Layout (AFL) with File Definition Manager.
The File Definition Manager creates the following files:
  • *_DG_Source.AFL.DM.txt
    — The AFL file that defines the record layout according to the ASCII layout.
  • *_zOS.AFL.DM.txt
    — The EBCDIC version of the AFL file that is used in Mainframe Datamaker.
  • XLS file —
    The spreadsheet contains one sheet per record type in the parsed copybooks parse. Use this spreadsheet to register the File Definition in Datamaker.
Register Parsed Copybook (Advanced File Layout)
Follow these steps:
  1. Verify the AFL files before you use them. Make sure you have edited the layouts to add information that the parser cannot derive. Make these edits before you register the layouts or transfer them to z/OS.
  2. Register the zOS.AFL.DM.txt files in Datamaker. These files are used to design file masking and subsetting.
  3. Transfer the zOS.AFL.DM.txt layouts to z/OS to execute file masking and subsetting. The masking and subsetting programs read these files in z/OS.
DG_Source.AFL.DM.txt facilitates file conversion between mainframe and windows code pages. For more information, see Mainframe File Conversion.
Register the Advanced File Layout in TDM Datamaker
  1. Launch CA TDM Datamaker.
  2. Select a project version context.
  3. Click
    Project, Register
    .
Select one of the following Copybook Parser Layouts to register to Datamaker:
  • File generation
    Register
    File Definition from G-T Excel file
  • File Masking and subsetting
    Register
    Advanced File Layout from G-T text file
Register File Definition from G-T Excel File
For files that contain a single record type:
  1. Open the *.csv file in the layout directory in Excel.
  2. Save as Excel 97-2003 workbook.
    Note:
    Save the file as *.xls, not as *.xlsx.
For files that contain multiple record types:
  1. Open
    LoadCSV.xls
    in the z/OS_CopybookParser directory. This spreadsheet contains an Import CSVs macro. Use this macro to import all CSVs in cell A1 of the directory.
  2. Enter the directory that contains the CSVs into cell A1, and select
    Import CSVs
    .
    A new spreadsheet opens with all of the CSVs that are imported in their own worksheet. An extra row in A1 is inserted in the first worksheet. This row provides extra information about the file.
    This file contains the following parameters:
    • PARAMETERS
    • HEADER=N
    • TRAILER=N
    • STYLE=COMPLEX,ID_OFFSET=
    • ID_LENGTH=1
  3. Modify the following parameters to define the record identifier in the file:
    OFFSET
    Defines the start position of the record identifier
    ID_LENGTH =
    Defines the length of the record identifier.
    Note:
    In this example, the record identifier is
    REC_ID
    , so
    OFFSET = 1
    and
    ID_LENGTH=1
    .
  4. Save the spreadsheet as" Excel 97-2003 workbook (*.xls, not as *.xlsx).
    To register single records or multiple record files into Datamaker with the correct Project in the context, select Projects, Register.
  5. Proceed to Register Advanced Layout.
Register Advanced File Layout from G-T Text File
  1. Select
    Advanced File Layout from G-T text file
    and click the green arrow to the right of
    Select Type
    .
  2. Click the ellipsis (...) at the end of the
    Please Select the File to Register
    field.
  3. Browse for the AFL to register, and select
    Show Record Types
    to show the records types within the selected AFL.
    You are prompted with options to clear and register each Record Type.
  4. Use the "
    Register
    " button to register all record types and proceed to the
    Advanced file Layout
    screen.
    Note:
    You can also access the
    Advanced file Layout
    screen through the
    Menu bar
    ,
    Tools
    ,
    Manage Advanced File Layouts
    .
  5. Select the first record type to display the Fields for that record.
  6. Highlight all the record fields that require masking. Use the arrows to move the fields within the
    Selected Fields
    list.
    Note:
    All fields are typically selected. If you moved multiple fields at once, the screen might refresh several times.
  7. Select the fields and click
    Register as table
    to register the record type and the fields as a table.
  8. Select the next Record Type and repeat the process.
Extract Data from IMS Database
Using a connection profile that establishes connection between the CA TDM FM for IMS Integrator and IMS database server, you extract the data from specified IMS Database so that you mask the data and then reload the masked data to the same database or to another database.
Follow these steps:
  1. Launch the CA TDM FM for IMS Integrator and open the
    Extract Data
    tab.
  2. Select a
    Connection Profile
    that establishes the remote connection to the source IMS Database.
  3. Specify the following parameters in the
    Fill Details to Extract Data
    section. Consult with your CA File Master Plus Admin.
    • Job Name
      Defines the job named needed for your JOB card. Limit: seven characters.
    • IMS Layout DSN
      (Optional) Defines the data set where copybooks (layouts for your database) are stored.
      Custom Record Layout DSN
      (Optional) Defines the DSN of the partitioned data set in which Custom Record Layouts are stored.
    • Segment(s)
      (Optional) Defines a comma separated list of only the segments which you want to extract.
    • Selection Criteria
      (Optional) Defines the selection criteria for optional filters. See CA File Master Plus documentation.
    • Segment CrossRef
      (Optional) Defines the Data Set Name of the partitioned data set in which Segment Cross-Reference parm members are stored.
    • DSN Lists
      (Optional) Defines the Data Set Name List PDS. Each DSN List is a list of DSNs saved to a member in the DSN List PDS. You use DSN Lists to resolve DSN fields on any of the product's panels.
  4. Click the
    Create Job Definition
    button. The Utility creates job definitions.
  5. Select the
    Job Definition ID
    in the
    Submit Extract Job
    section, and then click the
    Submit Job
    button.
  6. Verify the
    Job Status
    :
    • Job Definition ID
    • Job Instance ID
    • Job Status
  7. Click
    Refresh Job Status
    until the utility confirms that the Job Status is completed. The data is extracted on the mainframe.
  8. Click
    Get Job Output
    to save a copy of the job log in a text file on your local computer for further review.
  9. Click
    Open Directory
    to access the file directory where the job log is placed.
Mask the Extracted Data
You can now mask the data that you extracted from the IMS Database, according to the masking rules that you generated into the CSV file using TDM Datamaker.
Follow these steps:
  1. Open the
    Mask Data
    tab in the CA TDM FM for IMS Integrator.
  2. Select a Connection Profile that establishes the remote connection to the source IMS Database.
  3. Specify the following parameters under the
    Fill Details to Mask Data
    section:
    • Job Name
      Defines the job name needed for the JOB card.
    • TDM Masking Data Set Prefix
      Defines the high-level qualifier (HLQ) of data sets where Mainframe Datamaker is installed. For more information, see the Mainframe Installation and Upgrade section.
    • AFL Member Name
      Defines the target member name into which the AFL CSV file is transferred. eight characters.
    • Map Member Name
      Defines the target member name into which the transformation map CSV file is transferred. eight characters.
    • Common Options
      Specifies optional common options.
      • Primary Cyls Masked
        — defines the space for the output masked data file.
        Type: Integer
        Default: Define the default value in the
        tdmId
        .PROCLIB(GTMSKVS)
        JCL. 
      • Secondary Cyls Masked
        — defines the space for the output masked data file.
        Type: Integer
        Default: Define the default value in the
        tdmId
        .PROCLIB(GTMSKVS)
        JCL.
      • Trimmed XRef
        — specifies whether to trim values before cross-reference lookup.
        Type: Boolean
        Default: false
      • Trimmed HashLov
        — specifies whether to trim values before using in HASHLOV function.
        Type: Boolean
        Default: false
      • Case InsensitiveXRef
        — specifies whether to do cross-reference lookups case-insensitively.
        Type: Boolean
        Default: false
    • Advanced Options
      Specifies optional advanced options.
      • Primary Cyls Audit
        — defines the space for the output audit file. Type: Integer.
        Default: Define the default value in the
        tdmId
        .PROCLIB(GTMSKVS)
        JCL.
      • Secondary Cyls Audit
        — defines the space for the output audit file. Type: Integer.
        Default: Define the default value in the
        tdmId
        .PROCLIB(GTMSKVS)
        JCL.
      • Base Century
        — defines the base century of the year. Only required if you use abbreviated dates with a one-digit century. Type: Integer
      • AFL Quote Style
        — defines the quote style to use in the AFL file. Options are SINGLE or DOUBLE.
        Type: String
        Default: DOUBLE
      • Map Quote Style
        — defines the quote style to use in the map file. Options are SINGLE or DOUBLE.
        Type: String
        Default: DOUBLE
      • Language
        — defines the language for output messages. Options are EN, DE, ES, IT. FR is not available.
        Type: String
        Default: EN
      • Page Limit
        — defines the maximum number of pages produced in audit and report output files.
        Type: Integer
        Default: 50
      • Bad DateString
        — defines the value to use in place of fields which are supposed to contain a valid date but do not. Type: String
      • Current Date
        — defines the current date to use in processing. Format: CCYYMMDD.
        Type: String
        Default: the system date
      • Low Date
        — defines the minimum date to be output during masking.
        Format: dd/mm/yyyy
        Type: String
        Default: 01/01/1800
      • High Date
        — defines the maximum date to be output during masking.
        Format: dd/mm/yyyy
        Type: String
        Default: 31/12/2200
      • Hash Type
        — defines the type of hashing to use, either "ASM" or "JAVA". ASM is much faster. JAVA (written in COBOL) is slower, but matches the algorithm used by FDM.
        Type: String
        Default: "ASM"
      • Commit
        — defines the database commit frequency.
        Type: Integer
        Default: 1000
      • Process Count
        — defines the number of records to mask.
        Type: Integer
      • Case Insensitive Seed
        — specifies whether to match *LOV1 lookups case insensitively.
        Type: Boolean
        Default: false
      • Case Insensitive HashLov
        — specifies whether to return the same HashLov value for strings which match, regardless of case.
        Type: BooleanDefault: false
      • Case Insensitive Hash
        — specifies whether to return the same hash value for strings which match, regardless of case.
        Type: Boolean
        Default: false
      • Blank As Null
        — specifies whether to treat blank fields as null.
        Type: Boolean
        Default: false
      • Keep Invalid
        — specifies not to mask fields that contain invalid data.
        Type: BooleanDefault: false
    • Diagnostic Options
      Specifies optional diagnostic options.
      • Audit
        — defines the audit frequency. We recommended using ALL only when testing masking, due to the volume of output.
        Type: String
        Options: "
        ALL
        " or "
        ROW
        j
        " or "
        SAMPLE
        j
        ", where
        j
        is an integer
      • Report Invalid
        — specifies whether to report invalid values as they are found during masking.
        Type: Boolean
        Default false.
      • Validate Only
        — specifies not to apply masking, and to validate merely program inputs.
        Type: Boolean
        Default: false
  4. Click the
    Create Job Definition
    button.
  5. Specify the following parameters under the
    Submit Mask Job
    section:
    1. Job Definition ID
      Select the job.
    2. Advance File Layout Location
      Click
      Browse ADL File
      to select a file from your hard drive.
    3. Transformation Map File Location
      Click
      Browse Transformation Map
      and upload the Transformation Map File.
  6. Click
    Submit Job
    . The utility reads the files, saves them to the members on the mainframe, and submits the job.
  7. Verify the Job Status section for the following:
    • Job Definition ID
    • Job Instance ID
    • Job Status
  8. Click
    Refresh Job Status
    until the utility confirms that the Job Status is completed.
  9. Click
    Get Job Output
    to save the copy of job log in a text file on your local computer. Part of the job log is the masking report and audit.
  10. Click
    Open Directory
    to access the file directory where the copy of job log is placed.
Reload Data into IMS Database
After you have masked the data that you extracted from the IMS Database, you can reload that data into the same source database, or to another target database to maintain both the original data and the masked data.
Follow these steps:
  1. Open the
    Reload Data
    tab in the CA TDM FM for IMS Integrator.
  2. Select a
    Source Connection Profile
    that establishes the remote connection to the source IMS Database that has the masked data.
  3. Select a
    Target Connection Profile
    that establishes the remote connection to the target IMS Database where you want to place the masked data.
    Tip:
    If you want to overwrite the original data in the source database with the masked data, select the same connection profile for both source and target.
  4. Specify the following parameters under the
    Fill Details to Mask Data
    section:
    • Job Name
      Defines the name of the JOB card.
      Limit: seven characters.
    • IMS Layout DSN
      (Optional) Defines the data set where layouts for our database are stored.
    • Custom Record Layout DSN
      (Optional) Defines the data set name of the partitioned data set in which Custom Record Layouts are stored.
    • Segment CrossRef
      (Optional) Defines the data set name of the partitioned data set in which Segment Cross-Reference parm members are stored.
    • DSN Lists
      (Optional) Defines the Data Set Name List PDS. Each DSN List is a list of DSNs saved to a member in the DSN List PDS. You use DSN Lists to resolve DSN fields on any of the product's panels.
  5. Click
    Create Job Definition
    .
  6. Select the
    Job Definition ID
    under the
    Submit Reload Job
    section and click
    Submit Job
    .
  7. Verify the Job Status section for the following:
    • Job Definition ID
    • Job Instance ID
    • Job Status
  8. Click
    Refresh Job Status
    until the utility confirms that the Job Status is completed.
  9. Click
    Get Job Output
    to save the copy of the job log and report in a text file on your local computer.
  10. Click
    Open Directory
    to access the file directory where the copy of job log is placed.