Publish Data Using the CA TDM Portal

Publishing data means creating data in the target database schema or files. When you publish data, the data is generated based on the data generation rules that you define, and the data is added to the tables in the target database schema or files. The target database schema is the connection profile that you select at the time of publishing. The target table must already exist in the schema for the publish to work. 
tdm48
Publishing data means creating data in the target database schema or files. When you publish data, the data is generated based on the data generation rules that you define, and the data is added to the tables in the target database schema or files. The target database schema is the connection profile that you select at the time of publishing. The target table must already exist in the schema for the publish to work. 
Select Project and Data Generator
  1. Access the CA TDM Portal.
  2. Select an appropriate project and its corresponding version from the
    Project
    drop-down list in the top blue bar.
    This selection sets the required project and version context for all the related operations that you perform in the Portal.
  3. Click
    Generators
     in the left pane.
    The
     Generators
    page opens.
    Note:
    If the left pane is not visible, click the icon (represented by three horizontal bars) in the top left corner.
  4. Click the data generator that you want to use to publish data.
    The
    <Data Generator Name>
    pages opens. This page includes the details about the data generator; for example, associated project and project version.
  5. Click the
    Select Tables
    button to open the 
    Registered Tables
     dialog.
    The dialog lists the registered tables (
    used
    and
    unused
    tables) based on the project version that is associated to the selected data generator.
    Note:
     Publish from a Teradata database with special characters in table or column names fails with an "Inconsistent Table definitions" error. Ensure that all characters are UTF-8 encoded.
  6. Click the row corresponding to the
    used
    table that you want to use for data publishing. 
    Note:
     To identify a
    used
    table, verify the presence of
    Yes
    in the
    Used
    column of the table. A
    used
    table is a table that includes data or for which you have already defined data generation rules.
    All the table rows are listed in the 
    <Data Generator Name>
    page, including columns.
  7. Review the data generation rules added to the table.
  8. Click the
    Publish
    button.
    A dialog with the publishing options opens.
Configure Publish Options and Publish
Expressions in tables that use metafunctions such as @execsql with the "T" argument use the selected
target
profile when publishing. Expressions with the "S" argument use the selected
source
profile when publishing. 
  1. Select the appropriate schema from the
    Publish From
    drop-down list. This drop-down list includes all the schemas that are available for the selected data generator. 
  2. Specify the following for each table:
    • Include In Publish
      Specifies whether to include the table while publishing data or not. Only the tables that you checked are included in publish and the tables unchecked are excluded. You can generate and publish the data only for the included tables.
    • Table Count
      Specifies the number of times that you want to repeat each table while generating data. Enter a number to specify the table repeat. You can also use functions and variables to specify the Repeat count. Click the Data Painter icon to open the Data Painter dialog. The dialog lets you click on objects in the Functions and Variables sections. These objects transfer to the edit section where you manipulate them.
      Currently negative values are not supported in this field. In TDM Datamaker, you specify negative value in table repeat count field to publish a limited number of rows from large volume of data created within a table.
    • Stored Columns
      (Optional) Specifies a comma-separated list of stored columns from registered tables and publish data. These tables are saved to the repository for later reuse in another publish job. 
    • Publish Options
      (Optional) Specifies what CA TDM Portal does if it encounters a record with the same key in the target database. Select one of the following options:
      • Use publish level setting
        Specifies that the publish job should use global setting for the selected data generator. This is the default behavior.
      • Continue
        Specifies the publish job skips duplicate records.
      • Update
        Specifies the publish job updates the duplicate record in the target database.
      • Exit
        Specifies that the publish job exits with an error. 
    • Publish to Location
      (Optional) Specifies different target locations for individual tables. If you want to publish to a different location than the registered location, you do not need to create a new connection profile. If the job is set to publish to a file then the location column is not shown.
      Default: The Connection Profile and Schema drop-downs specify the default target location for tables that do not have a Publish to Location set. 
  3. Review the information in the table.
    This table shows a preview of the registered
    used
    tables in the database schema that you selected in the previous step. This information lets you decide whether your publish adheres to the schema (in the target database) that you select in the next two steps.
  4. Select the format to which you want to publish the data from the
    Publish To
    drop-down list. The available options are:
    • Connection Profile
      Lets you publish the data to a 
      Target Connection Profile
      . If you use meta functions which specify the "S" argument, also define a 
      Source Connection Profile
      .
      Select a Schema that is available in the target database from the
      Select Schema
      drop-down list. The schema for the source connection profile is set in the query specified in calls to @execsql, if applicable.
      1.  If the Target is a
      Sybase IQ database:
      Ensure that you have modified the corresponding database configuration file (.cfg) in the Sybase IQ installed server for the following parameters:
      - Modify the parameter
      -c 48m
      to
      -c 64m
      - Modify the parameter
      -gm 10
      to
      -gm 30
      - Add the parameter
      -gn 45
      at the end of the file
      After modifying the configuration file (.cfg) of the corresponding database, restart the Sybase IQ database.
      2. If the target is a
      TestMatch data pool
      with
      'Clear Down Existing Data'
      option checked, in
      Enterprise mode
      :
      Ensure that the following line is present in your
      application.properties
      file:
      tdmweb.publish.enableTestMatchDataPoolCleardown=true
      For more information about connection profiles (for example, whether you can access a specific connection profile), see Create and edit Connection Profiles.
      You can set advanced connection pool settings for your Target Connection Profile. See Set Advanced Target Connection Settings below. These options are intended for Advanced/Network Admin users only.
    • File
      Lets you publish the data to a file. CSV, TXT, SQL, and FD text files are saved in Windows format beginning with a 2-byte BOM mark to indicate UTF-8 encoding, and terminated by \r\n. The file name is the same name as the generator but normalized, so that it makes an acceptable filename, for example, punctuation characters are converted to underscores.
      The following file types are supported:
      • CSV
        — Comma delimited table
      • TXT 
        — Tab delimited table
      • SQL 
        — SQL Statements File
      • FD 
        — Formatted Text File
        Note: 
        Publishing to the FD file works only for the tables that are created by the registration of the G-T Excel or CSV file object. Therefore, ensure that you select only these tables while publishing the data to the FD file. If you try to publish other tables, the publishing fails.
      • XLS 
        — Excel 97-2003 Workbook with one worksheet per table
      • XLSX
         — Excel Workbook with one worksheet per table
      • XML 
        — Well-forced XML file(s)
  5. Specify the number of rows that you want to publish in the
    Repeat
    field. You can also use functions and variables to specify the Repeat count. Click the Data Painter icon to open the Data Painter dialog. The dialog lets you click on objects in the Functions and Variables sections. These objects transfer to the edit section where you manipulate them.
  6. (Optional) Click Variables to expand the list of variables used in the selected generator. Currently variables table always shows default variables. if we select any other option than "Default", the UI table will not get refreshed with variables values. To override the values used for variables in publish job, follow one of the below methods:
    • Manually override the default values
      Select Default Values from the Variables drop-down. Identify the variable you want to override from the list and click the variable. In the Edit Variable Value dialog, modify the default value and click OK. Based on the Variable Type and Display Type you specified to create the variable, the Validate option is enabled. Click Validate to verify whether the modified value is part of the defined rule.
      When you modify the value of any variable, the dependent variable will resolve the hierarchy, if used. For example, a variable
      Var1
      includes
      HR
      as its value. Another variable
      Var2
      includes the variable
      Var1
      as its default value. So, the value of the variable
      Var2
      becomes
      HR
      . Now, you modify the value of
      Var1
      from
      HR
      to
      Finance
      . In this case, the value of the variable
      Var2
      also resolves to
      Finance
      .
    • Override the default values using a CSV file
      Select Values from a File from the Variables drop-down. Click the Save to CSV button to download a CSV file with the default variables. Modify the Variable values in the CSV file as necessary and save the file. Click the folder icon to open the Load Variable Values dialog. Drag the saved CSV file to the dialog and click the Load button. Uploads the CSV file and uses the variable values from the CSV file while publishing.
      If the uploaded CSV file includes only some of the used variables, then default values of the respective variables are only modified. For remaining used variables default values are applied. Any irrelevant variable information found in file is ignored.
    • Override default values using Generator
      Select Values from Generator from the Variables drop-down. Click the Generators button next to Variable drop-down. In the Select Generator dialog expand the project version and select an applicable Generator. Click OK.
      Notes: 
      • The variables present in the selected Generator are not considered for evaluation. Only the variable names which exists as columns in used tables are considered as valid variables.
      • The selected generator is a dedicated variable container and can have only one used table. Any generator that has more than one used table will fail the validation.
  7. Specify the correct email ID in the
    Email
    field.
    The CA TDM Portal sends an email that includes log files (related to the publishing operation) to the email ID that you provide in this field.
  8. Select
    Now 
    to publish the data immediately. Otherwise, Select 
    Schedule
    and specify the applicable date and time to schedule the publishing.
  9. Click
    Options
    to specify the action to perform, if there are duplicate values in the generated data. Following are the available options:
    • On Duplicate In Data Target
      Specifies what CA TDM Portal does if it encounters a record with the same key in the target database. Select one of the following options:
      • Exit
        Specifies that the publish job exits with an error. This is the default behavior.
      • Continue
        Specifies the publish job skips duplicate records.
      • Update
        Specifies the publish job updates the duplicate record in the target database.
    • On Generated Duplicate
      Specifies what CA TDM Portal does if it encounters duplicate values in the generated data. That means same values are generated for more than one row. Select one of the following options:
      • Exit
        Specifies that the publish fails when the first duplicate is identified in the generated data.
      • Remove
        Specifies that the publish carries on and removes the row that corresponds to the duplicate value.
  10. Click
    Publish
    .
    A message with a Request ID for the publish operation is displayed. You also have the option to
    Cancel the Request
    , then the publish is created in a ‘Cancelled’ status. If you don't cancel, the publish job is added to the jobs queue in the
    Submitted Requests
     page. 
Review, Cancel, or Re-submit Scheduled Jobs
Click
Submitted Requests
in the navigation pane, or click a
Request ID
link in the submit message to open the 
Submitted Requests
 table. This
 
table includes all submitted jobs and additional details. 
  • Click the row in the
    Submitted Requests
    window that includes the Request ID.
    The
    Additional Information
    dialog opens. This dialog includes comprehensive information about the publish job.
  • Press the
    Cancel Request 
    button to unschedule a job that has not started yet.
    A job can only be canceled by its owner, or by an Administrator. You cannot cancel jobs that are already running.
  • Press the
    Re-submit Request 
    button to resume canceled jobs.
    A job can only be re-submitted by its owner, or by an Administrator.
    The re-submitted job goes into "Not Started" state and runs on the previously scheduled time. If the scheduled time is in the past, it starts immediately.
When the status of the job changes to
Completed
, the data has been published into the target database successfully.
Set Advanced Target Connection Settings
You can define parameters that control how 
Test Data Manager
manages database connections. These settings override Apache Tomcat's default settings.
Users should only make changes to these settings if they are confident that their database connection is problematic.
To set advanced connection pool settings for your Target Connection Profile, it is necessary to add the following lines to your
application.properties
file, located at
C:\Program Files\CA\CA Test Data Manager Portal\conf
in a standard installation:
tdmweb.TDMPublishService.db.spring.datasource.tomcat.initialSize=10
tdmweb.TDMPublishService.db.spring.datasource.tomcat.minIdle=10
tdmweb.TDMPublishService.db.spring.datasource.tomcat.maxIdle=100
tdmweb.TDMPublishService.db.spring.datasource.tomcat.maxActive=100
tdmweb.TDMPublishService.db.spring.datasource.tomcat.maxWait=30000
tdmweb.TDMPublishService.db.spring.datasource.tomcat.testWhileIdle=false
tdmweb.TDMPublishService.db.spring.datasource.tomcat.timeBetweenEvictionRunsMillis=5000
tdmweb.TDMPublishService.db.spring.datasource.tomcat.minEvictableIdleTimeMillis=600000
tdmweb.TDMPublishService.db.spring.datasource.tomcat.removeAbandoned=false
tdmweb.TDMPublishService.db.spring.datasource.tomcat.logAbandoned=false
tdmweb.TDMPublishService.db.spring.datasource.tomcat.removeAbandonedTimeout=60
tdmweb.TDMPublishService.db.spring.datasource.tomcat.abandonWhenPercentageFull=0
tdmweb.TDMPublishService.db.spring.datasource.tomcat.maxAge=2200
tdmweb.TDMPublishService.db.spring.datasource.tomcat.suspectTimeout=0
tdmweb.TDMPublishService.db.spring.datasource.tomcat.defaultTransactionIsolation=1
All values above are the parameters' default values.
These lines are not present in the standard application.properties file, as supplied with CA TDM.