Publish Data Using Datamaker

Publishing data implies creating data; that is, you create data based on the criteria defined for the data. You can create data directly into a database, flat file, API call, SOAP Harness, or REST protocol. 
tdm42
Publishing data implies creating data; that is, you create data based on the criteria defined for the data. You can create data directly into a database, flat file, API call, SOAP Harness, or REST protocol. 
  • The two
    Enterprise
    submission methods support all file formats that CA TDM Portal publish supports; for a list of supported file formats, see Publish Data Using the CA TDM Portal
  • You can use the 
    Immediate
    publish engine for quick, small, immediate publishes. The Immediate submission method supports all file formats listed below.
  • If you execute a .bat file that you have created in a previous version of Datamaker, Datamaker defaults to the non-enterprise publishing mode, and the publish job is executed by the Datamaker publish engine rather than the by the CA TDM Portal publish engine. For more information see Publish in Batch Mode.
    Do one of the following to enable Enterprise publishing mode for old .bat files:
    • Update the old publish job XML file from Datamaker by choosing
      Batch
      mode when publishing.
      The log will include the line "
      Changes to handle non supported Enterprise Publish Option
      ".
    • Edit the old publish job XML file manually and add the following two lines between the <GTDatamaker> and </GTDatamaker> tags:
      <use_enterprise_mode>true</use_enterprise_mode>
      <tdm_portal_url>
      https://YOUR_TDM_PORTAL_URL
      </tdm_portal_url>
Follow these steps:
  1. Open the Datamaker UI and select
    Projects, Projects Manager
    from the main menu.
  2. Navigate the hierarchy in the left pane and select the appropriate data pool in the left pane.
  3. Select
    Test Data, Publish Data
    from the main menu.
    The publish dialog opens. Select a 
    Submission Method
     for this publish job:
    • Enterprise Mode
      —Use this publish option to create large scale data. This option is the default.
    • Immediate
      —Use the publish engine built into Datamaker for quick, small publishes. 
    • Batch
      —Use this publish option to create scheduled data. 
    • Remote
      —Use this publish option to create scheduled large scale data. 
  4. Click the forward arrow icon in the top-right corner of the dialog to complete the publishing.
Note:
If you try to publish data to your Hadoop/Hive target database, Hadoop/Hive inserts NULL in a column if the generated value is not valid for the column. For example, for Hadoop/Hive, the valid value for the TINYINT data type is a 1-byte signed integer ranging from -128 to 127. Therefore, for any generated value outside of this range, Hadoop/Hive inserts NULL in the column.
Publish Data Multiple Times
Enter the number of times you want to repeat publish in the 
Repeat
 field. 
Note:
 If you are not using variables such as ~ITERATION~ or ~NEXT~, you can get duplicate values in your data output.
Repeat Table Data Multiple Times
If you want to repeat individual tables in a set of data:
  1. Right-click the table name and select
    Table repeat count
    from the context menu.
  2. Enter the number of times you want to repeat this table. You can also use functions, from the Functions pane, to vary the number of times a particular table is repeated. For example, @randrange(1,5)@ and @randnorm(1,100,70,10)@
    If you have created large volumes of data within a table, it is also possible to publish a small selection of rows. For example, enter -3 in the field publishes up to three rows from the selected table.
Publish Data with Substitution Variables
When you publish data, you can control the publish by importing lists of substitution variables from the sources that are mentioned in the
 Values for Variables
 section—default values, from a CSV file, from a test case, or from SQL. Different ways to use the substitution variables for publishing are as follows:
  • Importing Variables from CSV files
    Select the 
    from file
     option and browse for the CSV file you want. This file performs one publish per row. If you have a repeat set, each row in the CSV iterates multiple times. Ensure that the CSV file follows the following format:
    TRAVELDATE,CURRENCY,CREDITCARD,FLIGHTCOST,FLIGHTPRICE,TICKETTYPE
    2008-03-01,GBP,AX,100,110,900
    2008-03-02,GBP,VI,110,110,900
    2008-03-03,GBP,VI,120,110,900
    2008-03-04,GBP,AX,130,110,900
    2008-03-05,GBP,VI,140,110,900
    2008-03-01,USD,AX,150,110,900
     Each substitution variable that you want to modify must be in its own column. If you do not include a substitution variable, the default value is used.
  • Import Variables from a Test Case
    Select the
     from Test Case
     option to browse through all available test cases in the project to source your variables.
  • Import Variables from SQL
    Select the 
    from SQL
     option, then select whether to import from the target, source, or repository. A drop-down list of available SQL from which you can select your variables displays.
  • Use Repeat in CSV Control Files
     
    You can add a special reserved column that is named REPEAT to a CSV file. This column controls the number of iterations. Each row in the CSV is considered as a separate publish.
    An example CSV file that includes REPEAT is as follows:
    REPEAT,CREDITCARD,CURRENCY,FLIGHTCOST,FLIGHTPRICE,TAXRATE
    "10","AX","GBP","75","100","0.10"
    "5","VI","GBP","75","100","0.10"
    "50","MC","USD","80","100","0.10"
  • Using Excel Spread sheets to control publishes
    You can convert your CSV file to a spreadsheet and can use this file to publish the data. However, set the name of the worksheet and the location of the row header and starting data file. To do so, go to 
    Settings > Excel Parameter Settings
     from the main menu. Enter the name of the worksheet. You also need to specify the row with the column headers and the row number where your data starts. The following segment is an example of a spreadsheet that is used to control a publish:
    Please enter the test cases you want starting at row 5
    Name:                             Email:
    MyTestName     Loyality     Location     Sex     Shipping Class
    Huw Invalid1   1            FL           F       Class3
    Huw Valid1     2            CA           F       Class3
    Huw Valid11    2            CA           F       Class2
    Huw Valid12    1            FL           F       Class1
    The column headers are in row 4 and have been hidden from the user.
    Please enter the test cases you want starting at row 5
    Name:                             Email:
    MyTestName     Loyality     Location     Sex     Shipping Class
    TEST_NAME      Loyality     Location     Gender  Shipping 
    Huw Invalid1   1            FL           F       Class3
    Huw Valid1     2            CA           F       Class3
    Huw Valid11    2            CA           F       Class2
    Huw Valid12    1            FL           F       Class1
  • Storing CSV Control Files within CA TDM
     
    You can maintain the CSV file using Microsoft Excel as long as you save it back in the .csv format. You can also store the CSV file in CA TDM. You can then publish the CSV file and use it to control another publish.
Publish Data-Only Generators Without Resolving Metafunctions
You as Test Data Engineer may want to create data-only generators, so you can use them as an intermediate publish job to execute further activities. You expect to be able to put any text into the painter and it will not be resolved as an expression. For example, you want expressions in the table counter to be resolved, but data inside the data pool itself not.
In Datamaker, such a publish fails if the data-only generator contains expressions in columns which are not of string type, for example the expression ~NEXT~ in a numeric column.
In CA TDM Portal, such a publish works when the target is not a database. For example, publishing to a CSV file, or to another Data Pool. When publishing to a target database table, however, the publish displays an error, because an unresolved string expression cannot be inserted in a non-string table column.
Create a Starter CSV File Using the Datamaker UI
If you click the save icon in the bottom-right corner of the publish dialog, the
CSV Variables
dialog containing a list of the current substitution variables with their default values is displayed. If you click the save icon on the
CSV Variables
dialog, you can save a starter CSV file. You can then open the CSV with Microsoft Excel and create extra rows (each row being an individual publish) as you require.
Publish to Appropriate Formats
Select the appropriate format from the drop-down list in the top-right corner of the data publish dialog. Some of the options are as follows:
  • Publish to SQL
    SQL insert statements are built and various control statements are added depending on the database you are connected to. For example, if you are connected to Oracle, the file includes SQLPlus spool statements.
  • Publish to Excel (or XLSX)
    This option takes the multiple tables and creates a single Microsoft Excel/Open Format XML spreadsheet with multiple workbooks. 
  • Publish to XML
    This option creates each row as an XML object.
  • Publish to HTML
    This option creates a basic HTML page.
  • Publish to CSV
    This option creates a comma-separated file with headings. 
  • Publish to TXT
    This option creates a text file with columns separated by tabs.
  • DDL for Used Tables
    This option builds a SQL script containing all the table definitions to be published.
  • Publish to Target
    This option inserts the data directly into your target connection. If the data publishes correctly, the table name is prefixed with a tick mark icon in the data publish dialog. This icon represents that the columns match and that data publishes correctly. If mismatched tables exist, you can identify them by clicking the table name.
  • Publish to Test Case
    This option allows you to publish data from one test case to another. When you select
    Publish to Test Case
    and publish, a dialog opens to let you select the required test case. This feature lets you generate multiple rows of static data that a SoapUI or QTP can use to store actual and expected values. For example, you can generate a test case with all Zip Codes in the US. You can use this static test data to feed a SoapUI data harness issuing requests and storing responses for each static row of data.
Publish to Multiple Schemas
You can also publish individual tables to separate schemas in a single publish. In the publish dialog, right-click the table that you want to publish and select the
Specify table location
option from the context menu. A dialog opens and displays a choice of other connection profiles (containing connection properties to your alternative schema).