FASTLOAD

The FASTLOAD utility loads data into a non-SQL defined database for the first time.
idmscu19
The FASTLOAD utility loads data into a non-SQL defined database for the first time.
This article describes the following information:
Authorization
To
You need this privilege
On
FASTLOAD into a segment
DBAWRITE
The segment
Syntax
  ►►─── FASTLOAD ─┬──────────────────────┬──────────────────────────────────────►                 ├─ STEP ─┬─ step-name ─┘                 └─ FROM ─┘  ►─┬──────────────────────────────┬───────────────────────────────────────────►    └─ NOTIFY notify-record-count ─┘  ►─────┬─────────────────┬────────────────┬─────────────────────┬─────────────►◄        └── as SORTEXIT ──┘                └── REUSE workfiles ──┘  
Parameters
  • STEP
    Specifies that only one step of the entire FASTLOAD process should be executed.
    If you do not specify STEP, all steps are performed.
  • FROM
    Specifies that FASTLOAD processing should begin at a specified step and that all remaining steps should be completed.
  • step-name
    The name of the step to execute.
    The name must be one of the following:
    • SORT1
    • IDMSDBL2
    • SORT3
    • IDMSDBL3
    • SORT4
    • IDMSDBL4
  • NOTIFY
    Directs the FASTLOAD utility to send a message to the system console after a specified number of records are read in the current step.
    If you omit the NOTIFY option or specify 0, no messages are sent. Only the standard messages that go to the SYSLST file are sent when a step is completed.
  • notify-record-count
    The number of records to read before sending a message.
    If you specify zero, the only message sent is the message that is always sent to the SYSLST file at the end of each step. The message indicates the number of records processed during the step.
  • as SORTEXIT
    Causes each DBL
    x
    step in the utility to return its input data directly from the preceding sort instead of having the sort write the data to a workfile. This option eliminates one workfile for each sort and saves the I/O it takes to write then read the workfile.
  • REUSE workfiles
    Causes each step in the utility to reuse an existing workfile, if possible, when writing its output data, instead of writing to a new one for each step. This reduces the number of workfiles that need to be allocated.
Usage
How to submit the FASTLOAD statement
You submit the FASTLOAD statement only through the batch command facility. You must run the batch command facility in local mode.
When to use the FASTLOAD utility
Use the FASTLOAD utility to load a non-SQL-defined database for the first time.
When not to use the FASTLOAD utility
To reload a non-SQL-defined database that has been unloaded, use RELOAD.
To load an SQL-defined database, use LOAD.
FASTLOAD cannot process mixed page groups and will issue an error message if mixed page groups are encountered. If your subschema binds to multiple page groups, you must select a subset of areas to process that are all in the same page group. You must use multiple invocations of the utility to process different page groups.
Before running FASTLOAD
To use the FASTLOAD utility, follow these sectionlines:
  1. Write a format program to prepare input data and create a subschema that describes the records and sets to be loaded.
  2. Link edit the format program with IDMSDBLU and IDMS.
  3. Execute the format program.
  4. Execute the FASTLOAD utility to complete the loading process.
The format program
The format program builds record and set ownership descriptors for each record to be loaded. These descriptors are used when loading the records.
The subschema
The subschema must:
  • Include all records being loaded and all sets in which the records participate
  • Allow the areas being loaded, and all areas with set connections to those areas, to be readied in exclusive update mode
Run FASTLOAD all at once or in steps
The processing initiated by the FASTLOAD utility has six steps, which you can run one at a time or as a single process. Each step generates output for use by the next step. Three of the steps are sorts to prepare the data for use by the next step. You can use your own sorting program or direct the FASTLOAD utility to sort the data during processing. If you run the steps as a single process, sorting is performed automatically.
When to run FASTLOAD in steps
The most common reason to use the FASTLOAD utility in steps is to cut the work into pieces, each piece requiring less time to run than the whole process.
You may also decide to use the FASTLOAD utility in steps to use your own sorting programs between the steps, or you can run the sort steps on a different machine than that holding the database.
Restarting IDMSDBL2, IDMSDBL3 or IDMSDBL4
If a problem arises while running IDMSDBL2 or IDMSDBL3,
do not
simply fix the cause of the problem and rerun the step. In addition to fixing the cause of the problem, you must do one of the following:
  • Reinitialize the database and begin again at the IDMSDBL2 step
  • If you backed up the database before running the step, you can run the step again, using the backup
These steps change the database, and if a problem arises, you need to undo the changes before running a step over again.
If a correctable abend occurs while running IDMSDBL4 (for example, a time-out abend), you can restart the fastload operation at the IDMSDBL4 step after unlocking the areas involved. Because the IDMSDBL4 step modifies existing records, when it is restarted it will simply modify the same records in the same way.
SORTEXIT and FROM/STEP
When using the FROM and STEP options with the SORTEXIT option, each pair of SORT
n
and DBL
x
steps are considered to be one step. If either half of the SORT
n
/DBL
x
is specified on a FROM or STEP option, processing will start with the SORT
n
step and the DBL
x
step will also be executed. For example:
  • FROM IDMSDBL3 will start with step SORT3 and will continue to the end.
  • STEP SORT3 will run steps SORT3 and IDMSDBL3.
SORTEXIT/REUSE WORKFILE restart considerations
Since SORTEXIT combines each SORT
n
step with the DBL
x
step that follows it, if a failure occurs in the DBL
x
step, a restart (if a restart is possible) must begin with the sort step and the input to the step will be resorted. Non-SORTEXIT mode will take longer to run but can be restarted after the sort in this case. Therefore, if restart time is more critical than normal runtime, do not run the utility as a sortexit.
If the REUSE WORKFILE option is used with SORTEXIT, some input workfiles will be used as output files in the same step. Therefore, if these two options are used together and a failure occurs, the utility must be restarted from the beginning.
Workfile Considerations for restarting a failed FASTLOAD
If the FASTLOAD command fails, depending on the reason for failure, restart the command at the failing step using the "FROM step-name" syntax. You can restart a step only if the input files to that step are intact and valid.
To prepare for a possible restart when running a one-step FASTLOAD, the Intermediate work files should have a disposition that preserves the data set in the event of an abend, for example, "DISP=(NEW,CATLG,CATLG)."
To restart FASTLOAD at a particular step, the input files to that step must have a disposition to specify that the files already exist, for example, "DISP=OLD."
To determine which files were input to a given step, refer to the "Intermediate Work File" tables under "JCL Considerations." Partially created output files should be deleted before restarting the job, and the original disposition should be used in the restart job, for example, "DISP=(NEW,CATLG,CATLG)."
The SYSPCH file contains sort parameter information for sort steps. It is an output file to IDMSDBL
n
steps, but is not read unless restarting or running in step mode. So during a normal run, the SYSPCH file should be treated as a normal output file, for example, "DISP=(NEW,CATLG,CATLG)." However, restarting is not as straightforward. If the previous job failed in an IDMSDBL
x
step, the SYSPCH file was an output file and should be deleted before restarting. But if the failure occurred in a SORT
x
step, the contents of the SYSPCH file should contain the same values that were input to the SORT
x
step. In this case, the SYSPCH file should be preserved and defined as a SYS001 input file to the restart step.
When the SORTEXIT option is used, the SORT
x
and IDMSDBLx steps are combined. If a failure occurs in this mode, the SYSPCH file should normally be preserved and used as a SYS001 input file to the restart. However, there is a small window at the end of an IDMSDBL
x
step where the SYSPCH file is opened for output and new SORT parameters are written. If the job fails at this point, the entire SORT
x
/IDMSDBL
x
step must be restarted, but the SYSPCH file will not be valid as a SYS001 input file. In this case, the sort parameters must be recreated by hand or the job must be restarted at an earlier IDMSDBL
x
step if possible. One way to avoid this situation is to run in step mode when running SORTEXIT mode.
The RELDCTL data set is always an input file to the first step of a FASTLOAD whether being restarted or not.
The steps of FASTLOAD
The FASTLOAD utility consists of the following steps which you can run separately or as a single operation:
Step
Description
SORT1
Sorts the output file from the execution of the format program.
IDMSDBL2
Stores records in the database using the output from SORT1,but does not connect any sets.
Creates an intermediate work file for use by SORT3.
Prints statistics on the records written to the database.
SORT3
Sorts the file produced by IDMSDBL2.
IDMSDBL3
Establishes pointers for each chained setin which each record participates (that is, builds the record prefix) but does not write the prefixes to the database.
Creates an intermediate work file for use by SORT4.
Builds indexes in the database.
SORT4
Sorts the file produced by IDMSDBL3.
IDMSDBL4
Inserts the record prefixes by performing a serial sweep of the database.
Each step has input and output
Step
Input
Output
SORT1
SYS001 contains sort control parameters from the SYSPCHfile after running the format program
SYS002 from running the format program
SYS004 contains the sorted contents of SYS002
IDMSDBL2
SYS004 from SORT1
RELDCTL file from the format program;RELDCTL contains control and set information
SYSPCH contains sort control parameters
SYS005 contains set membership information
SYSLST contains statistics on records written to the database
Note:
Overflow statistics may be fewer than expected. To improve performance during a fastload, IDMSDBL2 uses the dbkey of the previously stored record as a 'direct' dbkey if the next record to be stored has the same new target page. This reduces the number of overflow conditions.
SORT3
SYS001 contains sort control parameters from IDMSDBL2.If running all steps at once, no sort parameters are needed.
SYS005 from IDMSDBL2
SYS009 contains the sorted contents of SYS005
IDMSDBL3
SYS009 from SORT3
RELDCTL file from the format program;RELDCTL contains control and set information
SYSPCH contains sort control parameters
SYS010 contains pointer descriptors
SORT4
SYS001 contains sort control parameters from IDMSDBL3.If running all steps at once, no sort parameters are needed.
SYS010 from IDMSDBL3
SYS011 contains the sorted contents of SYS010
IDMSDBL4
SYS011 from SORT4
RELDCTL file from the format program;RELDCTL contains control and set information
 
This table describes the input and output files as if FASTLOAD were executing without the SORTEXIT and REUSE options. For the effect of these parameters, see "JCL Considerations" later in this section.
Sort output after each step
If you run the FASTLOAD utility a step at a time, you must sort the contents of the intermediate work files. You can use your own sort program or direct the FASTLOAD utility to perform the sorts for you. If you use your own sort program, do not execute the FASTLOAD utility sort steps.
You can use the sort parameters in SYSPCH from the format programs IDMSDBL2 and IDMSDBL3 as the starting point for coding your sort parameters.
Sort the intermediate work files as follows:
Sort name
File to sort
Sort order
Sort on
Begins at
SORT1
SYS002
Descending
16 bytes
Byte 5
SORT3
SYS005
Ascending
16 + (2 x
n
)
n
= the length of the longest sort or CALC key in the subschema
Byte 5
SORT4
SYS010
Ascending
12 bytes
Byte 5
The format program
Functions
The format program must perform the following functions for each record to be loaded:
  • Builds a record occurrence descriptor
  • Builds one owner descriptor for each set in which the record is an automatic member
  • Builds one owner descriptor for each set in which the record is a section member if the record is to be connected to the set at load time
  • Calls IDMSDBLU, passing the record occurrence descriptor as the first argument and the owner descriptors as the remaining arguments
Output
IDMSDBLU uses the information provided by the format program to create an output file for use as input by the FASTLOAD utility.
The format program specifies the subschema, SEGMENT, and DMCL
As part of preparation for FASTLOAD operations, the format program calls IDMSDBLU. On the FIRST CALL ONLY, the subschema, segment name, and the DMCL name must be identified for use by IDMSDBLU.
For example,
01  PARMLIST-1.     02  SUBSCHEMA-NAME  PIC X(8)  VALUE 'EMPSS01'.     02  SEGMENT-NAME    PIC X(8)  VALUE 'EMPDEMO'.     02  DMCL-NAME       PIC X(8)  VALUE 'EMPDMCL'. CALL 'IDMSDBLU' USING PARMLIST-1.
The subschema, segment name, and DMCL must exist and be accessible at the time that the format program passes their names to IDMSDBLU, and when they are to be used by the FASTLOAD utility.
Record descriptors
Record occurrence descriptors built by the format program must be aligned on a doubleword and must contain the following fields:
Field
Usage
Size
Description
1
Char
18 bytes
The record name of the record being loaded. The format program must initialize this field before calling IDMSDBLU.
2
Char
6 bytes
Binary zeros.
3
Binary
4 bytes
Record ID of the record occurrence being loaded. The format program must initialize this field before calling IDMSDBLU.
4
Binary
4 bytes
Suggested page number for storage of the record occurrence being loaded. Responsibility for supplying the page number depends on the location mode of the record:
If the location mode is either CALC or VIA a CALC record, IDMSDBLU determines and returns the suggested page number.
If the location mode is DIRECT, the format program must initialize this field either with an actual page number or with the value -1. In the latter case, IDMSDBLU returns as the suggested page number the number of the first page in the range to which the record is assigned.
If the location mode is VIA a VIA record, the format program must process the owner record first and save the suggested page number of the owner record to initialize this field.
5
Binary
4 bytes
Binary zeros.
6
Binary
4 bytes
Serial number that uniquely identifies the record occurrence being loaded. IDMSDBLU generates and returns this value.
7
Character
4 bytes
Error status. IDMSDBLU returns error-status codes that parallel those returned by CA IDMS/DB. The format program must ensure that the returned value is 0000 before proceeding.
8
Character
Size of the largest record to be loaded
Actual record occurrence, left justified. The format program must initialize this field before calling IDMSDBLU.
After the last record is passed to IDMSDBLU, it is necessary to call IDMSDBLU one final time. This call must have the RECORD descriptor with the record ID field set to a -1 as its parameter.
Owner descriptors
Owner descriptors built by the format program must be aligned on a fullword and must contain the following fields:
Field
Usage
Size
Description
1
Character
16 bytes
Name of the set in which the record being loaded is a member, left justified. The format program must initialize this field before calling IDMSDBLU.
2
Binary
4 bytes
Serial number of the owner record. The format program must initialize this field only if the owner record has a location mode of CALC and duplicates are allowed. In this case, the format program must process the owner record first and save the returned serial number (field 4 of the record occurrence descriptor) to initialize the field.
The format program does not need to initialize this field for system-owned indexes.
3
CALC owner: as defined in the schema
1 through 256 bytes
CALC key of the owner record, left justified. The format program must initialize this field before calling IDMSDBLU.
 
Non-CALC owner: binary
4 bytes
Serial number of the owner record. In this case, the format program must process the owner record first and save the returned serial number (field 4 of the record occurrence descriptor) to initialize the field.
The format program does not need to initialize this field for system-owned indexes.
JCL Considerations
When you submit a FASTLOAD utility through the batch command facility, the JCL to execute the facility must include statements to define:
  • Files containing the areas to be loaded
  • The intermediate work files
  • Sort space
For more information about the generic JCL used to execute the batch command facility, see the section for your operating system in this section.
Intermediate work files
The following tables indicate which work files are created and read by the different utility steps depending on the use of the SORTEXIT and REUSE WORKFILE options.
Step
Input
Output
FASTLOAD: NOT sortexit mode and NOT reusing workfiles
SORT1
SYS002
SYS004
IDMSDBL2
SYS004
SYS005
SORT3
SYS005
SYS009
IDMSDBL3
SYS009
SYS010
SORT4
SYS010
SYS011
IDMSDBL4
SYS011
 
FASTLOAD: NOT sortexit mode and REUSING workfiles
SORT1
SYS002
SYS004
IDMSDBL2
SYS004
SYS005
SORT3
SYS005
SYS004
IDMSDBL3
SYS004
SYS005
SORT4
SYS005
SYS004
IDMSDBL4
SYS004
 
FASTLOAD: SORTEXIT mode and NOT reusing workfiles
SORT1/IDMSDBL2
SYS002
SYS005
SORT3/IDMSDBL3
SYS005
SYS010
SORT4/IDMSDBL4
SYS010
 
FASTLOAD: SORTEXIT mode and REUSING workfiles
SORT1/IDMSDBL2
SYS002
SYS005
SORT3/IDMSDBL3
SYS005
SYS005
SORT4/IDMSDBL4
SYS005
 
The RELDCTL file is read in steps IDMSDBL2, IDMSDBL3, and IDMSDBL4.
Work file JCL Considerations for STEP mode
FASTLOAD normally runs as a single step but runs as separate steps using the "STEP step-name" syntax. When running in step mode, input files should have dispositions to state that the file already exists, for example, "DISP=OLD."
Preserve output files on successful completion but not when the job fails, for example, "DISP=(NEW,CATLG,DELETE)."
See the "Intermediate Work File" table to determine which files are input and which files are output and when they are used.
The RELDCTL file is always input to every step.
The SYSPCH file is created by an IDMSDBL
x
step and used as input to a SORT
x
step. When used as input, it is defined as SYS001.
Work file record lengths:
  • The RELDCTL file is a fixed length file with a record length of 60 bytes.
  • The SYSPCH file is a fixed length file with a record length of 80 bytes.
  • All SYS
    xxx
    files are variable length files. The record length can vary from one step to the next, from one job to the next. Do not code an LRECL value in the JCL, just code a BLKSIZE value. A BLKSIZE value should be chosen based on the optimal size for the device being used, for example, 1/2 track if disk or 32k if tape.
Example
The following example directs the FASTLOAD utility to perform an initial load of a sample CA IDMS/DB non-SQL-defined database.
fastload;
The following command directs FASTLOAD to run all steps as a sortexit and to reuse workfiles:
fastload as sortexit reuse workfiles;
Sample Output
After successful completion of the FASTLOAD utility, the CA IDMS Batch Command Facility produces the following listing:
IDMSBCF                                              IDMS Batch Command Facility                            mm/dd/yy   PAGE 1  SET BATCH WIDTH PAGE 80; Status = 0    FASTLOAD; UT010002 BEGINNING PROCESSING FOR STEP SORT1 UT009001 IDMSDBLY RELEASE nn.n TAPE volser SORT STARTED UT009002    12 RECORDS WERE READ FROM SYS002 UT009003    12 RECORDS WERE WRITTEN TO SYS004 UT009004 IDMSDBLY RELEASE nn.n SORT COMPLETED SUCCESSFULLY UT010003 STEP SORT1    HAS COMPLETED SUCCESSFULLY UT010002 BEGINNING PROCESSING FOR STEP IDMSDBL2 UT005001 IDMSDBL2 RELEASE nn.n TAPE volser PROCESSING STARTED UT005002 DATABASE LOAD STATISTICS  DATABASE LOADED ON mm/dd/77 AT 154852 PAGES READ ...............             4 PAGES WRITTEN ............             5 PAGES REQUESTED ..........             7 CALC RCDS IN TARGET PAGE .             5 CALC RCDS OVERFLOWED .....             1 VIA RCDS IN TARGET PAGE ..             3 VIA RCDS OVERFLOWED ......             0 LINES REQUESTED BY IDMS ..            26 RCDS MADE CURRENT OF R/U .             9 CALLS TO IDMS ............            13 FRAGMENTS STORED .........             0 RECORDS RELOCATED ........             0 UT005003    12 INTERMEDIATE RECORDS WERE WRITTEN TO SYS005 UT005004    SYS005   RECORD LENGTH IS 44 UT005005    NO DATABASE ERRORS WERE ENCOUNTERED UT005006 IDMSDBL2 RELEASE nn.n PROCESSING COMPLETED UT010003 STEP IDMSDBL2 HAS COMPLETED SUCCESSFULLY UT010002 BEGINNING PROCESSING FOR STEP SORT3 UT009001 IDMSDBLY RELEASE nn.n TAPE volser SORT STARTED UT009002    12 RECORDS WERE READ FROM SYS005 UT009003    12 RECORDS WERE WRITTEN TO SYS009 UT009004 IDMSDBLY RELEASE nn.n SORT COMPLETED SUCCESSFULLY UT010003 STEP SORT3    HAS COMPLETED SUCCESSFULLY UT010002 BEGINNING PROCESSING FOR STEP IDMSDBL3 UT006001 IDMSDBL3 RELEASE nn.n TAPE volser PROCESSING STARTED UT006007    12 INTERMEDIATE RECORDS WERE READ FROM SYS009 UT006002    12 INTERMEDIATE RECORDS WERE WRITTEN TO SYS010 UT006005    NO DATABASE ERRORS WERE ENCOUNTERED UT006006 IDMSDBL3 RELEASE nn.n PROCESSING COMPLETED UT010003 STEP IDMSDBL3 HAS COMPLETED SUCCESSFULLY UT010002 BEGINNING PROCESSING FOR STEP SORT4 UT009001 IDMSDBLY RELEASE nn.n TAPE volser SORT STARTED UT009002    12 RECORDS WERE READ FROM SYS010 UT009003    12 RECORDS WERE WRITTEN TO SYS011 UT009004 IDMSDBLY RELEASE nn.n SORT COMPLETED SUCCESSFULLY UT010003 STEP SORT4    HAS COMPLETED SUCCESSFULLY UT010002 BEGINNING PROCESSING FOR STEP IDMSDBL4 UT007001 IDMSDBL4 RELEASE nn.n TAPE volser PROCESSING STARTED UT007004    NO DATABASE ERRORS WERE ENCOUNTERED UT007005    NO LOGIC ERRORS WERE ENCOUNTERED UT007002    12 RECORDS WERE READ FROM SYS011 UT007006 IDMSDBL4 RELEASE nn.n PROCESSING COMPLETED UT010003 STEP IDMSDBL4 HAS COMPLETED SUCCESSFULLY UT010001 DATABASE LOAD HAS COMPLETED SUCCESSFULLY Status = 0 AutoCommit will COMMIT transaction Command Facility ended with no errors or warnings
For more information about loading a non-SQL defined database see the
CA IDMS Database Administration Section
.