Define a non-MSI Input Format Routine (DYfdafmt)

The non-MSI input format routine is generated during the GEN GENSHELL process that is executed with testing the fdaGENIN.
micsrm140
Required roles: systems programmer, security administrator, database administrator
There are several methods that are used by
MICS
standard components for structuring input data format routines. A sample input format routine shell that is generated from the UNPGENIN module follows. Details about writing your specific FDA's input format code follows this sample.
/*-------------------------------------------------------------------*/ /* Copyright (C) 2015
Broadcom
. All rights reserved. */ /*-------------------------------------------------------------------*/ /* CHANGE | PROBLEM|MMYY|VENDOR CHANGE DESCRIPTION */ /*--------+--------+----+--------------------------------------------*/ /*-------------------------------------------------------------------*/ /* CHANGE |MMYY|USER CHANGE DESCRIPTION */ /*--------+----+-----------------------------------------------------*/ /*-------------------------------------------------------------------*/ /* ******************************************************************* * COMPONENT: UNP USER COMPONENT * * MODULE GENERATED = DYUNPFMT * * DATE GENERATED Thu, Apr 23, YYYY TIME GENERATED 6:29:59 * *
MICS
COMPONENT GENERATOR - BAS6000 * *********************************************************************/ /* ******************************************************************* * THIS MEMBER PROCESSES THE RAW INPUT DATA USED * * TO UPDATE THE
MICS
DATABASE. THE INPUT * * DATA IS READ, VERIFIED, INTERPRETED, AND * * ORGANIZED TO BUILD THE RECORDS USED TO UPDATE * * THE
MICS
DATABASE DETAIL (OR DAYS) TIMESPAN. * * * * ADDITIONALLY, THIS MEMBER ENSURES THAT THE NECESSARY * * AUDIT TRAILS ARE BUILT, DUPLICATE DATA ELIMINATION * * IS PERFORMED, AND COMPONENT CHECKPOINT UPDATING * * IS ACCOMPLISHED. * *********************************************************************/ TITLE "
MICS
UNP COMPONENT INPUT FORMAT ROUTINE"; DATA WORK.UNPUNA00 (KEEP=%UNAKEEP(TS=DETAIL )) &ADMX..CKPTDATA (LABEL='INPUT STATISTICS FOR CURRENT DATA' KEEP=%WHLKEEP(TS=DETAIL)) %USRFMT1 ;;;; /*EXIT RETURN INDICATOR SET */ %EXITCK; /* COMMON LENGTH/FORMAT AND LABELS DEFINITION */ %UNPCLEN(TS=DETAIL); /* FIELD LENGTH/FORMAT/INFORMAT DEFS BY FILE */ /* USER NCP ACTIVITY FILE */ %UNALEN(TS=DETAIL ); %UNAFMT(TS=DETAIL ); %UNALBL; LENGTH BLANKS $200.; %LET BLANKS = 1; LENGTH ORGSYSID $4 COMPT $8; RETAIN BLANKS ' ' ZERO 0 ONE 1 HUN 100 COMPT 'UNP ' ROUTINE 'DYUNPFMT' LINKSW 0 ; /* Invoke CHECKPOINT retrieval routine */ %CKRTN1; /* INPUT FILE DEFINITION */ /* The DD name specified must be INPUTSAS when fdaGENIN specifies */ /* INPUTSOURCE SMF, and INPUTfda when fdaGENIN specifies */ /* INPUTSOURCE NONSMF. In this FDA's prefix.MICS.PARMS library, */ /* a corresponding member named INPUTSAS or INPUTfda must be */ /* supplied to specify this DD name for JCL generation. */ INFILE INPUTSAS STOPOVER EOF=EOF_1ST LENGTH=RECLEN COLUMN=COL ; INPUT @; INSMF+1; COMPT='UNP'; /* USER INPUT STATEMENT DEFINITION EXIT */ /* INPUT ORGSYSID AND ENDTS, FOR THE PROPER OPERATION */ /* OF CKRTN2 - DATA RANGE CHECKING OF INPUT. */ %USRFMT2; /* COMMON RECORD PROCESS ROUTINE */ COMPROC: ; _USRSEL ; /* INVOKE USER SELECT EXIT ROUTINE IF CODED */ /*** PROCESS THE RECORD ***/ /*** PROCESS THE RECORD ***/ ENDTS=TODAY(); UNAAGTCT=8; OUTPUT UNPUNA00; /*** PROCESS THE RECORD ***/ /*** PROCESS THE RECORD ***/ /* Invoke CHECKPOINT routine to verify the data (ENDTS) as to one */ /* of the following conditions: */ /* ENDTS within CHECKPOINT range - drop duplicates */ /* ENDTS below CHECKPOINT range - drop prior data */ /* ENDTS above today's date - drop future data */ %CKRTN2; /* Invoke common fields derivation routine. */ /* YEAR, MONTH, WEEK, ZONE, DAY, DAYNAME, and HOUR */ /* are built from the ENDTS field. */ %CMNFLDS; /* Invoke SYSID lookup routine to build logical SYSID, MIPS rate, */ /* CPUMODEL, SCP level, and RMF CPU conversion factor. */ %SYSID; /* Input statements are defined for each record to be used in the */ /* update procedure along with any other edit or manipulative */ /* testing of the data. The RETURN statement must not be used. */ %USRFMT3; RETURN; /* ******************************************************************* * AN OUTPUT ROUTINE IS ESTABLISHED FOR EACH DETAIL * * OR DAYS FILE TO BE PROCESSED. * *********************************************************************/ /* OUTPUT ROUTINE FOR USER NCP ACTIVITY FILE */ OUTUNA: /* USER OUTPUT LOGIC FOR USER NPC ACTIVITY FILE */ /* PERFORM SFD TEST AND OUTPUT FILE */ IF &UNASFD THEN DO; %UNADERV; /* USER COSTING AND MANIPULATION EXIT */ %FXIT(NAME=USRSUNA); OUTPUT WORK.UNPUNA00; END; RETURN; /* OPTION DELETE ROUTINE PROVIDES METHOD TO */ /* ADJUST RECORD COUNTS FOR USER DELETED */ /* ACTION */ OPTN_DEL: RECKPT=RECKPT-1; RECODLT+1; RETURN; /* USER END OF FILE ROUTINE EOF_1ST */ /* USRFMT4 IS FOR USER END OF FILE CODE */ EOF_1ST:; %USRFMT4; GOTO EOF_RTN; /* EOF_RTN IS DEFINED IN CKRTN3. */ %CKRTN3; RUN; /* INVOKE CHECKPOINT ROUTINE TO UPDATE THE */ /* CHECKPOINT TIME-RANGES FOR THE SYSIDS AND */ /* COMPONENT UPDATED, AND RECORD THE DATA */ /* REQUIRED FOR UPDATING THE INPUT HISTORY */ /* LOG ON THIS COMPONENT'S UPDATE ACTIVITY. */ %CKRTN4; RUN; /* USER POST-FORMAT AND PRE-SORT EXIT */ %USRFMT5; /* ******************************************************************* * EACH OF THE FILES ARE SORTED TO ORDER THE DATA FOR * * ITS NEXT USE IN THE UPDATE PROCESS AND * * TO AUTOMATICALLY ELIMINATE DUPLICATE DATA * * AS DEFINED BY THE SEQUENCE MACROS. FINALLY, THE * * IN-PROCESS FILE IN THE DETAIL TIMESPAN * * IS BUILT AND THE TEMPORARY WORK FILE IS * * DELETED IF DETAIL IS ACTIVE. * *********************************************************************/ /* USER NCP ACTIVITY FILE */ PROC SORT NODUP DATA=WORK.UNPUNA00 OUT=&UNPX..UNPUNA00(%UNAFILE(TS=DETAIL,OP=FILEOPTS)); BY %UNASEQ(TS=DETAIL ); RUN; PROC DATASETS DDNAME=WORK NOLIST NOWARN; DELETE UNPUNA00; RUN; QUIT; /*USER END OF FORMAT ROUTINE EXIT */ %USRFMT6; RUN;
The points where your FDA's input format code executes are defined by SAS macro references. The macros themselves, named USRFMTn, reside in member sharedprefix.MICS.SOURCE($fdaEXIT). You put your code in this member. The macros perform the following tasks:
USRFMT1
Work file definition code
USRFMT2
Input statement definition code
USRFMT3
Input/output processing code
USRFMT4
Input end-of-file code
USRFMT5
Post-format, pre-sort code
USRFMT6
End-of-format routine code
The first four macros are referenced in the main SAS data step. The input format routine begins with a DATA step that inputs source data and outputs observations to work file versions of DETAIL timespan files you defined in sharedprefix.MICS.GENLIB(fdaGENIN).
After the main DATA step, the USRFMT5 macro is referenced. The code that sorts the work file versions of your FDA's DETAIL timespan files into the order that is needed and writes the DETAIL database follows. After the sort completes, the USRFMT6 macro is referenced.

Define Non-Database Files (USRFMT1)

The USRFMTn macros are executed as part of the main SAS DATA step in DYfdaFMT.
Use the USRFMT1 macro to add non-database oriented files to the SAS work files that are written by the input format routine. (One work file is created for each database file that exists in the DETAIL timespan.)
If you need an additional work file, code its file name and the KEEP list in USRFMT1.
Example:
%MACRO USRFMT1; WORK.MYFILE (KEEP ENDTS STARTTS INTERVLS) %MEND USRFMT1; resolves to DATA WORK.UNPUNA00 (KEEP=%UNAKEEP (TS=DETAIL)) &ADMX..CKPTDATA (LABEL='INPUT STATISTICS FOR CURRENT DATA' KEEP=%WHLKEEP (TS=DETAIL)) WORK.MYFILE (KEEP=ENDTS STARTTS INTERVLS);
And enables you to generate observations to WORK.MYFILE when necessary.

Define Input Record Validation (USRFMT2)

The USRFMT2 macro is used to do initial validation of each input record immediately after it is read. In addition, some essential data elements must be populated now.
Each record entering your USRFMT2 routine must experience one of the following outcomes:
  • The record is rejected by incrementing the WEIRDREC counter and branching as explained here.
  • The record is found to be unwanted, is counted as such, and is dropped.
  • The record is accepted, validated, and so on; and reaches the end of your USRFMT2 routine. This action allows the code that contains the _USRSEL exit point, common field determination, and observation checkpointing to be executed next. This must occur for every observation of every FDA file created.
Consider the following items and code your routine appropriately:
  • Occasionally, an input data stream contains troublesome records. One such event may involve unrecognizable data. These records must be handled by incrementing the MICS-supplied counter that is named WEIRDREC and then causing the next input record to be read:
    WEIRDREC+1; GOTO EOF_CK;
  • Another occasional occurrence is that the total length of the incoming record is not proper for the INPUT process that follows. Logic in your USRFMT2 routine should validate the input record length.
    The SAS variable RECLEN is specified in the INFILE statement in DYfdaFMT. RECLEN contains the current input record's length. Ensure that USRFMT2 checks the value of RECLEN before code that performs an INPUT of fields from the input record to avoid the error of attempting to read beyond the end of the record.
    When USRFMT2 determines that the current input record is not the expected length or is shorter than some minimum length, again the MICS-supplied counter that is named WEIRDREC must be incremented and the next input record read:
    WEIRDREC+1; GOTO EOF_CK;
  • Your USRFMT2 logic must assign the variables ENDTS and ORGSYSID to non-missing values for use by the
    MICS
    checkpoint process. If you do not assign these variables properly, the prefix.MICS.CHECKPT.DATA data set is not updated with valid ORGSYSID/COMPONENT checkpoint information. If the timestamp to be associated with the end of an event is not immediately apparent given the input data source, call
    Broadcom Support
    for assistance in determining what value you should assign to ENDTS.
    The variable ENDTS is used to compute values for common
    MICS
    variables such as MONTH, YEAR, HOUR, and DAY. These variables are computed in the %SYSID macro, which is invoked between the %USRFMT2 and %USRFMT3 macros in DYfdaFMT.
  • If your FDA is processing SMF records, you are probably interested in only one or two SMF record types. Read the record type information from the input record and examine it in USRFMT2. When a record type is dropped, increase the MICS-supplied counter that is named RECNOTUS to report these dropped records. Add the following SAS statements:
    RECNOTUS+1; GOTO EOF_CK;
    Note:
    Something similar to this action may be accomplished in the common _USRSEL input record selection exit. For more information, see General Operational Exits.
  • The MICS-supplied label COMPROC can be used when it is necessary for USRFMT2 to branch to the end of its processing. This label begins the routine that contains the _USRSEL exit point, common field determination, and observation checkpointing.
  • Do not
    issue commands that cause the DATA step to return prematurely. (RETURN and DELETE cause premature returns.) Each pass of an input record must "fall through" to the bottom of USRFMT2 or result in a GOTO to a label as described previously.
    Do not write output files in USRFMT2 because duplicate data is not eliminated until later processing steps execute
    MICS
    -supplied CKRTN2. If the DAILY job that invokes your format routine and exits is processing the same data as a previous run of the DAILY, CKRTN2 eliminates the records that have been processed before. Records output from within USRFMT2 bypasses the duplicate data checking that is performed by CKRTN2.

Define Input/Output Processing (USRFMT3)

Use the USRFMT3 macro to read fields from the input data, validate each input field, and format these fields (data elements) according to the needs of your field developed application. Thereafter, build output file observations and direct the writing of output files.
Consider the following items and code your routine appropriately:
  • Validate all data that you input. Use either standard or user-defined SAS INFORMATs to validate input data fields, or use logic in your USRFMT3 macro. A few areas to examine are:
    • Impossibly high or low values
    • Unexpected negative numbers
    • Check the data against a list of permissible values
  • Remember to use a trailing "@" (at sign) on INPUT statements that follow a subsequent INPUT statement.
  • Create output file data elements from input record fields and other processing decisions.
    The values of calculated data elements (elements whose TYPE statement is XC, C, or a variation) are calculated through the %fffDERV macro. The file's data element calculation macro is invoked by the format routine shell code directly before the file's constructed observation is output, so the code you write in USRFMT3 does not need to duplicate these operations.
  • Some input record processing might require you to read a subtype field in the front of the input record and then branch to a particular logic subroutine based on the subtype value.
  • If your input records have a segmented structure such that a single input record results in multiple observations in the database, use USRFMT2 to decode the common information from the record (including ENDTS and ORGSYSID) and use USRFMT3 to decode each segment. Use the SAS LINK statement to branch to the output processing subroutine OUTfff in DYfdaFMT and then return to input the next segment.
  • Further in the USRFMT3 logic, if it is determined that the current input record does not contribute to ANY FDA output file, this decision must be handled by branching to label OPTN_DEL so that this record is counted as 'option deleted'.
  • Input records whose length is not sufficient for the INPUT process that follows might be encountered. Logic in USRFMT3 might need to validate the (remaining) input record length more than once.
    When USRFMT3 determines that the current input record is not the expected length or not long enough, branch to label SHORTREC so that this record is counted as 'short record deleted'.
At some point, USRFMT3 reaches an output decision point. One observation suitable for output to one or more work files has been built. Its timestamp has been found to allow it into the database that is based on checkpointing algorithms, and the record's data elements have been filled in. The output branch logic is now referenced.
DYfdaFMT's shell includes code that is generated from the following template:
OUTfff: IF &fffSFD THEN DO; %fffDERV; /* USER COSTING AND MANIPULATION EXIT */ %FXIT(NAME=USRSfff); OUTPUT WORK.iiifff00; END; RETURN;
DYfdaFMT contains one of these routines for each file that appears in its main DATA step. To write an observation to a file, reference this routine using the code in USRFMT3. Choose one of the following methods to reference this code:
  • Link to it if you have further processing to perform on the input record. If you have segmented records and each segment is an output observation or if you are building two different records in the database for each input record, linking is the best technique to use. The statement format is as follows:
    LINK OUTfff;
  • Branch to it if no further processing is to be done on the current input record. This action causes the observation that you have built to be put to a work file and the format routine to return for another record. The statement format is as follows:
    GOTO OUTfff;
This illustration shows two equivalent USRFMT3 macros.
%MACRO USRFMT3; detailed processing IF condition-favorable THEN LINK OUTfff; ELSE GOTO OPTN_DEL; %MEND USRFMT3; %MACRO USRFMT3; detailed processing IF condition-unfavorable THEN GOTO OPTN_DEL; %MEND USRFMT3;
Remember, if you do not select to use an input record to build an observation in your database, code GOTO OPTN_DEL. OPTN_DEL adjusts the input records processed count for the audit trail and causes the format routine to get the next input record, if any.

Define Additional Process Code (USRFMT4-6)

The USRFMT4, USRFMT5, and USRFMT6 macros exist to support site-specific requirements. Use them if you need to perform a function that is not directly related to translating input record fields to output observations.
USRFMT4 - Input end-of-file code
USRFMT4 is referenced in DYfdaFMT's main SAS DATA step. The code that you put in USRFMT4 is executed at the time an end of file on the input data set has been detected by SAS and the last input observation has been processed.
USRFMT4 code should contain valid SAS logic suitable for inclusion in a DATA step and must not begin with a label. This macro is typically used to produce a run status report. A run status report provides the number of records of each type and subtype that were processed, records validity check counts, and so on.
USRFMT5 - Post-format, pre-sort code
After the main DATA step, USRFMT5 is referenced. The reference is followed by code that sorts the work file versions of the component's files into the prescribed order onto the DETAIL database.
Because this macro is referenced outside of an existing SAS DATA or PROC step, it must contain DATA or PROC statements of its own.
It can be used to analyze work files through SAS PROCs, such as PROC CONTENTS or PROC FREQ, or it can:
  • Contain DATA statements to process work files further,
  • Incorporate more control information, or
  • Produce more database files from the original work files that are created in the main DATA step.
On DATA statements, remember to use macros of the form.
... &iiit..iiifffnn (%fffFILE(OP=FILEOPTS,TS=timespanname)) ...
And on PROC statements, remember to use macros of the form.
PROC SORT DATA=WORK.iiifffnn OUT=&iiit.iiifffnn (%fffFILE(OP=FILEOPTS,TS=timespanname));
When writing DETAIL timespan files to the database.
USRFMT6 - End of format routine code
Like the USRFMT5 macro, the USRFMT6 macro is referenced outside of an existing SAS DATA or PROC step, so it must contain DATA or PROC statements of its own.
It can be used to analyze work files through SAS PROCs, such as PROC CONTENTS or PROC FREQ, or it can:
  • Contain DATA statements to process work files further,
  • Incorporate more control information, or
  • Produce more database files from the original work files that are created in the main DATA step.
On DATA statements, remember to use macros of the form.
... &iiit..iiifffnn (%fffFILE(OP=FILEOPTS,TS=timespanname)) ...
And on PROC statements, remember to use macros of the form.
PROC SORT DATA=WORK.iiifffnn OUT=&iiit.iiifffnn (%fffFILE(OP=FILEOPTS,TS=timespanname));
When writing DETAIL timespan files to the database.

Validate Your Code

Two kinds of tests are needed for your DYfdaFMT routine. First is a syntax and operational test and second is a content test, which checks the actual content of data on the database files produced by DYfdaFMT.
The best way to test the syntax and operation of DYfdaFMT is to install the FDA into a test database unit, preferably one constructed solely for new product development. The advantages of this method follow:
  • The checkpointing routines operate exactly as they do in a normal production environment.
  • You have full control over allocation parameters in your test unit.
  • Database data sets, including CHECKPT.DATA and the DETAIL, DAYS, WEEKS, MONTHS, and YEARS timespans, are permanent data sets, enabling you to interrogate the FDA's files in depth.
  • The test unit's database files are available to the
    MICS
    online database access tools, including MSAS (SAS/DM with
    MICS
    macros available) and MICF (if your site permits it through the Test
    MICS
    user profile parameter). MICF knows about the field-developed application and is able to generate inquiries into the new product's data.
To test your DYfdaFMT routine in a test database unit:
  • Change sharedprefix.MICS.GENLIB(fdaGENIN) from GEN GENSHELL to GEN GENFILES.
  • Install the field-developed application in a test database unit.
    Use the checklist in the Adding a Product to an Existing Database Unit section if you have an existing test database unit. Use the checklist in the Adding a New Product to a New Database Unit section if you need to create a test database unit.
    Running the first daily might require several iterations. You can delete the database data sets at any time before the FDA enters a production status and can reallocate them for the next test using the standard ALLOCATE or RESTORE jobs. Changes to file structures require resubmitting the sharedprefix.MICS.CNTL(fdaCGEN) job to take effect.
    In reviewing the output of the DAILY job, verify that:
    • Data elements contain reasonable values o data elements are summarized correctly o exception conditions are identified o management objective report is formatted correctly o checkpoint data set is valid
      In reviewing the output of the WEEKLY and MONTHLY jobs, verify that:
    • Data elements contain reasonable values o archive files contain valid data o management objective report is formatted correctly
    • In reviewing the output of the YEARLY job, verify that:
    • The MICSLOG and SAS Log do not contain error messages
    In reviewing the output of the fdaCHECK job:
    • Verify that the file keys for your higher timespan files are constructed correctly.
    • Consider running the job in the fdaCNTS member in sharedprefix.MICS.SOURCE. This job performs a PROC CONTENTS on each cycle of each file in your FDA. Be sure that the MONTHLY job has been run before you run the fdaCNTS job.
    • Consider running the job in the fdaDATAS member in sharedprefix.MICS.SOURCE. This job performs a PROC DATASETS on each timespan in the unit that contains your FDA. Be sure that the MONTHLY job has been run before you run the fdaDATAS job.