Bulk Data Export

As a systems administrator or architect, you want to export polled rate data from Data Aggregator as a continuous CSV export to your own reporting tool for analysis and custom reporting purposes.
capm300
As a systems administrator or architect, you want to export polled rate data from Data Aggregator as a continuous CSV export to your own reporting tool for analysis and custom reporting purposes.
The bulk data export feature is unsupported in environments with fault tolerant Data Aggregators.
To export specific data, or to run a one-time export, the OpenAPI is a more flexible. For more information, see OpenAPI.
Data Aggregator provides data export capabilities that are done at the frequency of polling rate. When you start data export, all poll responses are written to CSV files. You can configure inclusive lists for metric families so that you only export the data you need.
The CSV files contain rate data and internal names, which are mapped to readable display names. If exported metric families include suppressed metrics, the CSV shows a null value for those metrics.
The size of CSV output files that are generated can come close to or can exceed the following limits:
Environment
Limit per hour (uncompressed)
Large scale
50 GB
Medium scale
25 GB
Small scale
5 GB
To avoid exhausting the available disk space, regularly process the output CSV files. For example, copy the files to another system and remove the files from the Data Aggregator host.
For performance reasons, create a separate disk partition on the Data Aggregator system. Select this partition for the output of the CSV rate data.
To export data, follow this process:
Configure the Export File Output Options
Configure the output file options using the
streamexport.csvoutwriter.cfg
file. This configuration file is copied to the directory during the Data Aggregator installation.
Example of the path:
/opt/IMDataAggregator/apache-karaf-2.4.3/etc
Follow these steps:
  1. Change the directory to /opt/IMDataAggregator/apache-karaf-2.4.3/etc.
  2. Edit the
    streamexport.csvoutwriter.cfg
    file as needed:
    • output.filenameExtension
      Specifies the extension or suffix of the CSV files.
    • output.csvFileDelimiter
      Specifies the column delimiter that is used in the output CSV file. For example, if the bundle is started and this parameter is changed, a new file is written immediately using this new column delimiter.
    • output.filenameLocationPath
      Specifies the file path and the prefix of the output file name. (The output file also consists of the date and time.)
      This file path must be on a different partition than the Data Aggregator installation. If
      CA Performance Management
      was installed with a sudo user, change the ownership of the directory to enable access to the sudo user.
      The syntax of the complete file name is:
      output.
      filenameLocationPath=
      <
      DC_host
      >_
      yyyy
      -
      MM
      -
      dd
      -T
      hh
      -
      mm
      -
      sec
      -
      ms
      .
      output.filenameextension
      Example:
      The file is written on April 2, 2013 at 8:42:04 a.m. and 123 ms. The Data Collector hostname is server.abc.com, and the following parameters are configured:
      output.filenameLocationPath=
      /opt/export_data/mydata
      output.filenameExtension=
      .csv
      The file name is:
      mydata_server.abc.com_2013-04-02T08-42-04-123.csv
      You can configure files with an absolute name, such as /myOutputDir/mydata. If the parent folders in the absolute path do not exist, the folders are created.
    • output.filesize
      Specifies the file size in bytes using a valid integer greater than 0. If the file size is exceeded, a new output file gets written.
      If the value is -1, this parameter is ignored and
      all
      the data is written into a single file (infinite).
    • output.duration
      Specifies the number of minutes using a valid integer greater than 1. If a file is older than
      x
      minutes, then a new file is written.
      If the value is 1, then this parameter is ignored. If the value is -1, then
      all
      the data is written into a single file (infinite).
    The output.filesize and output.duration parameters affect each other. If the output.filesize is exceeded
    or
    the file is older than output.duration, then a new file is written.
    Example: streamexport.csvoutwriter.cfg
    output.filesize=100000000
    output.filenameLocationPath=/opt/data_export/ratedata_
    feature.enabled=on
    output.duration=60
    output.csvFileDelimiter=,
    output.filenameExtension=.csv
  3. Save your changes.
Configure an Inclusive List of Metric Families
You can specify the metric families whose data you do want to export by configuring an inclusive list (whitelist)
.
Data is collected only for the listed metric families. This configuration is optional.
Two files pertain to scoping by metric family:
  • streamexport.allMetricFamilies.out
    This file is auto-generated at the start-up of the Data Aggregator system. The file includes all the available metric families.
    This file is periodically updated to include new metric families that are added to the system.
    Location:
    $KARAF_HOME/etc folder
    This file has the following format:
    metricFamilyInternalName
    =
    metricFamilyDisplayNameinEnglish
    Example:
    #Metric Family Name List for Customer Reference
    #Mon Jun 17 11:18:50 EDT 2013
    normalizedmemoryinfo=Memory
    normalizedavailabilityinfo=Availability
    normalizedcpuinfo=CPU
    normalizedportinfo=Interface
  • streamexport.metricFamilyWhiteList.cfg
    Specifies all the metric families to export.
    Location:
    $KARAF_HOME/etc folder
    This file has the following format:
    whitelist.
    number
    =
    metricFamilyInternalName
    Example:
    #This is the whitelist file for the metric families.
    #Mon Jun 10 17:25:46 EDT 2013
    feature.enabled=on
    whitelist.1 = NormalizedMemoryInfo
    whitelist.2 = NormalizedPortInfo
Follow these steps:
  1. On the local Data Aggregator system, open
    streamexport.allMetricFamilies.out
    using a text editor.
  2. On the local Data Aggregator system, also open
    streamexport.metricFamilyWhiteList.cfg
    using a text editor.
  3. Copy one or more metric family internal names from the
    streamexport.allMetricFamilies.out
    file. Paste the copied text into the
    streamexport.metricFamilyWhiteList.cfg
    file.
    Example:
    whitelist.1=<paste here>
    whitelist.2=<paste here next metric family name>
    Set feature.enabled=on
  4. Save your changes to the
    streamexport.metricFamilyWhiteList.cfg
    file.
Export Extra Columns for Components
To include more information about component items, configure the export to include the ifAlias and ifDescr columns.
If you configure this option after you start the data export, the output file might contain mixed data. Lines that are exported before you add the columns do not include the extra columns. Lines that are exported after the change include the columns. To avoid a mixed file, stop the Data Aggregator before you change the configuration.
Follow these steps:
  1. Edit the following file:
    etc/streamexport.exportInfoResolver.cfg
  2. Set the following parameter:
    enableInterfaceOutput=true
  3. Save the changes.
    The Data Aggregator adds the ifAlias and ifDescr columns to the output file. This change occurs immediately.
Start the Rate Data Export Feature
Start the Data Aggregator Rate Data Export Service (export feature) using Data Aggregator REST web services. You can use any REST client tool or an HTTP tool that can send requests and can get responses. For this scenario, we use a REST client.
Follow these steps:
  1. Set up a REST client with a connection to the Data Aggregator server.
  2. Set the REST Content-type to application/xml.
  3. Enter the following URL:
    GET http://
    da_host
    :8581/rest/dataexport/
  4. Take note of the id of the data export profile you want to modify. By default, there is only one profile.
  5. Enter text in the Body tab of the HTTP Request pane. At a minimum, set Enabled to true. For example:
    <DataExportInfo version="1.0.0">
    <Enabled>true</Enabled>
    </DataExportInfo>
  6. Review other options that can be set at the following URL:
    http://
    da_host
    :8581/rest/dataexport/xsd/get.xsd
  7. Save and start the rate data export feature by entering the following URL:
    PUT http://
    da_host
    :8581/rest/dataexport/
    id
    • id
      Specifies the ID of the Data Export service.
      The following URL retrieves the ID of the Data Export service:
      GET http://
      da_host
      :8581/rest/dataexport
  8. To verify that your changes took effect, enter the following URL:
    GET http://
    da_host
    :8581/rest/dataexport/
    id
    The data export starts automatically and temporary export files are created.
    When the export file is ready, the exporter automatically renames it to the previously configured file extension, such as .csv.
    You do not need to restart the services for a newly written file.
    After the data is exported, copy the data to your other system using the method required by that other system.
Stop the Bulk Data Export
To stop the data rate export at any time, set the Enabled attribute to
false
.
Example:
<DataExportInfo version="1.0.0">
<Enabled>false</Enabled>
</DataExportInfo>