Bulk Data Export

As a systems administrator or architect, you want to export polled rate data from Data Aggregator as a continuous CSV export to your own reporting tool for analysis and custom reporting purposes.
As a systems administrator or architect, you want to export polled rate data from the data aggregator as a continuous CSV export to your own reporting tool for analysis and custom reporting purposes.
You cannot bulk data export in fault-tolerant data aggregator environments.
To export specific data, or to run a one-time export, the OpenAPI is a more flexible. For more information, see OpenAPI.
You can data export using the data aggregator. These are done at the frequency of polling rate. When you start data export, the poll responses are written to CSV files. You can configure inclusive lists for metric families so that you only export the data you need.
The CSV files contain rate data and internal names, which are mapped to readable display names. If exported metric families include suppressed metrics, the CSV shows a null value for those metrics.
The size of CSV output files that are generated can come close to or can exceed the following limits:
Environment
Limit per hour (uncompressed)
Large scale
50 GB
Medium scale
25 GB
Small scale
5 GB
To avoid exhausting the available disk space, regularly process the output CSV files. For example, copy the files to another system and remove the files from the data aggregator host.
For performance reasons, create a separate disk partition on the data aggregator system. Select this partition for the output of the CSV rate data.
To export data, follow this process:
Configure the Export File Output Options
Configure the output file options using the
streamexport.csvoutwriter.cfg
file. This configuration file is copied to the directory during the data aggregator installation.
Follow these steps:
  1. Change to the
    <installation_directory>
    /IMDataAggregator/
    apache-karaf-*
    /etc
    directory.
    • installation_directory
      The installation directory of the data aggregator.
      Default:
      /opt
    • apache-karaf-*
      The installation directory for Apache Karaf.
      Example:
      apache-karaf-4.2.6
  2. Edit the
    streamexport.csvoutwriter.cfg
    file as needed:
    • output.filenameExtension
      Specifies the extension or suffix of the CSV files.
    • output.csvFileDelimiter
      Specifies the column delimiter that is used in the output CSV file. For example, if the bundle is started and this parameter is changed, a new file is written immediately using this new column delimiter.
    • output.filenameLocationPath
      Specifies the file path and the prefix of the output file name. (The output file also consists of the date and time.)
      This file path must be on a different partition than the data aggregator installation. If
      DX NetOps Performance Management
      was installed with a sudo user, change the ownership of the directory to enable access to the sudo user.
      The syntax of the complete file name is:
      output.
      filenameLocationPath=
      <
      DC_host
      >_
      yyyy
      -
      MM
      -
      dd
      -T
      hh
      -
      mm
      -
      sec
      -
      ms
      .
      output.filenameextension
      Example:
      The file is written on April 2, 2013 at 8:42:04 a.m. and 123 ms. The Data Collector hostname is server.abc.com, and the following parameters are configured:
      output.filenameLocationPath=
      /opt/export_data/mydata
      output.filenameExtension=
      .csv
      The file name is:
      mydata_server.abc.com_2013-04-02T08-42-04-123.csv
      You can configure files with an absolute name, such as /myOutputDir/mydata. If the parent folders in the absolute path do not exist, the folders are created.
    • output.filesize
      Specifies the file size in bytes using a valid integer greater than 0. If the file size is exceeded, a new output file gets written.
      If the value is -1, this parameter is ignored and
      all
      the data is written into a single file (infinite).
    • output.duration
      Specifies the number of minutes using a valid integer greater than 1. If a file is older than
      x
      minutes, then a new file is written.
      If the value is 1, then this parameter is ignored. If the value is -1, then
      all
      the data is written into a single file (infinite).
    The
    output.filesize
    and
    output.duration
    parameters affect each other. If
    output.filesize
    is exceeded
    or
    the file is older than
    output.duration
    , then a new file is written.
    Example:
    streamexport.csvoutwriter.cfg
    output.filesize=100000000
    output.filenameLocationPath=/opt/data_export/ratedata_
    feature.enabled=on
    output.duration=60
    output.csvFileDelimiter=,
    output.filenameExtension=.csv
  3. Save your changes.
Configure an Inclusive List of Metric Families
You can specify the metric families whose data you do want to export by configuring an inclusive list (whitelist)
.
Data is collected only for the listed metric families. This configuration is optional.
The following files pertain to scoping by metric family:
  • streamexport.allMetricFamilies.out
    This file is auto-generated at the start-up of the Data Aggregator system. The file includes all the available metric families.
    This file is periodically updated to include new metric families that are added to the system.
    Location:
    <installation_directory>
    /IMDataAggregator/
    apache-karaf-*
    /etc
    directory
    • installation_directory
      The installation directory of the data aggregator.
      Default:
      /opt
    • apache-karaf-*
      The installation directory for Apache Karaf.
      Example:
      apache-karaf-4.2.6
    This file has the following format:
    metricFamilyInternalName
    =
    metricFamilyDisplayNameinEnglish
    Example:
    #Metric Family Name List for Customer Reference
    #Mon Jun 17 11:18:50 EDT 2013
    normalizedmemoryinfo=Memory
    normalizedavailabilityinfo=Availability
    normalizedcpuinfo=CPU
    normalizedportinfo=Interface
  • streamexport.metricFamilyWhiteList.cfg
    Specifies all the metric families to export.
    Location:
    <installation_directory>
    /IMDataAggregator/
    apache-karaf-*
    /etc
    directory.
    • installation_directory
      The installation directory of the data aggregator.
      Default:
      /opt
    • apache-karaf-*
      The installation directory for Apache Karaf.
      Example:
      apache-karaf-4.2.6
    This file has the following format:
    whitelist.
    number
    =
    metricFamilyInternalName
    Example:
    #This is the whitelist file for the metric families.
    #Mon Jun 10 17:25:46 EDT 2013
    feature.enabled=on
    whitelist.1 = NormalizedMemoryInfo
    whitelist.2 = NormalizedPortInfo
Follow these steps:
  1. On the local Data Aggregator system, open
    streamexport.allMetricFamilies.out
    and
    streamexport.metricFamilyWhiteList.cfg
    using a text editor.
  2. Copy one or more metric family internal names from the
    streamexport.allMetricFamilies.out
    file. Paste the copied text into the
    streamexport.metricFamilyWhiteList.cfg
    file.
    Example:
    whitelist.1=<paste here>
    whitelist.2=<paste here next metric family name>
    Set feature.enabled=on
  3. Save your changes to the
    streamexport.metricFamilyWhiteList.cfg
    file.
Export Extra Columns for Components
To include more information about component items, configure the export to include the ifAlias and ifDescr columns.
If you configure this option after you start the data export, the output file might contain mixed data. Lines that are exported before you add the columns do not include the extra columns. Lines that are exported after the change include the columns. To avoid a mixed file, stop the Data Aggregator before you change the configuration.
Follow these steps:
  1. Edit the
    etc/streamexport.exportInfoResolver.cfg
    file.
  2. Set the
    enableInterfaceOutput=true
    parameter.
  3. Save the changes.
    The data aggregator adds the
    ifAlias
    and
    ifDescr
    columns to the output file. This change occurs immediately.
Export the Rate Data
Start the Data Aggregator Rate
dataexport
Service (export feature) using Data Aggregator REST web services. You can use any REST client tool or an HTTP tool that can send requests and can get responses. This example uses a REST client.
Follow these steps:
  1. Set up a REST client with a connection to the data aggregator server.
  2. Set the REST Content-type to application/xml.
  3. Enter the following URL for the
    dataexport
    Data Aggregator REST web service:
    GET http://
    da_host
    :8581/rest/dataexport/
  4. Take note of the ID of the data export profile you want to modify. By default, there is only one profile.
  5. Enter text in the
    Body
    tab of the HTTP Request pane. At a minimum, set
    Enabled
    to
    true
    .
    For example:
    <DataExportInfo version="1.0.0">
    <Enabled>true</Enabled>
    </DataExportInfo>
  6. Review other options that can be set at the following URL:
    http://
    da_host
    :8581/rest/dataexport/xsd/get.xsd
  7. Save and start the rate data export feature by entering the following URL:
    PUT http://
    da_host
    :8581/rest/dataexport/
    id
    • id
      Specifies the ID of the
      dataexport
      service.
      The following URL retrieves the ID of the
      dataexport
      service:
      GET http://
      da_host
      :8581/rest/dataexport
  8. To verify that your changes took effect, enter the following URL:
    GET http://
    da_host
    :8581/rest/dataexport/
    id
    The
    dataexport
    service starts automatically and temporary export files are created.
    When the export file is ready, the exporter automatically renames it to the previously configured file extension, such as .csv.
    You do not need to restart the services for a newly written file.
    After the data is exported, copy the data to your other system using the method required by that other system.
Stop the Bulk Rate Data Export
To stop the rate data export at any time, set the
Enabled
attribute to
false
.
Example:
<DataExportInfo version="1.0.0">
<Enabled>false</Enabled>
</DataExportInfo>