Manage Data Source Providers

Data source providers are sets of JavaScript code that implement the extensible data source framework. Add one when you require a connection to a data source that 
Layer7 Live API Creator
 does not support.
lac53
As a system administrator (
sa
), you can add data source providers when API developers require connections to data sources that 
Layer7 Live API Creator
 does not support. Dats source providers are pluggable persistence (read/write) mechanisms. They expose system-level objects. These objects can leverage 
Layer7 Live API Creator
 services, such as resources, logic, filters, and sorts in your API. Because 
Layer7 Live API Creator
 exposes data source providers at the system level, all TeamSpaces can use them.
Data source providers are sets of JavaScript code that implement the extensible data source framework. This framework includes interfaces where you can create functions that establish database connectivity, discover database objects, define the schema structure, implement required operations, and expose these operations in 
Layer7 Live API Creator
.
In this article:
The Lifecycle of a Data Source Provider
The following diagram shows the lifecycle for adding a data source provider and adding a data source that uses the data source provider:
DataSourceProviders2Col
API developers use the following process to add a data source provider, and then create a data source that uses the data source provider:
  1. In their local development environment, logs in as a system administrator (
    sa
    ) user, and then adds a data source provider.
  2. From an existing API, creates a data source that uses this data source provider by completing the following:
    1. Clicks 
      Add
      , and then in the 
      Create data source window
      , selects 
      Data source provider
       from the 
      Create data source using
       drop-down.
    2. Selects the 
      data source x
       data source provider from the 
      Data source provider
       drop-down. The drop-down displays the data source providers that use the extensible data source framework.
      The 
      Connection
       tab for the data source displays.
      For more information about how to create a data source, see Database Connectivity.
    3. Enters the parameter values for the data source. The parameters that are displayed are specific to the data source provider.
      The 
      ConfigInfo
       function specifies the configuration parameters that are listed on this tab.
    4. Activates the data source by selecting the 
      Active
       checkbox, and then clicks 
      Save
      .
Layer7 Live API Creator
 does the following:
  1. Tests and verifies that the data source is configured correctly and ready to service data requests by calling the 
    testConnection
     function.
  2. If the connection is successful, configures the data source by calling the 
    open
     function.
  3. Retrieves the definition of the structure of the underlying data source by calling the 
    structure
     function. 
    Layer7 Live API Creator
     calls this function when you reload the schema.
Layer7 Live API Creator
 calls the data source provider whenever it must interaction with the data.
Data Source Provider Examples
Layer7 Live API Creator
 includes the following data source provider examples that you can use as a starting point to creating your own data source provider:
  • JDBCExample:
     This example demonstrates connecting to a Derby SQL database. It can connect to the Derby demo, finance, or the sample databases that the single-user demonstration package includes.
  • MongoDB:
     This example demonstrates connecting to a NoSQL database. Use it to connect to MongoDB as a data source. It scans the first row of each MongoDB collection to guess the data model structure. It requires non-null records and values to determine an accurate model. You can also manually create this model (see the 
    MongoDBAudit
     data source provider example’s 
    structure_code.js
     file).
  • MongoDBAudit:
     This example demonstrates connecting to a NoSQL database for auditing user transaction details. It is a specialized version of the MongoDB data source provider example and is used in the audit user transaction details functionality. The predefined schema writes audit records to a collection in a MongoDB audit database.
Add a Data Source Provider
You can add a data source provider using API Creator or by adding the data source provider definition to the 
data_source_providers
 directory within the 
system
 directory of the admin repository. The following procedure explains how to add a data source provider using API Creator.
For more information:
Prerequisites:
 
  • You understand the API that includes a data source that uses this data source provider (Java in most cases) and how to interact with it from the JavaScript code.
  • You have added the JDBC driver (the JAR file) that you want to implement and call from the data source provider's JavaScript code. If you are writing to MongoDB, you have added the MongoDB client JAR.
    For more information about the drivers that 
    Layer7 Live API Creator
     supports, see Installation Requirements and Supported Platforms.
  • You have installed the JDBC driver or the JAR files and restarted API Server.
    • If you have installed the self-contained, single-user version of 
      Layer7 Live API Creator
       that is based on Jetty, copy the JDBC driver into the 
      %JETTY_HOME%/caliveapicreator/lib/ext
       directory.
    • If you have installed 
      Layer7 Live API Creator
       on Tomcat, copy the JDBC driver (JAR file) into the 
      %{CATALINA_HOME}/lib
       directory.
  • You have system administrator (
    sa
    ) user privileges in 
    Layer7 Live API Creator
    .
Follow these steps:
  1. Log in to API Creator as a system administrator (
    sa
    ) user.
  2. Click the 
    Data Source Providers
     tab.
  3. Select one of the following data source provider examples that you want to use as a starting point to creating your own data source provider, and then click 
    Duplicate
    .
    • JDBCExample:
       Select this example if you are adding a data source provider by connecting to a SQL database.
    • MongoDB:
       Select this example if you are adding a data source provider by connecting to a NoSQL database.
    • MongoDBAudit:
       Select this example if you want to customize your API to audit user transaction details.
      For more information about how to customize your API to audit user transactions, see Customize your API to Audit User Transactions.
  4. Edit the data source provider name, add a comment for the data source provider, and then save your changes.
  5. Modify the JavaScript code for each group of data source provider functions.
Modify the JavaScript Code for Each Group of Data Source Provider Functions
Modify the JavaScript code for each group of data source provider functions to accommodate the implementation of the data source provider that you just added. The functions expose capabilities. For example, allow only some queries or accept only certain data types. You expose capabilities by way of implementing functions. A minimal data source provider can consist of a few functions, whereas a data source provider that implements the full interface can require many hundreds of lines of code
The functions for the data source provider are grouped by behaviors that 
Layer7 Live API Creator
 implements for the data source provider. The functions that you must implement are noted as required in the following tables. You must define the JavaScript code for the required functions and the JavaScript code for at least one optional function. For example, if you want the ability to create objects, the data source provider must implement the insert function.
While defining the JavaScript code for each group of data source provider functions, you can do the following:
  • Test by calling the data source endpoints in the REST Lab (in another browser window). This process can be iterative.
    For more information about the REST Lab, see Test your API Using the REST Lab.
  • Debug by printing out debug statements to the Java console.
Initialization
Initialize the data source providers that you want to use. This process includes adding JavaScript code for configuring the user interface to get data source connection parameters, configuring a connection using these parameters, getting the structure of the data source, closing a connection, and testing a connection. Navigate through the sections for initializing your data source provider by clicking the section indicators.
Function Name
File name in admin repository
Description
configInfo
config_info_code.js
Specifies the configuration parameters that are required when an API developer creates a data source that uses this data source provider.
The configuration definition is a JavaScript object that contains the following:
  • ui_config:
     The JavaScript array that represents the user interface that is required to procure details that this data source provider requires to establish a connection.
  • env:
     The JavaScript object that represents the environment containing properties that are global to the code steps in this data source provider.
  • options:
     The JavaScript object that represents additional capabilities that defines the behavior of this data source provider.
Layer7 Live API Creator
 returns the configuration code as the 
configInfo
 Java Script object in other initialization code types.
The configuration parameters that you define in this code are displayed in the 
preview
 function section.
Required:
 Yes
open
open_code.js
This code creates a connection to a database. 
Layer7 Live API Creator
 hands the data source provider the configuration values that the API developer enters, and the function prepares for interacting with the data source. For most data source providers, this means opening a connection object.
This code creates and returns an open connection object that 
Layer7 Live API Creator
 passes to each of the data retrieval and data manipulation JavaScript. This can be a JDBC connection or a global object that contains values.
Required:
 Yes
structure
structure_code.js
Layer7 Live API Creator
 executes this code whenever it requires a definition of the structure of the underlying data source.
This code creates the model that 
Layer7 Live API Creator
 uses to define the entities, attributes (including the attribute names and types), keys, relationships, functions and procedures (if applicable).
Required:
 Yes
close
close_code.js
Layer7 Live API Creator
 executes this code whenever it must close a connection. Some data sources are stateless and, therefore, do not need to be "cleaned up".
Required:
 No
testConnection
test_connection_code.js
Layer7 Live API Creator
 executes this code whenever the API user tests the data source (clicks 
Test Connection
). This code returns an object reflecting the status of the connection. If the status is not OK, then the data source is considered to be in a bad state.
Testing the connection to the data source (by clicking 
Test Connection
 in the 
preview
 function section) uses the parameters that you define in this code.
Required:
 Yes
You can preview the required configuration parameters when an API developer connects their API to a data source that uses this data source provider by clicking the 
preview
 section indicator.
Data Retrieval
Define this data source provider to query the data in a data source using a unique identifier for the data or by plainly executing a query. 
Layer7 Live API Creator
 translates the structure of the connected data store into a schema so that an API user can perform a REST GET call.
Navigate through the sections for configuring the data retrieval by clicking the section indicators.
Function Name
File name in admin repository
Description
byKey
by_key_code.js
Layer7 Live API Creator
 executes this code whenever it requires an object and must find it using its primary key as it pertains to the underlying data source. The code for this section is optional. If you do not provide code in this section, 
Layer7 Live API Creator
 uses the query code.
Required:
 No
query
query_code.js
Layer7 Live API Creator
 executes this code whenever it must run a query against the underlying data source.
Includes the definition to support for filters, sorts, and joins.
Required:
 No
attributeValue
attribute_value_code.js
Layer7 Live API Creator
 executes this code for GET requests to fetch binary or clob/blob data associated with an attribute in an entity.
This function uses the 
/data/
 fragment instead of 
/rest/
 on the URL.
For more information about how to use binary data, see Use Binary Data.
Required:
 No
Data Manipulation
You can configure this data source provider to change the data in a data source by way of insertion, update, or deletion of data. 
Layer7 Live API Creator
 translates the structure of the connected data store into a schema so that an API user can perform a REST operation on the data, such as POST, PUT, and DELETE, by way of REST URLs.
Navigate through the sections for configuring the data manipulation by clicking the section indicators.
Function Name
File name in admin repository
Description
insert
insert_code.js
Layer7 Live API Creator
 executes this code whenever it must insert objects in the data source.
Complete this code if your data source is read/write.
Required:
 No
update
update_code.js
Layer7 Live API Creator
 executes this code whenever it must update objects in the data source.
Complete this code if the underlying data source supports updates.
Required:
 No
delete
delete_code.js
Layer7 Live API Creator
 executes this code whenever it must delete objects in the underlying data source.
Complete this code if the underlying data source supports deletes.
Required:
 No
invoke
invoke_code.js
Layer7 Live API Creator
 executes this code whenever it processes a request for a stored procedure or function.
Complete this code if the underlying database includes facilities that are exposed as stored procedures or functions and you want to expose them.
Required:
 No
Data Transaction
You can configure this data source provider to support transactions like SQL databases. Define JavaScript code that dictates how the data source provider performs commits or rollbacks on opened connections.
Navigate through the sections for configuring the data transaction by clicking the section indicators.
Function Name
File name in admin repository
Description
commit
commit_code.js
Layer7 Live API Creator
 executes this code whenever it must flush to disk the data changes so far to this data source in the request.
Complete the code for this section if the underlying data source supports the concept of commit.
Required:
 No
rollback
rollback_code.js
Layer7 Live API Creator
 executes this code whenever it must to undo the data changes so far to this data source in the request.
Complete the code for this section if the underlying data source supports the concept of rollback.
Required:
 No
Other Data Source Provider Examples
The following data source provider examples are additional examples that are available in gitHub:
  • PetStore:
     This OpenAPI (formerly Swagger) data source provider example demonstrates the ability to manually create a model for each of the objects (pets, orders, and users) using procedures to define specific type of endpoint functions (for example, pets/findByStatus).
  • InMemoryExample:
     This data source provider example demonstrates how to manually create a model (or structure) that includes a couple of tables and a relationship.
  • MySQL:
     This data source provider example is a complete JDBC example for MySQL.
For more information about these data source provider examples and other examples, see GitHub.
Data Source Provider Limitations
The following are known limitations for data sources that use the data source providers:
  • You can filter the results of endpoints only using system filters (
    sysfilter
    ).
    For more information about system filters, see Structured Filters.
  • You can pass sorting information only using system sorts (
    sysorder
    ).
    For more information about system sorts, see Structured Sorts.
  • If you have defined system filters on subresources that are connected to data sources that use a data source provider, you can apply filters 
    only
     using the following syntax:
    sysfilter.<resourcename>.<subresourcename>
    For example:
    sysfilter.CustomerBusObject.Orders=equal(paid:true)
  • If your API is connected to a NoSQL database that uses the 
    MongoDB
     data source providers:
    • Layer7 Live API Creator
       discovers the columns within MongoDB collection using the first row and returns the schema. It assumes all rows have identical attribute names and data types.
    • When creating table-based resources, 
      Layer7 Live API Creator
       attempts to discover the columns within the MongoDB collection.
  • If your data source uses a data source provider that is connected by way of JDBC (for example, the 
    JDBCExample
     data source provider example), you cannot use FreeSQL resources.
  • If your API is connected to two data sources and one of them uses a data source provider, some rules, such as sums and counts, might not work when transactions have to roll back.
Next Steps
Now that you have added the data source provider, you can do the following:
  • Add a data source that uses this data source provider to your API.
    For more information, see Database Connectivity.
  • Iterate and define the JavaScript code for the functions in the data source provider until they fulfill your implementation requirements.
  • Deploy the data source provider as part of deploying the API Server-level configuration files.
    For more information about this step in the DevOps workflow, see Team Development.
  • (Recommended) Include a logger entry for the data source provider log type in the logging configuration file while configuring external logging. Use this entry to log the events to an external logger.
    For more information about how to configure external logging, see External Logging.