XOG Governor Node Limit

ccppmop1591
When you import or export many instances using the XOG, you can encounter out-of-memory or timeout exceptions. The 
Maximum XML Node Limit
setting helps handle these issues. This limit specifies the number of XML nodes that can be imported or exported at any given time. Often called a
XOG governor
limit, the default value for the limit is 150,000 nodes.
Do not confuse this
node
limit with similar
row
limits such as the 50,000-row NSQL limit in a list view or grid portlet.
2
XML Element and Attribute Nodes
A node is the sum of the XML elements and all their attributes. For example, the following fictional example shows XML code for project output. This example represents 22 nodes for 21 attributes plus the project element.
<Project active="true"  allowAssignedTasksOnly="0"  approved="true"  approvedForBilling="1"  assgnPool="0"  billingCurrencyCode="USD"  billingType="S"  calculateFinancialMetrics="true" costType="OPERATING"   investmentDefinedReinvestmentRate="0"  investmentDefinedTotalCostOfCapital="0" laborExchageRateType="AVERAGE" lastUpdatedBy="admin"  projectID="pitcProject001"  setBudgetValuesEqualToPlannedValues="true"  start="2013-08-01T08:00:00" status="1" syncInvestmentAndBudgetDates="true"  template="false"  trackMode="2"  useSystemDefinedReinvestmentRate="true"  useSystemDefinedTotalCostOfCapital="true">
XOG Read
For a XOG read request, the server processes the instances until the node limit is reached. The response is sent back to the client in chunks using pagination. For the next set of instances, send a new read request to the server with the appropriate information using pagination.
After processing the instances, the number of processed instances is updated in the skip element of the XOG output. Before Release 13.3, the entire requested information was sent in a single response which resulted in out-of-memory or time-out exceptions.
The following example explains the pagination process. The read request is for 1000 project instances and the node limit is reached after processing the 24th instance. This sample code shows the XML output:
<NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../xsd/nikuxog_read.xsd"> <Projects> <Project ...> .... </Project> <XOGOutput> <Object type="project" /> <Status state="SUCCESS" /> <Skip value="24"/> <XOGTotalFilteredRecords value="267"/> <Statistics failureRecords="0" insertedRecords="0" totalNumberOfRecords="24" updatedRecords="0"/> </XOGOutput>
The
<Skip value="24"/> 
 element in the XML signifies the number of instances that are processed in the request. To request the next set of instances, add the skip argument in the header of the XOG read request. In this example, 24 instances are processed. To request the next set, add the 
<args name="skip" value="24"/>
 argument to the header in the next read request as shown in this example:
<?xml version="1.0" encoding="UTF-8"?> <NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  xsi:noNamespaceSchemaLocation="../xsd/nikuxog_read.xsd"> <Header version="6.0.11" action="read" objectType="project" externalSource="NIKU"> <args name="skip" value="24"/> </Header> <Query> <Filter name="projectID" criteria="EQUALS">ank00_1*</Filter> </Query> </NikuDataBus>
If the <XOGOutput> element in the XML output does not contain the skip argument, this indicates that the server has processed all the requested instances.
The pagination process works differently for WSDL, GEL, and XOG.
  • WSDL and GEL
    Add the 
    <args name="skip" value="n"/>
     element to the header element in the next read request manually. This sample code shows the GEL script for pagination:
    <gel:script xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:core="jelly:core" xmlns:gel="jelly:com.niku.union.gel.GELTagLibrary" xmlns:soap="jelly:com.niku.union.gel.SOAPTagLibrary" xmlns:file="jelly:com.niku.union.gel.FileTagLibrary" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:sql="jelly:sql" xmlns:xog="http://www.niku.com/xog" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <gel:parameter default="http://clrtstage02:80" var="XOGURL"/> <gel:parameter default="addsr02" var="XOGUsername"/> <gel:parameter default="admin" var="XOGPassword"/> <!-- Log into XOG and get a session ID --> <soap:invoke endpoint="${XOGURL}/niku/xog" var="auth"> <soap:message> <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xog="http://www.niku.com/xog"> <soapenv:Header/> <soapenv:Body> <xog:Login> <xog:Username>${XOGUsername}</xog:Username> <xog:Password>${XOGPassword}</xog:Password> </xog:Login> </soapenv:Body> </soapenv:Envelope> </soap:message> </soap:invoke> <!-- Checking whether a sessionID is returned. If not, it means that Login was unsuccessful --> <gel:set asString="true" select="$auth/SOAP-ENV:Envelope/SOAP-ENV:Body/xog:SessionID/text()" var="sessionID"/> <core:choose> <core:when test="${sessionID == null}"> <gel:out>Couldn't Log in. Check the username/password.</gel:out> </core:when> <core:otherwise></core:otherwise> </core:choose> <core:set var="skipValue" value="0"/> <core:set var="inc" value="1"/> <core:choose> <core:while test="${skipValue != null}"> <!--Run XOG and attach an input file...alternatively the Body section can be the NikuDatabus section of an input file--> <soap:invoke endpoint="${XOGURL}/niku/xog" var="runresult"> <soap:message> <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xog="http://www.niku.com/xog"> <soapenv:Header> <xog:Auth> <xog:SessionID>${sessionID}</xog:SessionID> </xog:Auth> </soapenv:Header> <soapenv:Body> <!--Paste your read XML here--> </soapenv:Body> </soapenv:Envelope> </soap:message> </soap:invoke> <!-- Read the output and extract some information from it --> <gel:set asString="true" select="$runresult/SOAP-ENV:Envelope/SOAP-ENV:Body/NikuDataBus/XOGOutput/Status/@state" var="XOGoutcome"/> <core:switch on="${XOGoutcome}"> <core:case value="SUCCESS"> <gel:set asString="false" select="$runresult/SOAP-ENV:Envelope/SOAP-ENV:Body/NikuDataBus/XOGOutput/XOGTotalFilteredRecords/@value" var="XOGTotalFilteredRecords"/> <gel:set asString="false" select="$runresult/SOAP-ENV:Envelope/SOAP-ENV:Body/NikuDataBus/XOGOutput/Statistics/@totalNumberOfRecords" var="totalNumberOfRecords"/> <core:choose> <core:when test="${skipValue == 0}"> <gel:out>Total number of Instances = <gel:expr select="$XOGTotalFilteredRecords"/></gel:out> </core:when> <core:otherwise> </core:otherwise> </core:choose> <gel:out>------------------</gel:out> <gel:out>Instances XOGed-out this request <gel:expr select="$totalNumberOfRecords"/></gel:out> <gel:set asString="false" select="$runresult/SOAP-ENV:Envelope/SOAP-ENV:Body/NikuDataBus/XOGOutput/Skip" var="skip"/> <gel:set asString="false" select="$runresult/SOAP-ENV:Envelope/SOAP-ENV:Body/NikuDataBus/XOGOutput/Skip/@value" var="skipToken"/> <core:set var="skipValue" value="${skipToken.getValue()}"/> <core:choose> <core:when test="${skipValue == null}"> <gel:out>Skip Token is null</gel:out> </core:when> <core:otherwise> <gel:out>Skip Token = <gel:expr select="$skip/@value"/></gel:out> </core:otherwise> </core:choose> </core:case> <core:case value="FAILURE"> <gel:set asString="false" select="$runresult/SOAP-ENV:Envelope/SOAP-ENV:Body/NikuDataBus/XOGOutput/Statistics" var="stats"/> <gel:out>XOG failed. Out of <gel:expr select="$stats/@totalNumberOfRecords"/> records, <gel:expr select="$stats/@failureRecords"/> failed.</gel:out> </core:case> </core:switch> <core:catch var="XOGoutcome"> <gel:serialize fileName="C:\GEL_TEST\testGel_${inc}.xml" var="${runresult}"/> </core:catch> <core:set var="inc" value="${inc+1}"/> <core:if test="${XOGoutcome != null}"> <gel:out>Caught Exception: ${XOGoutcome}</gel:out> </core:if> </core:while> <core:otherwise></core:otherwise> </core:choose> <!-- Log out of the XOG --> <soap:invoke endpoint="${XOGURL}/niku/xog" var="logoutresult"> <soap:message> <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xog="http://www.niku.com/xog"> <soapenv:Header> <xog:Auth> <xog:SessionID>${sessionID}</xog:SessionID> </xog:Auth> </soapenv:Header> <soapenv:Body> <xog:Logout/> </soapenv:Body> </soapenv:Envelope> </soap:message> </soap:invoke> </gel:script>
    The GEL script version does not incorporate the skip value into the subsequent requests in the loop. A placeholder indicates where the read request goes. A variable holds this value, and that variable should have a gel:set applied to it. The variable updates or inserts the args value for skip on each iteration. Without it, the script continuously fetches the initial page of results over and over again in an endless loop.
  • XOG client
     
    The XOG client handles the pagination automatically. The requested instances are processed in iterations internally and the output contains all the requested instances. This process eliminates adding the 
    <skip>
     argument manually in the header element of the read request.
    A progress bar displays the status of the XOG client.
If any instance in the read request exceeds the node limit, that instance is not processed. The following error message appears in the XOG output for that instance:
XOG-9059: The system cannot export this instance with the current configuration. Increase the 'Maximum XML Nodes' value in CSA and try again.
However, all the other instances are processed successfully. You can use the following methods to export the unprocessed instances:
  • Change the Maximum XML Nodes limit.
  • Use filters so that the requested information does not exceed the node limit.
Additional Information
The output file is not indented. You can use any third-party plug-in to convert the output to an XML format.
The default session timeout is one hour. When the node limit is high and the internal XOG request is taking more than the Global Session Timeout set in CSA, this error appears:
XOG-9062: The requested data failed to export because the processing time exceeded the session timeout interval. Contact your administrator to update the XML Node Limit.
The node limit is not enabled for the following XOG categories:
  • Company Class
  • Content Pack
  • Documents
  • Forms
  • Notifications
  • Portfolios
  • Process Notifications
  • Resource Class
  • Timesheets
  • Transaction Class
  • UI Theme
  • Vendor
  • WIP Class
As a result, you can encounter an out-of-memory exception when you export many instances.
XOG Write
When you import instances, the server processes 20 instances for each iteration. If the node limit is reached before the 20 instances are loaded, the server processes only the loaded instances for the current iteration. The remaining instances are considered automatically for the next iteration.
If you use SOAP and GEL, there is no change in the XOG write process. If the server goes down while processing the instances, the latest instances in the current iteration are not saved. However, all the instances in the earlier iterations are saved.
The progress bar does not display for the XOG client.
If the input XML content has any instance that exceeds the defined node limit, the XOG write does not process this instance. This message displays in the XML output:
XOG-9060: The system cannot import this instance with the current configuration. Increase the 'Maximum XML Nodes' value in the CSA and try again.
For all the XOG categories (except the content pack) that do not have the Governor Limit enabled, if the input XML content exceeds the defined node limit, then the input is not processed. This message displays in the XML output:
XOG-9061: The system cannot import the contents of the file with the current configuration. Either increase the 'Maximum XML Nodes' value in the CSA or reduce the file size and try again.
XOG Governor Limit Benchmarks
Use the recommended node limits from the following table. This table bases values on the percentage of the JVM heap space that you want to allocate for XOG processing.
Used JavaHeap Space
XML Node Counts
RecommendedNode Limit
Custom Objects
Cost Plans
Users
Projects
192
4,800
0
36,000
22,000
4,800
256
42,000
56,000
190,000
128,000
42,000
512
160,000
247,000
700,000
780,000
160,000
1024
375,000
609,600
1,360,000
2,000,000
375,000
1536
620,000
960,000
0
0
620,000
2048
860,000
1,300,000
0
0
860,000
For example, an instance is running on the JVM whose total heap space is 2 GB. The administrator wants to allocate 10 percent of the total heap space for XOG processing. In this case, the JVM heap space is 204 (10 percent of 2048). Select the nearest heap space value from the table, which is 192. For 192 MB JVM heap space, the recommended node limit is 4800.
Calculate the Node Limit for any Object
  1. Update the logger.xml file with the following category and appender.
    <category additivity="false" name="xog.usage.memory"> <priority value="debug"/> <appender-ref ref="XOGMEMORY"/> </category> <appender class="org.apache.log4j.RollingFileAppender" name="XOGMEMORY"> <param name="File" value="../logs/xog-usage-memory.xml"/> <param name="Append" value="true"/> <param name="MaxFileSize" value="5MB"/> <param name="MaxBackupIndex" value="3"/> <layout class="com.niku.union.log.NikuLayout"> <param name="ConversionPattern" value="%m\r\n"/> </layout> </appender>
  2. Update jvmParamters in properties.xml as shown in this example:
    Xms64m -Xmx1536m  -XX:MaxPermSize=192m  -XX:-UseGCOverheadLimit  -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.port=14010  -Dsun.lang.ClassLoader.allowArraySyntax=true  -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=C:\ca\clarity\trunk\build\install\logs  -Xloggc:C:\ca\clarity\trunk\build\install\logs\xog_gc.log  -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
    Change the HeapDumpPath & loggc directory location accordingly.
  3. Restart the application and perform garbage collection from Jconsole before you start the XOG request.
    Back up all the log files for each combination.
  4. Verify that the app-xog-usage-memory.xml and xog_gc.log is created in the log folder of the 
    Classic PPM
     installation directory.
  5. Check the node limits for the different JVM heap sizes. This example shows the logging information that is related to benchmarking in the
    app-xog-usage-memory.xml
    file:
    Starting Xog Request..... XML Node count : 0 Initial Used memory : 480 MB ***************************************************** ----- Loading the new XQLObjectSet ------- Before start doing the read:252 MB ----- Loaded the new XQLObjectSet ------- Query: objectInstances Page : 0 : Records fetched: 1000 : Total records: 126534 The used memory after loading set and running GC: 250 MB ----- End of pre-processing ------- ----- XML Node Count after processing this paged records = 35005 And used memory = 372 MB
  6. When you encounter an out-of-memory exception, capture the XML Node Count ( 
    _xogOutputCnt
    in the
    UtilityThreadLocal
    class) from the heap dump using a Java profiler.