Using the IS Repository for Data Storage: A Use Case By Ray Moser

---

Introduction

I’d like to welcome everyone back to the webMethods Ezine. This article highlights an actual development challenge and solution using the Integration Server 4.6 repository store. The names and processes were changed to protect the innocent (and to keep my job). However, this article represents a real, Production-deployed webMethods solution.

---

The Use Case Description

A client requested the ability to drop a large batch file containing XML documents into a local file server directory. The Integration Server must poll that directory at scheduled intervals and, if a document is found, begin processing it. Normally, a developer works with only one inbound XML document at a time. For this client, though, the large batch file would contain between 1500 and 2500 sub-documents -- each an XML document itself.

The basic requirements:

1. Break the large XML document into individual XML sub-documents
2. Validate each sub-document
3. Envelope each valid sub-document, persist each invalid sub-documents
4. Send each valid sub-document to an exchange via HTTP
5. Summarize the entire process with an acknowledgement message. The acknowledgement message must contain the following information:
   • The name of the original large XML document
   • The total sub-document count
   • The total valid sub-document count
   • The total invalid XML sub-document count
   • The total number of sub-documents sent via HTTP to the exchange
   • The start and finish time for the entire processing sequence
   • For each invalid sub-document, the document name and description of the errors found.

A sample acknowledgement message follows:

File Name: AM_ST_999_20020805T135107286
 (AM = Ack Msg; ST = Statement)

File Name: AM_ST_999_20020805T135107286
File Count: 218
Actual File: Count: 218
Valid File Count: 217
Invalid File Count: 1
Files Sent To Exchange: 217
*************************************

FILE ERROR COUNT: 1
----------
File Name: ST_999_20020805T135107286-146_20020805T135532517
Number of errors in file: 7

StatementID: 569635

Error # 1: /StatementHeader/StatementType
VV-001
[B2BCORE.0082.9025] Missing Object
Error # 2: /StatementHeader/PurposeCoded
VV-001
[B2BCORE.0082.9025] Missing Object
Error # 3: /StatementHeader/StatementtYpe
VV-002
[B2BCORE.0082.9026] Undefined Object found
Error # 4: /StatementHeader/PurposecOded
VV-002
[B2BCORE.0082.9026] Undefined Object found
Error # 5: /StatementDetail/DeliveryIdentification/BuyerLineItemNumber
VV-001
[B2BCORE.0082.9025] Missing Object
Error # 6: /StatementDetail/DeliveryIdentification/ContractRate
VV-002
[B2BCORE.0082.9026] Undefined Object found
Error # 7: /StatementSummary/TotalNetWeight
VV-002
[B2BCORE.0082.9026] Undefined Object found

---

Tackling the Use Case

At first, this may seem like a simple task. Once the Integration Server processes are defined, however, the seemingly easy task becomes difficult.

The three main processes are as follows:

1. Process Batch XML Document
   • Poll a local directory for one or more batch documents
   • If multiple batch documents are found, compile file list of batch documents to process
   • Break apart batch XML document into XML sub-documents
   • Persist sub-documents to hard drive.

2. Validate XML Sub-Documents
   • Scan hard drive for XML sub-documents
   • If sub-documents are found, compile list of file names
   • Loop over list of sub-documents and validate each one
   • If sub-document is valid, envelope it and persist it to the hard drive.
   • If sub-document is invalid, trap the errors found and move sub-document to errors folder.

3. Send XML Sub-Documents to Exchange
   • Scan hard drive for sub-documents. Remember that the invalid sub-documents have already been moved to new location and will be ignored in this scan.
   • If sub-documents are found, compile list of file names
   • Loop over list of sub-documents and send each one to the exchange via HTTP
   • For each delivered document, trap the HTTP status from the HTTP header
   • If HTTP status is equal to 200, move the sub-document to the final archive folder
   • If HTTP status is not equal to 200, consider the transaction to be in error and leave the sub-document in the current directory. Generate an email for Production Support stating that the sub-document was not sent.

To manage the information required for the Acknowledgement Message, an IData record (ProcessRecord) was created:

IData Record: ProcessRecord
 -- String: Original Batch XML Document Name
 -- String: Start Time (ISO-8601 Format)
 -- String: End Time (ISO-8601 Format)
 -- String: Stated Sub-Document Count
 -- String: Actual Sub-Documents Count
 -- String: Number of Valid Sub-Documents
 -- String: Number of Invalid Sub-Documents
 -- String: Total Sub-Documents Sent to Exchange via HTTP
 -- RecordList:	Specific Sub-Document Errors
		 -- String: StatementID
		 -- String: Sub-Document name
 -- RecordList: 	ActualErrorRecordList
		 -- String: ErrorCode
		 -- String: ErrorMsg
		 -- String: Path

Note that the RecordList ActualErrorRecordList is a copy of the error array provided by the pub.record:validation service.

---

[1]  2  Next>>

Go Deeper on the Subject: The wMUsers Discussion Forums