Topics: |
A work order processes data for a given subject through a series of work order items. A work order is associated with a transaction ID. Records marked with the work order transaction ID will be included in subsequent processing.
The following list describes the types of work orders.
The following syntax shows the REST API service:
/server/processInstance
Relevant work order items include:
The following syntax shows the REST API service:
/server/quality/reprocess/{subject}
The following syntax shows the REST API service:
/remediation/PropertyOverride
The following syntax shows the REST API service:
/remediation/MatchOverride
The following syntax shows the REST API service:
/server/subjectGroups
The following syntax shows the REST API service:
/consumption/work/{ods}
The following syntax shows the REST API service:
/server/quality/cleanse/{subject}
The following syntax shows the REST API service:
/server/quality/match/{subject}
The following syntax shows the REST API service:
/server/master?responseType=oid|masterId
In a DELETE_STALE work order, all INACTIVE records of a given subject that were last updated before the Purge Inactive Age are deleted. This includes source, instance, master, history, and master-master reference records. The Purge Inactive Age is a Runtime setting to specify the number of days after which INACTIVE records will be deleted.
/api/v1/server/reset/{subject}
The following list describes the work order states.
This section lists and describes the work order items.
In this step, the work order state moves from SCHEDULED to ACTIVE. This is the first item in every work order.
The following processes describe the steps involved in the RAMP_TO_SOURCE work order item, which is invoked for mastered or cleansed subjects.
When all data in the ramp batch has been processed, a root source record not yet in the transaction will be added to the transaction if any of its collection items have the current transaction ID.
For more information on the Ramp Load policy, see the Omni-Gen Integration Services User's Guide.
A list of codes and the fields which reference those codes are also collected in memory during ramp processing. checkForMissingCodeXRefs identifies the new code-field references and adds them to the os_source_code_xref table.
Note: When a SourceInstanceID is trimmed of leading and trailing whitespace, a warning message is logged. If the trimmed SourceInstanceID contains a space or a colon, the record will not be moved to the source table and an error is logged.
Parent records are not auto-generated for orphan records.
The following steps describe TXN_ON_SOURCE and TXN_ON_SEL_SOURCES.
This section describes the step involved in the SOURCE_TO_MODEL work order item, which is invoked for mastered or cleansed subjects.
Source Processing
In this step, data of a subject is copied from the source tables to the instance tables based on the transaction ID. Each subject collection is copied independently on its own thread. The instance record and/or its transaction ID and Omni-Gen modified date are updated if the source data (business fields only) or status differs from the instance data or status, or if the ChangeDetection IGNORE ramp load option was specified.
When all source data in the transaction has been processed, a root instance record not yet in the transaction will be added to the transaction if any of its collection items have the current transaction ID.
The instance data in the transaction is now pre-cleansed.
The following processes describe the steps involved in the RAMP_TO_MODEL work order item, which is invoked for non-mastered, non-cleansed subjects.
For more information on the Ramp Load policy, see the Omni-Gen Integration Services User's Guide.
A list of codes and the fields which reference those codes are also collected in memory during ramp processing. checkForMissingCodeXRefs identifies the new code-field references and adds them to the os_source_code_xref table.
Note: When a SourceInstanceID is trimmed of leading and trailing whitespace, a warning message is logged. If the trimmed SourceInstanceID contains a space or a colon, the record will not be moved to the source table and an error is logged.
Parent records are not auto-generated for orphan records.
This step only applies to mastered subjects. The instance records in the transaction are traversed for other mastered instances that are referred to by the participating instances. A record is inserted into the os_master_reference table for each new mastered instance reference. These records are used by the FILL_RELOAD_QUEUE work order item in preparation for a RELOAD work order.
In this step, data for a given subject is cleansed. For a non-mastered subject, this step will only be included if the cleanse attribute in the IDS is true.
The DQ Cleansing service is called to start processing. DQ calls back to Omni-Gen Server to retrieve the data to cleanse, identifying the fields in the Cleansing service definition.
The Cleansing service runs the rules defined in the Cleanse plan and returns the cleansed data. The set of values returned is independent of the values that were sent, though generally there is a correlation.
The following steps describe TXN_ON_INSTANCE and TXN_ON_SEL_INSTANCES.
In this step, records of a mastered subject are processed by match rules to generate the master ID. The DQ Matching service is called to start processing. DQ calls back to Omni-Gen Server to retrieve the data to use in the match plan, identifying the fields in the Matching service definition.
The Matching service runs the rules defined in the Match plan and returns the results. The number of records received may be larger than the number of records sent because all instances of an affected master ID are returned.
If there is a change to the master ID of an instance, any instances of the previous master ID are added to the transaction. This ensures the complete set is provided to the MERGE process.
A white or black list is maintained for each participating instance on a white or black list, referencing the other list participants.
This step will run for a mastered subject if the Enable/Disable Match Post Processing for Inactive runtime setting is true. Turning off this feature may result in unreliable master data.
The status of an ACTIVE master record is set to INACTIVE if it has no active instances. The prev_master_id of instance records in the transaction is used to optimize this process.
This step will run for a mastered subject if the Enable/Disable Match Post Processing for Delete runtime setting is true.
The status of a master record is set to DELETE if it is in a PREDELETE state and has no active instances.
In this step, the os_reload_queue table is populated with all mastered instances that are referenced by a mastered instance in the current transaction, as found in the os_master_reference table, in preparation for a RELOAD work order.
The following list shows the related work order items:
In this step, master records are created for a given subject. The DQ Merging service is called to start processing. DQ calls back to Omni-Gen Server to retrieve the data to use in the Merge plan, identifying the fields in the Merging service definition (all fields, plus the match response fields).
The Merging service runs the rules defined in the Merge plan and returns the master record(s).
In this step, instructions are applied to promote master collection data to the master record root.
In this step, remediation tickets are created for a given subject. For a non-mastered subject, this step will only be included if the remediate attribute in the IDS is true.
The DQ Remediation service is called to start processing. DQ calls back to Omni-Gen Server to retrieve the data to use in the remediation plan, identifying the fields in the Remediation service definition (all fields, plus the match response fields).
The Remediation service runs the rules defined in the Remediation plan and returns cleansing and/or matching remediation tickets.
The values of the above attributes are used to build the cleansing ticket records, which are continued to the omni_remediation_ticket table.
The values of the above attributes are used to build the matching ticket records, which are persisted to the omni_remediation_ticket table along with the master ID to which the remediation ticket applies.
Related work order item: AUTO_CLOSE
This step determines if any open remediation tickets can be closed. It also updates the status of tickets to indicate they are ready to be sent to the OGC Remediation Service.
All remediation tickets of instances or master records in the current transaction that are not in a closed state and were not updated in the current transaction will be closed. If the ticket has not yet been sent to the OGC Remediation Service, it is closed directly. Otherwise, the ticket status and destination are updated and the ticket will be closed by the TicketOutboundService after it is sent to OGC.
Remediation tickets from the omni_remediation_ticket table are sent to OGC in a background process.
Remediation Ticket States
ACK is set by the TicketOutboundService after successful transmission of an open ticket to the OGC Remediation Service.
CLOSED is set by the TicketOutboundService based upon response from the OGC Remediation Service of a close ticket request.
The following steps will be included if the captureHistory attribute in the instance and/or master IDS is true.
In this step, a JSON representation of records along with statistics is compiled and sent to the Elastic index. For the _data index, the entire record is replaced. For the _hist index, the new record is added, and the _endDate is set on the previous version of the record.
In this step, a RELOAD work order is created for os_reload_queue records added by the FILL_RELOAD_QUEUE step in the current transaction. A separate RELOAD work order is created for each relevant subject.
Related work order items: FILL_RELOAD_QUEUE, MASTER_REFERENCE_RELOAD
In this step, the transaction ID of instance records is updated based on the contents of the os_reload_queue table for a particular RELOAD work order.
In this step, records are deleted from the os_reload_queue table after the corresponding instance records have been processed by the RELOAD work order.
This step runs only if the Enable/Disable CDC Notification runtime setting is true.
The os_cdc_change table is populated with the XML representation (Omni Interface Document) of instance and/or master records in the current transaction for which a CDC subscription exists.
This step runs in a SUBJECT_GROUP_PROCESS work order. A SUBJECT_GROUP_PROCESS work order is launched only under the following conditions:
Subject Group Process
The /relations endpoint of the DQ Merging service is invoked, which runs a DQ plan that generates data for the os_subject_group_relations table in its entirety (data from previous iterations is deleted).
In this step, the status, result type, and end date of the work order are updated. This is the last item in every work order.
Topics: |
The ability to automate a sequential set of batches to process was added to Omni-Gen.
The following is a high-level summary of the process:
The automation job is a work order of its own. It is composed of a series of work order items that may also create other work orders. With this in mind, you are able to track the progress of the automation from the work order processing screen.
Each automation work order item will be attempted until one fails or the work is complete. This is no different than regular work order processing.
New Functions
To support automation, the following new functions were added:
The following image shows how work orders appear in the console processing screen.
You are required to use a text editor and a Swagger interface for automation purposes.
Automation creates a work order and corresponding work order items from the provided JSON document. The tags match the role and purpose of the tags in their respective tables.
The following syntax shows a sample automation JSON document.
{ "sourceType":"USER_DEFINED", <-- Required. "subject":"dynamic", <-- Required, value up to user, should not be the name of a subject. "sourceName":"devops", <-- Required, value up to user. "omniSystemType":"SERVER", <-- Only SERVER supported at the moment. "workOrderItem":[ { "workOrderItemName":"START_BATCH", <-- Required as is. "workOrderItemNameExtension":"car-batch", <-- Used to differentiate different batches. "processorOrder":1, <-- Required and must be sequential across work order items. "jsonContext":{ <-- These are the parameters passed in to starting the batch. "subject":"Car", "batchId":"abb5bca6-0565-4344-b85a-fa975456ec11", "sourceSubType":"MERGE_PRESERVE_ON_NULL", "source" : "TestSource” } }, { "workOrderItemName":"WAIT_FOR_ALL_COMPLETE", <-- This task has an 1 min idle time, be patient. "processorOrder":2 <-- Make sure to increment this. }, { "workOrderItemName":"START_BATCH", "workOrderItemNameExtension":"pet-batch", "processorOrder":3, "jsonContext":{ "subject":"Pet", "batchId":"e24f889a-bc4d-473b-85e3-3cd97d06dd42", "sourceSubType":"MERGE_PRESERVE_ON_NULL", "source" : "TestSource" } }, { "workOrderItemName":"WAIT_FOR_ALL_COMPLETE", "processorOrder":4 } ] }
To start the automated work order:
https://<hostname>:9500/swagger-ui.html#!/WorkOrder/postWorkOrderUsingPOST
View the automation executing on the console work order screen.