Work Orders

Topics:

A work order processes data for a given subject through a series of work order items. A work order is associated with a transaction ID. Records marked with the work order transaction ID will be included in subsequent processing.

Work Order Types

The following list describes the types of work orders.

  • BULK. In a BULK work order, data is loaded into Omni-Gen from the ramp tables for a given subject. This work order is created when an os_ramp_control record is processed or when an Omni Interface Document (OID) is loaded from the File Input Location.
  • IMMEDIATE. In an IMMEDIATE work order, data is loaded into Omni-Gen from an Omni Interface Document using the REST API service.

    The following syntax shows the REST API service:

    /server/processInstance
  • RELOAD. In a RELOAD work order, master records are recreated for select instance records of a given subject. This work order is created through a MASTER_REFERENCE_RELOAD work order item.

    Relevant work order items include:

    • SET_RELOAD_TRANSACTION
    • MERGE
    • CLEAR_RELOAD_QUEUE
    • HISTORY_MASTER (optional)
    • CDC_RECORD (optional)
  • MASTER_PLAN_CHANGE. In a MASTER_PLAN_CHANGE work order, all data for a given subject is reprocessed from the source tables. This work order is created using the Reprocess subject option on the Deployment screen in the console.

    The following syntax shows the REST API service:

    /server/quality/reprocess/{subject}
  • OVERRIDE_RELOAD. In an OVERRIDE_RELOAD work order, an instance record is updated with specific value(s) and reprocessed beginning with CLEANSE. This work order is initiated through a request from OGC.

    The following syntax shows the REST API service:

    /remediation/PropertyOverride
  • MANUAL_MASTER_OVERRIDE. In a MANUAL_MASTER_OVERRIDE work order, records in a match override are reprocessed beginning with MATCH. This work order is initiated through a request from OGC.

    The following syntax shows the REST API service:

    /remediation/MatchOverride
  • SUBJECT_GROUP_PROCESS. In a SUBJECT_GROUP_PROCESS work order, data is generated for subject groups. For more information, see subject.

    The following syntax shows the REST API service:

    /server/subjectGroups
  • LOAD_ODS. In a LOAD_ODS work order, consumption tables are populated with data from instance or master records. This work order is initiated using the Consumption View console.

    The following syntax shows the REST API service:

    /consumption/work/{ods}
  • CLEANSE_PLAN_CHANGE. In a CLEANSE_PLAN_CHANGE work order, records of a given subject are processed using CLEANSE.

    The following syntax shows the REST API service:

    /server/quality/cleanse/{subject}
  • MATCH_PLAN_CHANGE. In a MATCH_PLAN_CHANGE work order, records of a given subject are processed using MATCH and MERGE.

    The following syntax shows the REST API service:

    /server/quality/match/{subject}
  • READ_ONLY_MATCH. This is used to find a master record based on specific data.

    The following syntax shows the REST API service:

    /server/master?responseType=oid|masterId
  • ADTEvent. In an ADTEvent work order, non-IDS xml that represents ADT (HL7) is translated into an OID. This is relevant only to OHD.
  • SETUP_DELETE_STALE/DELETE_STALE. A SETUP_DELETE_STALE work order is created at the Purge Inactive Time (Runtime setting), an interval specified as a cron expression. It will only become ACTIVE if there are no other work orders in progress. When a SETUP_DELETE_STALE work order becomes ACTIVE, a DELETE_STALE work order will be created for each subject in the model that is populated with data.

    In a DELETE_STALE work order, all INACTIVE records of a given subject that were last updated before the Purge Inactive Age are deleted. This includes source, instance, master, history, and master-master reference records. The Purge Inactive Age is a Runtime setting to specify the number of days after which INACTIVE records will be deleted.

  • RESET_SUBJECT. In a RESET_SUBJECT work order, all data for a given subject is purged from the system. This work order is created using the Reset button on the Deployment screen in the console. It will only become ACTIVE if there are no other work orders in progress. All tables associated with the subject are truncated, including ramp, source, instance, master, and history tables. Other records such as remediation cases and tickets, overrides and master-master, and code-field references are deleted. If applicable, Elastic indexes are dropped and recreated. For a mastered subject, the DQ match index is reinitialized as follows:
    • The Matching service is stopped.
    • The DQ index tables (repos_subject_*) are dropped.
    • The Matching service is restarted, which in turn, recreates the tables.
    • REST API service:
      /api/v1/server/reset/{subject}

Work Order States

The following list describes the work order states.

  • NEW. The work order is created, but not ready to process until all work order items are added. This state is managed in general code.
  • READY. The work order is ready to be scheduled. Generally, a subject-based work order will remain in READY state until prior work orders of this subject are complete. This state is managed in general code.
  • SCHEDULED. This is a temporary state, set by the scheduler, indicating the work order will be executed immediately.
  • ACTIVE. The work order is actively running. The first step of every work order is the START work order item, which moves the work order to ACTIVE state.
  • PAUSED. The Pause console menu item on the Work Orders screen can be used to pause an ACTIVE work order. Note that the work order will not go into PAUSED state until the currently executing work order item can safely come to a stop. A PAUSED work order must be resumed using the Resume console menu item. The work order will then resume processing from the last active work order item.
  • COMPLETE. The work order has finished executing. The result can be PASS or FAIL. A failed work order must be restarted or ignored to proceed. The Restart console menu item on the Work Orders screen can be used to restart work order processing beginning with the failed work order item. The Ignore console menu item will set the result of the work order to IGNORE and allow the next READY work order of this subject to be scheduled.

Work Order Items

START

In this step, the work order state moves from SCHEDULED to ACTIVE. This is the first item in every work order.

RAMP_TO_SOURCE

The following processes describe the steps involved in the RAMP_TO_SOURCE work order item, which is invoked for mastered or cleansed subjects.

  • Ramp Processing. In this step, data of a subject is copied from ramp tables to source tables based on a ramp batch ID and optionally, a source name. Each subject collection is copied independently on its own thread. The source record and/or its transaction ID and Omni-Gen modified date are updated if the ramp data differs from the source data (business fields only), if there is a change in the status of the record, or if the ChangeDetection IGNORE ramp load option is specified. The ramp load policy can cause a record status change. The default ramp load policy is UPSERT. The default record status is ACTIVE and it can become INACTIVE under REPLACE or DELETE ramp load policies.

    When all data in the ramp batch has been processed, a root source record not yet in the transaction will be added to the transaction if any of its collection items have the current transaction ID.

    For more information on the Ramp Load policy, see the Omni-Gen Integration Services User's Guide.

  • Code Processing. All unique source codes are collected in memory during ramp processing. checkForMissingCodes determines which of those codes are new. A ramp batch is created for the new codes and a SourceCodeSet BULK work order is submitted for processing.

    A list of codes and the fields which reference those codes are also collected in memory during ramp processing. checkForMissingCodeXRefs identifies the new code-field references and adds them to the os_source_code_xref table.

    Note: When a SourceInstanceID is trimmed of leading and trailing whitespace, a warning message is logged. If the trimmed SourceInstanceID contains a space or a colon, the record will not be moved to the source table and an error is logged.

    Parent records are not auto-generated for orphan records.

TXN_ON_SOURCE and ON_SEL_SOURCES

The following steps describe TXN_ON_SOURCE and TXN_ON_SEL_SOURCES.

  • TXN_ON_SOURCE. This step in a MASTER_PLAN_CHANGE work order sets the transaction ID on all the source records for the subject.
  • TXN_ON_SEL_SOURCES. This step in an OVERRIDE_RELOAD work order sets the transaction ID on select source records.

SOURCE_TO_MODEL

This section describes the step involved in the SOURCE_TO_MODEL work order item, which is invoked for mastered or cleansed subjects.

Source Processing

In this step, data of a subject is copied from the source tables to the instance tables based on the transaction ID. Each subject collection is copied independently on its own thread. The instance record and/or its transaction ID and Omni-Gen modified date are updated if the source data (business fields only) or status differs from the instance data or status, or if the ChangeDetection IGNORE ramp load option was specified.

When all source data in the transaction has been processed, a root instance record not yet in the transaction will be added to the transaction if any of its collection items have the current transaction ID.

The instance data in the transaction is now pre-cleansed.

RAMP_TO_MODEL

The following processes describe the steps involved in the RAMP_TO_MODEL work order item, which is invoked for non-mastered, non-cleansed subjects.

  • Ramp Processing. In this step, data of a subject is copied from ramp tables to instance tables based on a ramp batch ID and optionally, a source name. Each subject collection is copied independently on its own thread. The source record and/or its transaction ID and Omni-Gen modified date are updated if the ramp data differs from the source data (business fields only), if there is a change in the status of the record, or if the ChangeDetection IGNORE ramp load option is specified. The ramp load policy can cause a record status change. The default ramp load policy is UPSERT. The default record status is ACTIVE and it can become INACTIVE under REPLACE or DELETE ramp load policies.

    For more information on the Ramp Load policy, see the Omni-Gen Integration Services User's Guide.

  • Code Processing. All unique source codes are collected in memory during ramp processing. checkForMissingCodes determines which of those codes are new. A ramp batch is created for the new codes and a SourceCodeSet BULK work order is submitted for processing.

    A list of codes and the fields which reference those codes are also collected in memory during ramp processing. checkForMissingCodeXRefs identifies the new code-field references and adds them to the os_source_code_xref table.

    Note: When a SourceInstanceID is trimmed of leading and trailing whitespace, a warning message is logged. If the trimmed SourceInstanceID contains a space or a colon, the record will not be moved to the source table and an error is logged.

    Parent records are not auto-generated for orphan records.

MASTER_REFERENCE

This step only applies to mastered subjects. The instance records in the transaction are traversed for other mastered instances that are referred to by the participating instances. A record is inserted into the os_master_reference table for each new mastered instance reference. These records are used by the FILL_RELOAD_QUEUE work order item in preparation for a RELOAD work order.

CLEANSE

In this step, data for a given subject is cleansed. For a non-mastered subject, this step will only be included if the cleanse attribute in the IDS is true.

The DQ Cleansing service is called to start processing. DQ calls back to Omni-Gen Server to retrieve the data to cleanse, identifying the fields in the Cleansing service definition.

  • Cleansing Sender. ACTIVE source records with the current transaction ID are sent to the Cleansing service. For code fields requested by the Cleanse plan, if the source code is mapped to a standard code, the standard code is transmitted. Otherwise, the source code is transmitted.

    The Cleansing service runs the rules defined in the Cleanse plan and returns the cleansed data. The set of values returned is independent of the values that were sent, though generally there is a correlation.

  • Cleansing Receiver. The instance record is updated with the cleansed value if it differs from its current value, and the meta_tags column is updated with the tags returned by the service or with an empty representation. The transaction ID and Omni-Gen modified date are also updated. The Cleansing service generally does not return code values so the source code is the value persisted on the instance record.
  • Overrides. Property overrides, performed by an OVERRIDE_RELOAD work order, are applied during the CLEANSE process. This includes overrides of fields that are not defined in the cleanse plan. If the field is defined in the cleanse plan, the cleansed value supersedes the override value.

TXN_ON_INSTANCE and ON_SEL_INSTANCES

The following steps describe TXN_ON_INSTANCE and TXN_ON_SEL_INSTANCES.

  • TXN_ON_INSTANCE. This step in a MASTER_PLAN_CHANGE work order sets the transaction ID on all the instance records for the subject.
  • TXN_ON_SEL_INSTANCES. This step in an OVERRIDE_RELOAD or MANUAL_MASTER_OVERRIDE work order sets the transaction ID on select instance records.

MATCH

In this step, records of a mastered subject are processed by match rules to generate the master ID. The DQ Matching service is called to start processing. DQ calls back to Omni-Gen Server to retrieve the data to use in the match plan, identifying the fields in the Matching service definition.

  • Matching Sender. ACTIVE and INACTIVE instance records with the current transaction ID are sent to the Matching service.

    The Matching service runs the rules defined in the Match plan and returns the results. The number of records received may be larger than the number of records sent because all instances of an affected master ID are returned.

  • Matching Receiver. The instance record is updated with results of the match plan, as follows:
    • master_id. Current master_id last assigned to this instance by the matching engine.
    • prev_master_id. Previous master_id assigned to this instance by the matching engine. In the event that the instance changed its master during the current work order, this data element is used to determine whether to re-merge the remaining instances assigned to prev_master_id, or soft delete the master from the system in the event that there are no instances tied to it.
    • match_rule. Human readable string describing the matching rule used in identifying the master_id. Values will be determined by the mastering resource implementing the Matching Rules.
    • match_candidate_id. Identifier of the pre-matching group, defined during the unification step in the matching plan.
    • match_quality. Contains digital representation of the match quality (how good the match was). In this case, the lower the overall score, the better the match becomes.
    • match_quality_detail. Additional supporting metadata for match_quality.
    • match_role. The role that the instance record played in the match. The following list describes the valid values:
      • N. Record has no regular key for candidate grouping.
      • M. Best record of one candidate group. It represents the center of the initially established matching group within the candidate group.
      • I. Next selected centers of other matching groups in the candidate group.
      • S. Slaves or records similar to some center and attached to its matching group.
      • R. Renegades or records not similar to any center in a candidate group.
    • transaction_id. Affected instance records not previously in the transaction.

    If there is a change to the master ID of an instance, any instances of the previous master ID are added to the transaction. This ensures the complete set is provided to the MERGE process.

  • Overrides. Match overrides, performed by a MANUAL_MASTER_OVERRIDE work order, are applied during the MATCH process.
    • Whitelist. A space delimited string of instance IDs that should be placed in the same match group, though they may not be considered a match by the match rules.
    • Blacklist. A space delimited string of instance IDs that should never be placed in the same match group, though they may be considered a match by the match rules.

      A white or black list is maintained for each participating instance on a white or black list, referencing the other list participants.

    • Pre-mastered survivor contract. An agreement that effectively whitelists items from the source system. This does not involve a MANUAL_MASTER_OVERRIDE work order, but is specified when data is loaded from the ramp.

MATCH_SET_INACTIVE

This step will run for a mastered subject if the Enable/Disable Match Post Processing for Inactive runtime setting is true. Turning off this feature may result in unreliable master data.

The status of an ACTIVE master record is set to INACTIVE if it has no active instances. The prev_master_id of instance records in the transaction is used to optimize this process.

MATCH_SET_DELETE

This step will run for a mastered subject if the Enable/Disable Match Post Processing for Delete runtime setting is true.

The status of a master record is set to DELETE if it is in a PREDELETE state and has no active instances.

FILL_RELOAD_QUEUE

In this step, the os_reload_queue table is populated with all mastered instances that are referenced by a mastered instance in the current transaction, as found in the os_master_reference table, in preparation for a RELOAD work order.

The following list shows the related work order items:

  • MASTER_REFERENCE
  • MASTER_REFERENCE_RELOAD

MERGE

In this step, master records are created for a given subject. The DQ Merging service is called to start processing. DQ calls back to Omni-Gen Server to retrieve the data to use in the Merge plan, identifying the fields in the Merging service definition (all fields, plus the match response fields).

  • Merging Sender. ACTIVE instance records with the current transaction ID are sent to the Merging service. For code fields, if the source code is mapped to a standard code, the standard code is transmitted. Otherwise, the source code is transmitted. For link fields, the master ID of the referenced instance record is transmitted. If it does not exist, an empty string is sent.

    The Merging service runs the rules defined in the Merge plan and returns the master record(s).

  • Merging Receiver. All previously existing master records returned by the Merge plan are updated and new master records are created. New collection items are created and previous records are deleted afterwards. For code fields, the standard code is persisted to the master record. For link fields, the master ID of the referenced record is persisted.

PROMOTE_MASTER

In this step, instructions are applied to promote master collection data to the master record root.

REMEDIATE

In this step, remediation tickets are created for a given subject. For a non-mastered subject, this step will only be included if the remediate attribute in the IDS is true.

The DQ Remediation service is called to start processing. DQ calls back to Omni-Gen Server to retrieve the data to use in the remediation plan, identifying the fields in the Remediation service definition (all fields, plus the match response fields).

  • Remediation Sender. ACTIVE instance records with the current transaction ID are sent to the Remediation service.

    The Remediation service runs the rules defined in the Remediation plan and returns cleansing and/or matching remediation tickets.

  • Cleansing Ticket Receiver. The Remediation plan returns the following attributes:
    • ID of the instance record to which the remediation ticket applies.
    • Name of the field whose value requires remediation.
    • Reason code.
    • Reason code description.
    • Severity.
    • Subject or entity to which the remediation ticket applies.

    The values of the above attributes are used to build the cleansing ticket records, which are continued to the omni_remediation_ticket table.

  • Matching Ticket Receiver. The Remediation plan returns the following attributes:
    • Subject to which the remediation ticket applies.
    • Reason code.
    • Reason code description.
    • References to instance records to which the remediation ticket applies.
    • Severity.

    The values of the above attributes are used to build the matching ticket records, which are persisted to the omni_remediation_ticket table along with the master ID to which the remediation ticket applies.

    Related work order item: AUTO_CLOSE

AUTO_CLOSE

This step determines if any open remediation tickets can be closed. It also updates the status of tickets to indicate they are ready to be sent to the OGC Remediation Service.

All remediation tickets of instances or master records in the current transaction that are not in a closed state and were not updated in the current transaction will be closed. If the ticket has not yet been sent to the OGC Remediation Service, it is closed directly. Otherwise, the ticket status and destination are updated and the ticket will be closed by the TicketOutboundService after it is sent to OGC.

Remediation tickets from the omni_remediation_ticket table are sent to OGC in a background process.

Remediation Ticket States

  • PENDING_NEW. Initial status of remediation tickets created in the REMEDIATE step.
  • PENDING_CURRENT. Intermediate state for PENDING tickets updated in the REMEDIATE step of the current transaction.
  • PENDING. Set by the AUTO_CLOSE process for tickets in PENDING_NEW or PENDING_CURRENT state. It is also set for ACK tickets to be closed. Tickets in this state will be sent to the OGC Remediation Service.
  • ACK_CURRENT. Intermediate state for ACK tickets updated in the REMEDIATE step of current transaction.
  • ACK. Set by the AUTO_CLOSE process for tickets in ACK_CURRENT state. Tickets in this state were already sent to the OGC Remediation Service.

    ACK is set by the TicketOutboundService after successful transmission of an open ticket to the OGC Remediation Service.

  • CLOSED. Set by the AUTO_CLOSE process if the PENDING ticket is repaired in the current transaction.

    CLOSED is set by the TicketOutboundService based upon response from the OGC Remediation Service of a close ticket request.

  • ERROR. Set by the TicketOutboundService when a failure occurs in the transmission of a ticket to the OGC Remediation Service.

HISTORY

The following steps will be included if the captureHistory attribute in the instance and/or master IDS is true.

  • HISTORY_INSTANCE. All instance records in the current transaction are copied to the corresponding history table. A snapshot of the entire record is recorded in history. Each entity is copied independently on its own thread.
  • HISTORY_MASTER. All master records in the current transaction are copied to the corresponding history table. A snapshot of the entire record is recorded in history. Each entity is copied independently on its own thread. History records of collections are not related to each other by ID because master records are recreated in entirety every time.
  • HISTORY_INSTANCE_MASTER. This step runs when history is enabled for both instance and master records of a mastered subject.

PUBLISH

In this step, a JSON representation of records along with statistics is compiled and sent to the Elastic index. For the _data index, the entire record is replaced. For the _hist index, the new record is added, and the _endDate is set on the previous version of the record.

  • PUBLISH_SOURCES. All source records in the current transaction are processed.
  • PUBLISH_INSTANCES. All instance records in the current transaction are processed.
  • PUBLISH_MASTERS. All master records in the current transaction are processed.

MASTER_REFERENCE_RELOAD

In this step, a RELOAD work order is created for os_reload_queue records added by the FILL_RELOAD_QUEUE step in the current transaction. A separate RELOAD work order is created for each relevant subject.

Related work order items: FILL_RELOAD_QUEUE, MASTER_REFERENCE_RELOAD

SET_RELOAD_TRANSACTION

In this step, the transaction ID of instance records is updated based on the contents of the os_reload_queue table for a particular RELOAD work order.

CLEAR_RELOAD_QUEUE

In this step, records are deleted from the os_reload_queue table after the corresponding instance records have been processed by the RELOAD work order.

CDC_RECORD

This step runs only if the Enable/Disable CDC Notification runtime setting is true.

The os_cdc_change table is populated with the XML representation (Omni Interface Document) of instance and/or master records in the current transaction for which a CDC subscription exists.

SUBJECT_GROUP_PROCESS

This step runs in a SUBJECT_GROUP_PROCESS work order. A SUBJECT_GROUP_PROCESS work order is launched only under the following conditions:

  • The mastering/services/relationships directory exists.
  • The server has been idle for the Work Order Scheduler idle interval (default is 5 minutes) after a successful MERGE in a completed BULK or MASTER_PLAN_CHANGE work order.
  • A SUBJECT_GROUP_PROCESS work order has not been executed since the last successful MERGE in a completed BULK or MASTER_PLAN_CHANGE work order.

Subject Group Process

The /relations endpoint of the DQ Merging service is invoked, which runs a DQ plan that generates data for the os_subject_group_relations table in its entirety (data from previous iterations is deleted).

STOP

In this step, the status, result type, and end date of the work order are updated. This is the last item in every work order.

Work Order Automation

Topics:

The ability to automate a sequential set of batches to process was added to Omni-Gen.

The following is a high-level summary of the process:

  1. The integrator creates a series of batches to process with batch control records.
  2. Using the batch IDs, create a work order automation JSON document that includes entries for each batch to process.
  3. Copy and paste the JSON document into the work order Swagger API.
  4. Click Try it out to test the work order.
  5. View the automation executing on the console work order screen.

Executing the Automation

The automation job is a work order of its own. It is composed of a series of work order items that may also create other work orders. With this in mind, you are able to track the progress of the automation from the work order processing screen.

Each automation work order item will be attempted until one fails or the work is complete. This is no different than regular work order processing.

New Functions

To support automation, the following new functions were added:

  • START_BATCH. The start batch takes batch the information and starts the batch as a BULK work order. In the work order processing screen, the batch will be visible as normal and the batch parameters are provided in the jsonContext aggregate.
  • WAIT_FOR_ALL_COMPLETE. Using the automation work order as a starting point, this function polls the work order table looking for any new work orders to complete. It will wait a configurable number of iterations with configurable n second pause between iterations. If any subsequent work order fails, then the automation work order will also fail. This function will also wait for an additional duration, making sure no other work comes into the system. This is also a configurable wait. This combination is intended to support not only the immediately created batch, but also any work orders it may create, for example a RELOAD.

The following image shows how work orders appear in the console processing screen.

Creating an Automation

You are required to use a text editor and a Swagger interface for automation purposes.

Automation creates a work order and corresponding work order items from the provided JSON document. The tags match the role and purpose of the tags in their respective tables.

The following syntax shows a sample automation JSON document.

{
    "sourceType":"USER_DEFINED",  <-- Required.
    "subject":"dynamic", <-- Required, value up to user, should not be the name of a subject.
    "sourceName":"devops", <-- Required, value up to user.
    "omniSystemType":"SERVER", <-- Only SERVER supported at the moment.
    "workOrderItem":[
        {
            "workOrderItemName":"START_BATCH",  <-- Required as is.
            "workOrderItemNameExtension":"car-batch", <-- Used to differentiate different batches.
            "processorOrder":1, <-- Required and must be sequential across work order items.
            "jsonContext":{ <-- These are the parameters passed in to starting the batch.
                "subject":"Car",
                "batchId":"abb5bca6-0565-4344-b85a-fa975456ec11",
                "sourceSubType":"MERGE_PRESERVE_ON_NULL",
                "source" : "TestSource”
            } 
        },
        {
            "workOrderItemName":"WAIT_FOR_ALL_COMPLETE", <-- This task has an 1 min idle time, be patient.            
            "processorOrder":2 <-- Make sure to increment this.
        },
        {
            "workOrderItemName":"START_BATCH",
            "workOrderItemNameExtension":"pet-batch",
            "processorOrder":3,
            "jsonContext":{
                "subject":"Pet",
                "batchId":"e24f889a-bc4d-473b-85e3-3cd97d06dd42",
                "sourceSubType":"MERGE_PRESERVE_ON_NULL",
                "source" : "TestSource"
            } 
        },
        {
            "workOrderItemName":"WAIT_FOR_ALL_COMPLETE",            
            "processorOrder":4
        }
    ]
}

Starting the Automated Work Order (Submitting the Job)

To start the automated work order:

  1. Use Swagger on the controller. For example:
    https://<hostname>:9500/swagger-ui.html#!/WorkOrder/postWorkOrderUsingPOST
  2. Copy and paste the JSON document into the work order Swagger API.
  3. Click Try it out.

    View the automation executing on the console work order screen.