Key Features and Considerations

The Omni Console Downloads section, available as of Omni Version 3.14, enables you to download the Omni Designer, as well the necessary security certificates and profiles to connect to a remote Omni Designer Repository server. This enables Omni Designer in Cloud installations and access to remote Omni Designer repositories.

In the left pane of the Omni Console, click Downloads, as shown in the following image.

The Downloads page provides support for those users working with a remote instance of Omni-Gen who need the Omni Designer functionality in their local environment. It provides two different options, depending on the use case.

Configuration. The download configuration option is for users who already have Omni Designer installed on their local machine, but need to update their configuration. It contains all of the settings, profiles, and necessary security certificates for accessing a remote Omni Designer Repository.

Executable. The download executable option is for users who do not have Omni Designer installed. It contains all of the configurations mentioned for Configuration, in addition to the actual Omni Designer application. You specify the appropriate operating system (Windows or Linux/UNIX) after clicking the Downloads button.

For more information on downloading the zip file to update an existing version of Omni Designer or set up a new installation of Omni Designer, see the Omni Console User's Guide.

As of Omni Version 3.14, the Ramp Quality Gate service creates a QUALITY_GATE work order which flags erroneous records in a ramp batch, so they are not included in Omni-Gen work order processing. A new column, rec_quality, on the _r tables captures the error and warning status detected by the Quality Gate.

The service has the following three parameters, all of which are required:

• batchId
• sourceName
• subject

The QUALITY_GATE work order does the following:

• Scans all ramp tables of a subject.
• Generates system warnings for columns that require trimming.
• Generates system errors for columns that contain embedded white spaces or a colon (:).
• Updates the rec_quality column on the ramp record. If the record generated system errors, rec_quality = 'E'. If the record generated system warnings, rec_quality = 'W'.

Note: The measures of this work order show three values:

• Number of the records scanned (Processed).
• Number of the records that did not generate a system error (Results).
• Number of the records that generated a system error (Errors).

When the work order completes, click the System Messages menu item on the work order to see the warnings and errors.

At this time, all flagged records should be fixed, and the rec_quality column should be set to NULL before the batch is submitted for processing. This applies even to those records marked with a warning, because NATIVE_SQL loads can only trim spaces.

The following image shows an example of the parameters for Ramp Quality Gate.

As of Omni Version 3.14, the console provides information explaining why a work order that is ready to process is blocked from execution.

The possible blocked reasons are shown in the following table.

The following image shows an example of a blocked work order with the SUBJECT_RUNNING explanation.

As of Omni Version 3.14, the Batch Split feature allows for more optimized database resource usage and for the progress of the load to be visible to the user.

This is only applicable to loads of a transactional subject, where the Data Transfer Mode is NATIVE_SQL. In addition, the batch process must be bound to a single source

If the set of instances in a work order is too large, resource constraints can result in a batch processing failure. Using the Batch Split Size setting, multiple NATIVE_SQL operations are executed, each targeting a subset of the batch. The Measures are updated when each operation completes, showing how many records were processed and how many records were new or updated.

When Batch Split is off, all the records in the batch are processed by a single NATIVE_SQL operation and the measures are only updated when the operation is complete.

The default value of Batch Split Size is -1, which means not to process in chunks. If this value is greater than 0, the batch will be processed in Batch Split Size chunks. For example, if there are 10 million records in the table and the size is set to 1 million, approximately one million rows will be processed at a time.

The following image shows the measures updating while ramp to instance is processing. This only happens with Batch Split. Otherwise, measures are updated once they are completed.

As of Omni-Gen version 3.12, the Elastic components have been removed from the product. This change should not affect the migration of the existing implementations into Omni-Gen version 3.12.

As of Omni-Gen version 3.12, the Data Quality Workbench has been disabled in the Omni-Gen Master Data (MD) and Data Quality (DQ) Editions. The Data Quality Workbench is still available for personal use as part of the Omni-Gen Personal Edition.

Omni-Gen version 3.12 introduces a set of new APIs to facilitate Data Lineage and Governance. The APIs to manage the metadata and relevant services are available as part of the swagger interface for quick reference. The APIs are also fully documented in the Omni-Gen™ API Service Reference Guide. This is a phased-release and the introduction of the initial set of services which will be expanded in the future releases of Omni-Gen.

The Deployment History option enables you to see your prior Deployment History at a glance, with the following new features available for each Deployment.

• Current Deployment. Enables you to download the currently Deployed Data Model and Data Quality Plans as a Deployment Bundle, for ease of propagation to other environments.
• Deployment Bundle. Enables you to download the Bundle that was used to affect a specific Update Bundle, Update Data Model, or Update Data Quality Plans operation.

During the course of on-going incremental processing, instances, masters, and their children can be marked for soft-delete from the system. This feature enables you to schedule the automatic physical deletion of stale soft-deleted records from the system on a configurable interval.

For more details, see the Omni-Gen™ Operation and Management User's Guide.

In order to meet tighter processing windows for large volumes of data, several new processing options have been added that optimize the configuration of os_ramp_control:

• batch_type = INSERT_ONLY. Omni processing is optimized to skip internal Change Data Capture processes to facilitate large initial loads, assuming direct inserts.
• data_transfer_mode = NATIVE_SQL. A performance optimization that shifts internal processing to the Database Server for significantly large batches.

  Note: In version 3.12, the Native SQL processing option is only supported with Microsoft SQL Server and PostgreSQL.

• change_detection = IGNORE. Standard work order processing is performance-optimized to skip steps of the Work Order when the parent and child records of the instance have not been changed.

  This option forces the Omni Server to bypass this optimization, and is generally used as a recovery step in the event that an error occurs during processing of a given batch.

  It ensures that all instances in the ramp batch propagate to instance and quality operations, even if they have not changed.

  Important: For key updates on using the new os_ramp_control options, while converting from the older, deprecated os_ramp_control options, and additional information on bulk processing improvements, see the Omni-Gen™ Integration Services User's Guide.

By decoupling Mastering processes from Ramp to Instance processing, and allowing you to make a minor configuration setting to execute groups of mastered instances in parallel, overall processing time can been further reduced. Instead of one call to the matching engine for each Work Order, you can now execute the Mastered Instances from multiple sources in parallel, and make a single call to the Matching engine when all are complete. This option is available in the Work Order step management for the subject, as shown in the following image.

For more information on setting up parallel processing options, see the Omni-Gen™ Operation and Management User's Guide.

As of Omni-Gen version 3.8, the operations user has access to descriptive system-level error codes, which can be found in the Omni Console, as shown in the following image.

The error codes will continue to be updated and expanded in subsequent Omni-Gen releases, but this update available in Omni-Gen version 3.8 provides an initial, descriptive explanation of the error codes.

As of Omni-Gen version 3.8, you can change the log level that is used by the Omni-Gen Governance Console (OGC) and deployed web applications.

As of Omni-Gen version 3.6, enhanced security for the 360 Viewer and Remediation applications in the Omni Governance Console (OGC) is available in two forms (Column-based access security and Row-based access security).

• Column-based access security provides the ability to restrict access to any column in any table to any user.

  For example, all data stewards would see six columns of data in a 360 view of a subject, while finance managers would also see a seventh, restricted access data column, such as Revenues Received.

• Row (or column criteria)-based access security provides the ability to restrict the display of rows in any table where the value of the content in any column is a configurable value.

  For example, the rows of customer master records displayed for regional managers can be restricted to the rows where the Region column contains a value that is specific to the region of the manager.

By default, Column-based access security and Row-based access security are disabled and can be enabled independently as required. For more information, see the Omni Governance Console Administration User's Guide.

As of Omni-Gen version 3.6, the installer supports the configuration of a port range to be used by the Omni-Gen product. You can provide the initial port, which will be used as a starter to allocate the needed ports for the product services. You are prompted to provide a Base Port Number during installation, as shown in the following image.

Throughout the installation, you have an option to change the default ports or accept the recommended values. Each applicable installation step provides you with an option to reconfigure the port assignment. However, it is recommended to keep the defaults to prevent any conflicts. The following image shows an example of a dialog box where you can change the default ports.

As of Omni-Gen version 3.6, the installer enables you to select whether the Omni-Gen Server should use HTTP or HTTPS. You can update this setting at a later time using the Omni-Gen Server Console. The default is HTTPS, as shown in the following image.

As of Omni-Gen version 3.6, you have an option to track history of data changes for both Instance and Master records. The purpose of history data tables is to enable a safe place where all modifications of data can be tracked, if necessary. Currently, it is possible to track a history of instance records and/or master records. The naming convention for history tables is to include an '_h' suffix to an instance or master table name. For example, person → person_h; person_m → person_m_h.

A history table replicates all tracked table column names, types, and order, but contains a few additional columns. Both instance and master history tables have a time stamp type column, 'hist_start_date'. In addition, instance history tables have a 'u_transaction_id' column.

The work of writing history records is captured in a separate data load step. Depending on data flow, one of the distinct work items: HISTORY_INSTANCE, HISTORY_MASTER, or HISTORY_INSTANCE_MASTER, could be created and executed. In the Omni Console measures, you can see how much time is spent on the operation and how many new history records are created.

As of Omni-Gen version 3.6, you have an option on how to reset the Deployment environment. There are two options available where only Model related tables are removed, or where both Model and System tables are removed. For more information, see the Omni-Gen™ Operation and Management Guide.

The domain name has certain limitations in the naming convention. The use of certain characters is not permitted and should be avoided in the domain name definition, as well as other field and attribute definitions. The information for model is being retrieved at runtime by an array of RESTful and other services, which do not allow and might incorrectly respond to requests for domains containing special characters. Below is the specification of allowed (white list) of characters used for domain specification.

Allowed are letters, numbers, and underscore (_). First character must be [a-z, A-Z], subsequent characters may be alphanumeric or underscore [a-z, A-Z, 0-9, _]

As of Omni-Gen version 3.4, you can now import a project into Omni Designer from a release bundle that was previously exported or generated.

Right-click anywhere in the Project Explorer tab, select Import Project, and then click from Release Bundle from the context menu, as shown in the following image.

You are prompted to select an existing repository into which the project will be imported.

In the Omni Designer Project dialog box, provide a project name and then click Finish.

The Export Project functionality in Omni Designer works the same as in previous releases.

As of Omni-Gen version 3.2, you can modify the configuration of your repository using the Omni Console. However, be advised that you are still responsible for providing correct third-party driver .jar files for the repository connection, to be accessible from the following path:

<OmniGenHome>\OmniDesignerRepository\lib

In addition to the Update Bundle functionality in prior releases, the following options have been added:

• Update Data Model. Enables you to upgrade the model without affecting the already deployed Data Quality Plans, making it easier to implement data model changes.
• Update Data Quality Plans. Enables you to upgrade the Data Quality plans without affecting the already deployed Data Model, making it easier to deploy incremental customer-specific Data Quality plan changes. This pushes only Data Quality updates and bypasses the code generation and data base processing. This option assumes that the IDS directory in the deployment bundle will be empty for the Data Quality only update.

This feature was originally introduced in Omni-Gen version 3.1.2, but is also included in this set of release notes, since it is considered a key feature.

The support for mastering intra-subject relations is also known as households. This feature allows for the creation of named relationships between master documents. Relationships are stored in the newly introduced internal table os_subject_group_relation, and are created by Data Quality plans present in the merging server. The online services for each subject group plan are defined as for cleansing, matching, etc., except that the location of the service is the name of the relationship process.

The list of available group relationship processes are defined in a text file whose location is given by the Subject groups file setting in the DQ runtime settings of the Omni Console. Each line in the file contains the process name, the originating ("from") subject, the destination ("related to") subject, and a comma separated list of all other involved subjects. For example:

banking;Customer;Account;Organization,Employee

This feature with all related configuration is optional. If the relationships file exists, then appropriate subject group processes are automatically invoked as part of any master work order. Processing is otherwise unaffected by the addition of this feature.

As of Omni-Gen version 3.5, there is an available feature that disables the submission of tickets to the Remediation Service (Omni Governance Console). The system can continue to process records, and Remediation tickets can be generated, but they will not be submitted to the Remediation Service. When enabled, the tickets will be submitted to the Remediation Service.

This setting is in the Omni Console (expand Configuration in the left pane, select Runtime, and then click the Server Remediation tab), as shown in the following image.

As of Omni-Gen version 3.5, there are various ways to optimize the system for performance depending on the application scenario and requirements. One way is to disable the Match Post Processing database operations. This step is executed after the Match step and checks the masters table to see if any should be inactive or marked as deleted. This is a resource-intensive operation, as it must scan the entire table.

As of Omni-Gen version 3.5, the following additional settings are made available where you can choose to enable or disable this Match Post Processing:

• When Enable/Disable Match Post Processing for Inactive is false, the work order does not include the MATCH_SET_INACTIVE step.
• When Enable/Disable Match Post Processing for Delete is false, the work order does not include the MATCH_SET_DELETE step.

These settings are available in the Omni Console under the Runtime section, as shown in the following image.

As of Omni-Gen version 3.5, when the application is started, the controller and the server test the connection with the host. ​If an invalid host name is provided during the installation, it is ignored. Once the controller and the server are up and running, use the following URLs to test the communication between these two components:

https://<hostname>.<domain>:9500/api/vi/pingServer

https://<hostname>.<domain>:9512/server/api/vi/pingController

As of Omni-Gen version 3.5, a new Operations section is available in the Omni Console.

The Operations section provides a detailed view into system resource utilization, transaction processing, execution statistics, and other relevant data to optimize the Omni-Gen system and isolate any performance bottlenecks. This new area is separated into relevant sub-sections, based on the product area that is most resource intensive.

The Operations section is meant to be a supplement to other performance monitoring tools, such as Java monitoring, database monitoring, tuning facilities, and other related third-party tools. It is not meant to be a replacement, but rather a supplemental tool to help identify and isolate potential problems in the Omni-Gen system.

For more information on using this feature, see the Omni Console User's Guide.