Configuring an Ordered Queue Listener

Topics:

Reference:

The Ordered Queue listener is used to manage an ordered queue. In this section, the term queue refers to the entire set of messages that are awaiting transfer to a subchannel.

The ordered queue is a collection of queuelets. Each queuelet collects the messages of one group in the order they were received. A new queuelet is created automatically when a message comes in with a key that has no associated queuelet. A queuelet is automatically deleted when it becomes empty, that is, when all the messages in that group have completed execution. Messages are deleted from the queuelet after their execution is completed. This ensures that messages cannot be lost when the queue is configured to be persistent, and maintains the ordering should a message arrive for the group while execution of the last message is proceeding.

This section will address differences in configuration between the simpler internal queue and the ordered queue. For more information on how the ordered queue system works for the ordering of messages, see Introducing the Ordered Queue Facility.

To configure an Ordered Queue listener, you must create a channel for your application project using the Channel Builder in iWay Integration Tools (iIT) and select Ordered Queue as the listener type for your inlet. For more information, see Configuring a Listener Using iWay Integration Tools.

For a complete description of the configuration parameters that are available for the Ordered Queue listener, see Ordered Queue Listener Configuration Parameters.

For a complete description of the Ordered Queue listener Special Registers (SREGs), see Ordered Queue Listener Special Registers.

Pended Messages

Pended messages are always executed in the proper order within the group. The implication is that it is the group itself that is pended rather than the individual message. Consistent with the normal server rules, if the pended message times out or fails, then it is considered to be complete and the next message in the queuelet is made available for dispatch.

Immediate Mode Queues

In immediate mode, the messages are available for dispatch to the subchannel (process flow) immediately upon their arrival at the listener.

Batch Mode Queues

In batching mode, a new message is put in its queuelet. However, the queuelet simply collects the messages without making them available for dispatch. When the end of group signal is received for that group, the queuelet becomes available for execution, just like in immediate mode.

A delete group message can be sent to delete an unended group. A delete message cannot be used for an ended group.

In persistent mode, the end of group signal is pended along with the messages, thus instructing the channel to correctly end the group when reloading it from safe store. An unended group is reloaded as an unended group. Application logic must decide whether to end that group or delete it.

Control signals, such as delete group, are sent through the Ordered emit service (com.ibi.agents.XDOrderedEmitAgent), which is configured using the Queue Out object in the iWay Process Flow Designer and selecting the iWay queue (ordered) - send a message action. Available signals are listed and described in the following table:

Signal

Description

Document {doc}

Current document message is to be enqueued for its group key.

Delete {del}

Batch queue is to be deleted.

End Signal {end}

Final message has been enqueued for this group, and the batch queue is to be made available for dispatching.

Keep Alive {keepalive}

Resets the timeout period for a batch queue.

Last Document {docend}

Effects a document and end operation. Signals that this message is the final message and the queue is to be made available for dispatching.

The following table lists and describes the general set of special registers (SREGs):

Special Register

Description

endofgroup

Set to true when the last message of a batching queuelet is dispatched to the process flow. This register is set to true for this final message. Emitted signals, such as the end of group signal are not dispatched to the associated process flow.

msgkey

Group key associated with this message.

sortkey

Sort key for non-immediate processing. Not applicable for non-sorted immediate processing.

The ordered listener has a parameter to define the timeout before a batching group must be closed. This is the maximum time between the last message of a group and receipt of the end of the group signal. The timeout does not apply to queues in immediate mode. The Expired Group Event Flow parameter holds the name of a process flow to run when the group timeout expires. If no process flow is specified, then the server writes an error message in the log. The messages are enqueued to the dead letter destination (assuming one is configured) and the group is deleted. If a process flow is specified, then the server runs that process flow. The process flow can send new messages, the delete group message, the end of group signal, or the keep alive signal to reset the timer for the specified group. If the process flow returns a success condition, then the server considers the condition to be handled, and no further action is taken. If the process flow returns a condition other than success, then the server reverts to the default behavior as if there were no process flow.

The expired group flow receives a signal message in the event of the timeout expiration. This flow can elect to emit a keep alive or a delete signal message to the group.

<signal errors="0" protocol="ordered" timestamp="2018-10-24T17:56:30.730Z" 
type="expired" version="2">
    <parms count="2">
      <parm name="queue">ordered</parm>
      <parm name="key">patient1</parm>
    </parms>
</signal>

The type of the signal document is expired and specifies the queue and group names, which in this example are ordered and patient1.

Stopping the Server

If the channel in immediate mode is stopped, then the messages behave as defined for the Internal Queue listener.

  • A persistent queue is stopped immediately upon completion of any messages that are in process. The persisted messages are available once the server is restarted.
  • A non-persistent queue will complete messages that are in the queue before stopping the channel.

Restarting the Server

When the server starts, it searches for persisted messages awaiting execution. The queuelets are reconstructed to their status prior to the when the server was stopped. Immediate mode and ended batch mode queuelets begin dispatching messages immediately, while unended batch mode queuelets wait for an end of group signal.

Application designers are reminded that the timeouts applied to batch mode queues are based on wall clock time. This can result in the execution of timeout flows, and the application logic should take this into account.

Reference: Ordered Queue Listener Configuration Parameters

The following table lists and describes parameters for the Ordered Queue listener.

Note: Parameters that are common to queue listeners are described in Listener Configuration Parameters.

Property

Description

Name of Ordered Queue (required)

Name of the ordered queue that is used to identify the ordered message listener destination.

Queuing Mode

Determines how received messages are handled. Available modes include:

  • immediate. As messages for a group are received, they become available for dispatching.
  • batch. A message becomes available for dispatching only when the group is closed.

The default value is immediate.

Sorting Mode

Applies to batch queues only, and only within the batch itself. Available options include:

  • Chronological (default). No intra-batch sorting is performed, and the order of presentation to the application is arrival order.
  • Lexical. The subsort key is sorted in lexical order. For strings, this is generally considered alphabetic order.
  • Numerical. The subsort key is sorted numerically by value rather than as a string.

The default value is Chronological.

Group Timeout

Indicates the time that a batching queue can remain unavailable for dispatching following the receipt of the latest message. In effect, this is an inactivity timeout. The default value is 300 seconds.

Expired Group Flow

Name of a published process flow that receives control if an unended group times out.

Persistent

If enabled, messages are persisted. Persistent messages are held in the safestore until completion and can be recovered if the server is restarted.

Select one of the following options from the drop-down list:

  • none {false}
  • rdbms {rdbms}
  • file {true}

The default value is none {false}.

The rdbms option enables the application to persist messages to a remote database location where, in the event of a failover situation, a secondary server running remotely can continue processing persisted messages without any business interruption.

To use RDBMS persistence, you must create the iway_queues table in the JDBC provider, using the DDL script in the etc/setup directory. A hot backup channel can be configured to use the same RDBMS safestore.

Safestore Location

If persistent, this is the location in the file system to which documents are safestored.

Compress Persistent

If the queue is persistent, the documents written to the safestore can be compressed. Select one of the following options from the drop-down list:

  • smallest {best}
  • none {none}
  • fastest {speed}
  • standard {std}

The default option is to not compress (none).

Low Mark

If the size of the named queue falls below this value, then the named listeners that are specified in the Control List parameter are sent an activate message.

The low mark value applies to the queue as a whole, and not to the individual quelelets. The default value is 0.

High Mark

If the size of the named queue goes above this value, then a passivate message is sent to the listeners that are specified in the Control List parameter.

The high mark value applies to the queue as a whole, and not to the individual quelelets. The default value is 0.

Control List

If a high or low mark value is crossed, the appropriate message is sent to each listed listener. For example, if two listeners LISA and LISB are feeding the internal queue, listing them as LISA,LISB will cause each to receive the appropriate message.

Inhibit Add

If set, the queue will not accept new messages when its size reaches the high mark, and will resume accepting messages when the number of messages on the queue reaches the low mark. The effect of this inhibition can cascade through the application, controlling overall performance. The default value is false.

External Mark

When set to a number greater than 0, this causes the message to be stored in memory up to the specified number. When the queue grows larger than the specified number, the message is stored on disk. This is equivalent to splitting the memory and performance optimization for the messages in the queue, and is effective for large lists that may fill memory. The default value is 0.

Duration

Maximum time in seconds (allows xhxxmxxs format) that a document can remain in this channel, starting from when the document is added to the queue, including pends and retries. The default value is 24 hours.

Support Pending

If set, messages can be pended for later execution if the process flow calls for a Fail/Pending operation. Pending messages persist for a specified time and are retried at a specific interval. You might use a pending operation in the event that a message cannot be processed because an external resource is currently unavailable. The default value is false.

Retry Interval

Determines the interval (in seconds) between retrying pending requests. The default value is 600 seconds (10 minutes).

Reference: Ordered Queue Listener Special Registers

The following table lists and describes the Special Registers (SREGs) available on the Ordered Queue listener. These values can be used in the application for message routing and processing.

Name

Source

Level

Type

Description

endofgroup

Listener

Document

Boolean

Indicates the last message of a queuelet group.

iway.channel

Listener

System

String

Full name of the channel (may include channelname.inlet.listener).

iway.channelname

Listener

System

String

Channel name portion of the name from the full channel name of channelname.inlet.listener.

iway.inletname

Listener

System

String

Inlet name portion of the name from the full channel name of channelname.inletname.listener

iway.listener

Listener

System

String

Name of the listener.

iway.pid

System

System

String

Process ID of the server, if available.

iway.serverfullhost

System

System

String

Full host name of the server (includes domain).

iway.serverhost

System

System

String

Host name of the server.

iwayconfig

System

System

String

Current active configuration name.

iwayhome

System

System

String

Base at which the server is loaded.

iwayversion

System

System

String

Release version of the server.

iwayworkdir

System

System

String

Path to the base of the current configuration.

msgkey

Listener

Document

String

Batch key being dispatched.

msgsize

Listener

Document

Integer

Physical length of the message payload.

name

Listener

System

String

Assigned name of the master (same as iway.channel).

protocol

Listener

System

String

Protocol on which the message was received.

sortkey

Listener

Document

String

Values being sorted for ordered dispatch, if applicable.