Understanding iWay Service Manager Thread Management

Topics:

iWay Service Manager (iSM) is engineered as a modern, multi-threaded server. It accepts messages on threaded inlets and processes many messages in parallel. Computers use threads as a means of allocating computational resources. Operating systems take responsibility for allocating those resources among threads. Although different operating systems may implement the heuristics of thread/resource allocation in different ways, the general purpose is to get as much work as possible through the application in a given period of time. The usual heuristic is to let one thread execute while another thread is awaiting completion of some slow task such as awaiting user input or the completion of an I/O event.

iSM is designed to maximize the work able to be performed on arriving messages to provide high throughput, or the number of messages that can be processed in a given period of time. Messages are isolated from each other during their processing, insuring that inadvertent interaction does not take place and that a failure of one message does not accidentally affect other messages.

In multiprocessor hardware, the threads are executed by separate CPUs, and the ability of the application to take advantage of multiple CPUs is referred to as scalability. iSM works to avoid the interlocks that reduce scalability in order to take maximum advantage of the resources available to its execution.

In iSM, some thread management is automatic; for example, the use of threads to watch for run-away process flows or to respond to console requests. These threads have minimal impact of the actual message execution and are not considered further in this section.

User configuration of listeners affects the use of threads in iSM. The following section explains how these settings work together and how they can be expected to affect the use of threads.

How Listeners Use Threads

Topics:

Inlets are the portion of a channel that accepts and prepares messages for execution. In an inlet, the portion that accepts and prepares messages for execution is called a listener. Listeners implement the specific protocol needed to obtain the message.

Each listener executes in its own thread, and never shares resources with other listeners. The manager starts and stops listeners, and gathers and distributes statistics on their operation. Listeners are designed to be reasonably inactive; they mostly await events and prepare them for execution. The execution is delegated to a worker thread, which in iSM is called a worker. When you set the thread count for a listener, you are declaring how many workers that listener has to accept and process work. A message is only bound to a worker during its execution, from when it is ready to be processed until that processing is complete. Completion of process is not meant in a business sense -- rather it means the handling on the message until the current operation is completed.

Because workers take time to start up and shut down, the server allocates to each listener a pre-initialized set of workers, called the listener's worker pool. The Multithreading field on the configuration console instructs iSM startup on how many workers are to be allocated to the pool. When a message arrives, the listener selects a worker from the pool, assigns the message to it, and then moves on to await the arrival of the next message.

The consequence of the multithreading number varies from protocol to protocol. Protocols that hold queues, such as any of the queue listeners like MQ or internal, grab a message and request a worker for its execution. The listener waits until a worker is available and then hands off the work. If the number of workers is less than the number in the queue, then the messages wait in the queue until a worker is available for their execution. Given an expected arrival distribution and estimated time of service for a message, you can calculate how long a message will wait.

Maximum Listener Thread Settings

There are other listeners that cannot wait for any period of time. An example is TCP or HTTP. The listener must accept the message and pass it to a worker for execution or else the message gets lost.

If there are no workers available to handle the message, you can turn many of the listeners into queue listeners by writing the message onto an internal queue and then getting back to the sender with an "I got it" message. For example, this technique is used (although not with the internal listener) for AS2 -- listeners take the message, return an MDN and then proceed to handle the message. In this way, the time for a worker to become available is minimized.

You may want to increase the number of workers running in parallel to handle the actual traffic. For example, in HTTP you might usually expect five simultaneous messages (five workers) but once in a while you need ten workers. To do this, set threads to 5 and max to 10. When the sixth message arrives, the worker pool "sees" that there are no workers available and creates a sixth worker to get the message. In this way, the message is not lost. If the sixth worker finishes, it goes back into the pool which now has six. However, because each worker takes resources, the system watches the pool and after some time, it destroys the sixth worker, since all you need is five. The extra was to handle the peak situation.

Some listeners do not offer the ability to grow threads, and some do.

You cannot set the max threads to a number which is less than the multithreading count. The threading count is the number of waiting workers. They are pre-started to avoid setup time when messages arrive, so the size of the pool is the multi thread count. Do not confuse this with OS threads; it is not guaranteed that they map one to one, although in most JVMs they do.

The Internal Listener and its Affect on the Threading Model

The internal protocol is an iSM-managed queuing protocol that accepts messages from other channels for asynchronous execution. The main purposes of the internal protocol are:

  • to offer the opportunity to change from one threading model to another.
  • to alleviate stress on another channel by executing work independently of the response to the sender.
  • to break up (modularize) the application for simpler development and maintenance.

When the Internal Emit Service is used in a process flow, the message is immediately added to a named internal listener queue. For more information, see the iWay Cross-Channel Services Guide.

When the internal emit service sends the message to the internal queue, it immediately completes, passing the document out for any further processing. A common use of the internal listener is to have the first channel perform needed preliminary processing and then immediately send a receipt message (called an MDN for Message Delivery Notification) while the actual business processing proceeds without reference to the actual source. In the AS2 protocol, the logic in the internal listener channel could prepare an asynchronous notification of the result of the business processing. The recipient would need to correlate an asynchronously received result with the original message. Because the internal listener channel displays the original transaction ID, the asynchronous message can contain the needed information to assist in the correlation.

The diagram shows the link between two channels. The inlet and outlet handling is shown as purple boxes. The top channel process flow sends the message to the internal listener channel and then immediately proceeds to complete operation of the first stage.

The internal listener channel can operate on a different number of threads than the original recipient of the message. For example, it may be necessary to receive the messages on a single thread to preserve input message number tracking. However, once the message is determined to be in the correct sequence, the actual execution can proceed on multiple threads.

Alternatively, it may be required to receive on a large number of threads, but for a final stage of execution only one thread might be possible. This type of requirement can occur because of restrictions on a resource (for example, a fixed number of connections to an online system, such as SAP) or the need to control output sequence numbering in some manner.

The internal listener offers two forms of application flow throttling control:

  • Passivation

    Passivation refers to sending instructions to other listeners (channels) to cease acquiring messages from their own input sources. The internal listener does this by allowing the configuration of a high and low mark.

    When the number of messages on the internal listeners queue exceeds the high mark, the passivation is sent to the identified channels. The internal listener continues to acquire messages from its queue, and when the number in the queue reaches the low mark, the identified channels are reactivated. Channels not passivated can continue to acquire messages for their processing and add messages to the internal queue.

  • Inhibition

    Inhibition can be configured for the channel. It causes the internal queue itself to reject attempts to add to the queue while the number of messages on the queue exceeds the high mark. The inhibition state is reset when the queue reaches the low mark. The flows attempting to add to the inhibited queue enter a pause state, and the add attempt may time out. In such a timeout case, a status document is passed down the timeout edge of the feeding flow, if available. Use of inhibition can have a cascading effect on the application, causing a general pause in processing while the internal channel catches up with the required work.

The internal channel also supports message priority. Usually, the message is placed on the queue at a middle priority. Using priorities allows the sender to reorder message execution through the channels as required. As a best practice, iWay recommends using priorities only if needed. Use priorities one up or one down from the offered value. Avoid using the highest priority value (9). Priorities are relative to other messages, and higher numbers do not necessarily result in faster processing.

The Ordered Listener

A specialized form of the internal listener is the ordered listener. This is a patented facility that automatically sorts and relates messages to ensure that the messages are presented to the listener in a desired order (time, lexical or numeric compare, or application-specific). In addition, an ordered listener ensures that the messages in any ordered group are processed sequentially while also processing multiple groups in parallel. This facility simplifies the development of applications that require identified processing orders.

Getting Thread Information

When operating iWay Service Manager (iSM) in a command window or Telnet console, many commands exist to assist in understanding (and possibly debugging) the current state of the server. One of the more useful commands is threads, as shown in the following image.

In this particular display of an iSM instance with two listeners, SOAP1 and file1, you can see the thread running each protocol listener (the channel) and one thread for each message handler (worker) that it controls. There will be one thread for each currently available worker in the pool.

The SOAP listener's sub-threads include one for handling HTTP and one for file management. There is also a repository control thread that in this example is related to the HSQL data base.

The threads are controlled by the manager thread which is in charge of the server. The manager controls five console threads, two channels, and the server's heartbeat clock used to coordinate time-related issues such as detecting run-away process flows.

Best Practices

The flexibility of iWay's threading models enables the server to be tailored to a wide variety of application needs. While no set of suggestions are valid for all applications, some overarching guidelines should be considered as best practices.

  1. Decide before you start configuring what performance you need. What constitutes success? Just going fast for its own sake takes up resources that might be better spent elsewhere.
  2. Use as few threads as possible to keep up with the application's needs.
  3. Sometimes fewer threads give better performance than many threads. For example, applications configured using only a few threads in reading from an IBM Websphere MQ queue have been clocked as completing work faster than when the listener is configured with many threads.
  4. Try to keep the number of worker threads in active use to a small multiple of the number of available processors.
  5. Process flows with long wait times built in, such as those that address external systems over communications lines, will need more threads that those that simply work in memory doing such things as transformations.
  6. Don't be afraid to try various configurations. Measure as you go. And remember that threads are not the only resource that affects performance: consider memory availability and other processes running on the computer. And when measuring, remember that Java optimizes code after many transactions run through the server. Do not start to measure until you have run at least 1000 transactions.