Using Qpid Dispatch

You can use Dispatch Router to flexibly route messages between any AMQP-enabled endpoints, including clients, servers, and message brokers. Dispatch Router provides the following benefits:

Dispatch Router supports the following industry-recognized standards and network protocols:

Before using Dispatch Router, you should be familiar with AMQP and understand some key concepts about Dispatch Router.

In this document, sudo is used for any command that requires root privileges. You should always exercise caution when using sudo, as any changes can affect the entire system.

Dispatch Router is distributed as a set of RPM packages, which are available for yum/dnf-based Linux distributions. Alternatively, you can build the Dispatch Router from source.

The router’s configuration file (qdrouterd.conf) controls the way in which the router functions. The default configuration file contains the minimum number of settings required for the router to run. As you become more familiar with the router, you can add to or change these settings, or create your own configuration files.

By default, the router configuration file defines the following settings for the router:

After installing Dispatch Router, you start the router by using the qdrouterd command.

After starting the router, send some test messages to see how the router can connect two endpoints by distributing messages between them.

This procedure demonstrates a simple configuration consisting of a single router with two clients connected to it: a sender and a receiver. The receiver wants to receive messages on a specific address, and the sender sends messages to that address.

A broker is not used in this procedure, so there is no "store and forward" mechanism in the middle. Instead, the messages flow from the sender, through the router, to the receiver only if the receiver is online, and the sender can confirm that the messages have arrived at their destination.

Apache Qpid Proton Python must be installed. For more information, see https://qpid.apache.org/proton/.

After you successfully install a standalone router and use it to distribute messages between two clients, you can configure the router to make it production-ready, and add additional routers to form a router network.

The router’s configuration is defined in the router configuration file. You can access this file to view and modify that configuration.

Before you can make changes to a router configuration file, you should understand how the file is structured.

The configuration file contains sections. A section is a configurable entity, and it contains a set of attribute name-value pairs that define the settings for that entity. The syntax is as follows:

The router configuration file supports pattern matching and wildcards to enable you to match multiple values for certain attributes. However, the syntax varies based on the type of entity that you are configuring.

You can use different methods for changing a router’s configuration based on whether the router is currently running, and whether you want the change to take effect immediately.

The router’s configuration file controls the way in which the router functions. The default configuration file contains the minimum number of settings required for the router to run. As you become more familiar with the router, you can add to or change these settings, or create your own configuration files.

When you installed Dispatch Router, the default configuration file was added at the following path: /etc/qpid-dispatch/qdrouterd.conf. It includes some basic configuration settings that define the router’s operating mode, how it listens for incoming connections, and routing patterns for the message routing mechanism.

The router’s default configuration settings enable the router to run with minimal configuration. However, you may need to change some of these settings for the router to run properly in your environment.

Dispatch Router provides listeners that accept client connections. A client connecting to a router listener uses the same methods that it would use to connect to a broker. From the client’s perspective, the router connection and link establishment are identical to broker connection and link establishment.

Several types of listeners are defined by their role.

You can configure Dispatch Router to create outbound connections to messaging brokers or other AMQP entities using connectors. A connector is defined with the network address of the broker and the name or names of the resources that are available in that broker. When a router connects to a broker through a connector, it uses the same methods a normal messaging client would use when connecting to the broker.

Several types of connectors are defined by their role.

When a router connects to a broker, the broker might provide backup connection data that the router can use if the primary connection fails. If the primary connection fails, the router attempts to reconnect by using a combination of the primary and — if provided — backup connections in round-robin fashion until the connection is successful. For more information about viewing the backup connection data provided by the broker, see Managing Connectors.

In Dispatch Router, a connector attempts to maintain an open network transport connection to the configured remote host and port. If the connection cannot be established, the connector will continually retry until the connection is established. If an established connection is lost, the connector shall immediately attempt to re-establish the connection.

Connection Failover is a mechanism by which the remote host can provide alternate connection information for the connector to use in the event the established connection is lost. In this case, rather than attempting to re-establish the connection to the same host, the connector shall try the alternate hosts as well. This is useful in the case where the remote host is formed by a cluster or array of servers providing the same service.

Dispatch Router can participate in Connection Failover as both a client (initiator of transport connections) and a server (recipient of transport connections). In the client role, connectors shall always honor the failover lists provided by connected servers. As a server, a listener may be configured to provide a failover list to the clients that connect to it.

Listener attribute failoverUrls is an optional component that contains a comma-separated list of URLs to be used as backups for this listener. Each URL is of the form:

When a client establishes a connection to this listener, it will be provided with this list of backup URLs to be used in the event that this connection is lost.

As an example, a listener may be configured like this:

You can configure Dispatch Router to communicate with clients, routers, and brokers in a secure way by authenticating and encrypting the router’s connections. Dispatch Router supports the following security protocols:

You can configure policies to secure messaging resources in your messaging environment. Policies ensure that only authorized users can access messaging endpoints through the router network, and that the resources on those endpoints are used in an authorized way.

Dispatch Router provides the following types of policies:

The resource limits defined in global and vhost policies are applied to user connections only. The limits do not affect inter-router connections or router connections that are outbound to waypoints.

While you can use either message routing or link routing to deliver messages to a destination, they differ in several important ways. Understanding these differences will enable you to choose the proper routing approach for any particular use case.

With message routing, routing is performed on messages as producers send them to a router. When a message arrives on a router, the router routes the message and its settlement based on the message’s address and routing pattern.

With message routing, you can do the following:

Link routing provides an alternative strategy for brokered messaging. A link route represents a private messaging path between a sender and a receiver in which the router passes the messages between end points. You can think of a link route as a "virtual connection" or "tunnel" that travels from a sender, through the router network, to a receiver.

With link routing, routing is performed on link-attach frames, which are chained together to form a virtual messaging path that directly connects a sender and receiver. Once a link route is established, the transfer of message deliveries, flow frames, and dispositions is performed across the link route.

There are three router operating modes, as specified in the router section of the configuration:

When deploying a Dispatch Router network, it is very important that the security and integrity of that network be considered. This is particularly true of the interior portion of the network. The integrity, security, and availability of a network will be severely compromised if unauthorized routers are permitted to join the interior network, or worse, unauthorized endpoints assuming the role of a router by establishing an inter-router connection into the network. For this reason, it is recommended that the interior routers of a network use a strong authentication mechanism to identify themselves to each other.

A good way to establish solid interior router security is to create an x.509 certificate authority that oversees the network with individual certificates generated for each interior router in the network. The inter-router listeners are then configured to use the certificate authority as the authenticator for each incoming inter-router connection (using SASL mechanism EXTERNAL to enable x.509 client certificate authentication). Edge connections and direct endpoint connections can use different levels of security, depending on the needs of the deployment.

Using this method, a new router cannot join the network until the owner of the certificate authority issues a new certificate for the new router. And an intruder wishing to spoof an interior router cannot do so unless it has a valid x.509 certificate issued by the network’s certificate authority.

When a connection is established between routers, message traffic flows in both directions across that connection. Each connection has a client side (connector) and a server side (listener) for the purposes of connection establishment, but once the connection is established, the two sides become equal participants in a bi-directional connection. For the purposes of routing AMQP traffic across the network, the direction of connection establishment is not relevant.

When establishing inter-router connections, the deployer must choose which router will be the listener and which will be the connector. There should be only one connection between any pair of routers. There is no need, and it is in fact undesirable, to create two connections in opposite directions between two routers. Don’t do it.

The selection of connector vs. listener is sometimes arbitrary because it doesn’t matter. In other cases, the direction may be important because of IP network boundaries and firewalls. For example, to connect a router in a private IP network to another router in a public location (i.e. a public cloud provider), the router in the private network must have the connector and the router in the public location must have the listener. This is because the public location cannot reach the private location via TCP/IP without the use of VPNs or other firewall features designed to allow public-to-private access.

Sometimes the choice of connector vs. listener is dictated by the shape of the topology. For example, a star-topology that has a series of routers connected to one or two central "hub" routers would most easily be set up with listeners on the hub and connectors on the spokes. This way, new spoke routers may be added without changing the configuration of the hub.

Dispatch Router logs are broken into different categories called logging modules. Each module provides important information about a particular aspect of Dispatch Router.

You can specify the types of events that should be logged, the format of the log entries, and where those entries should be sent.

You may need to view log entries to diagnose errors, performance problems, and other important issues. A log entry consists of an optional timestamp, the logging module, the logging level, and the log message.

You can use qdstat to view the status of routers on your router network. For example, you can view information about the attached links and configured addresses, available connections, and nodes in the router network.

You can use qdmanage to view and modify the configuration of a running router at runtime. Specifically, qdmanage enables you to create, read, update, and delete the sections and attributes in the router’s configuration file without having to restart the router.

The Dispatch Router management schema provides a set of management entities that you can use to configure and operate a router.

For details about the management entities, see Dispatch Router Management Schema.

Offering path redundancy means designing the network topology in a way that even when one or more routers go down or even connections between them, each destination is always reachable following alternate paths through the routers that are still part of the network.

Consider the following simple scenario :

The "Router.A" configuration is something like following.

There is only one listener in order to accept client connections and two connector entities for connecting to the other two routers.

The "Router.B" configuration is the following.

It has two listener entities in order to listen for connections from clients and from other routers in the network (in this case from the "Router.A" and "Router.C").

Finally, quite similar is the "Router.C" configuration.

It has two listener entities in order to listen for connections from clients and from other routers in the network (in this case from the "Router.A") and finally it has a connector (for connecting to the "Router.B")

Consider a sender client connected to "Router.A" and attached to my_address address which start to send messages (that is, 10 messages) and a receiver client connected to the "Router.B" and attached to the same address.

Starting the receiver, it waits for messages with no output on the console.

Starting the sender, all the messages flow through "Router.A" and "Router.B" reaching the receiver; at this point the messages are all confirmed at sender side.

At same time, the receivers shows the messages received through the "Router.B".

The path redundancy is provided by the other available path through the "Router.A", "Router.C" and then "Router.B". It means that if the connection between "Router.A" and "Router.B" goes down, the alternative path is used to reach the receiver.

Now, consider a fault on the "Router.B"; the receiver is not reachable anymore on that path but it can connect to the "Router.C" in order to continue to receive messages from the sender which does not know what’s happened and it can continue to send messages to the "Router.A" in order to reach the receiver.

The receiver is still reachable in order to get messages from the sender as displayed in the console output.

In order to have temporal decoupling in a solution based on Dispatch Router, adding one or more brokers is a must for its "store and forward" feature. Choosing the right topology, it is possible to have a solution which offers reliability with both path redundancy and permanent storing for messages.

Consider the following simple scenario :

The receiver client can be offline when the sender starts to send messages because they’ll be stored into the queue permanently; coming back online, the receiver can get messages from the queue itself without message loss.

The "Router.A" configuration is something like following.

It has a listener for accepting incoming connections from clients and two connector entities in order to connect to the other routers. The queue named my_queue on the broker is exposed by a waypoint.

The "Router.B" configuration is the following.

It can accept incoming connections from clients and from other routers (in this case the "Router.A") and connects to the broker. The queue named my_queue on the broker is exposed by a waypoint with the related auto-links in both directions in order to send and receive messages to/from the queue itself.

Finally, the simple "Router.C" configuration.

It can accept incoming connections from clients and from other routers (in this case the "Router.A"). Initially there is no connection between this router and the broker.

First of all, thanks to the broker and its "store and forward" feature, the sender can connect to the "Router.A" and start to send messages even if the receiver is not online in that moment. Using the Python sample from the Qpid Proton library, the console output is like following.

All messages are confirmed because they reached the queue inside the broker through "Router.A" and "Router.B"; it is confirmed using the qdstat tool.

For the "Router.B", there are 10 messages as output (from the router to the broker) on the my_queue address.

Starting the receiver connected to the "Router.A", it gets all the available messages from the queue.

Using the qdstat tool on the "Router.B" another time, the output is like following.

For the "Router.B", there are 10 messages as input (from the broker to the router) on the my_queue address.

Now, consider a fault on the "Router.B"; in this case the broker is not reachable but it is possible to set up path redundancy through the "Router.C".

Using the qdmanage tool, it is possible to configure the waypoint on my_queue address, the related auto-links in both directions and finally the connector instance in order to enable the connection to the broker.

The "Router.C" configuration changes in the same way as "Router.B". It can accept incoming connections from clients and from other routers (in this case the "Router.A") and connects to the broker. The queue named my_queue on the broker is exposed by a waypoint with the related auto-links in both directions in order to send and receive messages to/from the queue itself.

At this point, the sender can connect to the "Router.A" for sending messages to the queue in the broker thanks to the "Router.C".

All messages are confirmed because they reached the queue inside the broker through "Router.A" and "Router.C"; it is confirmed using the qdstat tool.

For the "Router.C", there are 10 messages as output (from the router to the broker) on the my_queue address.

Starting the receiver connected to the "Router.A", it gets all the available messages from the queue.

Using the qdstat tool on the "Router.C" another time, the output is like following.

For the "Router.C", there are 10 messages as input (from the broker to the router) on the my_queue address.

Every broker has limits in terms of queue size but in order to overcome this problem, one possible solution is "sharding" queues : in that way a single queue is divided in more "shards" (chunks) each on a different broker. It means that such solution needs more than one broker instance in order to host a shard on each of them. Of course, a sender connected to one of these brokers can send messages to the shard hosted only on that broker. At same time, a receiver connected to a broker can get messages from the shard that is hosted on that broker and can not see available messages in the shards hosted on the other brokers, even if they are all parts of the same queue.

The big problem in this scenario, designed only with brokers, is that a receiver can be stucked on an empty shard without reading any messages while the shards on the other brokers have messages to deliver. it is a real problem because the receiver is interested in receiving messages from the whole queue and it does not take care if it is shared or not. Because of this problem, the receiver sees the queue as empty even if it is not so true due to the sharding and the messages available on the other shards.

The above problem can be solved adding a Dispatch Router instance in the network in front of the brokers and leverage on its waypoint feature with related auto-links.

Consider the following simple scenario :

With such solution and connecting to the "Router.A", sender and receiver do not know anything about sharding; they want send and receive messages to/from the whole queue that is the only thing they are aware of. They are both connected to the router and see only one address (related to the queue).

The "Router.A" configuration is something like following.

The router has a listener for incoming connection from clients and two connector instances in order to connect to both brokers. The whole queue is named my_queue hosted in terms of shards on both brokers and the router is configured with a waypoint for that address. Finally, there are two auto-links in both directions for that queue on both brokers.

Using the Python sample from the Qpid Proton library, the sender can connect to the "Router.A" and start to send messages to the queue; the console output is like following.

All messages are confirmed because they reached the queue and, thanks to the default balanced distribution on the address, the messages are delivered to both shards on the brokers (5 messages per shard). Using the qdstat tool on the router, the distribution is clear.

There are the out links (from router to brokers) for the my_queue address (id values 7 and 9) which have each 5 deliveries. It shows messages distributed across brokers and related shards for the queue; it is confirmed by the different connections they are tied (conn id values 1 and 2).

Starting the receiver connected to the "Router.A", it gets all the available messages from the queue.

As for the sender, they are received through both the brokers and related shards. it is confirmed using the qdstat tool.

There are the in links (from brokers to router) for the my_queue address (id values 6 and 8) which have each 5 deliveries. It shows messages distributed across brokers and related shards for the queue; it is confirmed by the different connections they are tied (conn id values 1 and 2).

One disadvantage of sharded queues is that the receiver might receive messages "out of order" even with very good performance.

Dispatch Router should, in theory, work with any client that is compatible with AMQP 1.0. The following clients have been tested:

Dispatch Router is an AMQP router and as such, it provides extensions, code-points, and semantics for routing over AMQP. This page documents the details of Dispatch Router’s use of AMQP.

To generate a SASL database to store credentials, enter the following command:

This command creates or updates the specified SASL database, and adds the specified user name to it. The command also prompts you for the user name’s password.

The full user name is the user name you entered plus the domain name (USER_NAME@DOMAIN_NAME). Providing a domain name is not required when you add a user to the database, but if you do not provide one, a default domain will be added automatically (the hostname of the machine on which the tool is running). For example, in the command above, the full user name would be user1@domain.com.

To view the user names stored in the SASL database:

To use the SASL database to provide authentication in Dispatch Router:

Revised on 2019-03-29 09:45:01 EDT