Broker memory usage

It is possible for the broker to recieve frames at a rate faster than it can process. When this occurs a large number of Jobs and Events are produced. This can further slow down the system by increasing memory usage, causing the GC to to run frequently and generally compound the issue. This is undesireable.

High level solution

Ultimately, the broker needs to decide to cease creating new jobs until those that already exist have been processed. The broker will stop reading frames from the network layer. The servers network buffer will fill, and for OS will cease to read the socket as TCP flow control kicks in. The corresponding client side buffer to fill, and then writes to it will block.

When memory usage falls as the events are processed, the broker will start to process frames again, and normal operation will resume. However, if the broker does not recover sufficently quickly it is possible that the socket will time out and the connection will be closed.

Required changes

To completely implement the above solution, a number of changes are required.
The broker needs to be able to determine currently used / available memory. This can be obtained via JMX.
The threads which process Jobs and Events need to be signalled to pause, and to resume.
Protectio in the MINA layer of both the client and the broker needs to be enabled by default.
When a memory threshold is reached, the broker should fire an event which signals the job processing threads to pause. In future this event should be listened for by other mechanisms designed to mitigate the issue - such as flow to disk.

Apache Qpid

Resources

About Qpid

About AMQP

Broker memory usage

High level solution

Required changes