Flow to disk limits the amount of direct and heap memory that can be occupied by messages. Once this limit is reached any new transient messages and all existing transient messages will be transferred to disk. Newly arriving transient messages will continue to go to the disk until the cumulative size of all messages falls below the limit once again.
By default the Broker makes 75% of the max direct available memory for messages. This memory is divided between all the queues across all virtual hosts defined on the Broker with a percentage calculated according to their current queue size. These calculations are refreshed periodically by the housekeeping cycle.
For example if there are two queues, one containing 75MB and the second 100MB messages respectively and the Broker has 1GB direct memory with the default of 75% available for messages. The first queue will have a target size of 320MB and the second 430MB. Once 750MB is taken by messages, messages will begin to flow to disk. New messages will cease to flow to disk when their cumulative size falls beneath 750MB.
Flow to disk is configured by Broker context variable
broker.flowToDiskThreshold. It is expressed as a size in bytes and defaults
to 75% of the JVM maximum heap size.
A number of statistics attributes are available on the Broker to allow monitoring
of the amount of utilized direct memory by the enqueued messages.
The total amount of allocated direct memory by the Broker can be determined by
checking Broker statistics usedDirectMemorySize. There is another
Broker level statistics directMemoryTotalCapacity to get the total amount
of allocated direct memory. Usually, the values reported by both statistics attributes
usedDirectMemorySize and directMemoryTotalCapacity are the same
or do not differ much.
The direct memory consumed by the VirtualHost messages is reported as
VirtualHost statistics inMemoryMessageSize. The current value of
VirtualHost direct memory threshold is exposed with statistics attribute
inMemoryMessageThreshold. When the value of inMemoryMessageSize is
greater than inMemoryMessageThreshold, the flow to disk is triggered to bring the amount of
direct memory consumed by the VirtualHost messages in-line with the
inMemoryMessageThreshold.
The Flow to Disk events are not reported as operational logs or
INFO logs due to quite frequent triggering of Flow to Disk
for messaging use cases requiring holding messages on the Broker side for some time.
As result, the Flow to Disk logs can quickly dominate the broker logs and cause unnecessary
disk consumption.
Though, if required, the Flow to Disk DEBUG logs can be enabled by adding
the following logging rule into the corresponding Broker logger.
Example 9.1. Flow to Disk logging rule
{
"name" : "DirectMemory",
"type" : "NameAndLevel",
"level" : "DEBUG",
"loggerName" : "org.apache.qpid.server.directMemory.*"
}
Please note, that the logger org.apache.qpid.server.directMemory.broker
is used by the Broker to report conditions when direct memory utilization exceeds the pred-defined
Broker threshold, whilst the logger org.apache.qpid.server.directMemory.virtualhost
is used to report conditions when direct memory utilization by the VirtualHost
messages exceeds the current value of the VirtualHost threshold.
Apache Qpid, Messaging built on AMQP; Copyright © 2015 The Apache Software Foundation; Licensed under the Apache License, Version 2.0; Apache Qpid, Qpid, Qpid Proton, Proton, Apache, the Apache feather logo, and the Apache Qpid project logo are trademarks of The Apache Software Foundation; All other marks mentioned may be trademarks or registered trademarks of their respective owners