Appendix C. Operational Logging
Prev		Next

Appendix C. Operational Logging

The Broker will, by default, produce structured log messages in response to key events in the lives of objects within the Broker. These concise messages are designed to allow the user to understand the actions of the Broker in retrospect. This is valuable for problem diagnosis and provides a useful audit trail.

Each log message includes details of the entity causing the action (e.g. a management user or messaging client connection), the entity receiving the action (e.g. a queue or connection) and a description of operation itself.

The log messages have the following format:

    [Actor] {[Subject]} [Message Id] [Message Text]

Where:

Actor is the entity within the Broker that is performing the action. There are actors corresponding to the Broker itself, Management, Connection, and Channels. Their format is described in the table below.
Subject (optional) is the entity within the Broker that is receiving the action. There are subjects corresponding to the Connections, Channels, Queues, Exchanges, Subscriptions, and Message Stores. Their format is described in the table below.
Some actions are reflexive, in these cases the Actor and Subject will be equal.
Message Id is an identifier for the type of message. It has the form three alphas and four digits separated by a hyphen AAA-9999.
Message Text is a textual description

To illustrate, let's look at two examples.

CON-1001 is used when a messages client makes an AMQP connection. The connection actor (con) provides us with details of the peer's connection: the user id used by the client (myapp1), their IP, ephemeral port number and the name of the virtual host. The message text itself gives us further details about the connection: the client id, the protocol version in used, and details of the client's qpid library.

[con:8(myapp1@/127.0.0.1:52851/default)] CON-1001 : Open : Destination : AMQP(127.0.0.1:5672) : Protocol Version : 0-10 : Client ID : myapp1 : Client Version : 8.0.6 : Client Product : qpid

QUE-1001 is used when a queue is created. The connection actor con tells us details of the connection performing the queue creation: the user id used by the client (myapp1), the IP, ephemeral port number and the name of the virtual host. The queue subject tells use the queue's name (myqueue) and the virtualhost. The message itself tells us more information about the queue that is being created.

[con:8(myapp1@/127.0.0.1:52851/default)/ch:0] [vh(/default)/qu(myqueue)] QUE-1001 : Create : Owner: myapp1 Transient

The first two tables that follow describe the actor and subject entities, then the later provide a complete catalogue of all supported messages.

Table C.1. Actors Entities

Actor Type	Format and Purpose
Broker	[Broker]
Broker	Used during startup and shutdown
Management	[mng:`userid`(`clientip`:`ephemeralport`)]
Management	Used for operations performed by the Web Management interfaces.
Connection	[con:`connectionnumber`(`userid`@/`clientip`:`ephemeralport`/`virtualhostname`)]
Connection	Used for operations performed by a client connection. Note that connections are numbered by a sequence number that begins at 1.
Channel	[con:`connectionnumber`(`userid`@/`clientip`:`ephemeralport`/`virtualhostname`/ch:`channelnumber`)]
Channel	Used for operations performed by a client's channel (corresponds to the JMS concept of Session). Note that channels are numbered by a sequence number that is scoped by the owning connection.
Group	[grp(/`groupname`)/vhn(/`virtualhostnode name`)]
Group	Used for HA. Used for operations performed by the system itself often as a result of actions performed on another node..

Table C.2. Subject Entities

Subject Type	Format and Purpose
Connection	[con:`connectionnumber`(`userid`@/`clientip`:`ephemeralport`/`virtualhostname`)]
Connection	A connection to the Broker.
Channel	[con:`connectionnumber`(`userid`@/`clientip`:`ephemeralport`/`virtualhostname`/ch:`channelnumber`)]
Channel	A client's channel within a connection.
Subscription	[sub:`subscriptionnumber`(vh(/`virtualhostname`)/qu(`queuename`)]
Subscription	A subscription to a queue. This corresponds to the JMS concept of a Consumer.
Queue	[vh(/`virtualhostname`)/qu(`queuename`)]
Queue	A queue on a virtualhost
Exchange	[vh(/`virtualhostname`)/ex(`exchangetype`/`exchangename`)]
Exchange	An exchange on a virtualhost
Binding	[vh(/`virtualhostname`)/ex(`exchangetype`/`exchangename`)/qu(`queuename`)/rk(`bindingkey`)]
Binding	A binding between a queue and exchange with the giving binding key.
Message Store	[vh(/`virtualhostname`)/ms(`messagestorename`)]
Message Store	A virtualhost/message store on the Broker.
HA Group	[grp(/`group name`)]
HA Group	A HA group

The following tables lists all the operation log messages that can be produced by the Broker, and the describes the circumstances under which each may be seen.

Table C.3. Broker Log Messages

Message Id	Message Text / Purpose
BRK-1001	Startup : Version: `version` Build: `build`
BRK-1001	Indicates that the Broker is starting up
BRK-1002	Starting : Listening on `transporttype` port `portnumber`
BRK-1002	Indicates that the Broker has begun listening on a port.
BRK-1003	Shutting down : `transporttype` port `portnumber`
BRK-1003	Indicates that the Broker has stopped listening on a port.
BRK-1004	Qpid Broker Ready
BRK-1004	Indicates that the Broker is ready for normal operations.
BRK-1005	Stopped
BRK-1005	Indicates that the Broker is stopped.
BRK-1006	Using configuration : `file`
BRK-1006	Indicates the name of the configuration store in use by the Broker.
BRK-1008	`delivered\|received` : `size` kB/s peak : `size` bytes total
BRK-1008	Statistic - bytes delivered or received by the Broker.
BRK-1009	`delivered\|received` : `size` msg/s peak : `size` msgs total
BRK-1009	Statistic - messages delivered or received by the Broker.
BRK-1010	Platform : JVM : `vendor` version: `version` OS : `operating system vendor` version: `operating system version`} arch: `processor architecture` cores: `number of CPU cores`
BRK-1010	Key information about the environment hosting the Broker
BRK-1011	Maximum Memory : Heap : `size` bytes Direct : `bytes` size
BRK-1011	Configured memory paramters for the Broker.
BRK-1012	Management Mode : User Details : `management node user id` / `management mode password`
BRK-1012	Used when Broker is started in management mode to indicate the management credentials that may be used connect to the Broker.
BRK-1016	Fatal error : `root cause` : See log file for more information
BRK-1016	Indicates that broker was shut down due to fatal error.
BRK-1017	Process : PID `process identifier`
BRK-1017	Process identifier (PID) of the Broker process.
BRK-1018	Operation : `operation name`
BRK-1018	Indicates that the named operation has been invoked

Table C.4. Management Log Messages

Message Id	Message Text / Purpose
MNG-1001	`type` Management Startup
MNG-1001	Indicates that a Management plugin is starting up. Supported by Web management plugin.
MNG-1002	Starting : `type` : Listening on `transporttype` port `port`
MNG-1002	Indicates that a Management plugin is listening on the given port.
MNG-1003	Shutting down : `type` : port `port`
MNG-1003	Indicates that a Management plugin is ceasing to listen on the given port.
MNG-1004	`type` Management Ready
MNG-1004	Indicates that a Management plugin is ready for work.
MNG-1005	`type` Management Stopped
MNG-1005	Indicates that a Management plugin is stopped.
MNG-1007	Open : User `username`
MNG-1007	Indicates the opening of a connection to Management has by the given username.
MNG-1008	Close : User `username`
MNG-1008	Indicates the closing of a connection to Management has by the given username.

Table C.5. Virtual Host Log Messages

Message Id	Message Text / Purpose
VHT-1001	Created : `virtualhostname`
VHT-1001	Indicates that a virtualhost has been created.
VHT-1002	Closed
VHT-1002	Indicates that a virtualhost has been closed. This occurs on Broker shutdown.
VHT-1005	Unexpected fatal error
VHT-1005	Virtualhost has suffered an unexpected fatal error, check the logs for more details.
VHT-1006	Filesystem is over `size in %` per cent full, enforcing flow control.
VHT-1006	Indicates that virtual host flow control is activated when the usage of file system containing Virtualhost message store exceeded predefined limit.
VHT-1007	Filesystem is no longer over `size in %` per cent full.
VHT-1007	Indicates that virtual host flow control is deactivated when the usage of file system containing Virtualhost message falls under predefined limit.
VHT-1008	Operation : `operation name`
VHT-1008	Indicates that the named operation has been invoked

Table C.6. Queue Log Messages

Message Id	Message Text / Purpose
QUE-1001	Create : Owner: `owner` `AutoDelete` [`Durable`] `Transient` Priority: `numberofpriorities`
QUE-1001	Indicates that a queue has been created.
QUE-1002	Deleted
QUE-1002	Indicates that a queue has been deleted.
QUE-1003	Overfull : Size : `size in bytes`, Capacity : `resumesize in bytes`, Messages : `size in messages`, Message Capacity : `resumesize in messages`
QUE-1003	Indicates that a queue has exceeded its permitted capacity when Producer Flow Control overflow policy is used. See Section 4.7.5, “Controlling Queue Size” for details.
QUE-1004	Underfull : Size : `size in bytes`, Capacity : `resumesize in bytes`, Messages : `size in messages`, Message Capacity : `resumesize in messages`
QUE-1004	Indicates that a queue has fallen to its resume capacity when Producer Flow Control overflow policy is used. See Section 4.7.5, “Controlling Queue Size” for details.
QUE-1005	Dropped : `number` messages, Depth : `size` bytes, `size` messages, Capacity : `limit` bytes, `limit` messages
QUE-1005	Indicates that a given number of messages is deleted when Ring overflow policy is used and any of queue capacity limits is breached . See Section 4.7.5, “Controlling Queue Size” for details.
QUE-1016	Operation : `operation name`
QUE-1016	Indicates that the named operation has been invoked

Table C.7. Exchange Log Messages

Message Id	Message Text / Purpose
EXH-1001	Create : [`Durable`] Type: `type` Name: `exchange name`
EXH-1001	Indicates that an exchange has been created.
EXH-1002	Deleted
EXH-1002	Indicates that an exchange has been deleted.
EXH-1003	Discarded Message : Name: `exchange name` Routing Key: `routing key`
EXH-1003	Indicates that an exchange received a message that could not be routed to at least one queue. queue has exceeded its permitted capacity. See Section 4.6.4, “Unrouteable Messages” for details.
EXH-1004	Operation : `operation name`
EXH-1004	Indicates that the named operation has been invoked

Table C.8. Binding Log Messages

Message Id	Message Text / Purpose
BND-1001	Create : Arguments : `arguments`
BND-1001	Indicates that a binding has been made between an exchange and a queue.
BND-1002	Deleted
BND-1002	Indicates that a binding has been deleted

Table C.9. Connection Log Messages

Message Id	Message Text / Purpose
CON-1001	Open : Destination : `target port` : Protocol Version : `protocol version` : Client ID : `clientid` : Client Version : `client version` : Client Product : `client product`
CON-1001	Indicates that a connection has been opened. The Broker logs one of these message each time it learns more about the client as the connection is negotiated.
CON-1002	Close
CON-1002	Indicates that a connection has been closed. This message is logged regardless of if the connection is closed normally, or if the connection is somehow lost e.g network error.
CON-1003	Closed due to inactivity
CON-1003	Used when heart beating is in-use. Indicates that the connection has not received a heartbeat for too long and is therefore closed as being inactive.
CON-1004	Connection dropped
CON-1004	Indicates that a connection has been unexpectedly closed by the peer. This usually occurs if a machine hosting an application fails or the application's process is abruptly terminated.
CON-1005	Client version `version` logged by validation
CON-1005	Indicates that a connection has been received from client with a version number that is configured to be logged. This feature may help teams manage software currency.
CON-1006	Client version `version` rejected by validation
CON-1006	Indicates that a connection attempt has been received from client with a version number that is configured to be rejected. This feature may help manage software currency.
CON-1007	Connection close initiated by operator
CON-1007	Indicates that a connection has been closed by the actions of an Operator using manangement.
CON-1009	Uncommitted transaction(s) contains `size` bytes of incoming message data exceeding `size` bytes limit. Messages will be flowed to disk.
CON-1009	Warns that the transactions associated with this connection contain so much uncommitted data that a threshold has been breached. The connection responds by flowing the messages already associated with the transactions and any new messages to disk. The connection reverts to normal behaviour once the quantity of uncommitted data falls beneath the threshold. Normally this happens when the transactions commit or rollback.
CON-1010	Open Transaction : `time` ms
CON-1010	Indicates that a messaging transaction has been open for longer than that permitted. See Section 9.3, “Transaction Timeout” for more details.
CON-1011	Idle Transaction : `time` ms
CON-1011	Indicates that a messaging transaction has been idle for longer than that permitted. See Section 9.3, “Transaction Timeout” for more details.

Table C.10. Channel Log Messages

Message Id	Message Text / Purpose
CHN-1001	Create
CHN-1001	Indicates that a channel (corresponds to the JMS concept of Session) has been created.
CHN-1002	Flow Started
CHN-1002	Indicates message flow to a session has begun.
CHN-1003	Close
CHN-1003	Indicates that a channel has been closed.
CHN-1004	Prefetch Size (bytes) `size` : Count `number of messages`
CHN-1004	Indicates the prefetch size in use by a channel.
CHN-1005	Flow Control Enforced (Queue `queue name`)
CHN-1005	Indicates that producer flow control has been imposed on a channel owning to excessive queue depth in the indicated queue. Produces using the channel will be requested to pause the sending of messages. See Section 4.7.5, “Controlling Queue Size” for more details.
CHN-1006	Flow Control Removed
CHN-1006	Indicates that producer flow control has been removed from a channel. See Section 4.7.5, “Controlling Queue Size” for more details.
CHN-1009	Discarded message : `message number` as no alternate exchange configured for queue : `queue name`{1} routing key : `routing key`
CHN-1009	Indicates that a channel has discarded a message as the maximum delivery count has been exceeded but the queue defines no alternate exchange. See Section 9.4.2, “Maximum Delivery Count” for more details. Note that `message number` is an internal message reference.
CHN-1010	Discarded message : `message number` as no binding on alternate exchange : `exchange name`
CHN-1010	Indicates that a channel has discarded a message as the maximum delivery count has been exceeded but the queue's alternate exchange has no binding to a queue. See Section 9.4.2, “Maximum Delivery Count” for more details. Note that `message number` is an internal message reference.
CHN-1011	Message : `message number` moved to dead letter queue : `queue name`
CHN-1011	Indicates that a channel has moved a message to the named dead letter queue
CHN-1012	Flow Control Ignored. Channel will be closed.
CHN-1012	Indicates that a channel violating the imposed flow control has been closed
CHN-1014	Operation : `operation name`
CHN-1014	Indicates that the named operation has been invoked

Table C.11. Subscription Log Messages

Message Id	Message Text / Purpose
SUB-1001	Create : [`Durable`] Arguments : `arguments`
SUB-1001	Indicates that a subscription (corresponds to JMS concept of a MessageConsumer) has been created.
SUB-1002	Close
SUB-1002	Indicates that a subscription has been closed.
SUB-1003	SUB-1003 : Suspended for `time` ms
SUB-1003	Indicates that a subscription has been in a suspened state for an unusual length of time. This may be indicative of an consuming application that has stopped taking messages from the consumer (i.e. a JMS application is not calling receive() or its asynchronous message listener onMessage() is blocked in application code). It may also indicate a generally overloaded system.
SUB-1004	Operation : `operation name`
SUB-1004	Indicates that the named operation has been invoked

Table C.12. Message Store Log Messages

Message Id	Message Text / Purpose
MST-1001	Created
MST-1001	Indicates that a message store has been created. The message store is responsible for the storage of the messages themselves, including the message body and any headers.
MST-1002	Store location : `path`
MST-1002	Indicates that the message store is using `path` for the location of the message store.
MST-1003	Closed
MST-1003	Indicates that the message store has been closed.
MST-1004	Recovery Start
MST-1004	Indicates that message recovery has begun.
MST-1005	Recovered `number of messages` messages.
MST-1005	Indicates that recovery recovered the given number of messages from the store.
MST-1006	Recovered Complete
MST-1006	Indicates that the message recovery is concluded.
MST-1007	Store Passivated
MST-1007	The store is entering a passive state where is it unavailable for normal operations. Currently this message is used by HA when the node is in replica state.
MST-1008	Store overfull, flow control will be enforced
MST-1008	The store has breached is maximum configured size. See Section 9.2.1, “Disk quota-based flow control” for details.
MST-1009	Store overfull condition cleared
MST-1009	The store size has fallen beneath its resume capacity and therefore flow control has been rescinded. See Section 9.2.1, “Disk quota-based flow control” for details.

Table C.13. Transaction Store Log Messages

Message Id	Message Text / Purpose
TXN-1001	Created
TXN-1001	Indicates that a transaction store has been created. The transaction store is responsible for the storage of messages instances, that is, the presence of a message on a queue.
TXN-1002	Store location : `path`
TXN-1002	Indicates that the transaction store is using `path` for the location of the store.
TXN-1003	Closed
TXN-1003	Indicates that the transaction store has been closed.
TXN-1004	Recovery Start
TXN-1004	Indicates that transaction recovery has begun.
TXN-1005	Recovered `number` messages for queue `name`.
TXN-1005	Indicates that recovery recovered the given number of message instances for the given queue.
TXN-1006	Recovered Complete
TXN-1006	Indicates that the message recovery is concluded.

Table C.14. Configuration Store Log Messages

Message Id	Message Text / Purpose
CFG-1001	Created
CFG-1001	Indicates that a configuration store has been created. The configuration store is responsible for the storage of the definition of objects such as queues, exchanges, and bindings.
CFG-1002	Store location : `path`
CFG-1002	Indicates that the configuration store is using `path` for the location of the store.
CFG-1003	Closed
CFG-1003	Indicates that the configuration store has been closed.
CFG-1004	Recovery Start
CFG-1004	Indicates that configuration recovery has begun.
CFG-1005	Recovered Complete
CFG-1005	Indicates that the configuration recovery is concluded.

Table C.15. HA Log Messages

Message Id	Message Text / Purpose
HA-1001	Created
HA-1001	This HA node has been created.
HA-1002	Deleted
HA-1002	This HA node has been deleted
HA-1003	Added : Node : '`name`' (`host:port`)
HA-1003	A new node has been added to the group.
HA-1004	Removed : Node : '`name`' (`host:port`)
HA-1004	The node has been removed from the group. This removal is permanent.
HA-1005	Joined : Node : '`name`' (`host:port`)
HA-1005	The node has become reachable. This may be as a result of the node being restarted, or a network problem may have been resolved.
HA-1006	Left : Node : '`name`' (`host:port`)
HA-1006	The node is no longer reachable. This may be as a result of the node being stopped or a network partition may be preventing it from being connected. The node is still a member of the group.
HA-1007	HA-1007 : Master transfer requested : to '`name`' (`host:port`)
HA-1007	Indicates that a master transfer operation has been requested.
HA-1008	HA-1008 : Intruder detected : Node '`name`' (`host:port`)
HA-1008	Indicates that an unexpected node has joined the group. The virtualhost node will go into the ERROR state in response to the condition.
HA-1009	HA-1009 : Insufficient replicas contactable
HA-1009	This node (which was in the master role) no longer has sufficient replica in contact in order to complete transactions.
HA-1010	HA-1010 : Role change reported: Node : '`name`' (`host:port`) : from `role` to `role`
HA-1010	Indicates that the node has changed role within the group.
HA-1011	HA-1011 : Minimum group size : `new group size`
HA-1011	The quorum requirements from completing elections or transactions has been changed.
HA-1012	HA-1012 : Priority : `priority`
HA-1012	The priority of the object node has been changed. Zero indicates that the node cannot be elected master.
HA-1013	HA-1013 : Designated primary : `true\|false`
HA-1013	This node has been designated primary and can now operate solo. Applies to two node groups only. See Section 10.4.5, “Allow to Operate Solo”
HA-1014	HA-1014 : Diverged transactions discarded
HA-1014	This node is in the process of rejoining the group but has discovered that some of its transactions differ from those of the current master. The node will automatically roll-back (i.e. discard) the diverging transactions in order to be allowed to rejoin the group. This situation can only usually occur as a result of use of the weak durability options. These allow the group to operate with fewer than quorum nodes and therefore allow the inconsistencies to develop. On encountering this condition, it is strongly recommendend to run an application level reconcilation to determine the data that has been lost.

Table C.16. Port Log Messages

Message Id	Message Text / Purpose
PRT-1001	Create
PRT-1001	Port has been created.
PRT-1002	Open
PRT-1002	Port has been open
PRT-1003	Close
PRT-1003	Port has been closed
PRT-1004	Connection count `number` within `warn limit` % of maximum `limit`
PRT-1004	Warns that number of open connections approaches maximum allowed limit
PRT-1005	Connection from `peer` rejected. Maximum connection count (`limit`) for this port already reached.
PRT-1005	Connection from given host is rejected because of reaching the maximum allowed limit
PRT-1007	Unsupported protocol header received `header bytes`, replying with `AMQP version`
PRT-1007	Ususualy indicates an attempt to make an non-AMQP connection on an AMQP port, for instance, with a web browser.
PRT-1008	Connection from `address` rejected
PRT-1008	Incoming connection is rejected because the port's connection limits are already reached.
PRT-1009	FAILED to bind `name` service to `port number`
PRT-1009	The given port number could not be bound because it is already in-use.
PRT-1010	Operation : `operation name`
PRT-1010	Indicates that the named operation has been invoked

Prev		Next
Appendix B. System Properties	Home	Appendix D. Statistics Reporting

Apache Qpid, Messaging built on AMQP; Copyright © 2015 The Apache Software Foundation; Licensed under the Apache License, Version 2.0; Apache Qpid, Qpid, Qpid Proton, Proton, Apache, the Apache feather logo, and the Apache Qpid project logo are trademarks of The Apache Software Foundation; All other marks mentioned may be trademarks or registered trademarks of their respective owners

Project

Messaging APIs

Servers and tools

Resources

Appendix C. Operational Logging