Azure Service Bus queues are often used to decouple parts of connected systems and to offload heavy-duty processing to background tasks. Loose coupling improves the overall reliability and performance of the system. However, with a level of abstraction offered by the Service Bus, it is harder to notice when message processing fails or doesn’t keep up with the demand.
Netreo provides a number of ways to verify that ASB messages are processed successfully and in a timely manner.
In this article we’ll explain what metrics should be monitored in systems using Azure Service Bus queues and how they can be configured in Netreo:
When message processing services fail to process a message for a number of times, Azure Service Bus can move that message to a dead-letter queue. This ensures unprocessable ASB messages do not get stuck.
It is important to monitor dead-letter queues to understand if any of the ASB messages can’t be processed. Netreo by default offers metrics and alerts that will monitor deadletters. Default alert will trigger if it detects any dead-letters for any of the queues in the monitored Service Bus namespace.
To familiarize yourself with how dead-letter queues are monitored:
The “Deadletters Detected” alert is triggered when Netreo detects any dead-lettered messages or are already present in the dead-letter queue. This alert sends a notification when any of the dead-letter queues in the monitored namespace have messages.
It may be also important to receive individual notifications when specific queues have dead-letter messages. Using Netreo users can also track and alert on dead-letters for a specific queue:
It is also important to validate that message processors are working and are actually picking up messages. One way to do this is to alert when the message ID of the next-to-be-processed message does not change over time, which indicates the same message is stuck in the queue for a long period of time.
To alert when message id of the oldest message does not change over time:
How to ensure ASB messages are being processed
Sometimes messages are being picked up, but the system may still be impacted if services that are pulling messages from queues or topic-subscriptions can’t keep up with the incoming messages rate. It’s a good idea to monitor queue lengths and oldest message age, and alert when they are above the expected thresholds.
The easiest way to monitor queues lengths, assuming the acceptable length is the same for all queues, is using the “Queues” metric provided in the default template. Alternatively, you can track the length of an individual queue using “AzureServiceBusActiveMessageCount” metric type. It’ll be more practical if various queues have different acceptable length thresholds.
To monitor all queues lengths:
In this article, we explained how to ensure that messages in Azure Service Bus queues are being processed successfully in a timely manner.
Apart from monitoring and automatic notifications, Netreo also offers powerful automation. For example, you can auto-scale Cloud Services and Web Apps based on ASB queues lengths. Learn more here.