Introduction to Azure Messaging and Queues

Witali Stinski - August 13, 2019

Why use queue?

In programming, especially in the era of microservices, we encounter situations when we need to process more data than we can handle in one moment. In such a situation, we can stop receiving new data until we finish processing the current one, or we just add each next piece of data to the collection and continue processing it one by one. We can do it sequentially or use multiple threads and then things will go faster. Actually, it doesn’t matter how much data we need to process, we just pick one element from the collection and process it, then pick the next one. This is very intuitive. We can see examples of such processes everywhere in real life.

In programming, such type of collection is called a Queue. Some Danish engineer and mathematician even created a Queueing theory, which could be very useful when we need to design complex systems.

How skillfully developer is able to use collections, and especially queues, to solve problems, defines his level of competence.

Queues and messaging

In case of a simple in-memory queue, adding and popping items from the queue is not a big deal. But if we already have more sophisticated architecture, consisting of more than one service and we need to guarantee that the queue doesn’t disappear on server restart, then delivering new items in the queue requires an advanced interface. A messaging system could act as such an interface. With messaging systems, apart from simple functionality such as adding and popping items from the queue, you get additional benefits such as delivery guarantee, durability, routing and more. Often these two concepts - queue and messaging - are used together or even interchangeably. ​​

Event and Message

​We can distinguish different types of items in queue depending on their purpose and content.

It is important to understand what kinds of items you will use, because it will determine the type of the queue and messaging system.

Event - this is just a notification. It just says “Hello! Something happened.” Producer of the event doesn’t care about what the consumer does with it. Also, don’t care when the event processed. Their task is just to notify.

Typically, an event contains information about what happened. For example, a user changes his status from ‘online’ to ‘offline’, the event could contain the user’s id, status, and time of change. If a particular event is part of a bigger change, it could contain some unique identifier of change, to make it possible to connect separate events to the big picture.

Events are delivered in the same order they were received. This fact is important, especially if it’s logs or bank transactions.

Message - this is a command. It says “Please do this for me.” A message has specific consumer and precise action to do. For example: “Add a new user”. It should contain all the required information to complete the task. Producer of the command has an expectation of how and when the command will be handled. For example, it could expect a notification when the user is added.

However, there are scenarios where a producer doesn’t expect any response from a consumer. For example, if a message is used to pass the data to the next step in some processing flow.

Queue in the cloud

Nowadays Azure platform offers many queue messaging services. You may be asking why a single platform offers so many services for queue and messaging. Each solution has a slightly different set of features and, depending on your needs and the problem you are solving, you can choose the most suitable. In this article, I will briefly cover three most popular products: Azure Queue, Service Bus, and Event Grid. I hope it will help you to choose a proper solution which best meets your needs.

Azure Queue is a simple queue and messaging service. It is built on top of Azure Storage. This is the first queue service introduced in the Azure platform.

What is good about this product:

  • Big single queue size, limited only by the Storage quota, up to 500 TB;
  • At-Least-Once delivery guarantee;
  • RESTful and HTTP/HTTPS interfaces;
  • Client library for .NET, as well as for other languages: Java, PHP, Node.js;
  • Detailed logs and aggregated metrics are available out of the box.

What you need to consider:

  • It doesn’t guarantee FIFO order;
  • It doesn’t support publish/subscribe and routing;
  • There are no transactions;
  • Long-polling is not supported, so there is no instant delivery.

Ideal for simple scenarios where you need reliable messaging between application components, to leverage load or build process workflow.

Due to lack of publish/subscribe model and FIFO order guarantee, it’s more suitable to deal with “messages” rather than “events”.

It’s worth to mention the Azure storage emulator, which could be very useful in testing and development. What is more, in Visual Studio it’s possible to manage the queues, view, add and update messages (in Server Explorer).

Azure Service Bus is a more sophisticated solution, which would satisfy the most demanding cases. It’s like a Swiss Army Knife.

What is good about this product:

  • Thanks to a TCP-based full-duplex protocol, long polling is available and hence “almost immediate delivery”;
  • At-Least-Once and At-Most-Once delivery guarantee;
  • Duplicate detection;
  • FIFO order guaranteed within session;
  • Transactions and atomicity for many messages;
  • Publish/Subscribe model;
  • Messages routing and filtering.

What you need to consider:

  • No out-of-the-box detailed logs;
  • No emulator;
  • No build-in Visual Studio explorer/browser tool;
  • 80 GB limit per queue;
  • In higher tiers and big numbers of messages, it could be quite costly.

It could be used for critical features like processing banking transactions, orders, bookings, etc. Also, thanks to long polling, it could be useful in applications like chats or messengers. Thanks to a wide range of features Service Bus could be configured to deal with “messages” as well as with “events”.

Please see the detailed comparison of Azure Queue and Azure Service Bus.

Event Grid: This service uses the Publish/Subscribe model. It’s intended to work with big amount of emitted “events”, not correlated with each other. So it’s not designed to process sequences of events (streams) but rather separate, independent events. Many subscribers could consume events simultaneously, depending on context or filtering rules.

What is good about this product:

  • Deeply integrated with the Azure infrastructure, so it’s easy to integrate with other Azure services, such as serverless applications or other queues.
  • At-Least-Once delivery guarantee.
  • Pay-per-event.
  • Designed with a focus on a great scale.
  • Advanced filtering and routing.

What you need to consider:​

  • No back-pressure so event consumers should be prepared for high load and protect themselves from crashing. ​

Summary

This article is a brief introduction to a broader concept of the infrastructure of Azure Messaging. I recommend exploring other message-driven services like Event Hub. It’s also worthwhile to understand how to make the Azure services work together, using such products like Logic Apps, Notification Hub or Azure Function.

About the author

Witali Stinski

Software Engineer

Vitali is a software developer with a passion for building scalable web applications, APIs, and backend solutions using Microsoft technologies such as Azure and .NET. Vitali has been consistently implementing best practices in coding to produce high-quality products that are always on time. He loves learning new technologies, diving deep into the details of any subject he’s unfamiliar with, and helping others learn as well.

Tempted to work
on something
as creative?

That’s all we do.

join nexocode

Find us on

Need help with implementing AI in your business?

Let's talk blue circle

This site uses cookies for analytical purposes.

Accept Privacy Policy

In the interests of your safety and to implement the principle of lawful, reliable and transparent processing of your personal data when using our services, we developed this document called the Privacy Policy. This document regulates the processing and protection of Users’ personal data in connection with their use of the Website and has been prepared by Nexocode.

To ensure the protection of Users' personal data, Nexocode applies appropriate organizational and technical solutions to prevent privacy breaches. Nexocode implements measures to ensure security at the level which ensures compliance with applicable Polish and European laws such as:

  1. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (published in the Official Journal of the European Union L 119, p 1); Act of 10 May 2018 on personal data protection (published in the Journal of Laws of 2018, item 1000);
  2. Act of 18 July 2002 on providing services by electronic means;
  3. Telecommunications Law of 16 July 2004.

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet.

1. Definitions

  1. User – a person that uses the Website, i.e. a natural person with full legal capacity, a legal person, or an organizational unit which is not a legal person to which specific provisions grant legal capacity.
  2. Nexocode – NEXOCODE sp. z o.o. with its registered office in Kraków, ul. Generała Henryka Kamieńskiego 51, 30-644 Kraków, entered into the Register of Entrepreneurs of the National Court Register kept by the District Court for Kraków-Śródmieście in Kraków, 11th Commercial Department of the National Court Register, under the KRS number: 0000686992, NIP: 6762533324.
  3. Website – website run by Nexocode, at the URL: nexocode.com whose content is available to authorized persons.
  4. Cookies – small files saved by the server on the User's computer, which the server can read when when the website is accessed from the computer.
  5. SSL protocol – a special standard for transmitting data on the Internet which unlike ordinary methods of data transmission encrypts data transmission.
  6. System log – the information that the User's computer transmits to the server which may contain various data (e.g. the user’s IP number), allowing to determine the approximate location where the connection came from.
  7. IP address – individual number which is usually assigned to every computer connected to the Internet. The IP number can be permanently associated with the computer (static) or assigned to a given connection (dynamic).
  8. GDPR – Regulation 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of individuals regarding the processing of personal data and onthe free transmission of such data, repealing Directive 95/46 / EC (General Data Protection Regulation).
  9. Personal data – information about an identified or identifiable natural person ("data subject"). An identifiable natural person is a person who can be directly or indirectly identified, in particular on the basis of identifiers such as name, identification number, location data, online identifiers or one or more specific factors determining the physical, physiological, genetic, mental, economic, cultural or social identity of a natural person.
  10. Processing – any operations performed on personal data, such as collecting, recording, storing, developing, modifying, sharing, and deleting, especially when performed in IT systems.

2. Cookies

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet. The Website, in accordance with art. 173 of the Telecommunications Act of 16 July 2004 of the Republic of Poland, uses Cookies, i.e. data, in particular text files, stored on the User's end device.
Cookies are used to:

  1. improve user experience and facilitate navigation on the site;
  2. help to identify returning Users who access the website using the device on which Cookies were saved;
  3. creating statistics which help to understand how the Users use websites, which allows to improve their structure and content;
  4. adjusting the content of the Website pages to specific User’s preferences and optimizing the websites website experience to the each User's individual needs.

Cookies usually contain the name of the website from which they originate, their storage time on the end device and a unique number. On our Website, we use the following types of Cookies:

  • "Session" – cookie files stored on the User's end device until the Uses logs out, leaves the website or turns off the web browser;
  • "Persistent" – cookie files stored on the User's end device for the time specified in the Cookie file parameters or until they are deleted by the User;
  • "Performance" – cookies used specifically for gathering data on how visitors use a website to measure the performance of a website;
  • "Strictly necessary" – essential for browsing the website and using its features, such as accessing secure areas of the site;
  • "Functional" – cookies enabling remembering the settings selected by the User and personalizing the User interface;
  • "First-party" – cookies stored by the Website;
  • "Third-party" – cookies derived from a website other than the Website;
  • "Facebook cookies" – You should read Facebook cookies policy: https://www.facebook.com/policy/cookies
  • "Other Google cookies" – Refer to Google cookie policy: www.google.com/policies/technologies/types/

3. How System Logs work on the Website

User's activity on the Website, including the User’s Personal Data, is recorded in System Logs. The information collected in the Logs is processed primarily for purposes related to the provision of services, i.e. for the purposes of:

  • analytics – to improve the quality of services provided by us as part of the Website and adapt its functionalities to the needs of the Users. The legal basis for processing in this case is the legitimate interest of Nexocode consisting in analyzing Users' activities and their preferences;
  • fraud detection, identification and countering threats to stability and correct operation of the Website.

4. Cookie mechanism on the Website

Our site uses basic cookies that facilitate the use of its resources. Cookies contain useful information and are stored on the User's computer – our server can read them when connecting to this computer again. Most web browsers allow cookies to be stored on the User's end device by default. Each User can change their Cookie settings in the web browser settings menu: Google ChromeOpen the menu (click the three-dot icon in the upper right corner), Settings > Advanced. In the "Privacy and security" section, click the Content Settings button. In the "Cookies and site date" section you can change the following Cookie settings:

  • Deleting cookies,
  • Blocking cookies by default,
  • Default permission for cookies,
  • Saving Cookies and website data by default and clearing them when the browser is closed,
  • Specifying exceptions for Cookies for specific websites or domains

Internet Explorer 6.0 and 7.0
From the browser menu (upper right corner): Tools > Internet Options > Privacy, click the Sites button. Use the slider to set the desired level, confirm the change with the OK button.

Mozilla Firefox
browser menu: Tools > Options > Privacy and security. Activate the “Custom” field. From there, you can check a relevant field to decide whether or not to accept cookies.

Opera
Open the browser’s settings menu: Go to the Advanced section > Site Settings > Cookies and site data. From there, adjust the setting: Allow sites to save and read cookie data

Safari
In the Safari drop-down menu, select Preferences and click the Security icon.From there, select the desired security level in the "Accept cookies" area.

Disabling Cookies in your browser does not deprive you of access to the resources of the Website. Web browsers, by default, allow storing Cookies on the User's end device. Website Users can freely adjust cookie settings. The web browser allows you to delete cookies. It is also possible to automatically block cookies. Detailed information on this subject is provided in the help or documentation of the specific web browser used by the User. The User can decide not to receive Cookies by changing browser settings. However, disabling Cookies necessary for authentication, security or remembering User preferences may impact user experience, or even make the Website unusable.

5. Additional information

External links may be placed on the Website enabling Users to directly reach other website. Also, while using the Website, cookies may also be placed on the User’s device from other entities, in particular from third parties such as Google, in order to enable the use the functionalities of the Website integrated with these third parties. Each of such providers sets out the rules for the use of cookies in their privacy policy, so for security reasons we recommend that you read the privacy policy document before using these pages. We reserve the right to change this privacy policy at any time by publishing an updated version on our Website. After making the change, the privacy policy will be published on the page with a new date. For more information on the conditions of providing services, in particular the rules of using the Website, contracting, as well as the conditions of accessing content and using the Website, please refer to the the Website’s Terms and Conditions.

Nexocode Team