Message Queue
A message queue is a waiting line in which messages are buffered between systems until the recipient is ready to process them. It decouples sender and receiver, absorbs load spikes and ensures that no message is lost – in an ERP context, for example, orders, stock changes or documents moving between shop, ERP and shipping.
A message queue is a buffer in which messages are temporarily stored between two or more systems until the recipient is ready to process them. A sending system – the producer – places a message in the queue, and a receiving system – the consumer – picks it up later. Neither has to be available at the same time, and neither has to keep the same pace. The principle resembles a post office with a mailbox: the sender drops in a letter and moves on; the recipient empties the box whenever it suits them. The queue usually works on the FIFO principle – whatever is placed in first is processed first.
In an ERP context, the message queue is the building block that enables a reliable, asynchronous flow of data between systems. If an order comes in from the online shop while the ERP is busy with a batch run, it is not lost: it waits in the queue until the ERP processes it. In the same way, stock changes, documents or shipping orders are transported between shop, ERP, marketplace and logistics via queues. That keeps every order, every status change and every posting intact, even when a system is briefly overloaded or unreachable.
At a glance
- Message queue = a waiting line that buffers messages between systems until the recipient is ready
- Decouples producer (sender) and consumer (receiver): no simultaneous availability and no matching pace required
- Works asynchronously and usually on the FIFO principle (First In, First Out)
- Absorbs load spikes and prevents data loss during outages or overload
- In the ERP it reliably transports orders, stock changes, documents and shipping orders between systems
How a message queue works
A message queue sits between the systems involved and accepts messages instead of passing them straight through. The producer writes a message into the queue and immediately receives confirmation that it has been accepted – without having to wait until the recipient has actually processed it. The queue holds the message until a consumer picks it up. After successful processing, the consumer confirms receipt (acknowledgement), whereupon the message is removed from the queue. If the confirmation fails to arrive, for example because the consumer crashes, the message is delivered again – nothing is lost.
The decisive effect is decoupling in time and load. Sender and receiver work independently of one another: the shop can submit a hundred orders in a second while the ERP processes them at its own pace. The queue acts like a shock absorber that smooths out peaks. If the recipient goes down, the messages pile up in the queue and are delivered in order once it restarts.
Producer, consumer, broker and topic
Four terms come up again and again: the producer creates and sends messages. The consumer receives and processes them. The message broker is the software that manages the queues, stores messages and monitors their delivery – well-known examples are RabbitMQ, Apache Kafka or Amazon SQS. In the publish-subscribe model, a single queue is replaced by a topic to which several consumers subscribe: a message is then distributed to all interested recipients instead of being picked up by just one. This way, a single order notification can reach accounting, warehouse and shipping at the same time.
Why a message queue matters
Without a queue, a sending system would have to wait directly for the recipient’s response (synchronous processing). If the recipient is slow, overloaded or unreachable, that blocks the entire process – in the worst case the operation aborts and the data is gone. A message queue solves this problem by separating the handover from the processing. The sender is safely rid of its message as soon as it is in the queue; whether and when the recipient processes it is a separate matter.
This yields tangible benefits for reliability. Load spikes – for instance on a promotion day with many orders – are cushioned, because the queue serves as a buffer and does not overwhelm the backend. Systems can be maintained and restarted independently of one another without messages being lost. And because every message is stored persistently and only disappears after confirmation, a robust "exactly-once" or "at-least-once" delivery is established. For an ERP in which every order and every posting counts, that is the foundation of clean data quality.
Message queue in the ERP data flow
In day-to-day operation, the message queue transports the events surrounding the ERP. An order from the shop is placed as a message and travels into the ERP, which creates the sales order and posts the stock. The stock change is in turn distributed as a message and mirrored back across all sales channels. Shipping orders flow to the logistics provider, and tracking numbers and status updates come back. Each of these steps runs asynchronously via queues, so that no employee has to transfer data by hand and no operation is lost when a system briefly fails to respond.
Message queues are therefore a central building block of modern integration architectures. In an event-driven architecture, systems react to events that are distributed via queues and topics – "order received", "stock changed", "shipment dispatched". Middleware and an enterprise service bus use queues internally to buffer messages between the connected systems and forward them reliably.
Idempotency and dead-letter queue
Because messages in distributed systems can be delivered more than once, processing must be idempotent: the same order notification must not end up as two separate sales orders in the ERP if it arrives again. The consumer detects duplicates via a unique identifier and processes them only once. Messages that cannot be processed despite repeated attempts – for example because of faulty data – end up in a separate dead-letter queue. There they do not block the regular flow, yet remain available for analysis and manual reprocessing.
Distinction: message queue, API, webhook and ESB
Message queue, API and webhook denote different patterns of data exchange. An API is the docking point of a system and is typically called synchronously: the caller makes a request and waits for the response. A webhook reverses the direction – a system actively calls a stored address as soon as an event occurs. Both are designed for immediate delivery. The message queue, by contrast, deliberately relies on asynchronicity and buffering: it guarantees that the message is preserved even when the recipient is not ready. In practice the patterns complement each other – a webhook can trigger a message that lands in a queue and is robustly processed onward from there.
The queue differs from the enterprise service bus (ESB) and from middleware in its scope. It is the pure transport and buffer mechanism; ESB and middleware are higher-level layers that additionally translate formats, map fields and orchestrate processes – and use message queues as an internal building block for that. In short: the queue transports and buffers messages, while the middleware coordinates the substantive interplay of many systems.
Message queue when selecting an ERP
For users, the message queue is usually not directly visible – it works in the background of the integration layer. When selecting an ERP, it is nonetheless worth looking at how a system handles asynchronous processing: does it buffer incoming orders reliably, even under load? Are failed operations retried automatically and collected in a dead-letter queue instead of quietly disappearing? How transparent is the monitoring of the queues, and can stuck messages be reprocessed in a targeted way? Systems that build cleanly on queues cope significantly better with load spikes and partial outages.
For the DACH region it is additionally relevant where the messages are buffered. Because the queues hold personal customer and order data – often even persistently, until the recipient picks them up – the same GDPR requirements apply as for the rest of the data flow. With cloud-based message-broker services, the storage location, access protection and retention of the buffered data are part of data processing and should be governed contractually.
Example
Example: message queue on a promotion day in online retail
A mid-sized online retailer launches a discount campaign. Within a few minutes, several hundred orders come in from the shop and marketplaces – far more than the ERP can post at that same moment. Instead of passing the orders straight to the ERP and overloading it, the integration layer places each order as a message into a message queue.
The ERP pulls the messages from the queue at its own pace and creates the sales orders one after another. Customers receive their confirmation immediately, because acceptance into the queue already counts as a success. When the ERP is briefly restarted for a release update, the orders received in the meantime simply wait in the queue and are then processed without any gaps. An order with faulty data lands in the dead-letter queue and is checked manually later – without holding up the rest of the flow. No order is lost, and no customer notices the load spike.
Frequently asked questions
Related services
Questions about Message Queue in your ERP project?
We advise vendor-neutrally – and implement it ourselves on request.