Implement durable pub/sub?

Let's say I have a publisher and mutiple listeners. When the publisher sends a message, it has to be received by all listeners. If one of the listeners is down, he should get the message as soon he is back up again.

How can I implement this?

I was thinking about using queues: Each listener makes it's own queue and sends a subscrtiption message to the publisher with the location of it's queue. The publisher saves the location to a file or DB, and starts sending it's messages to that queue.

So, this would be the timeline:

Publisher is started. No listeners yet.

Publisher sends message 1.

Publisher sends message 2.

Publisher sends message 3.

Listener 1 starts and subscribes with publisher.

Publisher sends message 4.

Listener 1 receives message 4.

Listener 2 starts and subscribes with publisher.

Publisher sends message 5.

Listener 1 receives message 5.

Listener 2 receives message 5.

Listener 2 chrashes.

Publisher sends message 6.

Listener 1 receives message 6.

Publisher sends message 7.

Listener 1 receives message 7.

Listener 2 comes back up, no need to subscribe again.

Listener 2 receives message 6.

Listener开发者_运维知识库 2 receives message 7.

The bottom line is I need one queue per listener, and one queue or channel to send and receive messages for 'start listening' and 'stop listening'. Am I thinking in the right direction, or am I completely wrong?

---

You shouldn't need a separate queue per subscriber, but you will want at least two queues. The initial key to scalability is making sure that when the publisher delivers it's initial message you don't try to "fan it out" to all subscribers at that point in time. Instead you put it on a received queue and return immediately letting the publisher know it's succeeded. From there you have workers which are fed by the main receive queue and whose responsibility it is to "fan out" the message to the various subscribers. It does this by figuring out who those subscribers are and generating N messages containing the original message from the publisher with each listener's address/binding information and plopping those onto a delivery queue. Finally you have workers who are responsible for pulling the messages off the delivery queue and attempting delivery using the address/binding information.

How you handle delivery faults could be by moving the message to a retry queue where the messages will be put to sleep for X amount of time before being replaced on the delivery queue again. Then you of course have to handle poison messages where you've retried say 5 times and the listener just throws you errors every time. Those would need to be moved to some kind of dead letter queue for error reporting.

---

You are right, and this is exactly how NServiceBus (for example) implements pub-sub on top of MSMQ. Read more about it here: http://docs.particular.net/samples/pubsub/.