What is the best way to send messages to Socket.IO clients from various back-ends

My Setup: I have an existing python script that is using Tweepy to access the Twitter Streaming API. I also have a we开发者_C百科bsite that shows aggregate real-time information from other sources from various back-ends.

My Ideal Scenario: I want to publish real-time tweets as well as real-time updates of my other information to my connected users using Socket.IO.

It would be really nice if I could do something as simple as an HTTP POST (from any back-end) to broadcast information to all the connected clients.

My Problem: The Socket.IO client implementation is super straight forward... i can handle that. But I can't figure out if the functionality I'm asking for already exists... and if not, what would be the best way to make it happen?

[UPDATE]

My Solution: I created a project called Pega.IO that does what I was looking for. Basically, it lets you use Socket.IO (0.8+) as usual, but you can use HTTP POST to send messages to connected users.

It uses the Express web server with a Redis back-end. Theoretically this should be pretty simple to scale -- I will continue contributing to this project going forward.

Pega.IO - github

To install on Ubuntu, you just run this command:

curl http://cloud.github.com/downloads/Gootch/pega.io/install.sh | sh

This will create a Pega.IO server that is listening on port 8888.

Once you are up and running, just:

HTTP POST http://your-server:8888/send

with data that looks like this:

channel=whatever&secretkey=mysecret&message=hello+everyone

That's all there is to it. HTTP POST from any back-end to your Pega.IO server.

---

The best way I've found for this sort of thing is using a message broker. Personally, I've used RabbitMQ for this, which seems to meet the requirements mentioned in your comment on the other answer (socket.io 0.7 and scalable). If you use RabbitMQ, I'd recommend the amqp module for node, available through npm, and the Pika module for Python.

An example connector for Python using pika. This example accepts a single json-serialized argument:

    def amqp_transmit(message):
        connection = pika.AsyncoreConnection(pika.ConnectionParameters(host=settings.AMQP_SETTINGS['host'],
            port=settings.AMQP_SETTINGS['port'],
            credentials=pika.PlainCredentials(settings.AMQP_SETTINGS['username'],
                    settings.AMQP_SETTINGS['pass'])))
        channel = connection.channel()
        channel.exchange_declare(exchange=exchange_name, type='fanout')
        channel.queue_declare(queue=NODE_CHANNEL, auto_delete=True, durable=False, exclusive=False)
        channel.basic_publish(exchange=exchange_name,
                routing_key='',
                body=message,
                properties=pika.BasicProperties(
                        content_type='application/json'),
                )
        print ' [%s] Sent %r' %(exchange_name, message)
        connection.close()

Very basic connection code on the node end might look like this:

    var connection = amqp.createConnection(
        {host: amqpHost,
        port: amqpPort,
         password: amqpPass});

    function setupAmqpListeners() {
        connection.addListener('ready', amqpReady)
        connection.addListener('close', function() {
            console.log('Uh oh! AMQP connection failed!');
        });
        connection.addListener('error', function(e) {throw e});
    }

    function amqpReady(){
        console.log('Amqp Connection Ready');
        var q, exc;
        q = connection.queue(queueName,
            {autoDelete: true, durable: false, exclusive: false},
            function(){
                console.log('Amqp Connection Established.');
                console.log('Attempting to get an exchange named: '+exchangeName);
                exc = connection.exchange(exchangeName,
                    {type: 'fanout', autoDelete: false},
                    function(exchange) {
                        console.log('Amqp Exchange Found. ['+exchange.name+']');
                        q.bind(exc, '#');
                        console.log('Amqp now totally ready.');
                        q.subscribe(routeAmqp);
                    }
                );
            }
        );
    }


    routeAmqp = function(msg) {
            console.log(msg);
            doStuff(msg);
    }

Edit: The example above uses a fan-out exchange that does not persist messages. Fan-out exchange is likely going to be your best option since scalability is a concern (ie: you are running more than one box running Node that clients can be connected to).

---

Why not write your Node app so that there are two parts:

1. The Socket.IO portion, which communicates directly with the clients, and
2. An HTTP API of some sort, which receives POST requests and then broadcasts appropriate messages with Socket.IO.

In this way, your application becomes a "bridge" between your non-Node apps and your users' browsers. The key here is to use Socket.IO for what it was made for--real time communication to browsers--and rely on other Node technologies for other parts of your application.

[Update]

I'm not on a development environment at the moment, so I can't get you a working example, but some pseudocode would look something like this:

http = require('http');
io   = require('socket.io');

server = http.createServer(function(request, response) {
  // Parse the HTTP request to get the data you want
  io.sockets.emit("data", whatever); // broadcast the data to Socket.IO clients
});
server.listen(8080);
socket_server = io.listen(server);

With this, you'd have a web server on port 8080 that you can use to listen to web requests (you could use a framework such as Express or one of many others to parse the body of the POST request and extract the data you need).