RESTful Rails controller ans net/http as client - slow on many requests

I have an Array populated with potentially lots of records. Every records shall be saved by the server in a RESTful RecordsController. My Solution in the moment looks like this:

def self.send! options = nil
  records = fetch_records
  records.each do |r|
    send_data!(r) ? records = records.delete_if{|rec| rec == r } : break
  end
  storage.save! records

  true
end

private

def self.send_data! record, options = nil
  begin
    response = Net::HTTP.Proxy(configuration.proxy_host, configuration.proxy_port).start(configuration.host, configuration.port) do |http|
      request = Net::HTTP::Post.new(request_path options)
      request.body = record.to_json
      http.request request
    end
    raise StandardError unless response.code == "200"
  rescue Exception => e
    return false
  end

  true
end

The advantage of that solution is, that if a ConnectionError, ConnectionTimeout or ServerError occurs, the unsent records get stored locally and can be submitted again later. The matching controller is a standard Rails controller.

My Problem now is, that in production mode this seems very slow. It's about 4 requests/s where there the server is not the bottleneck.

The question now is, wether it could help to instantiate the HTTP client only once and send all the records using the same connection. I did not find a solution to implement that, cause I need the save_or_store behavior of the code up here.

Another solution could be not to transfer the records done after another but group them and create a new controller that accepts groups of my records to store them.

So this questions boils down t开发者_如何学JAVAo a more architectural one than a technical. Either way I wonder if I can keep a HTTP connection open and speed up my solution that way.

Any ideas?

Regards Felix

---

Switch to em-http-request - an EventMachine-based HTTP client - and use its Multi interface to send several request at the same time could raise your throughput a bit.

But the benefit of a better HTTP client is negligible. Since you don't do anything with the HTTP response you get from your controller -- which means you're not using HTTP as an application protocol - the best solution for your problem is to actually send the whole Array for processing on the server side. After all, you don't have to pay the price of making a single request for every single record.

Eventually, if you so desire, it would be nice to expose a URI with the list of records that failed to process, so you could re-submit then or just ignore, at your will.

And I would throw MessagePack in the mix, to serialize the Array and reduce the size of the payload and speed the message exchange.

---

If i am reading the send! method correctly, you are looping through each record, and inside that loop just removing the current record from the array and sending everything else? To save a bit of effort on the computing side, use the delete() method of the Array class instead of the delete_if.

def self.send! options = nil
  records = fetch_records
  records.each do |r|
    if send_data!(r)
      records.delete(r)
    else
      break
    end
  end
  storage.save! records

  true
end

Though i'm not sure where you're passing the record parameter for the send_data! method. Since the terinary would be using the result from that method.

Net http can take some time to run through all its requests. I know when i've ran a few hundred URLs to check their response codes it's taken anywhere between 5-10 minutes to complete. I haven't used it much myself, but take a look at: https://github.com/jnunemaker/httparty. Maybe it will work better for you.