Optimization on huge generating xml?

Currently, I'm developing a rails app that are heavy generating xml for restful webservice. My xml representation of web service use nokogiri gem to generates xml format that match expected format from client. But the problem is data is quite big around 50, 000 records to pull out from the table(millions records). I just test in my local machine, it takes about 20 minutes to get the response from the request.

Do you have any ideas on how to optimize this problem? Another option, I'm not sure if we don't use ActiveRecord, and we just use pure sql statement to pull out the data for generating xml, the开发者_高级运维n the performance is huge faster or not?

---

One possible solution is to look into the idea of pagination. Pagination will return a subset of your original results. Callers to your service would make multiple requests to your service to retrieve all the desired records.

Pagination is going to have a number of benefits, here are just a few

1. Initial Faster Response time to API calls
2. The server-side process is going to be less memory intensive as you are not storing 50K results in memory at one time
3. Allows callers to pull only a subset of data if desired

Example of pagination in practice

Users of the Twitter API encounter a similar situation when they want to return all of the followers of a given user. Twitter uses the concept of a cursor to provide the results in a paginated fashion. Using the cursor parameter allows for all of the followers to be returned, without generating huge requests.

Twitter API Link:

http://apiwiki.twitter.com/w/page/22554748/Twitter-REST-API-Method:-statuses%C2%A0followers

From the API docs:

cursor. Optional. Breaks the results into pages. A single page contains 100 users. This is recommended for users who are followed by many other users. Provide a value of -1 to begin paging. Provide values as returned to in the response body's next_cursor and previous_cursor attributes to page back and forth in the list.