Fast datacollection for price comparison

I have a price comparis开发者_运维技巧on site where some of the vendors dont have any current API for fetching price information. This far I have solved it with cURL, and since it is really slow I stored them in a database where the end user fetch them from.

For reasons I want to allow full real time and thus get the prices on end-users command and also avoid keeping a huge price-info database.

Which alternative would be faster and lighter than the current cURL solution?

Have anyone here done something similar and have experience in this?

---

The time you need to fetch the prices manually won't change no matter what you do. It's going to be a HTTP request + parsing the response no matter how you do it. Also, caching is a must if you have any significant volume of users.

However, you might try to indirectly attack the problem with techniques such as:

• When you want to display live data, make an AJAX call to your server (who will cURL and send the parsed results back to you to update the page) instead of doing a page reload. This will allow you to customize (and hopefully improve) the user experience during the wait.
• Run a background process that scrapes your vendors every so often and caches the results. If a user then requests this information before the cached data expires, you will be able to satisfy the request immediately.
• Tweak the above process to scrape intelligently (scrape more often during the hours when your visitor count peaks, scrape vendors and/or products more or less often based on how often your users actually request this information, etc).