I开发者_Go百科 am prepping up for the open graph release.
One of the features from open graph is to crawl the site’s meta data and pull relevant info for the timeline. My sitemap.xml runs into gigs and is protected by rate limiting and ip whitlisting for popular crawlers like Googlebot & Slurp. Can someone pass me the robot names and ips that will be crawling for facebook? This is not just for sitemap.xml, but for general rate limit(whole site) white listing too.The current user agent is: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Facebook publishes their IP range here.
精彩评论