Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

At Theneeds we use a "sliding windows" approach to deal with polling. Say you run the scraper every hour. Each feed F_i is scraped once every n_i polls. If the feed returns more than his average news, then n_i gets decreased, while if the feed returns no news, it gets increased.

Perhaps with a similar trick you can run your scraper more frequently on some feeds, still keeping the cost under control.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: