I'm looking for someone with serious scraping experience for a large-scale data collection project. This isn't your average "let me grab some product info from a website" gig - we're talking industrial-strength, performance-optimized scraping that can handle millions of data points.
What I need:
- Someone who's battle-tested with high-volume scraping challenges
- Experience with parallel processing and distributed systems
- Creative problem-solver who can think outside the box when standard approaches hit limitations
- Knowledge of handling rate limits, proxies, and optimization techniques
- Someone who enjoys technical challenges and finding elegant solutions
I have the infrastructure to handle the actual scraping once the solution is built - I'm looking for someone to develop the approach and architecture. I'll be running the actual operation, but need expertise on the technical solution design.
Compensation: Fair and competitive - depends on experience and the final scope we agree on. I value expertise and am willing to pay for it.
If you're the type who gets excited about solving tough scraping problems at scale, DM me with some background on your experience with high-volume scraping projects and we can discuss details.
Thanks!