Data Crawler Project
Built a crawler cluster for financial news and new-listing data; I led data engineering to ensure data timeliness.
Key stakeholders: Huatong International news team, Trading & Research business lines
- Automated collection across four major data sources including financial news, gray-market quotes and new listings
- Tasks run stably after the microservice split, meeting real-time data needs
- Established a library of data collection compliance and anti-crawling strategies
- Defined task scheduling, alerting and log tracing mechanisms