Back to Projects

Data Crawler Project

WatonGroup
N/A
Head of Data Engineering
Automated collection covering four major data sources including financial news, grey-market trades, and IPO data Tasks run stably after the microservice split, meeting real-time data needs

Project Overview

At the end of 2020, to enrich the data dimensions of the company's products, you initiated and developed a series of data crawler projects. The main goal of the project was to automatically scrape information - including 7x24 flash news, financial headlines, IPO data, and grey-market trading data - from major financial websites (such as Wallstreetcn) and data sources, providing data support for the company's market quotes, news, and trading businesses.

Technical Architecture

  • Core crawler framework: Python, Scrapy
  • Data processing and service layer: Kotlin, Spring Boot
  • Data storage: Elasticsearch (for news content)

Key Contributions

1. Technology Selection and Framework Adoption

  • Crawler framework selection: After a comparative analysis of Java crawler frameworks such as Nutch, Crawler4j, and WebMagic against Python's Scrapy framework, you ultimately decided to adopt Scrapy as the core crawler framework for its powerful features, active community, and excellent performance.
  • Scrapy in practice: You studied Scrapy in depth and mastered it, successfully implementing data fetching, page parsing, and structured data storage for target websites (such as Wallstreetcn).

2. Core Data Scraping

  • News: Developed crawlers targeting 7x24 flash news and financial headlines, enabling real-time scraping and storage of news content.
  • IPO and grey-market data: Developed and maintained crawlers for IPO listing information (such as sponsors and margin subscription data) and grey-market trading data, providing an important data supplement for the trading system.

3. Servitization and Refactoring

  • Microservice split: In the later stage of the project, you refactored the crawler services. To improve system cohesion and maintainability, you extracted the grey-market data scraping logic from the original all-in-one crawler service into an independent microservice (darktrade) - a reflection of your continuous optimization and evolution in architecture design.

Project Summary

The data crawler project opened up a brand-new technical area for you in 2020. Through this project, you:

  • Mastered the important skill of web crawling, gaining the ability to acquire and integrate data from the internet - especially valuable in today's data-driven world.
  • Deepened your hands-on practice with the Python stack, especially the Scrapy framework, further broadening your technical range.
  • Demonstrated strong architectural evolution skills, refactoring the system toward a more rational, more maintainable design through the microservice split - one of an architect's core responsibilities.

This project opened the door to the field of data engineering and created more possibilities for your future technical growth.