Scrapy Tutorial

Execution Flow

• Logical flow

	graph LR
	A[URL]-->B[Request]
	B[Request]-->C[Response]
	C[Response]-->D[Item]
	D[Item]-->E[More]

• Code execution flow: where these show up in my code

	graph LR
	A[Spider.start_requests]-->B[Middleware.process_request]
	B[Middleware.process_request]-->C[Spider.parse]
	C[Spider.parse]-->D[Pipeline.process_item]
	D[Pipeline.process_item]-->E[MySQL]

Problems and Solutions

1. start_urls can only be set as a single array — what if I want to configure the crawl sources dynamically?

• Override the Spider’s start_requests function and set up the requests there

2. If I want to issue new requests based on a response, how do I write that?

• Issue requests directly from the callback with yield

3. If I issue new requests based on a response, how do I handle the new responses?

   • Add the handler function you need in the Spider and simply declare a response parameter.

     • yield scrapy.Request(next_page, callback=self.parse_new)

     • def parse_new(self, response, **kwargs)

4. How do I pass parameters from the request through to the response?

   • Send a dictionary over via the Request’s meta parameter.
   • Retrieve the values you stored via response.meta.

5. How do I configure spider-specific middleware and pipelines?

   • custom_settings
     • ITEM_PIPELINES
     • SPIDER_MIDDLEWARES
     • DOWNLOADER_MIDDLEWARES

6. How do I handle pages that load content asynchronously?

   • Option 1: splash

     • Start a Splash proxy service with docker:
       • docker run -it -p 8050:8050 --rm scrapinghub/splash
     • settings configuration:
       SPLASH_URL = 'http://host:port' DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter' HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage' SPIDER_MIDDLEWARES = { 'scrapy_splash.SplashDeduplicateArgsMiddleware': 100, } DOWNLOADER_MIDDLEWARES = {
       'scrapy_splash.SplashCookiesMiddleware': 723, 'scrapy_splash.SplashMiddleware': 725, 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810 }

     • Install the Python dependency pip3 install scrapy-splash
     • Import it in the Spider from scrapy_splash import SplashRequest
     • Swap out the request
       • From: scrapy.Request(stock_url, self.parse, meta=params, dont_filter=True)
       • To: SplashRequest(url, self.parse, meta=params, args={'wait': 0.5})
     • Then let parse handle the results as usual

   • Option 2: selenium

     • Install the Python dependency

       • pip3 install selenium

     • Configure custom_settings:

       'DOWNLOADER_MIDDLEWARES': { 'ht_msg.middlewares.SeleniumDownloadMiddleware': 500 }

     • Creating the Middleware

       from selenium.webdriver.chrome.options import Options
       from selenium import webdriver
       from scrapy.http import HtmlResponse
       
       class SeleniumDownloadMiddleware(object):
         def __init__(self):
          	chrome_options = Options()
          	chrome_options.add_argument('--headless')
          	chrome_options.add_argument('--disable-gpu')
          	chrome_options.add_argument("window-size=1024,768")
          	chrome_options.add_argument("--no-sandbox")
          	self.driver = webdriver.Chrome(chrome_options=chrome_options)
       
         def process_request(self, request, spider):
          	self.driver.get(request.url)
          	time.sleep(12)
          	source = self.driver.page_source
          	url = self.driver.current_url
          	return HtmlResponse(url=url, body=source, request=request, encoding="utf-8")
       

To be continued.....

References

• scrapy:

  • Documentation
    • https://doc.scrapy.org/en/latest/topics/request-response.html#response-objects
  • Tutorials
    • https://geek-docs.com/scrapy/scrapy-tutorials/scrapy-css-grammar.html
    • https://www.jianshu.com/p/df9c0d1e9087
  • Multiple options: crawling dynamic JS pages
    • https://zhuanlan.zhihu.com/p/130867872
  • splash tutorial
    • https://splash.readthedocs.io/en/stable/
  • Configuring dynamic start_urls
    • https://blog.csdn.net/sirobot/article/details/105360486
  • splash
    • https://github.com/scrapy-plugins/scrapy-splash
  • xpath/css
    • https://zhuanlan.zhihu.com/p/65143300
  • Different pipelines per spider:
    • https://blog.csdn.net/peiwang245/article/details/100071316
  • Passing parameters through response:
    • https://blog.csdn.net/killeri/article/details/80255500
  • Spider settings precedence
    • https://blog.csdn.net/he_ranly/article/details/85092065
  • Differences between the middlewares
    • https://www.jianshu.com/p/3a9a385b9483

• scrapy deployment

  • APscheduler
    • https://apscheduler.readthedocs.io/en/latest/
  • subprocess
    • https://www.runoob.com/w3cnote/python3-subprocess.html
  • APscheduler+scarpy
    • https://apscheduler.readthedocs.io/en/latest/userguide.html
    • https://www.cnblogs.com/jdbc2nju/p/9563761.html
    • https://www.jianshu.com/p/c37c46de3168