scrapy

Web-crawling framework.

• Create a project:

scrapy startproject {project_name}

• Create a spider (in project directory):

scrapy genspider {spider_name} {website_domain}

• Edit spider (in project directory):

scrapy edit {spider_name}

• Run spider (in project directory):

scrapy crawl {spider_name}

• Fetch a webpage as Scrapy sees it and print the source to stdout:

scrapy fetch {url}

• Open a webpage in the default browser as Scrapy sees it (disable JavaScript for extra fidelity):

scrapy view {url}

• Open Scrapy shell for URL, which allows interaction with the page source in a Python shell (or IPython if available):

scrapy shell {url}

Copyright © 2014—present the tldr-pages team and contributors.

This work is licensed under the Creative Commons Attribution 4.0 International License (CC-BY).