fetch [OPTION] URL¶

Fetch a URL using the Scrapy downloader

--headers

Print response HTTP headers instead of body

runspider [OPTION] spiderfile¶

Run a spider

--output=FILE

Store scraped items to FILE in XML format

settings [OPTION]¶

Query Scrapy settings

--get=SETTING

Print raw setting value

--getbool=SETTING

Print setting value, interpreted as a boolean

--getint=SETTING

Print setting value, interpreted as an integer

--getfloat=SETTING

Print setting value, interpreted as a float

--getlist=SETTING

Print setting value, interpreted as a float

--init

Print initial setting value (before loading extensions and spiders)

shell URL | file¶

Launch the interactive scraping console

startproject projectname¶

Create new project with an initial project template

--help, -h¶

Print command help and options

--logfile=FILE¶

Log file. if omitted stderr will be used

--loglevel=LEVEL, -L LEVEL¶

Log level (default: None)

--nolog¶

Disable logging completely

--spider=SPIDER¶

Always use this spider when arguments are urls

--profile=FILE¶

Write python cProfile stats to FILE

--lsprof=FILE¶

Write lsprof profiling stats to FILE

--pidfile=FILE¶

Write process ID to FILE

--set=NAME=VALUE, -s NAME=VALUE¶

Set/override setting (may be repeated)