Skip to content

Command Line Interface

spatula provides a command line interface that is useful for iterative development of scrapers.

Once installed within your Python environment, spatula can be invoked on the command line. E.g.:

  (scrape-venv) ~/scrape-proj $ spatula --version
  spatula, version 0.9.0

Or with poetry:

  ~/scrape-proj $ poetry run spatula --version
  spatula, version 0.9.0

The CLI provides four useful subcommands for different stages of development:

spatula

Usage:

spatula [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--version boolean Show the version and exit. False
--help boolean Show this message and exit. False

scout

Run first step of scrape & output data to a JSON file.

This command is intended to be used to detect at a first approximation whether or not a full scrape might need to be run. If the first layer detects any changes it is safe to say that the full run will as well.

This will work in the common case where a new subpage is added or removed. Of course in more advanced cases this depends upon the first page being scraped (typically a ListPage derivative) surfacing enough information (perhaps a last_updated date) to know whether any of the other pages have been scraped.

Usage:

spatula scout [OPTIONS] INITIAL_PAGE_NAME

Options:

Name Type Description Default
-s, --source text Provide (or override) source URL required
-o, --output-file text override default output file [default: scout.json]. scout.json
-ua, --user-agent text override default user-agent spatula 0.9.0
--rpm integer set requests per minute (default: 60) 60
--timeout integer set HTTP request timeout in seconds (default: 5) 5
--verify / --no-verify boolean control verification of SSL certs True
--retries integer configure how many retries to perform on HTTP request error (default: 0) 0
--retry-wait integer configure how many seconds to wait on HTTP request error (default: 10) 10
-H, --header text add a header to all requests. example format: 'Accept: application/json' required
-v, --verbosity integer override default verbosity for command (0-3) -1
--fastmode boolean use a cache to avoid making unnecessary requests False
--help boolean Show this message and exit. False

scrape

Run full scrape, and output data to disk.

Usage:

spatula scrape [OPTIONS] INITIAL_PAGE_NAME

Options:

Name Type Description Default
-o, --output-dir text override default output directory. required
--rmdir / --no-rmdir boolean remove output directory before scrape. False
-s, --source text Provide (or override) source URL required
--dump text Specify dump function json.dump
-ua, --user-agent text override default user-agent spatula 0.9.0
--rpm integer set requests per minute (default: 60) 60
--timeout integer set HTTP request timeout in seconds (default: 5) 5
--verify / --no-verify boolean control verification of SSL certs True
--retries integer configure how many retries to perform on HTTP request error (default: 0) 0
--retry-wait integer configure how many seconds to wait on HTTP request error (default: 10) 10
-H, --header text add a header to all requests. example format: 'Accept: application/json' required
-v, --verbosity integer override default verbosity for command (0-3) -1
--fastmode boolean use a cache to avoid making unnecessary requests False
--help boolean Show this message and exit. False

shell

Start a session to interact with a particular page.

Usage:

spatula shell [OPTIONS] URL

Options:

Name Type Description Default
-X, --verb text set HTTP verb such as POST GET
-ua, --user-agent text override default user-agent spatula 0.9.0
--rpm integer set requests per minute (default: 60) 60
--timeout integer set HTTP request timeout in seconds (default: 5) 5
--verify / --no-verify boolean control verification of SSL certs True
--retries integer configure how many retries to perform on HTTP request error (default: 0) 0
--retry-wait integer configure how many seconds to wait on HTTP request error (default: 10) 10
-H, --header text add a header to all requests. example format: 'Accept: application/json' required
-v, --verbosity integer override default verbosity for command (0-3) -1
--fastmode boolean use a cache to avoid making unnecessary requests False
--help boolean Show this message and exit. False

test

Scrape a single page and see output immediately.

This eases the common cycle of making modifications to a scraper, running a scrape (possibly with long-running but irrelevant portions commented out), and comparing output to what is expected.

test can also be useful for debugging existing scrapers, you can see exactly what a single step of the scrape is providing, to help narrow down where erroneous data is coming from.

Example:

$ spatula test path.to.ClassName --source https://example.com

This will run the scraper defined at path.to.ClassName against the provided URL.

Usage:

spatula test [OPTIONS] CLASS_NAME

Options:

Name Type Description Default
--interactive / --no-interactive boolean Interactively prompt for missing data. (Default: false) False
-d, --data text Provide input data in name=value pairs. required
-s, --source text Provide (or override) source URL required
--pagination / --no-pagination boolean Determine whether or not pagination should be followed or one page is enough for testing. (Default: true) True
--subpages / --no-subpages boolean Determine whether subpages should be scraped. (Default: false) False
-ua, --user-agent text override default user-agent spatula 0.9.0
--rpm integer set requests per minute (default: 60) 60
--timeout integer set HTTP request timeout in seconds (default: 5) 5
--verify / --no-verify boolean control verification of SSL certs True
--retries integer configure how many retries to perform on HTTP request error (default: 0) 0
--retry-wait integer configure how many seconds to wait on HTTP request error (default: 10) 10
-H, --header text add a header to all requests. example format: 'Accept: application/json' required
-v, --verbosity integer override default verbosity for command (0-3) -1
--fastmode boolean use a cache to avoid making unnecessary requests False
--help boolean Show this message and exit. False
Back to top