looter

To install, just run

$ pip install looter

features

• super-light-weight crawler
• automatically generate spider with template
• blazing fast speed with concurrent.futures
• provides shell to debug your crawler

example

Here's a very simple image crawler: First, open it with shell

$ looter shell konachan.net/post

Then, extract with css and grab images you want!

>>> imgs = tree.cssselect('a.directlink')
>>> save_imgs(imgs)

Or, just in a more crude way :d

>>> save_imgs(links(res, search='jpg'))

And of course, there are more functions for you to discover.

usage

Usage:
  looter genspider <name> <tmpl> [--async]
  looter shell [<url>]
  looter (-h | --help | --version)

Options:
  -h --help        Show this screen.
  --version        Show version.
  --async          Use async instead of concurrent.

Templates available: data, image. You can also view examples.

tutorial

以下是中文教程：猛戳这里