Scrapy Dynamic Configurable

A dynamic configurable news crawler based Scrapy. See the blogs for more detail.

##Requirements

  • Scrapy
  • MySQL
  • Redis
  • SQLAlchemy

##Install in development

Mac OS X, use Homebrew

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew install mysql
$ brew install redis
$ sudo pip install scrapy
$ sudo pip install SQLAlchemy
$ sudo pip install redis

Ubuntu*

$ sudo apt-get install redis-server mysql-server mysql-client
$ pip install scrapy
$ pip install SQLAlchemy
$ pip install redis

and restore datatables from 'db.sql'

It is for scrapy 1.0 . It is for scrapy 1.0 .



Scrapy Dynamic Configurable

基于Scrapy的动态可配置新闻抓取工具。有关详情,请参阅博客

##要求

  • Scrapy
  • MySQL
  • Redis
  • SQLAlchemy
安装在开发中

Mac OS X,使用Homebrew

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew install mysql
$ brew install redis
$ sudo pip install scrapy
$ sudo pip install SQLAlchemy
$ sudo pip install redis

Ubuntu *

$ sudo apt-get install redis-server mysql-server mysql-client
$ pip install scrapy
$ pip install SQLAlchemy
$ pip install redis

并从’db.sql’恢复数据表

这是为了刮擦1.0。 它是为了刮擦1.0。




相关问题推荐