By fcmax


2012-05-11 00:46:42 8 Comments

I can't change spider settings in parse method. But it is definitely must be a way.

For example:

class SomeSpider(BaseSpider):
    name = 'mySpider'
    allowed_domains = ['example.com']
    start_urls = ['http://example.com']
    settings.overrides['ITEM_PIPELINES'] = ['myproject.pipelines.FirstPipeline']
    print settings['ITEM_PIPELINES'][0]
    #printed 'myproject.pipelines.FirstPipeline'
    def parse(self, response):
        #...some code
        settings.overrides['ITEM_PIPELINES'] = ['myproject.pipelines.SecondPipeline']
        print settings['ITEM_PIPELINES'][0]
        # printed 'myproject.pipelines.SecondPipeline'
        item = Myitem()
        item['mame'] = 'Name for SecondPipeline'  

But! Item will be processed by FirstPipeline. New ITEM_PIPELINES param don't work. How can I change settings after start crawling? Thanks in advance!

1 comments

@sergiuz 2015-09-26 21:23:39

If you want that different spiders to have different pipelines you can set for a spider a pipelines list attribute which defines the pipelines for that spider. Than in pipelines check for existence:

class MyPipeline(object):

    def process_item(self, item, spider):
        if self.__class__.__name__ not in getattr(spider, 'pipelines',[]):
            return item
        ...
        return item

class MySpider(CrawlSpider):
    pipelines = set([
        'MyPipeline',
        'MyPipeline3',
    ])

If you want that different items to be proceesed by different pipelines you can do this:

    class MyPipeline2(object):
        def process_item(self, item, spider):
            if isinstance(item, MyItem):
                ...
                return item
            return item

Related Questions

Sponsored Content

2 Answered Questions

0 Answered Questions

1 Answered Questions

[SOLVED] Crawl and scrap a complete site with scrapy

2 Answered Questions

[SOLVED] Stop Scrapy spider when it meets a specified URL

  • 2016-09-10 11:13:11
  • Vladimir Sirovitskiy
  • 862 View
  • 1 Score
  • 2 Answer
  • Tags:   python scrapy

1 Answered Questions

Which parse method scrapy used to parse start_urls

2 Answered Questions

[SOLVED] When and how should use multiple spiders in one Scrapy project

  • 2014-08-01 02:19:17
  • user3337861
  • 3182 View
  • 18 Score
  • 2 Answer
  • Tags:   python scrapy

1 Answered Questions

[SOLVED] Crawling local files with Scrapy without an active project?

2 Answered Questions

2 Answered Questions

1 Answered Questions

Sequentially crawl website using scrapy

  • 2013-02-19 03:46:02
  • Praveer
  • 839 View
  • 0 Score
  • 1 Answer
  • Tags:   scrapy

Sponsored Content