I'v created a script in scrapy to parse the titles of different sites listed in start_urls. The script is doing it's job flawlessly.
What I wish to do now is let my script stop after two of the urls are parsed no matter how many urls are there.
I've tried so far with:
import scrapy
from scrapy.crawler import CrawlerProcess
class TitleSpider(scrapy.Spider):
name = "title_bot"
start_urls = ["https://www.google.com/","https://www.yahoo.com/","https://www.bing.com/"]
def parse(self, response):
yield {'title':response.css('title::text').get()}
if __name__ == "__main__":
c = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0',
})
c.crawl(TitleSpider)
c.start()
How can I make my script stop when two of the listed urls are scraped?
Currently I see the only one way to immediately stop this script - usage of os._exit force exit function:
import os
import scrapy
from scrapy.crawler import CrawlerProcess
class TitleSpider(scrapy.Spider):
name = "title_bot"
start_urls = ["https://www.google.com/","https://www.yahoo.com/","https://www.bing.com/"]
item_counter =0
def parse(self, response):
yield {'title':response.css('title::text').get()}
self.item_counter+=1
print(self.item_counter)
if self.item_counter >=2:
self.crawler.stats.close_spider(self,"2 items")
os._exit(0)
if __name__ == "__main__":
c = CrawlerProcess({'USER_AGENT': 'Mozilla/5.0' })
c.crawl(TitleSpider)
c.start()
Another things that I tried. But I didn't received required result (immediately stop script afted 2 scraped items with only 3 urls in start_urls):
Transfer CrawlerProcess instance into spider settings and calling
CrawlerProcess.stop ,(reactor.stop), etc.. and other methods
from parse method.
Usage of CloseSpider extension docs source ) with following CrawlerProcess definition:
c = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0',
'EXTENSIONS' : {
'scrapy.extensions.closespider.CloseSpider': 500,
},
"CLOSESPIDER_ITEMCOUNT":2 })
Reducing CONCURRENT_REQUESTS setting to 1 (with raise CloseSpider
condition in parse method). When application scraped 2 items and it
reaches code line with raise ClosesSpider - 3rd request already
started in another thread. In case of usage conventional way to stop
spider, application will be active until it process previously sent
requests and process their responses and only after that - it
closes.
As your application has relatively low numbers of urls in start_urls, application starts process all urls a long before it reaches raise CloseSpider.
As Gallaecio proposed, you can add a counter, but the difference here is that you export an item after the if statement. This way, it will almost always end up exporting 2 items.
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.exceptions import CloseSpider
class TitleSpider(scrapy.Spider):
name = "title_bot"
start_urls = ["https://www.google.com/", "https://www.yahoo.com/", "https://www.bing.com/"]
item_limit = 2
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.counter = 0
def parse(self, response):
self.counter += 1
if self.counter > self.item_limit:
raise CloseSpider
yield {'title': response.css('title::text').get()}
Why almost always? you may ask. It has to do with race condition in parse method.
Imagine that self.counter is currently equal to 1, which means that one more item is expected to be exported. But now Scrapy receives two responses at the same moment and invokes the parse method for both of them. If two threads running the parse method will increase the counter simultaneously, they will both have self.counter equal to 3 and thus will both raise the CloseSpider exception.
In this case (which is very unlikely, but still can happen), spider will export only one item.
Constructing on top of https://stackoverflow.com/a/38331733/939364, you can define a counter in the constructor of your spider, and use parse to increase it and raise CloseSpider when it reaches 2:
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.exceptions import CloseSpider # 1. Import CloseSpider
class TitleSpider(scrapy.Spider):
name = "title_bot"
start_urls = ["https://www.google.com/","https://www.yahoo.com/","https://www.bing.com/"]
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.counter = 0 # 2. Define a self.counter property
def parse(self, response):
yield {'title':response.css('title::text').get()}
self.counter += 1 # 3. Increase the count on each parsed URL
if self.counter >= 2:
raise CloseSpider # 4. Raise CloseSpider after 2 URLs are parsed
if __name__ == "__main__":
c = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0',
})
c.crawl(TitleSpider)
c.start()
I am not 100% certain that it will prevent a third URL to be parsed, because I think CloseSpider stops new requests from start but waits for started requests to finish.
If you want to prevent more than 2 items from being scraped, you can edit parse not to yield items when self.counter > 2.
Enumerate do jobs fine. Some changes in architecture and
for cnt, url in enumerate(start_urls):
if cnt > 1:
break
else:
parse(url)
Related
I am using Scrapy for scraping text from websites.
I would like Scrapy to scrape text from various URLs with different structure, without having to change the code for each website.
The following example works in my Jupyter Notebook for the given URL ( http://quotes.toscrape.com/tag/humor/ ). But it does not work for another (for ex.: https://en.wikipedia.org/wiki/Web_scraping ).
My question is, how to make it work for (most) other websites without manually inspecting every site and changing the code all the time? I guess I need to make a change under def parse(self, response), but so far I could not find a good example how to do that.
Modules:
import scrapy
import scrapy.crawler as crawler
from multiprocessing import Process, Queue
from twisted.internet import reactor
Spider:
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = ['http://quotes.toscrape.com/tag/humor/']
def parse(self, response):
for quote in response.css('div.quote'):
print(quote.css('span.text::text').extract_first())
A wrapper to make it run more times in Jupyter:
def run_spider(spider):
def f(q):
try:
runner = crawler.CrawlerRunner()
deferred = runner.crawl(spider)
deferred.addBoth(lambda _: reactor.stop())
reactor.run()
q.put(None)
except Exception as e:
q.put(e)
q = Queue()
p = Process(target=f, args=(q,))
p.start()
result = q.get()
p.join()
if result is not None:
raise result
Get the result:
print('Extracted text:')
run_spider(QuotesSpider)
Extracted text:
“The person, be it gentleman or lady, who has not pleasure in a good novel, ..."
I have a list of start 2000 urls and I'm using:
DOWNLOAD_DELAY = 0.25
For controlling the speed of the requests, But I also want to add a bigger delay after n requests.
For example, I want a delay of 0.25 seconds for each request and a delay of 100 seconds each 500 requests.
Edit:
Sample code:
import os
from os.path import join
import scrapy
import time
date = time.strftime("%d/%m/%Y").replace('/','_')
list_of_pages = {'http://www.lapatilla.com/site/':'la_patilla',
'http://runrun.es/':'runrunes',
'http://www.noticierodigital.com/':'noticiero_digital',
'http://www.eluniversal.com/':'el_universal',
'http://www.el-nacional.com/':'el_nacional',
'http://globovision.com/':'globovision',
'http://www.talcualdigital.com/':'talcualdigital',
'http://www.maduradas.com/':'maduradas',
'http://laiguana.tv/':'laiguana',
'http://www.aporrea.org/':'aporrea'}
root_dir = os.getcwd()
output_dir = join(root_dir,'data/',date)
class TestSpider(scrapy.Spider):
name = "news_spider"
download_delay = 1
start_urls = list_of_pages.keys()
def parse(self, response):
if not os.path.exists(output_dir):
os.makedirs(output_dir)
filename = list_of_pages[response.url]
print time.time()
with open(join(output_dir,filename), 'wb') as f:
f.write(response.body)
The list, in this case, is shorter yet the idea is the same. I want to have to levels of delays one for each request and one each 'N' requests.
I'm not crawling the links, just saving the main page.
You can look into using an AutoThrottle extension which does not give you a tight control of the delays but instead has its own algorithm of slowing down the spider adjusting it on the fly depending on the response time and number of concurrent requests.
If you need more control over the delays at certain stages of the scraping process, you might need a custom middleware or a custom extension (similar to AutoThrottle - source).
You can also change the .download_delay attribute of your spider on the fly. By the way, this is exactly what AutoThrottle extension does under-the-hood - it updates the .download_delay value on the fly.
Some related topics:
Per request delay
Request delay configurable for each Request
Here's a sleepy decorator I wrote that pauses after N function calls.
def sleepy(f):
def wrapped(*args, **kwargs):
wrapped.calls += 1
print(f"{f.__name__} called {wrapped.calls} times")
if wrapped.calls % 500 == 0:
print("Sleeping...")
sleep(20)
return f(*args, **kwargs)
wrapped.calls = 0
return wrapped
I'm new in Scrapy. I have thousands of url,xpath tuples and values in a database.
These urls are from different domains (not allways, there can be 100 urls from the same domain).
x.com/a //h1
y.com/a //div[#class='1']
z.com/a //div[#href='...']
x.com/b //h1
x.com/c //h1
...
Now I want to get these values every 2 hours as fast as possible but to be sure that I don't overload any of these.
Can't figure out how to do that.
My thoughts:
I could create one Spider for every different domain, set it's parsing rules and run them at once.
Is it a good practice?
EDIT:
I'm not sure how it would work with outputting data into database according to concurrency.
EDIT2:
I can do something like this - for every domain there is a new spider. But this is impossible to do having thousands of different urls and it's xpaths.
class WikiScraper(scrapy.Spider):
name = "wiki_headers"
def start_requests(self):
urls = [
'https://en.wikipedia.org/wiki/Spider',
'https://en.wikipedia.org/wiki/Data_scraping',
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
hxs = HtmlXPathSelector(response)
header = hxs.select('//h1/text()').extract()
print header
filename = 'result.txt'
with open(filename, 'a') as f:
f.write(header[0])
self.log('Saved file %s' % filename)
class CraigslistScraper(scrapy.Spider):
name = "craigslist_headers"
def start_requests(self):
urls = [
'https://columbusga.craigslist.org/act/6062657418.html',
'https://columbusga.craigslist.org/acc/6060297390.html',
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
hxs = HtmlXPathSelector(response)
header = hxs.select('//span[#id="titletextonly"]/text()').extract()
filename = 'result.txt'
with open(filename, 'a') as f:
f.write(header[0])
self.log('Saved file %s' % filename)
From the example you posted in edit2, it looks like all your classes are easily abstractable by one more level. How about this:?
from urllib.parse import urlparse
class GenericScraper(scrapy.Spider):
def __init__(self, urls, xpath):
super().__init__()
self.name = self._create_scraper_name_from_url(urls[0])
self.urls = urls
self.xpath = xpath
def _create_scraper_name_from_url(url):
'''Generate scraper name from url
www.example.com/foobar/bar -> www_example_com'''
netloc = urlparse(url).netloc
return netloc.replace('.','_')
def start_requests(self):
for url in self.urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
hxs = HtmlXPathSelector(response)
header = hxs.select(self.xpath).extract()
filename = 'result.txt'
with open(filename, 'a') as f:
f.write(header[0])
self.log('Saved file %s' % filename)
Next, you could group the data from database by xpaths
for urls, xpath in grouped_data:
scraper = GenericScraper(urls, xpath)
# do whatever you need with scraper
AD concurency: your database should handle concurent writes so I do not see a problem there
Edit:
Related to the timeouts: I Do not know how scrapy works under the hood i.e. if it uses some sort of paralelization and whether it runs asynchronously in the background. But from what you wrote I guess it does and when you fire up 1k scrapers each firing multiple requests at time your hardware cant handle that much traffic (disclaimer, this is just a guess!).
There might be a native way to do this, but a possible workaround is to use multiprocessing + Queue:
from multiprocessing import JoinableQueue, Process
NUMBER_OF_CPU = 4 # change this to your number.
SENTINEL = None
class Worker(Process):
def __init__(self, queue):
super().__init__()
self.queue = queue
def run(self):
# blocking wait !You have to use sentinels if you use blocking waits!
item = self.queue.get():
if item is SENTINEL:
# we got sentinel, there are no more scrapers to process
self.queue.task_done()
return
else:
# item is scraper, run it
item.run_spider() # or however you run your scrapers
# This assumes that each scraper is **not** running in background!
# Tell the JoinableQueue we have processed one more item
# In the main thread the queue.join() waits untill for
# each item taken from queue a queue.task_done() is called
self.queue.task_done()
def run():
queue = JoinableQueue()
# if putting that many things in the queue gets slow (I imagine
# it can) You can fire up a separate Thread/Process to fill the
# queue in the background while workers are already consuming it.
for urls, xpath in grouped_data:
scraper = GenericScraper(urls, xpath)
queue.put(scraper)
for sentinel in range(NUMBER_OF_CPU):
# None or sentinel of your choice to tell the workers there are
# no more scrapers to process
queue.put(SENTINEL)
workers = []
for _ in range(NUMBER_OF_CPU):
worker = Worker(queue)
workers.append(worker)
worker.start()
# We have to wait until the queue is processed
queue.join()
But please bear in mind that this is a vanilla approach for paralell execution completely ignoring Scrapy abilities. I have found This blogpost which uses twisted to achieve (what I think is) the same thing. But since I've never used twisted I can't comment on that
if you are thinking about scrapy can't handle multiple domains at once because of the allowed_domains parameters, remember that it is optional.
If no allowed_domains parameter is set in the spider, it can work with every domain it gets.
If I understand correctly you have map of domain to xpath values and you want to pull xpath depending on what domain you crawl?
Try something like:
DOMAIN_DATA = [('domain.com', '//div')]
def get_domain(url):
for domain, xpath in DOMAIN_DATA:
if domain in url:
return xp
def parse(self, response):
xpath = get_domain(response.url)
if not xpath:
logging.error('no xpath for url: {}; unknown domain'.format(response.url))
return
item = dict()
item['some_field'] = repsonse.xpath(xpath).extract()
yield item
I want to crawl a website with 2 parts and my script is not as fast as I need.
Is it possible to launch 2 spiders, one for scraping the first part and the second one for the second part?
I tried to have 2 different classes, and run them
scrapy crawl firstSpider
scrapy crawl secondSpider
but i think that it is not smart.
I read the documentation of scrapyd but I don't know if it's good for my case.
I think what you are looking for is something like this:
import scrapy
from scrapy.crawler import CrawlerProcess
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
process = CrawlerProcess()
process.crawl(MySpider1)
process.crawl(MySpider2)
process.start() # the script will block here until all crawling jobs are finished
You can read more at: running-multiple-spiders-in-the-same-process.
Or you can run with like this, you need to save this code at the same directory with scrapy.cfg (My scrapy version is 1.3.3) :
from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess
setting = get_project_settings()
process = CrawlerProcess(setting)
for spider_name in process.spiders.list():
print ("Running spider %s" % (spider_name))
process.crawl(spider_name,query="dvh") #query dvh is custom argument used in your scrapy
process.start()
Better solution is (if you have multiple spiders) it dynamically get spiders and run them.
from scrapy import spiderloader
from scrapy.utils import project
from twisted.internet.defer import inlineCallbacks
#inlineCallbacks
def crawl():
settings = project.get_project_settings()
spider_loader = spiderloader.SpiderLoader.from_settings(settings)
spiders = spider_loader.list()
classes = [spider_loader.load(name) for name in spiders]
for my_spider in classes:
yield runner.crawl(my_spider)
reactor.stop()
crawl()
reactor.run()
(Second Solution):
Because spiders.list() is deprecated in Scrapy 1.4 Yuda solution should be converted to something like
from scrapy import spiderloader
from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess
settings = get_project_settings()
process = CrawlerProcess(settings)
spider_loader = spiderloader.SpiderLoader.from_settings(settings)
for spider_name in spider_loader.list():
print("Running spider %s" % (spider_name))
process.crawl(spider_name)
process.start()
I have a list of start 2000 urls and I'm using:
DOWNLOAD_DELAY = 0.25
For controlling the speed of the requests, But I also want to add a bigger delay after n requests.
For example, I want a delay of 0.25 seconds for each request and a delay of 100 seconds each 500 requests.
Edit:
Sample code:
import os
from os.path import join
import scrapy
import time
date = time.strftime("%d/%m/%Y").replace('/','_')
list_of_pages = {'http://www.lapatilla.com/site/':'la_patilla',
'http://runrun.es/':'runrunes',
'http://www.noticierodigital.com/':'noticiero_digital',
'http://www.eluniversal.com/':'el_universal',
'http://www.el-nacional.com/':'el_nacional',
'http://globovision.com/':'globovision',
'http://www.talcualdigital.com/':'talcualdigital',
'http://www.maduradas.com/':'maduradas',
'http://laiguana.tv/':'laiguana',
'http://www.aporrea.org/':'aporrea'}
root_dir = os.getcwd()
output_dir = join(root_dir,'data/',date)
class TestSpider(scrapy.Spider):
name = "news_spider"
download_delay = 1
start_urls = list_of_pages.keys()
def parse(self, response):
if not os.path.exists(output_dir):
os.makedirs(output_dir)
filename = list_of_pages[response.url]
print time.time()
with open(join(output_dir,filename), 'wb') as f:
f.write(response.body)
The list, in this case, is shorter yet the idea is the same. I want to have to levels of delays one for each request and one each 'N' requests.
I'm not crawling the links, just saving the main page.
You can look into using an AutoThrottle extension which does not give you a tight control of the delays but instead has its own algorithm of slowing down the spider adjusting it on the fly depending on the response time and number of concurrent requests.
If you need more control over the delays at certain stages of the scraping process, you might need a custom middleware or a custom extension (similar to AutoThrottle - source).
You can also change the .download_delay attribute of your spider on the fly. By the way, this is exactly what AutoThrottle extension does under-the-hood - it updates the .download_delay value on the fly.
Some related topics:
Per request delay
Request delay configurable for each Request
Here's a sleepy decorator I wrote that pauses after N function calls.
def sleepy(f):
def wrapped(*args, **kwargs):
wrapped.calls += 1
print(f"{f.__name__} called {wrapped.calls} times")
if wrapped.calls % 500 == 0:
print("Sleeping...")
sleep(20)
return f(*args, **kwargs)
wrapped.calls = 0
return wrapped