I have a nice URL structure and I want to iterate through the URL's and download all the images from the URL's. I am trying to use BeautifulSoup to get the job done, along with the requests function.
Here is the URL - https://sixmorevodka.com/#&gid=0&pid={i}, and I want 'i' to iterate from say 1 to 100 for this example.
from bs4 import BeautifulSoup as soup
import requests, contextlib, re, os
#contextlib.contextmanager
def get_images(url:str):
d = soup(requests.get(url).text, 'html.parser')
yield [[i.find('img')['src'], re.findall('(?<=\.)\w+$', i.find('img')['alt'])[0]] for i in d.find_all('a') if re.findall('/image/\d+', i['href'])]
n = 100 #end value
for i in range(n):
with get_images(f'https://sixmorevodka.com/#&gid=0&pid={i}') as links:
print(links)
for c, [link, ext] in enumerate(links, 1):
with open(f'ART/image{i}{c}.{ext}', 'wb') as f:
f.write(requests.get(f'https://sixmorevodka.com{link}').content)
I think I either messed something up in the Yield line or in the very last write line. Someone help me out please. I am using Python 3.7
In looking at the structure of that webpage, your gid parameter is invalid. To see for yourself, open a new tab and navigate to https://sixmorevodka.com/#&gid=0&pid=22.
You'll notice that none of the portfolio images are displayed. gid can be a value 1-5, denoting the grid to which an image belongs.
Regardless, your current scraping methodology is inefficient, and puts undue traffic on the website. Instead, you only need to make this request once, and extract the urls actually containing the images using the ilb portfolio__grid__item class selector.
Then, you can iterate and download those urls, which are directly the source of the images.
Related
I'm actually trying to code a little "GPS" and actually I couldn't use Google API because of the daily restriction.
I decided to use a site "viamichelin" which provide me the distance between two adresses. I created a little code to fetch all the URL adresses I needed like this :
import pandas
import numpy as np
df = pandas.read_excel('C:\Users\Bibi\Downloads\memoire\memoire.xlsx', sheet_name='Clients')
df2= pandas.read_excel('C:\Users\Bibi\Downloads\memoire\memoire.xlsx', sheet_name='Agences')
matrix=df.as_matrix(columns=None)
clients = np.squeeze(np.asarray(matrix))
matrix2=df2.as_matrix(columns=None)
agences = np.squeeze(np.asarray(matrix2))
compteagences=0
comptetotal=0
for j in agences:
compteclients=0
for i in clients:
print agences[compteagences]
print clients[compteclients]
url ='https://fr.viamichelin.be/web/Itineraires?departure='+agences[compteagences]+'&arrival='+clients[compteclients]+'&arrivalId=34MTE1MnJ5ZmQwMDMzb3YxMDU1ZDFvbGNOVEF1TlRVNU5UUT1jTlM0M01qa3lOZz09Y05UQXVOVFl4TlE9PWNOUzQzTXpFNU5nPT1jTlRBdU5UVTVOVFE9Y05TNDNNamt5Tmc9PTBqUnVlIEZvbmQgZGVzIEhhbGxlcw==&index=0&vehicle=0&type=0&distance=km¤cy=EUR&highway=false&toll=false&vignette=false&orc=false&crossing=true&caravan=false&shouldUseTraffic=false&withBreaks=false&break_frequency=7200&coffee_duration=1200&lunch_duration=3600&diner_duration=3600&night_duration=32400&car=hatchback&fuel=petrol&fuelCost=1.393&allowance=0&corridor=&departureDate=&arrivalDate=&fuelConsumption='
print url
compteclients+=1
comptetotal+=1
compteagences+=1
All my datas are on Excel that's why I used the pandas library. I have all the URL's needed for my project.
Although, I would like to extract the number of kilometers needed but there's a little problem. In the source code, I don't have the information I need, so I can't extract it with Python... The site is presented like this:
Michelin
When I click on "inspect" I can find the information needed (on the left) but I can't on the source code (on the right) ... Can someone provide me some help?
Itinerary
I have already tried this, without succeeding :
import os
import csv
import requests
from bs4 import BeautifulSoup
requete = requests.get("https://fr.viamichelin.be/web/Itineraires?departure=Rue%20Lebeau%2C%20Liege%2C%20Belgique&departureId=34MTE1Mmc2NzQwMDM0NHoxMDU1ZW44d2NOVEF1TmpNek5ERT1jTlM0MU5qazJPQT09Y05UQXVOak16TkRFPWNOUzQxTnpBM01nPT1jTlRBdU5qTXpOREU9Y05TNDFOekEzTWc9PTBhUnVlIExlYmVhdQ==&arrival=Rue%20Rys%20De%20Mosbeux%2C%20Trooz%2C%20Belgique&arrivalId=34MTE1MnJ5ZmQwMDMzb3YxMDU1ZDFvbGNOVEF1TlRVNU5UUT1jTlM0M01qa3lOZz09Y05UQXVOVFl4TlE9PWNOUzQzTXpFNU5nPT1jTlRBdU5UVTVOVFE9Y05TNDNNamt5Tmc9PTBqUnVlIEZvbmQgZGVzIEhhbGxlcw==&index=0&vehicle=0&type=0&distance=km¤cy=EUR&highway=false&toll=false&vignette=false&orc=false&crossing=true&caravan=false&shouldUseTraffic=false&withBreaks=false&break_frequency=7200&coffee_duration=1200&lunch_duration=3600&diner_duration=3600&night_duration=32400&car=hatchback&fuel=petrol&fuelCost=1.393&allowance=0&corridor=&departureDate=&arrivalDate=&fuelConsumption=")
page = requete.content
soup = BeautifulSoup(page, "html.parser")
print soup
Looking at the inspector for the page, the actual routing is done via a JavaScript invocation to this rather long URL.
The data you need seems to be in that response, starting from _scriptLoaded(. (Since it's a JavaScript object literal, you can use Python's built-in JSON library to load the data into a dict.)
I'm not a programmer, but I'm trying to teach myself Python so that I can pull data off various sites for projects that I'm working on. I'm using "Automate the Boring Stuff" and I'm having trouble getting the examples to work with one of the pages I'm trying to pull data from.
I'm using Anaconda as my prompt with Python 3.65. Here's what I've done:
Step 1: create the beautiful soup object
import requests, bs4
res = requests.get('https://www.almanac.com/weather/history/zipcode/02111/2017-05-15')
res.raise_for_status()
weatherTest = bs4.BeautifulSoup(res.text)
type(weatherTest)
This works, and returns the result
<class 'bs4.BeautifulSoup'>
I've made the assumption that the "noStarchSoup" that was in the original text (in place of weatherTest here) is a name the author gave to the object that I can rename to something more relevant to me. If that's not accurate, please let me know.
Step 2: pull an element out of the html
Here's where I get stuck. The author had just mentioned how to pull a page down into a file (which I would prefer not to do, I want to use the bs4 object), but then is using that file as his source for the html data. The exampleFile was his downloaded file.
import bs4
exampleFile = open('https://www.almanac.com/weather/history/zipcode/02111/2017-05-15')
I've tried using weatherTest in place of exampleFile, I've tried running the whole thing with the original object name (noStarchSoup), I've even tried it with exampleFile, even though I haven't downloaded the file.
What I get is
"OSError: [Errno 22] Invalid argument:
'https://www.almanac.com/weather/history/zipcode/02111/2017-05-15'
The next step is to tell it what element to pull but I'm trying to fix this error first and kind of spinning my wheels here.
Couldn't resist here!
I found this page during my search but this answer didn't quite help... try this code :)
Step 1: download Anaconda 3.0+
Step 2: (function)
# Import Libraries
import bs4
import requests
def import_high_short_tickers(market_type):
if market_type == 'NADAQ':
page = requests.get('https://www.highshortinterest.com/nasdaq/')
elif market_type == 'NYSE':
page = requests.get('https://www.highshortinterest.com/nyse/')
else:
logger.error("Invalid market_type: " + market_type)
return None
# Parse the HTML Page
soup = bs4.BeautifulSoup(page.content, 'html.parser')
# Grab only table elements
all_soup = soup.find_all('table')
# Get what you want from table elements!
for element in all_soup:
listing = str(element)
if 'https://finance.yahoo.com/' in listing:
# Stuff the results in a pandas data frame (if your not using these you should)
data = pd.read_html(listing)
return data
Yes Yes its very crude but don't hate!
Cheers!
I am trying to write a Python script that can navigate to this page, fill up the form data, submit the form and download a ZIP file that is returned automatically.
the file I need is generated by selecting Bhavcopy in the form along with providing a valid date from the past.
[A sample input and result has been shown here2
So far I have tried to learn and implement several approaches using using Scrapy, webbrowser, requests, Beautifulsoup, Mechanize and Selenium by copying and modifying example codes from the documentations but haven't been able to make any progress.
So far the only code that worked without errors is :
import requests
from bs4 import BeautifulSoup
u = 'https://www.nseindia.com/products/content/all_daily_reports.htm'
p = requests.get(u)
soup = BeautifulSoup(p.text, 'html.parser')
for link in soup.find_all('a'):
d = requests.get(link.get('href'))
print(d)
This code is no where complete and I don't how to proceed.
Logically I know I should be:
fetching the page [done]
selecting form elements [not done] maybe doable in Scrapy
filling in data ( the 1st parameter is constant the date can be supplied in a loop) [have no clue how to do this programatically]
submitting the form [same as above]
clicking on the a tag to download the file in the href attribute [a request.get(on the href value) should do it
Any pointers to how I can achieve this would be greatly appreciated.
The webpage allows you to download a daily report called the Bhavcopy that contains the Open, Low, High, Close data of all stocks that traded in the National Stock Exchange (India) and i wish to accumulate as much historical data as possible.
If your aim is to download the zip file after providing the date, this will do the job.
If you inspect the zip file element, you can see that the href is /content/historical/EQUITIES/2018/FEB/cm02FEB2018bhav.csv.zip or /content/historical/EQUITIES/2017/DEC/cm06DEC2017bhav.csv.zip as per the date you enter.
If you look closer at the links, you can see that formatting the links is pretty easy. So, your program changes from going to the url, submitting the data and getting the zip file to simply formatting the url.
Using the url https://www.nseindia.com/content/historical/EQUITIES/2017/DEC/cm06DEC2017bhav.csv.zip, you can directly download the zip file.
import webbrowser
def download_zip(date, month, year):
url = 'https://www.nseindia.com/content/historical/EQUITIES/{2}/{1}/cm{0}{1}{2}bhav.csv.zip'.format(date, month, year)
webbrowser.open(url)
download_zip('02', 'FEB', '2018')
Calling the download_zip function will directly download the zip file.
The only thing you have to take care of is the format of the parameters. Date format is DD, month is MMM and year is YYYY. Obviously, this program will throw exceptions if the url is invalid. Take care of that and handle them using try/except.
If you don't want the browser to pop open for the download, you can download the file using requests module.
def download_zip(date, month, year):
url = 'https://www.nseindia.com/content/historical/EQUITIES/{2}/{1}/cm{0}{1}{2}bhav.csv.zip'.format(date, month, year)
r = requests.get(url)
location = 'C:/Users/username/Downloads/' # Change the location as per your OS and your needs.
filename = 'cm{}{}{}bhav.csv.zip'.format(date, month, year) # You can name this file anything you want, but with a .zip extension
with open(location + filename, 'wb') as f:
f.write(r.content)
I am trying to extract the 'Seller rank' from items on amazon using Python requests and lxml. So:
<li id="SalesRank">
<b>Amazon Bestsellers Rank:</b>
957,875 in Books (See Top 100 in Books)
from this example, 957875 is the number I want to extract.
(Please note, the actual HTML has about 100 blank lines between 'Amazon Bestsellers Rank:' and '957875'. Unsure if this is effecting my result.)
My current Python code is set up like so:
import re
import requests
from lxml import html
page = requests.get('http://www.amazon.co.uk/Lakeland-Expanding-Together-Compartments-Organiser/dp/B00A7Q77GM/ref=sr_1_1?s=kitchen&ie=UTF8&qid=1452504370&sr=1-1-spons&psc=1')
tree = html.fromstring(page.content)
salesrank = tree.xpath('//li[#id="SalesRank"]/text()')
print 'Sales Rank:', salesrank
and the printed output is
Sales Rank: []
I was expecting to receive the full list data including all the blank lines of which I would later parse.
Am I correct in assuming that /text() is not the correct use in this instance and I need to put something else?
Any help is greatly appreciated.
You are getting an empty list because in one call of the url you are not getting the complete data of the web page. For that you have to stream through the url and get all the data in small chunks. And then find out the required in the non-empty chunk. The code for the following is :-
import requests as rq
import re
from bs4 import BeautifulSoup as bs
r=rq.get('http://www.amazon.in/gp/product/0007950306/ref=s9_al_bw_g14_i1?pf_rd_m=A1VBAL9TL5WCBF&pf_rd_s=merchandised-search-3&pf_rd_r=1XBKB22RGT2HBKH4K2NP&pf_rd_t=101&pf_rd_p=798805127&pf_rd_i=4143742031',stream=True)
for chunk in r.iter_content(chunk_size=1024):
if chunk:
data = chunk
soup=bs(data)
elem=soup.find_all('li',attrs={'id':'SalesRank'})
if elem!=[]:
s=re.findall('#[\d+,*]*\sin',str(elem[0]))
print s[0].split()[0]
break
I tried to do the following: read trough a website choose the 18 th of link on this site, open that link and repeat that 7 times. But I am not really advanced in programming, so I stuck by trying to open the 18th link. How can I do that? My code is this:
import urllib.request
import io
u = urllib.request.urlopen("http://xxxxxxxx.com/tsugi/mod/python-data/data/known_by_Yong.html", data = None)
f = io.TextIOWrapper(u, encoding='utf-8')
text = f.read()
from bs4 import BeautifulSoup
soup = BeautifulSoup(text)
print (soup.find_all("a")
my result looks like this: e.g
[ href="http://xxxxxxxx.com/tsugi/mod/python-data/data/known_by_Keiva.html">Keiva</a>, href="http://xxxxx.com/tsugi/mod/python-data/data/known_by_Rowyn.</a>]
An HTML-document with names/links.
Till I not expect, that anybody is guiding me trough the whole code, where can I look up what I need?
Here are my main questions:
How can I make the program count the names/links?
How can I open the 18th link in the list?
How can I repeat that,7 times?
Thanks for your support in advance!!