We Keep Coding

How to use URLLib.request to loop through URL's and download images?

My current program looks like this
import os
import urllib.request
baseUrl = "https://website.com/wp-content/upload/xxx/yyy/zzz-%s.jpg"
for i in range(1,48):
url = baseUrl % i
urllib.request.urlretrieve(baseUrl, os.path.basename(url))
I haven't coded python in a long time, but I wrote this using urllib2 back when I used to use Python2.7.
It is supposed to replace the %s in the URL and loop through 1-48, and download all the images to the directory that the script is in. But i get alot of errors.
edit : Here is the error that is thrown.
Traceback (most recent call last):
File "download.py", line 9, in <module>
urllib.request.urlretrieve(url, os.path.basename(url))
File "C:\Program Files\Python37\lib\urllib\request.py", line 247, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "C:\Program Files\Python37\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Program Files\Python37\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Program Files\Python37\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Program Files\Python37\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Program Files\Python37\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Program Files\Python37\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

urllib.request is only available on Python 3 so you have to run the code in Python 3.

Try using the requests module:
import requests
baseUrl = "https://website.com/wp-content/upload/xxx/yyy/zzz-%s.jpg"
for i in range(1,48):
url = baseUrl % i
response = requests.get(url)
my_raw_data = response.content
with open(os.path.basename(url), 'wb') as my_data:
my_data.write(my_raw_data)
my_data.close()
Just to add, you must use url in the request, not the baseUrl as shown in your code :
import os
import urllib.request
baseUrl = "https://website.com/wp-content/upload/xxx/yyy/zzz-%s.jpg"
for i in range(1,48):
url = baseUrl % i
#urllib.request.urlretrieve(baseUrl, os.path.basename(url))
#Use This line :
urllib.request.urlretrieve(url, os.path.basename(url))
Run this in Python 3

Simple fix, if you pass the correct string:
urllib.request.urlretrieve(url, os.path.basename(url))
The documentation says urlretrieve is a Legacy carryover, so you might want to find a different way to do this.
I found this alternate approach modified from another SO answer:
import os
import requests
baseUrl = "https://website.com/wp-content/upload/xxx/yyy/zzz-%s.jpg"
for i in range(1,48):
url = baseUrl % i
r = requests.get(url)
open(os.path.basename(url), 'wb').write(r.content)

python cookielib error:403 Forbidden

I am learning python networking. I had learnt socket and now I want to learn python HTTP to connect to HTTPServer, extract cookies etc. I am facing this problem with cookie extraction. Tried google but didn't found a solution, here is the code:
import cookielib
import urllib
import urllib2
ID_USERNAME= 'id_username'
ID_PASSWORD = 'id_password'
USERNAME = 'you#email.com'
PASSWORD = 'mypassword'
LOGIN_URL = 'https://bitbucket.org/account/signin/?next=/'
NORMAL_URL = 'https://bitbucket.org/'
def extract_cookie_info():
cj=cookielib.CookieJar()
login_data= urllib.urlencode({ID_USERNAME : USERNAME,ID_PASSWORD:PASSWORD})
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
resp = opener.open(LOGIN_URL,login_data)
for cookie in cj:
print "First time cookie: %s ----> %s" %(cookie.name,cookie.value)
print "Headers: %s"%resp.headers
resp = opener.open(NORMAL_URL)
for cookie in cj:
print "Second time cookie: %s --> %s"%(cookie.name,cookie.value)
print "Headers : %s"%resp.headers
if __name__ == '__main__':
extract_cookie_info()
This is the error:
Traceback (most recent call last):
File "e.py",line 27,in <module>
extract_cookie_info()
File "e.py",line 16,in extract_cookie_info
resp=opener.open(LOGIN_URL,login_data)
File "C:\Python27\lib\urllib2.py",line 435, in open
response = meth(req,response)
File "C:\Python27\lib\urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 473, in error
return self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 407, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

You are sending your login details as POST data rather than as part of the url.
>>> url = 'https://bitbucket.org/account/signin/'
>>> user = 'foo#example.com'
>>> pwd = 'secret'
>>> d = urlencode({'ID_USERNAME': user, 'ID_PASSWORD': pwd})
>>> cj = cookielib.CookieJar()
>>> opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
>>> resp = opener.open(url + '?' + d)
>>> res.getcode()
200
>>> for cookie in cj:print cookie.name
...
csrftoken

Curl post request returning 500 server error in python

The following curl request works perfectly when I run it through Windows' CMD.
curl -XPOST -H"Content-Type:application/json" http://my_url:8082/druid/v2/sql/ -d "{\"query\":\"SELECT DISTINCT(event) FROM programs\"}"
I am trying to replicate the same call in python 3 using urllib.requests
import urllib.request
values = {'query':'SELECT DISTINCT(event) FROM programs'}
url = 'http://my_url:8082/druid/v2/sql'
data = urllib.parse.urlencode(values).encode("utf-8")
req = urllib.request.Request(url, data)
req.add_header("Content-Type","application/json")
response = urllib.request.urlopen(req)
the_page = response.read()
However the python version is returning a Server Error
response = urllib.request.urlopen(req)
File "C:\Python\Python35\lib\urllib\request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "C:\Python\Python35\lib\urllib\request.py", line 472, in open
response = meth(req, response)
File "C:\Python\Python35\lib\urllib\request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python\Python35\lib\urllib\request.py", line 510, in error
return self._call_chain(*args)
File "C:\Python\Python35\lib\urllib\request.py", line 444, in _call_chain
result = func(*args)
File "C:\Python\Python35\lib\urllib\request.py", line 590, in
http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Server Error
Can someone tell me what I'm doing wrong please?

You need to convert your data to json first, you are sending a dictionary.
import json
data = urllib.parse.urlencode(json.dumps(values)).encode("utf-8")
Or, use the requests library:
import requests
d = {'query':'SELECT DISTINCT(event) FROM programs'}
url = 'http://my_url:8082/druid/v2/sql'
r = requests.post(url, json=d)
r.raise_for_status()
print(r.text)

After some more experimentation, I managed to make the code work by replacing urllib.parse.urlencode with json.dumps. The code now reads as follows:
import urllib.request, json
values = {'query':'SELECT DISTINCT(event) FROM programs'}
url = 'http://my_url/druid/v2/sql'
data = json.dumps(values).encode("utf-8")
req = urllib.request.Request(url, data)
req.add_header("Content-Type","application/json")
response = urllib.request.urlopen(req)
the_page = response.read()

Post multilevel dict from python application to python webservice

I am working on an automation script (that I am using to automate the process of conversion of some videos). In this script after video conversion, I am calling my web service to update the clip status in database and sending the web service a list of clips in POST request. But the problem is this request is failing and causing 500 internal server error on server side.
Here is the code I am using to call the web service with sample data I am trying with:
post_body = {
'clips': [
{
'clip_id': 17555,
'db_url': '/720p/14555.mp4'
}
]
}
params = urlencode(post_body)
url = str(self.update_url)
req = urllib2.Request(url, params)
response = urllib2.urlopen(req)
res = response.read()
print res
And here is the code of my web service:
def update_conversion_clips(request):
print "Web service is called"
try:
clips = request.POST.get('clips', None)
print clips
return HttpResponse(True)
except:
return HttpResponse(False)
Even first print statement is not executing.
Here is the error stack trace on application side:
Traceback (most recent call last):
File "conversion_script.py", line 48, in <module>
conversion_script.run()
File "conversion_script.py", line 44, in run
self.clips.update_clips_info(None)
File "/home/abc/video_experiments/conversion/clips_manager.py", line 59, in update_clips_info
response = urllib2.urlopen(req)
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 444, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR
and this is error on server side:
[20/Feb/2014 04:13:15] "POST /update_conversion_clips HTTP/1.1" 500 68733
According to my research this is happening due to multilevel dict that I am sending in POST. But I could not find any solution to resolve it.
New code now sending data as json (still does not work):
values = dict()
values['clips'] = [
{
'clip_id': 17555,
'db_url': '/720p/14555.mp4'
}
]
req = urllib2.Request(self.update_url)
req.add_header('Content-Type', 'application/json')
response = urllib2.urlopen(req, json.dumps(values))
res = response.read()
print res
and on server side:
try:
data = json.loads(request.body)
clips = data['clips']
except:
print "Exception occured!"
HttpResponse(True)

urlencode isn't really a good format for this data. A much better one would be JSON.
req = urllib2.Request(self.update_url)
req.add_header('Content-Type', 'application/json')
response = urllib2.urlopen(req, json.dumps(data))
print response.read()
(you could make this part a lot simpler by using the third-party requests library).
And in the server:
clips = json.loads(request.body)

Make POST request using Python

I am trying to make a POST request But getting this error :
Traceback (most recent call last):
File "demo.py", line 7, in <module>
r = requests.post(url, data=payload, headers=headers)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 87, in post
return request('post', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 266, in request
prep = req.prepare()
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 215, in prepare
p.prepare_body(self.data, self.files)
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 338, in prepare_body
body = self._encode_params(data)
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 74, in _encode_params
for k, vs in to_key_val_list(data):
ValueError: too many values to unpack
This is my program :
import requests
url = 'http://www.n-gal.com/index.php?route=openstock/openstock/optionStatus'
payload = {'var:1945,product_id:1126'}
headers = {'content-type': 'application/x-www-form-urlencoded'}
r = requests.post(url, data=payload, headers=headers)
I have tried the same POST request through Advanced rest client using following data :
URL : http://www.n-gal.com/index.php?route=openstock/openstock/optionStatus
payload : var=1945&product_id=1126
Content-Type: application/x-www-form-urlencoded
And it is working fine can anyone help me please...

You have made payload a set, not a dictionary. You forgot to close the string.
Change:
payload = {'var:1945,product_id:1126'}
To:
payload = {'var':'1945','product_id':'1126'}
As it is a set, the request will thus fail.

Try this :
import requests
url = 'http://www.n-gal.com/index.php?route=openstock/openstock/optionStatus'
payload = 'var=1945&product_id=1126'
headers = {'content-type': 'application/x-www-form-urlencoded'}
r = requests.post(url, data=payload, headers=headers)
print r.json()

import requests
import json
url = 'http://www.n-gal.com/index.php?route=openstock/openstock/optionStatus'
payload = {'var:1945,product_id:1126'}
headers = {'content-type': 'application/x-www-form-urlencoded'}
r = requests.post(url, data=json.dumps(payload), headers=headers)

I know this is a really old question, but I had the same problem when using Docker recently, and managed to solve it by including the requests library in the requirements (or just update the library with pip install requests --upgrade).
When using the original version of the requests library, it raised that very same error on Python3.8, which only stopped after upgrading it.