I'm testing some things and i keep getting the error "write() argument must be str, not HTTPResponse" Here's my code:
import requests
image="http://www.casperdenhaan.nl/wp-content/uploads/2020/03/Logo.jpg"
savefile=open("image.png","w+")
savefile.write(requests.get(image).raw)
savefile.close()
I can get the raw data, but I can't write it to a new file. Is there a way i could get around this problem?
When you call .raw on the response object it returns an HTTPResponse object. You need to call .content to get a bytes object.
type(requests.get(image).raw)
urllib3.response.HTTPResponse
type(requests.get(image).content)
bytes
You need to open the file in write binary mode:
open("image.png","wb")
I suggest using a "with" block, this way you don't need to explicitly close the file. Here is a working version of the code:
import requests
url = "http://www.casperdenhaan.nl/wp-content/uploads/2020/03/Logo.jpg"
with open('image.png', 'wb') as f:
f.write(requests.get(url).content)
try this way of doing it
import requests
img_url = "http://www.casperdenhaan.nl/wp-content/uploads/2020/03/Logo.jpg"
img = requests.get(img_url)
with open('image.png', 'wb') as save_file:
save_file.write(img.raw)
This way you don't have to deal with closing the file. Also, the 'wb' opens the file in a writable binary mode.
Related
url = 'https://cdn.wsform.com/wp-content/uploads/2021/05/currency.csv'
f = open(url, 'r')
print(f.read())
Invalid argument: 'https://cdn.wsform.com/wp-content/uploads/2021/05/currency.csv'
Is there any way so that i can read file from url, instead of downloading it ??
Yes, the standard urllib.request.urlopen() does exactly that:
from urllib import request
with request.urlopen('https://cdn.wsform.com/wp-content/uploads/2021/05/currency.csv') as f:
print(f.read().decode()) # the decode turns the bytes into a string for printing
Just note that the file is opened as a file of bytes, since the function can't know if it's a text file or not.
I have this code for server
#app.route('/get', methods=['GET'])
def get():
return send_file("token.jpg", attachment_filename=("token.jpg"), mimetype='image/jpg')
and this code for getting response
r = requests.get(url + '/get')
And i need to save file from response to hard drive. But i cant use r.files. What i need to do in these situation?
Assuming the get request is valid. You can use use Python's built in function open, to open a file in binary mode and write the returned content to disk. Example below.
file_content = requests.get('http://yoururl/get')
save_file = open("sample_image.png", "wb")
save_file.write(file_content.content)
save_file.close()
As you can see, to write the image to disk, we use open, and write the returned content to 'sample_image.png'. Since your server-side code seems to be returning only one file, the example above should work for you.
You can set the stream parameter and extract the filename from the HTTP headers. Then the raw data from the undecoded body can be read and saved chunk by chunk.
import os
import re
import requests
resp = requests.get('http://127.0.0.1:5000/get', stream=True)
name = re.findall('filename=(.+)', resp.headers['Content-Disposition'])[0]
dest = os.path.join(os.path.expanduser('~'), name)
with open(dest, 'wb') as fp:
while True:
chunk = resp.raw.read(1024)
if not chunk: break
fp.write(chunk)
I am trying to download file from GitHub(raw file) and then run this file as .sql file.
import snowflake.connector
from codecs import open
import logging
import requests
from os import getcwd
import os
import sys
#logging
logging.basicConfig(
filename='C:/Users/abc/Documents/Test.log',
level=logging.INFO
)
url = "https://github.com/raw/abc/master/file_name?token=Anvn3lJXDks5ciVaPwA%3D%3D"
directory = getcwd()
filename = os.path.join(getcwd(),'VIEWS.SQL')
r = requests.get(url)
filename.decode("utf-8")
f = open(filename,'w')
f.write(str(r.content))
with open(filename,'r') as theFile, open(filename,'w') as outFile:
data = theFile.read().split('\n')
data = theFile.read().replace('\n','')
data = theFile.read().replace("b'","")
data = theFile.read()
outFile.write(data)
However I get this error
syntax error line 1 at position 0 unexpected 'b'
My converted sql file has b at the beginning and bunch of newline \n characters in the file. Also the entire output file is in single quotes 'text'. Can anyone help me get rid of these? Looks like replace isn't working.
OS: Windows
Python Version: 3.7.0
You introduced a b'.. prefix by converting the response.content bytes value to a string with str():
>>> import requests
>>> r = requests.get("https://github.com/raw/abc/master/file_name?token=Anvn3lJXDks5ciVaPwA%3D%3D")
>>> r.content
b'Not Found'
>>> str(r.content)
"b'Not Found'"
Of course, the specific dummy URL you gave in your question produces a 404 Not Found response, hence the Not Found content of the response body:
>>> r.status_code
404
so the contents in this demonstration are not actually all that useful. However, even for your real URL you probably want to test for a 200 status code before moving to write the data to a file!
What is going wrong in the above is that str(bytesvalue) converts a bytes object to its representation. You'd normally want to decode a bytes value with a text codec, using the bytes.decode() method. But because you are writing the data to a file here, you should instead just open the file in binary mode and write the bytes object without decoding:
r = requests.get(url)
if r.status_code == 200:
with open(filename, 'wb') as f:
f.write(r.content)
The 'wb' mode opens the file for writing in binary mode. Writing binary content to a binary file is the most efficient; decoding it first then writing to a text file requires that it is encoded again. Better to avoid doing double work.
As a side note: there is no need to join a local filename with getcwd(); relative paths always end up in the current working directory, and otherwise it's better to use os.path.abspath(filename).
You could also trust that GitHub sets the correct character set in the Content-Type headers and have response decode the value to str for you in the form of the response.text attribute:
r = requests.get(url)
if r.status_code == 200:
with open(filename, 'w') as f:
f.write(r.text)
but again, that's really doing extra work for nothing, first decoding the binary content from the request, then encoding again when writing to a text file.
Finally, for larger file responses it is better to stream the data and copy it directly to a file. The shutil.copyfileobj() function can take a raw response fileobject directly, provided you enable transparent transport decompression:
import shutil
r = requests.get(url, stream=True)
if r.status_code == 200:
with open(filename, 'wb') as f:
# enable transparent transport decompression handling
r.raw.decode_content = True
shutil.copyfileobj(r.raw, f)
Depending on your version of Python/OS it could be as simple as changing the file to read/write in binary (and if they're still there then altering where you have the replaces):
with open(filename,'rb') as theFile, open(filename,'wb') as outFile:
outfile.write(str(r.content))
data = theFile.read().split('\n')
data = data.replace('\n','')
data = data.replace("b'","")
outFile.write(data)
It would help to have a copy of the file and the line the error is occurring on.
With Python 3, I want to read an XML web page and save it in my local drive.
Also, if the file already exist, it must overwrite it.
I tested some script like :
import urllib.request
xml = urllib.request.urlopen('URL')
data = xml.read()
file = open("file.xml","wb")
file.writelines(data)
file.close()
But I have an error :
TypeError: a bytes-like object is required, not 'int'
First suggestion: do what even the official urllib docs says and don't use urllib, use requests instead.
Your problem is that you use .writelines() and it expects a list of lines, not a bytes objects (for once in Python the error message is not very helpful). Use .write() instead
import requests
resp = requests.get('URL')
with open('file.xml', 'wb') as foutput:
foutput.write(resp.content)
I found a solution :
from urllib.request import urlopen
xml = open("import.xml", "r+")
xml.write(urlopen('URL').read().decode('utf-8'))
xml.close()
Thanks for your help.
i have a function in my views.py. It is like this
from django.core.files.uploadedfile import SimpleUploadedFile
def get_file(self, url):
# pdb.set_trace()
result = urllib.urlretrieve(url)
fi = open(result[0])
fi_name = os.path.basename(url)
suf = SimpleUploadedFile(fi_name, fi)
return suf
while creating SimpleUploadedFile object i get the error saying
TypeError: file doesnot have buffer interface
I tried changing open mode to 'rb'. But still get the same error
Plz help me out
SimpleUploadedFile needs the actual file content, rather than a file object. So you could fix your code like this:
suf = SimpleUploadedFile(fi_name, fi.read())
I must say though I don't know why you are using urlretrieve, which saves the content to a local temp file which you then must open and read. Better to use urlopen and pass it directly:
result = urllib.urlopen(url)
fi_name = os.path.basename(url)
suf = SimpleUploadedFile(fi_name, result.read())
Had this problem on the dreaded El Capitan, in the requests lib.
It seems that passing unicode as HTTP content breaks stuff when converted to memoryview() on the socket layer.
Just passing everything as plain strings fixed it for me.