I have a XML-file that I need to send to some URL. I do that this way:
data = { 'file' : open('test.xml', 'rb') }
req = requests.post(URL, files=data)
This works, but problem is that first I need to generate XML, then I need to do this:
et = etree.ElementTree(root)
et.write('test.xml', encoding='utf8')
and then after that I do this:
data = { 'file' : open('test.xml', 'rb') }
req = requests.post(URL, files=data)
But I don't like this, I have XML-file, then I write it to disk, just to read it again from disk.
Is there a way to write that XML directly to file object (equivalent of open('test.xml', 'rb')) without writing it to file first?
Try using tostring
Ex:
et = etree.ElementTree(root)
req = requests.post(URL, data=etree.tostring(et.getroot()))
Related
This work fine but I want to upload multiple file (metadata) to ipfs under same CID using python.
import requests
import json
import os
import csv
header = ['image', 'IPFS']
images = os.listdir("./images/")
with open('ipfs.csv', 'w', encoding='UTF8') as f:
writer = csv.writer(f)
# write the header
writer.writerow(header)
for image in images:
# write the data
data=[]
name=image.replace(".png","").replace(".jpg","")
data.append(name)
url = "https://api.pinata.cloud/pinning/pinFileToIPFS"
payload={}
files=[
('file',('file',open("./images/"+image,'rb'),'application/octet-stream'))
]
headers = {
'pinata_api_key': 'APIKEY',
'pinata_secret_api_key': 'SECRETAPIKEY'
}
response = requests.request("POST", url, headers=headers, data=payload, files=files)
info=json.loads(response.text)
data.append("ipfs://"+info['IpfsHash'])
writer.writerow(data)
And if I get solution by using another api it fine.
And one more thing I'm running this code on Android pydroid3
You cant use the same cid for different files. Cid's hash the file and return a cid/the hash.
I am doing this for the first time and so far have setup a simple script to fetch 2 columns of data from an APIThe data comes through and I can see it with print commandNow I am trying to write it to CSV and setup the code below which creates the file but I can't figure out how to:1. Remove the blank lines in between each data row2. Add delimiters to the data which I want to be " "3. If a value such as IP is blank then just show " "I searched and tried all sorts of examples but just getting errorsMy code snippet which writes the CSV successfully is
import requests
import csv
import json
# Make an API call and store response
url = 'https://api-url-goes-here.com'
filename = "test.csv"
headers = {
'accept': 'application/json',
}
r = requests.get(url, headers=headers, auth=('User','PWD'))
print(f"Status code: {r.status_code}")
#Store API response in a variable
response_dict = r.json()
#Open a File for Writing
f = csv.writer(open(filename, "w", encoding='utf8'))
# Write CSV Header
f.writerow(["Computer_Name", "IP_Addresses"])
for computer in response_dict["advanced_computer_search"]["computers"]:
f.writerow([computer["Computer_Name"],computer["IP_Addresses"]])
CSV output I get looks like this:
Computer_Name,IP_Addresses
HYDM002543514,
HYDM002543513,10.93.96.144 - AirPort - en1
HYDM002544581,192.168.1.8 - AirPort - en1 / 10.93.224.177 -
GlobalProtect - gpd0
HYDM002544580,10.93.80.101 - Ethernet - en0
HYDM002543515,192.168.0.6 - AirPort - en0 / 10.91.224.58 -
GlobalProtect - gpd0
CHAM002369458,10.209.5.3 - Ethernet - en0
CHAM002370188,192.168.0.148 - AirPort - en0 / 10.125.91.23 -
GlobalProtect - gpd0
MacBook-Pro,
I tried adding
csv.writer(f, delimiter =' ',quotechar =',',quoting=csv.QUOTE_MINIMAL)
after the f = csv.writer line but that creates an error:TypeError: argument 1 must have a "write" method
I am sure its something simple but just can't find the correct solution to implement in the code I have. Any help is appreciated.
Also, does the file get closed automatically? Some examples suggest to use something like f.close() but that causes errors. Do I need it? The file seems to get created fine as-is.
I suggest you use pandas package to write .csv file, which is a most used package for data analysis.
For your problem:
import requests
import csv
import json
import pandas
# Make an API call and store response
url = 'https://api-url-goes-here.com'
filename = "test.csv"
headers = {
'accept': 'application/json',
}
r = requests.get(url, headers=headers, auth=('User','PWD'))
print(f"Status code: {r.status_code}")
#Store API response in a variable
response_dict = r.json()
#collect data to build pandas.DataFrame
data = []
for computer in response_dict["advanced_computer_search"]["computers"]:
# filter blank line
if computer["Computer_Name"] or computer["IP_Addresses"]:
data.append({"Computer_Name":computer["Computer_Name"],"IP_Addresses":computer["IP_Addresses"]})
pandas.DataFrame(data=data).to_csv(filename, index=False)
if you want use " " to separate value, you can set sep=" " in the last line output the .csv file. However, I recommend to use , as delimiters due to it's a common standard. Also much more configs could be set for DataFrame.to_csv() method, you can check the official docs. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html
As you said in comment, pandas is not a standard python package. You can simply open a file and write lines to that file, with the lines you build manually. For example:
import requests
import csv
import json
# Make an API call and store response
url = 'https://api-url-goes-here.com'
filename = "test.csv"
headers = {
'accept': 'application/json',
}
r = requests.get(url, headers=headers, auth=('User','PWD'))
print(f"Status code: {r.status_code}")
#Store API response in a variable
response_dict = r.json()
r = requests.get(url, headers=headers, auth=('User','PWD'))
print(f"Status code: {r.status_code}")
#Store API response in a variable
response_dict = r.json()
#Open a File for Writing
f = csv.writer(open(filename, "w", encoding='utf8'))
with open(filename, mode='w') as f:
# Write CSV Header
f.write("Computer_Name,"+"IP_Addresses"+"\n")
for computer in response_dict["advanced_computer_search"]["computers"]:
# filter blank line
if computer["Computer_Name"] or computer["IP_Addresses"]:
f.write("\""+computer["Computer_Name"]+"\","+"\""+computer["IP_Addresses"]+"\"\n")
Note that " around value was build by appending \". \n to change new line after each loop.
I would like to POST a .tgz file with the Python urllib2 library to a backend server. I can't use requests due to some licensing issues. There are some examples of file upload on stackoverflow but all relate to attaching a file in a form.
My code is the following but it unfortunately fails:
stats["random"] = "data"
statsFile = "mydata.json"
headersFile = "header-data.txt"
tarFile = "body.tgz"
headers = {}
#Some custom headers
headers["X-confidential"] = "Confidential"
headers["X-version"] = "2"
headers["Content-Type"] = "application/x-gtar"
#Create the json and txt files
with open(statsFile, 'w') as a, open(headersFile, 'w') as b:
json.dump(stats, a, indent=4)
for k,v in headers.items():
b.write(k+":"+v+"\n")
#Create a compressed file to send
tar = tarfile.open(tarFile, 'w:gz' )
for name in [statsFile,headersFile]:
tar.add(name)
tar.close()
#Read the binary data from the file
with open(tarFile, 'rb') as f:
content = f.read()
url = "http://www.myurl.com"
req = urllib2.Request(url, data=content, headers=headers)
response = urllib2.urlopen(req, timeout=timeout)
If I use requests, it works like a charm:
r = requests.post(url, files={tarFile: open(tarFile, 'rb')}, headers=headers)
I essentially need the equivalent of the above for urllib2. Does anybody maybe know it? I have checked the docs as well but I was not able to make it work..What am I missing?
Thanks!
I am using the pinnacle (betting) api which returns an XML file. At the moment, I save it to a .xml file as below:
req = urllib2.Request(url, headers=headers)
responseData = urllib2.urlopen(req).read()
ofn = 'pinnacle_feed_basketball.xml'
with open(ofn, 'w') as ofile:
ofile.write(responseData)
parse_xml()
and then open it in the parse_xml function
tree = etree.parse("pinnacle_feed_basketball.xml")
fdtime = tree.xpath('//rsp/fd/fdTime/text()')
I am presuming saving it as an XML file and then reading in the file is not necessary but I cannot get it to work without doing this.
I tried passing in responseData to the parsexml() function
parse_xml(responseData)
and then in the function
tree = etree.parse(responseData)
fdtime = tree.xpath('//rsp/fd/fdTime/text()')
But it doesn't work.
If you want to parse an in-memory object (in your case, a string), use etree.fromstring(<obj>) -- etree.parse expects a file-like object or filename -- Docs
For example:
import urllib2, lxml.etree as etree
url = 'http://www.xmlfiles.com/examples/note.xml'
headers = {}
req = urllib2.Request(url, headers=headers)
responseData = urllib2.urlopen(req).read()
element = etree.fromstring(responseData)
print(element)
print(etree.tostring(element, pretty_print=True))
Output:
<Element note at 0x2c29dc8>
<note>
<to>Tove</to>
<from>Jani</from>
<heading>Reminder</heading>
<body>Don't forget me this weekend!</body>
</note>
parse() is designed to read from file-like objects.
But you are passing a string in both cases - pinnacle_feed_basketball.xml string and responseData, which is also a string.
In the first case it should be:
with open("pinnacle_feed_basketball.xml") as f:
tree = etree.parse(f)
In the second case:
root = etree.fromstring(responseData) # note that you are not getting an "ElementTree" object here
FYI, urllib2.urlopen(req) is also a file-like object:
tree = etree.parse(urllib2.urlopen(req))
I'm using python to download a html file and store in a file.
Here's the code:
url = "http://www.nytimes.com/roomfordebate/2014/09/24/protecting-student-privacy-in-online-learning"
page = requests.get(url)
# save html content
file_name = url.split('/')[-1]
text_file = open(file_name, 'w+')
text_file.write(page.text())
text_file.close()
i got the following error:
File "scraper.py", line 15, in scrape_Page
text_file.write(page.text())
TypeError: 'unicode' object is not callable
Could anyone tell how could I successfully store the text or why I got this error?
Thanks
request.text is an attribute, not a method. You should not call it. You should not be using it to download a file, either, you should be using .content instead; you want the undecoded bytes, not the decoded Unicode value:
text_file.write(page.content)
To download content, you may want to stream it to the file instead:
import requests
import shutil
r = requests.get(url, stream=True)
file_name = url.rpartition('/')[-1]
with open(file_name, 'wb') as f:
r.raw.decode_content = True
shutil.copyfileobj(r.raw, f)