I'm currently working on a automated way to interface with a database website that has RESTful webservices installed. I am having issues with figure out the proper formatting of how to properly send the requests listed in the following site using python.
https://neesws.neeshub.org:9443/nees.html
Particular example is this:
POST https://neesws.neeshub.org:9443/REST/Project/731/Experiment/1706/Organization
<Organization id="167"/>
The biggest problem is that I do not know where to put the XML formatted part of the above. I want to send the above as a python HTTPS request and so far I've been trying something of the following structure.
>>>import httplib
>>>conn = httplib.HTTPSConnection("neesws.neeshub.org:9443")
>>>conn.request("POST", "/REST/Project/731/Experiment/1706/Organization")
>>>conn.send('<Organization id="167"/>')
But this appears to be completely wrong. I've never actually done python when it comes to webservices interfaces so my primary question is how exactly am I supposed to use httplib to send the POST Request, particularly the XML formatted part of it? Any help is appreciated.
You need to set some request headers before sending data. For example, content-type to 'text/xml'. Checkout the few examples,
Post-XML-Python-1
Which has this code as example:
import sys, httplib
HOST = www.example.com
API_URL = /your/api/url
def do_request(xml_location):
"""HTTP XML Post requeste"""
request = open(xml_location,"r").read()
webservice = httplib.HTTP(HOST)
webservice.putrequest("POST", API_URL)
webservice.putheader("Host", HOST)
webservice.putheader("User-Agent","Python post")
webservice.putheader("Content-type", "text/xml; charset=\"UTF-8\"")
webservice.putheader("Content-length", "%d" % len(request))
webservice.endheaders()
webservice.send(request)
statuscode, statusmessage, header = webservice.getreply()
result = webservice.getfile().read()
print statuscode, statusmessage, header
print result
do_request("myfile.xml")
Post-XML-Python-2
You may get some idea.
Related
I have to send a POST request to the /batch endpoint of : 'https://www.google-analytics.com'.
As mentioned in the Documentation I have to send the request to /batch endpoint and specify each payload on its own line.
I was able to achieve this using POSTMAN as follows:
My query is to make a POST request using Python's requests library
I tried something like this :
import requests
text = '''v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=65
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=15
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=95'''
response = requests.post('https://www.google-analytics.com/batch', data=text)
but it doesn't works.
UPDATE
I Tried this and it works !
import http.client
conn = http.client.HTTPSConnection("www.google-analytics.com")
payload = "v=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=63\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=11\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=23"
headers = {
'Content-Type': 'text/plain'
}
conn.request("POST", "/batch", payload, headers)
res = conn.getresponse()
But the question remains open, what's the issue with requests here.
You don't need to double-escape the newline symbol.
Moreover, you don't need the newline symbol at all for the multi-line string.
And also the indentations you put in your multi-line string are counted:
test = '''abc
def
ghi'''
print(test)
Here's an SO answer that explains this with some additional ways to make long stings: https://stackoverflow.com/a/10660443/4570170
Now the request body.
The documentation says
payload_data – The BODY of the post request. The body must include exactly 1 URI encoded payload and must be no longer than 8192 bytes.
So try uri-encoding your payload:
text = '''v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=65
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=15
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=95'''
text_final = requests.utils.quote(text)
response = requests.post('https://www.google-analytics.com/batch', data=text_final)
Finally , I figured out the solution myself.
Updating for others help.
The problem was I was working on AWS Cloud9 and as mentioned in the documentation
Some environments are not able to send hits to Google Analytics directly. Examples of this are older mobile phones that can't run JavaScript or corporate intranets behind a firewall.
So we just need to include the User Agent parameter
ua=Opera/9.80
in each of our payloads
It works !
I need to connect/send msg to http://localhost:8001/path/to/my/service, but I am not able to find how to do that. I know how to send if I only have localhost and 8001, but I need this specific path /path/to/my/service. There is where my service is running.
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(<full-url-to-my-service>)
s.sendall(bytes('Message', 'utf-8'))
Update
My service is running on localhost:8001/api/v1/namespaces/my_namespace/services/my_service:http/proxy. How can I connect to it with python?
As #furas told in the comments
socket is primitive object and it doesn't have specialized method for this - and you have to on your own create message with correct data. You have to learn HTTP protocol and use it to send
This is a sample snippet to send a GET request in python using requests library
import requests
URL = 'http://localhost:8001/path/to/my/service'
response_text = requests.get(URL).text
print(response_text)
This assumes the Content-Type that GET URL produces is text. If it is json, then a minor change is required
import requests
URL = 'http://localhost:8001/path/to/my/service'
response_json = requests.get(URL).json()
print(response_json)
There are other ways to achieve the same using other good frameworks like urllib, and so on.
Here is the documentation of requests library for reference
sendall() requires bytes, so String must be encoded.
s.sendall("foobar".encode())
Scope:
I am currently trying to write a Web scraper for this specific page. I have a pretty strong "Web Crawling" background using C#, but this httplib is beating me off.
Problem:
When trying to make a Http Get request for the page specified above I get a "Moved Permanently", that points to the very same URL. I can make a request using the requests lib, but I want to make it work using httplib so I can understand what I am doing wrong.
Code Sample:
I am completely new to Python, so any wrong language guideline or syntax is C#'s fault.
import httplib
# Wrapper for a "HTTP GET" Request
class HttpClient(object):
def HttpGet(self, url, host):
connection = httplib.HTTPConnection(host)
connection.request('GET', url)
return connection.getresponse().read()
# Using "HttpClient" class
httpclient = httpClient()
# This is the full URL I need to make a get request for : https://420101.com/strain-database
httpResponseText = httpclient.HttpGet('www.420101.com','/strain-database')
print httpResponseText
I really want to make it work using the httplib library, instead of requests or any other fancy one because I feel like I am missing something really small here.
The problem i've had too little or too much caffeine in my system.
To get a https, I needed the HTTPSConnection class.
Also, there is no 'www' in the address I wanted to GET. So, it shouldn't be included in the host.
Both of the wrong addresses redirect me to the correct one, with the 301 error code. If I were using requests or a more full featured module, it would have automatically followed the redirect.
My Validation:
c = httplib.HTTPSConnection('420101.com')
c.request("GET", "/strain-database")
r = c.getresponse()
print r.status, r.reason
200 OK
I am trying to add a lead to a Zoho CRM module with Python. I keep getting:
< response>< error>< code>4600< /code>< message>Unable to process your request. Please verify if the name and value is appropriate for the "xmlData" parameter.< /message>< /error>< /response>
from the server. I have no idea if I am posting correctly or if it is a problem with our Xml Data. I am using urllib and urllib2 to format the post request.
The post request looks like this.
url = ("https://crm.zoho.com/crm/private/xml/Leads/insertRecords?authtoken="
""+str(self.authToken)+"&scope=crmapi")
params = {"xmlData":self.xml}
data = urllib.urlencode(params)
request = urllib2.Request(url = url, data =data)
request.add_header("Content-Type",'application/xml')
response = urllib2.urlopen(request)
You cannot combine HTTP GET query parameters (ones in URL) and HTTP POST parameters.
This is limitation on the HTTP protocol level, not in Python or Zoho.
Most likely you are doing it wrong. Revisit Zoho documentation how it should be done.
Here is another old library doing Zoho + CRM, written in Python. You might want to check it for inspiration: https://github.com/miohtama/mfabrik.zoho
Is it possible to use Python's requests library to send a SOAP request?
It is indeed possible.
Here is an example calling the Weather SOAP Service using plain requests lib:
import requests
url="http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL"
#headers = {'content-type': 'application/soap+xml'}
headers = {'content-type': 'text/xml'}
body = """<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns0="http://ws.cdyne.com/WeatherWS/" xmlns:ns1="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<ns1:Body><ns0:GetWeatherInformation/></ns1:Body>
</SOAP-ENV:Envelope>"""
response = requests.post(url,data=body,headers=headers)
print response.content
Some notes:
The headers are important. Most SOAP requests will not work without the correct headers. application/soap+xml is probably the more correct header to use (but the weatherservice prefers text/xml
This will return the response as a string of xml - you would then need to parse that xml.
For simplicity I have included the request as plain text. But best practise would be to store this as a template, then you can load it using jinja2 (for example) - and also pass in variables.
For example:
from jinja2 import Environment, PackageLoader
env = Environment(loader=PackageLoader('myapp', 'templates'))
template = env.get_template('soaprequests/WeatherSericeRequest.xml')
body = template.render()
Some people have mentioned the suds library. Suds is probably the more correct way to be interacting with SOAP, but I often find that it panics a little when you have WDSLs that are badly formed (which, TBH, is more likely than not when you're dealing with an institution that still uses SOAP ;) ).
You can do the above with suds like so:
from suds.client import Client
url="http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL"
client = Client(url)
print client ## shows the details of this service
result = client.service.GetWeatherInformation()
print result
Note: when using suds, you will almost always end up needing to use the doctor!
Finally, a little bonus for debugging SOAP; TCPdump is your friend. On Mac, you can run TCPdump like so:
sudo tcpdump -As 0
This can be helpful for inspecting the requests that actually go over the wire.
The above two code snippets are also available as gists:
SOAP Request with requests
SOAP Request with suds
Adding up to the last answer, make sure you add to the headers the following attributes:
headers={"Authorization": f"bearer {token}", "SOAPAction": "http://..."}
The authorization is meant when you need some token to access the SOAP API,
Otherwise, the SOAPAction is the action you are going to perform with the data you are sending in,
So if you don't need Authorization, then you could pop that out of the headers,
That worked pretty fine for me,