So, I am trying to use the pyodata library in Python to access and download data from Odata.
I tried accessing the Northwind data and it worked. So, i guess the codes i used is ok.
import requests
import pyodata
url_t = 'http://services.odata.org/V2/Northwind/Northwind.svc'
# connection set up
northwind = pyodata.Client(url_t, requests.Session())
# This prints out a single value from the table Customers
for customer in northwind.entity_sets.Customers.get_entities().execute():
print(customer.CustomerID,",", customer.CompanyName)
break
# This will print out - ALFKI , Alfreds Futterkiste
I also tried connecting to Odata in excel to see if the codes above return the correct data, and it did.
Click to see the screenshot in excel for Odata connection
Now, using the same code to connect to the data source where I want to pull the data did not work:
#using this link to connect to Odata worked.
url_1 = 'https://batch.decisionkey.npd.com/odata/dkusers'
session = requests.Session()
session.auth = (user_name, psw)
theservice = pyodata.Client(url_1, session)
The above codes return this error message(is it something about security?):
Click to see error message
Connecting to the data in excel looks like this:
Click the view image
I am thinking about it might be security issue that is blocking me from accessing the data, or it could be something else. Please let me know if anything need to be clarify. Thanks.
First time asking question, so please let me know if anything I did not do right here. ^_^
You got HTTP 404 - Not Found.
The service "https://batch.decisionkey.npd.com/odata/dkusers" is not accessible from outside world for me to try it, so there is something more from networking point of view that happens in the second picture in the Excel import.
You can forget the pyodata at the moment, for your problem it is just wrapper around HTTP networking layer, the Requests library. You need to find a way initialize the Requests session in a way, that will return HTTP 200 OK instead.
Northwind example service is just plain and simple, so no problem during initialization of pyodata.Client
Refer to Requests library documentation- https://docs.python-requests.org/en/latest/user/advanced/
//sample script
url_1 = 'https://batch.decisionkey.npd.com/odata/dkusers'
session = requests.Session()
session.auth = (user_name, psw)
//??? SSL certificate needs to be provided perhaps?
//?? or maybe you are behind some proxy that Excel uses but python not.. try ping in CMD
response = session.get(url_1)
print(response.text)
Usable can be pyodata documentation about initialization, however you will not find there the reason why you get HTTP 404 - https://pyodata.readthedocs.io/en/latest/usage/initialization.html
Related
have tried looking into other similar question and search the web, but I cannot seem to find the answer, so hope some clever people here can help or guide me.
I have a proc http request in SAS which runs fine on my local machine, no problems:
filename lst temp;
proc http
url = "http://xxx/api/job/"
method = "get"
out = lst;
run;
libname lst json fileref=lst automap=create;
Trying to do the same in Python gives me error code 401.
import requests
response = requests.get("http://xxx/api/job/")
print(response)
print(response.status_code)
This is an API from a system running internally in our organization. One needs to log on the first time when accessing through a web browser, but then it works.
I have tried all the different auth= etc. I could find in the documentation giving my user and password. But, nothing seems to work.
Some how, SAS proc http must be working since my profile/user is somehow verified, but via Python it is not - or at least that is what I am thinking.
Any suggestions?
I'm doing a post request using python and confluence REST API in order to update confluence pages via a script.
I ran into a problem which caused me to receive a 400 error in response to a
requests.put(url, data = jsonData, auth = (username, passwd), headers = {'Content-Type' : 'application/json'})
I spent some time on this to discover that the reason for it was me not supplying an incremented version when updating the content. I have managed to make my script work, but that is not the point of this question.
During my attempts to make this work, I swapped from requests to an http.client connection. Using this module, I get a lot more information regarding my error:
b'{"statusCode":400,"data":{"authorized":false,"valid":true,"allowedInReadOnlyMode":true,"errors":[],"successful":false},"message":"Must supply an incremented version when updating Content. No version supplied.","reason":"Bad Request"}'
Is there a way for me to get the same feedback information while using requests? I've turned on logging, but this kind of info is never shown.
You're looking for
requests.json()
It outputs everything the requests item returns, as a dictionary.
I am trying to record HTTP GET/POST requests sent by my browser using the library scotch.
I am using their sample code: http://darcs.idyll.org/~t/projects/scotch/doc/recipes.html#id2
import scotch.proxy
app = scotch.proxy.ProxyApp()
import scotch.recorder
recorder = scotch.recorder.Recorder(app, verbosity=1)
try:
from wsgiref.simple_server import WSGIServer, WSGIRequestHandler
server_address = ('', 8000)
httpd = WSGIServer(server_address, WSGIRequestHandler)
httpd.set_app(app)
while 1:
httpd.handle_request()
finally:
from cPickle import dump
outfp = open('recording.pickle', 'w')
dump(recorder.record_holder, outfp)
outfp.close()
print 'saved %d records' % (len(recorder.record_holder))
So I ran above code, went over to google chrome, and visited a few sites to see if that would get recorded.
However, I do not see how the code should terminate. It seems that there has to be an error in httpd.handle_request() for the code to terminate.
I tried a variation of the code where I removed the try and finally syntax, and changed the while condition so that the loop ran for 30 seconds. However, that seems to be running forever as well.
Any ideas on how to get this working? I am also open to using other python libraries available for what I am trying to do: record my browser's GET/POST requests, including logons, and replay this within python.
Thanks.
Correct me if I'm wrong, but you're trying to log the activity of your local browser by setting a local proxy. If this is the case your browser needs to go through your proxy in order for your proxy server to log the activity.
The code that you've provided sets a proxy server at localhost:8000, so you need to tell your browser about this. The actual setting depends on the browser, I'm sure you'd be able to google it easily.
When I've asked to check if the code is running I actually mean whether your local proxy accepts some kind of request from the browser. Do you see the 'saved records' print out of your code at some point?
I wonder how can I send cookie and phpssid with urllib2 in python?
Actually I want to read a page I've logged in with my browser, but when I try to read it with this script I encounter a text which seems to say that you've missed something.
My script :
#!/usr/bin/python
import urllib2
f = urllib2.urlopen('http://mywebsite.com/sub/create.php?id=20')
content = f.read()
file = open('file.txt', 'w')
file.write(content)
file.close()
The error message I save instead of the real page :
Warning: session_start() [function.session-start]: Cannot send session cookie - headers already sent by (output started at /home/number/domains/1number.com/public_html/s4/app/mywidgets.php:1) in /home/number/domains/1number.com/public_html/s4/app/mywidgets.php on line 23
Warning: session_start() [function.session-start]: Cannot send session cache limiter - headers already sent (output started at /home/number/domains/1number.com/public_html/s4/app/mywidgets.php:1) in /home/number/domains/1number.com/public_html/s4/app/mywidgets.php on line 23
Warning: Cannot modify header information - headers already sent by (output started at /home/number/domains/1number.com/public_html/s4/app/mywidgets.php:1) in /home/number/domains/1number.com/public_html/s4/lib/webservice.php on line 0
What is the exact problem?(Please give me a simple way to implement what I want)
Thanks in advance
For the SID, one of the ways to send that is as part of the query string, and you're already doing that. At least I assume that's what the id=20 part of your URL is.
For cookies, everything you want is in cookielib.
Just creating a CookieJar to use for a session with the server is trivial. If you want to import cookies from your browser, there are three possibilities:
If your browser uses the old Netscape cookie file format, you can use FileCookieJar.
If your browser uses a sqlite database (as at least Firefox and Safari/Chrome do), use the sqlite3 module to read it, and populate a CookieJar manually.
If worst comes to worst, copy and paste the cookies from your browser into your script as hardcoded strings and popular a CookieJar manually.
If you don't want to read the docs on how to use cookielib, just see the examples at the end, which show how to use a CookieJar with urllib2, which is exactly what you want to do.
If you have a problem, read the docs.
Meanwhile, what you're showing us are (a) warnings, not errors, and (b) obviously a problem on the server side, not your script. The server should never be spewing out a bunch of warnings and an otherwise-blank page. If you, or one of your coworkers, is responsible for the server code, that needs to be fixed first (and your current simple Python script can serve as a great regression test case).
I'm trying to make a successful connection to the LinkPoint gateway using Python. For those of you unfamiliar with their API you get a .pem file you use for authentication purposes.
I'm having trouble using this file and creating a secure connection over SSL.
According to their API documentation (which leaves a lot to be desired, btw) I believe the configuration should look similar to below:
HOST = 'secure.linkpt.net'
API_URL = 'https://secure.linkpt.net/lpc/servlet/lppay'
PORT = 1129
cert_key = my_cert_key.pem
Using this information and a valid XML string how can I create this connection?
I'm pretty new to HTTP connections in Python. I've successfully implemented connections with other APIs using a POST with urllib2. Naturally, my first attempt started with a similar approach hoping I could stumble on to a solution.
Something like:
headers = { 'User-Agent' : 'Rico',
'Content-type' : 'text/xml; charset=\"UTF-8\"',
'Content-length' : len(self.xml_string),
}
# POST to First Data (Link Point)
req = urllib2.Request(API_URL, self.xml_string, headers)
response = urllib2.urlopen(req)
self.handleResponse(response.read())
I had little hopes this would work as I didn't provide anything about the cert_key or the PORT.
After this attempt I tried to use a similar approach as I found from a solution from another stackoverflow post. Unfortunately I wasn't able to get far with this as I don't have ca_certs or cert files (that I know of).
I've tried to use Requests but can't find the documentation/examples for me to make sense of it.
I've also tried to use Twisted, and I really hoped I could do something with this but this feels like trying to open a door with a wrecking ball. It just feels like overkill to me. I just need a simple connection/request/response...this seems overly complicated for that.
My next attempt was going to be PycURL, but have confronted enough despair during this process I thought I'd come here to see if someone had some good suggestions before diving into this.
If you think I should re-visit one of these tools please let me know. I didn't spend a great deal of time with any of these - just enough to get my feet wet. If you could also point me to a good example or detailed documentation that would be fantastic.
Also, I'd prefer not to use the standard SSL library to build the connection myself - I don't want to reinvent the wheel if I don't have to.
The solution I was able to use to get a valid connection was using httplib as follows:
import httplib
HOST = 'staging.linkpt.net'
API_URL = '/lpc/servlet/lppay'
PORT = 1129
CERTFILE = 'my_cert_file.pem'
headers = { 'User-Agent' : 'Rico',
'Content-type' : 'text/xml; charset=\"UTF-8\"',
'Content-length' : len(xml_str),
}
conn = httplib.HTTPSConnection(HOST, PORT, cert_file = CERTFILE)
conn.putrequest("POST", API_URL)
conn.putheader(headers)
conn.endheaders()
conn.send(xml_str)
response = conn.getresponse()
I have yet been able to generate a valid request. Apparently I interpreted the API documentation incorrectly and keep getting a Malformed or unrecognized request. but at least I'm making the connection.
I'll update this answer if I'm able to determine more useful information regarding this subject.
UPDATE: A Link Point customer service employee told me I was using old API documentation. I've since tried with the newer version and still cannot connect. I can't even get a response from their server. This is no longer a possible solution to this problem.
UPDATE 2: I was able to solve this problem in another post SSL Connection Using .pem Certificate With Python
Enjoy!