We Keep Coding

Using Python library pyodata to access data in Odata

So, I am trying to use the pyodata library in Python to access and download data from Odata.
I tried accessing the Northwind data and it worked. So, i guess the codes i used is ok.
import requests
import pyodata
url_t = 'http://services.odata.org/V2/Northwind/Northwind.svc'
# connection set up
northwind = pyodata.Client(url_t, requests.Session())
# This prints out a single value from the table Customers
for customer in northwind.entity_sets.Customers.get_entities().execute():
print(customer.CustomerID,",", customer.CompanyName)
break
# This will print out - ALFKI , Alfreds Futterkiste
I also tried connecting to Odata in excel to see if the codes above return the correct data, and it did.
Click to see the screenshot in excel for Odata connection
Now, using the same code to connect to the data source where I want to pull the data did not work:
#using this link to connect to Odata worked.
url_1 = 'https://batch.decisionkey.npd.com/odata/dkusers'
session = requests.Session()
session.auth = (user_name, psw)
theservice = pyodata.Client(url_1, session)
The above codes return this error message(is it something about security?):
Click to see error message
Connecting to the data in excel looks like this:
Click the view image
I am thinking about it might be security issue that is blocking me from accessing the data, or it could be something else. Please let me know if anything need to be clarify. Thanks.
First time asking question, so please let me know if anything I did not do right here. ^_^

You got HTTP 404 - Not Found.
The service "https://batch.decisionkey.npd.com/odata/dkusers" is not accessible from outside world for me to try it, so there is something more from networking point of view that happens in the second picture in the Excel import.
You can forget the pyodata at the moment, for your problem it is just wrapper around HTTP networking layer, the Requests library. You need to find a way initialize the Requests session in a way, that will return HTTP 200 OK instead.
Northwind example service is just plain and simple, so no problem during initialization of pyodata.Client
Refer to Requests library documentation- https://docs.python-requests.org/en/latest/user/advanced/
//sample script
url_1 = 'https://batch.decisionkey.npd.com/odata/dkusers'
session = requests.Session()
session.auth = (user_name, psw)
//??? SSL certificate needs to be provided perhaps?
//?? or maybe you are behind some proxy that Excel uses but python not.. try ping in CMD
response = session.get(url_1)
print(response.text)
Usable can be pyodata documentation about initialization, however you will not find there the reason why you get HTTP 404 - https://pyodata.readthedocs.io/en/latest/usage/initialization.html

Delete cookies / end session using cookiejar

I have written a python script fetching a State of Health web page from some hardware we are using. Looking into the HTML and javascript files, it was relatively easy to fetch what I wanted. The problem is that I am not able to end the session and delete the cookies so that the web interface is usable for other users before my session times out. I have no possibility to change anything on the server side.
What I am doing is basically:
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
url=mydataurl
headers={"Authorization:"username:encrytptedpassword","Cookie":"user=username; password="encryptedpassword"}
# Both Authorization and Cookie need to be set to go further
data="something"
req=urllib2.Request(url,data,headers)
connection=urllib2.urlopen(req)
response=connection.read()
# Now I have what I want in response and can work on that.
# But the server thinks I am still active and does not let anybody else in
# So I call what is called when I press logout on the web page:
url=logouturl
headers={}
data=""
req=urllib2.Request(url,None,headers)
connection=urllib2.urlopen(req)
logoutresponse=connection.read()
#and just in case
headers={}
cj.clear()
cj.clear_session_cookies()
url="http://myserver/index.htm"
req=urllib2.Request(url,None,headers)
connection=urllib2.urlopen(req)
logoutresponse=connection.read()
connection.close()
Am I doing something wrong to get rid of the cookies in this session? I have also tried to close all the three connections I have started but to no avail.
I can open the web page in a browser on the computer I am running the script on, then log out and immediately after opening it on another computer. If I run the script I have to wait some minutes to something times out on the server before being able to log on again.
It might, of course, be that the server is doing something else as well to keep the session alive, if so I may be out of luck.
I prefer to use built-in libraries and it is not possible for me at the moment to upgrade to use python 3.

A common way to "delete" cookies is to simply set the expiration date of the cookie to a time in the past (many systems can just set it to time=0 which should work). I am not familiar with cookiejar, but you might want to look into this method.

Python 3.6 Requests too long

I am trying to use requests to pull information from the NPI API but it is taking on average over 20 seconds to pull the information. If I try and access it via my web browser it takes less than a second. I'm rather new to this and any help would be greatly appreciated. Here is my code.
import json
import sys
import requests
url = "https://npiregistry.cms.hhs.gov/api/?number=&enumeration_type=&taxonomy_description=&first_name=&last_name=&organization_name=&address_purpose=&city=&state=&postal_code=10017&country_code=&limit=&skip="
htmlfile=requests.get(url)
data = htmlfile.json()
for i in data["results"]:
print(i)

This might be due to the response being incorrectly formatted, or due to requests taking longer than necessary to set up the request. To solve these issues, read on:
Server response formatted incorrectly
A possible issue might be that the response parsing is actually the offending line. You can check this by not reading the response you receive from the server. If the code is still slow, this is not your problem, but if this fixed it, the problem might lie with parsing the response.
In case some headers are set incorrectly, this can lead to parsing errors which prevents chunked transfer (source).
In other cases, setting the encoding manually might resolve parsing problems (source).
To fix those, try:
r = requests.get(url)
r.raw.chunked = True # Fix issue 1
r.encoding = 'utf-8' # Fix issue 2
print(response.text)
Setting up the request takes long
This is mainly applicable if you're sending multiple requests in a row. To prevent requests having to set up the connection each time, you can utilize a requests.Session. This makes sure the connection to the server stays open and configured and also persists cookies as a nice benefit. Try this (source):
import requests
session = requests.Session()
for _ in range(10):
session.get(url)
Didn't solve your issue?
If that did not solve your issue, I have collected some other possible solutions here.

Session variables like PHP in Python

i'm making a web application with python and I want to save some variables for the session, till the browser closes, like I would do with PHP:
<?php
session_start();
$_SESSION['size']='small';
?>
what's an easy yet safe way?
I'm using both lighttpd and apache so I want something that'll work with both.
also i there will be passwords saved so i need something safe.

When using session_start() in PHP, you are not using "pure code" either, it's also smoke and mirrors...
Leaving out all the caveats: What you can do is using a global dictionary to store session data. Once a client makes a request and passes the "session"-cookie, you look up all the session data in that dictionary. If there is no entry or the client has no session-cookie, you create a new session and pass the cookie to the client.
The session-cookie is made of a random, say sixteen character, string. Other clients are unable to guess another user's session because the keyspace is too large. From time to time, you prune the dictionary from session your server has not seen in a while.
You should really take a look at CherryPy's documentation on using sessions though.

I decided to do it with cookies, which is easier/safer. Here's the code for everyone interested:
# importing the libs
from http import cookies
import os
# setting the cookies
C = cookies.SimpleCookie()
C["cookie1"] = "some_text"
C["cookie2"] = "another_text"
print(C.output())
# sending the html header
print('Content-type: text/html;\n')
# reading the "cookie1" cookie
cookievalue = cookies.SimpleCookie(os.environ["HTTP_COOKIE"])
print (cookievalue["cookie1"].value)

How do I do JSONP with python on my webspace..?

I just checked my webspace and it's signature says: Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g
This give me hope that Python is somehow supported. Why is python listed twice? mod_python/3.3.1 AND Python/2.5.2 ???
There is a cgi-bin folder on my webspace.
What I want to do: I need to do a cross-site call to get some text-data from a server. The text-data is not JSON but I guess I should convert it to JSON (or is there an option to do cross-site without JSON?)
The python script gets the request for some JSONP. Depending on the request (I guess I should somehow parse the URL) the python script is to load the a requested text-data file from the webserver and wrap it in some JSON and return it.
Can somebody tell me how I do these three steps with python on my webspace?

First off, the signature isn't listing python twice. Its listing first the version of mod_python, which is an Apache web server plugin, then it is listing the version of the python interpreter on the system.
python cgi module - This is really an inefficient approach to writing python server code, but here it is. Ultimately you should consider one of the many amazing python web frameworks out there. But, using the cgi module, your response would always start with this:
print 'Content-Type: application/json\n\n'
Your python script would run on the server from an HTTP request. In that script you would check the request and determine the data you will want to serve from either the URL value or the query string.
At the very least you would just wrap your return value in a basic JSON data structure. The text data itself can just be a string:
import json
text_data = "FOO"
json_data = json.dumps({'text': text_data})
print json_data
# {"text": "FOO"}
For the JSONP aspect, you would usually check the query string to see if the request contains a specific name for the callback function the client wants, or just default to 'callback'
print "callback(%s);" % json_data
# callback({"text": "FOO"});
Returning that would be a JSONP type response, because when the client receives it, the callback is executed for the client.
And to conclude, let me add that you should be aware that python cgi scripts will need to start a brand new python interpreter process for every single request (even repeat requests from the same client). This can easily overwhelm a server under increased load. For this reason, people usually go with the wsgi route (mod_wsgi in apache). wsgi allows a persistant application to keep running, and handles ongoing requests.