I'm trying to read json from intranet site that's using windows authentication into pandas dataframe using read_json function but I'm getting 401 error.
A bit of googling showed that a similar issue with postman reading windows authenticated json was solved using Fiddler's "Automatically Authorize" function but it doesn't seem to work with pandas using anaconda.
import pandas as pd
df = pd.read_json(windows authenticated url)
btw the url works just fine it returns a perfectly formatted json in browser.
Thanks
Is the URL on your corporate intranet?
Do you normally enter it in the browser, then it pauses for a 10 .sec and you get the results without any password prompts?
If the above is true, it probably uses Kerberos authentication. You can certainly get it using python. Use here is the package that will help you with this https://github.com/requests/requests-kerberos
Note, some language environments maintain their own HTTP/Kerberos stack ( Java ) so you need to log in into Active Directory Domain there separately from your OS login.
I would suggest to do a HTTP GET request first using the requests package. The package provides a get() method that allows authentication as well as a json() method that returns the json-encoded content of a response. A code example could look like this:
import requests
r = requests.get('https://intranet.jsondata.com/xy.json', auth=('user', 'pass'))
json_content = r.json()
Related
I want to test my different env. like DEV,TEST,STAGE,PRODUCTION. The API call for the env. are different. For instance, http://dev.myclient.com, http://stage.myclient.com etc.
So, I want to write the test cases that will go to my specific URL make a search of specific thing and whatever the response come for example. I search for apples and I got 500 results related to that so, I want that result to print and save into text or Json. and same applies for the all different env.
then I will compare all the environments by one each other when I have a raw response data.
Any ideas how I can do that? In python specifically.
Thanks In advance!!
You could use the requests library for sending HTTP requests.This isn't a built-in library, so you should use pip install requests to install the library.
Here's an example:
import requests
url = "http://dev.myclient.com/"
response = requests.get(url).json()
print(response)
The console should now print the JSON response of the request you just sent. For this to work the URL provided should return a JSON string response.
My goal for this python code is to create a way to obtain job information into a folder. The first step is being unsuccessful. When running the code I want the url to print https://www.indeed.com/. However instead the code returns https://secure.indeed.com/account/login. I am open to using urlib or cookielib to resolve this ongoing issue.
import requests
import urllib
data = {
'action':'Login',
'__email':'email#gmail.com',
'__password':'password',
'remember':'1',
'hl':'en',
'continue':'/account/view?hl=en',
}
response = requests.get('https://secure.indeed.com/account/login',data=data)
print(response.url)
If you're trying to scrape information from indeed, you should use the selenium library for python.
https://pypi.python.org/pypi/selenium
You can then write your program within the context of a real user browsing the site normally.
I read the documentation of Facebook graph and trying to get the results from the oauth endpoint. The url I'm requesting is:
https://graph.facebook.com/oauth/access_token?client_id=<MY_CLIENT_ID>&client_secret=<MY_CLIENT_SECRET>&redirect_uri=<MY_REDIRECT_URL>&code=<CODE_RECEIVED_FROM_FB>
but I get a response like this:
access_token=CAAICzAsj0CgBAAQ7Go1k4NuG89mabLg6ZCpGoBoZCelRsLdQlcq1yvUbFZAKZBskTwMVmkTeZBZC9Thd4keYq0d3er3tGNTZCzR3TMnfEZABVfOpBqkOZBvZANZCny3XrDPDv7bTZB4ZAYjcPfvvMA4gRTcPrRJhht6XjIehV5gLtXW4YRvWaL4KuhYWB&expires=5169000
This happens from python requests library and also when I use curl from terminal or open in browser. Looks like it's returning response in form of get parameters. Why is it not returning JSON as documented? What's going on?
Ahh, my mistake, this is related to FB Graph API version. When I added the version as defined in the api, it worked fine.
https://graph.facebook.com/v2.3/oauth/access_token...
Reference:
https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow
I am new to python, and I am currently repeating all the solved examples from the book "python for data analysis"
in one example, I need to access the APIs from twitter search. Here is the code:
import requests
url='https://twitter.com/search?q=python%20pandas&src=typd'
resp=requests.get(url)
everything works okay up to here. problems comes when I use the json
import json
data=json.loads(resp.text)
then I received the error message ValueError: No JSON object could be decoded
I tried to switch to another url instead of the twitter, but I still receive the same error message.
does anyone have some ideas? thanks
You need a response with JSON content. To search Twitter this requires use of api.twitter.com, but you need to get an OAuth key. It's more complicated than the 5 lines of code you have. See http://nbviewer.ipython.org/github/chdoig/Mining-the-Social-Web-2nd-Edition/blob/master/ipynb/Chapter%201%20-%20Mining%20Twitter.ipynb
I'm a developer for a big GUI app and we have a web site for bug tracking. Anybody can submit a new bug to the bug tracking site. We can detect certain failures from our desktop app (i.e. an unhandled exception) and in such cases we would like to open the submit-new-bug form in the user predefined browser, adding whatever information we can gather about the failure to some form fields. We can either retrieve the submit-new-bug form using GET or POST http methods and we can provide default field values to that form. So from the http server side everything is pretty much OK.
So far we can successfully open a URL passing the default values as GET parameters in the URL using the webbrowser module from the Python Standard Library. There are, however, some limitations of this method such as the maximum allowed length of the URL for some browsers (specially MS IE). The webbrowser module doesn't seem to have a way to request the URL using POST. OTOH there's the urllib2 module that provides the type of control we want but AFAIK it lacks the possibility of opening the retrieved page in the user preferred browser.
Is there a way to get this mixed behavior we want (to have the fine control of urllib2 with the higher level functionallity of webbrowser)?
PS: We have thought about the possibility of retreiving the URL with urllib2, saving its content to a temp file and opening that file with webbrowser. This is a little nasty solution and in this case we would have to deal with other issues such as relative URLs. Is there a better solution?
This is not proper answer. but it also work
import requests
import webbrowser
url = "https://www.facebook.com/login/device-based/regular/login/?login_attempt=1&lwv=110"
myInput = {'email':'mymail#gmail.com','pass':'mypaass'}
x = requests.post(url, data = myInput)
y = x.text
f = open("home.html", "a")
f.write(y)
f.close()
webbrowser.open('file:///root/python/home.html')
I don't know of any way you can open the result of a POST request in a web browser without saving the result to a file and opening that.
What about taking an alternative approach and temporarily storing the data on the server. Then the page can be opened in the browser with a simple id parameter, and the saved partially filled form would be shown.
You could use tempfile.NamedTemporaryFile():
import tempfile
import webbrowser
import jinja2
t = jinja2.Template('hello {{ name }}!') # you could load template from a file
f = tempfile.NamedTemporaryFile() # deleted when goes out of scope (closed)
f.write(t.render(name='abc'))
f.flush()
webbrowser.open_new_tab(f.name) # returns immediately
A better approach if the server can be easily modified is to make POST request with partial parameters using urllib2 and open url generated by server using webbrowser as suggested by #Acorn.