I'm doing a post request using python and confluence REST API in order to update confluence pages via a script.
I ran into a problem which caused me to receive a 400 error in response to a
requests.put(url, data = jsonData, auth = (username, passwd), headers = {'Content-Type' : 'application/json'})
I spent some time on this to discover that the reason for it was me not supplying an incremented version when updating the content. I have managed to make my script work, but that is not the point of this question.
During my attempts to make this work, I swapped from requests to an http.client connection. Using this module, I get a lot more information regarding my error:
b'{"statusCode":400,"data":{"authorized":false,"valid":true,"allowedInReadOnlyMode":true,"errors":[],"successful":false},"message":"Must supply an incremented version when updating Content. No version supplied.","reason":"Bad Request"}'
Is there a way for me to get the same feedback information while using requests? I've turned on logging, but this kind of info is never shown.
You're looking for
requests.json()
It outputs everything the requests item returns, as a dictionary.
Related
So, I am trying to use the pyodata library in Python to access and download data from Odata.
I tried accessing the Northwind data and it worked. So, i guess the codes i used is ok.
import requests
import pyodata
url_t = 'http://services.odata.org/V2/Northwind/Northwind.svc'
# connection set up
northwind = pyodata.Client(url_t, requests.Session())
# This prints out a single value from the table Customers
for customer in northwind.entity_sets.Customers.get_entities().execute():
print(customer.CustomerID,",", customer.CompanyName)
break
# This will print out - ALFKI , Alfreds Futterkiste
I also tried connecting to Odata in excel to see if the codes above return the correct data, and it did.
Click to see the screenshot in excel for Odata connection
Now, using the same code to connect to the data source where I want to pull the data did not work:
#using this link to connect to Odata worked.
url_1 = 'https://batch.decisionkey.npd.com/odata/dkusers'
session = requests.Session()
session.auth = (user_name, psw)
theservice = pyodata.Client(url_1, session)
The above codes return this error message(is it something about security?):
Click to see error message
Connecting to the data in excel looks like this:
Click the view image
I am thinking about it might be security issue that is blocking me from accessing the data, or it could be something else. Please let me know if anything need to be clarify. Thanks.
First time asking question, so please let me know if anything I did not do right here. ^_^
You got HTTP 404 - Not Found.
The service "https://batch.decisionkey.npd.com/odata/dkusers" is not accessible from outside world for me to try it, so there is something more from networking point of view that happens in the second picture in the Excel import.
You can forget the pyodata at the moment, for your problem it is just wrapper around HTTP networking layer, the Requests library. You need to find a way initialize the Requests session in a way, that will return HTTP 200 OK instead.
Northwind example service is just plain and simple, so no problem during initialization of pyodata.Client
Refer to Requests library documentation- https://docs.python-requests.org/en/latest/user/advanced/
//sample script
url_1 = 'https://batch.decisionkey.npd.com/odata/dkusers'
session = requests.Session()
session.auth = (user_name, psw)
//??? SSL certificate needs to be provided perhaps?
//?? or maybe you are behind some proxy that Excel uses but python not.. try ping in CMD
response = session.get(url_1)
print(response.text)
Usable can be pyodata documentation about initialization, however you will not find there the reason why you get HTTP 404 - https://pyodata.readthedocs.io/en/latest/usage/initialization.html
have tried looking into other similar question and search the web, but I cannot seem to find the answer, so hope some clever people here can help or guide me.
I have a proc http request in SAS which runs fine on my local machine, no problems:
filename lst temp;
proc http
url = "http://xxx/api/job/"
method = "get"
out = lst;
run;
libname lst json fileref=lst automap=create;
Trying to do the same in Python gives me error code 401.
import requests
response = requests.get("http://xxx/api/job/")
print(response)
print(response.status_code)
This is an API from a system running internally in our organization. One needs to log on the first time when accessing through a web browser, but then it works.
I have tried all the different auth= etc. I could find in the documentation giving my user and password. But, nothing seems to work.
Some how, SAS proc http must be working since my profile/user is somehow verified, but via Python it is not - or at least that is what I am thinking.
Any suggestions?
I am using python 2.7 + requests & github3 libraries (modules) to find and replace some webhooks URL, in our Enterprise Github account. We have more than 500 organizations and 1000+ repositories (privates and publics).
Our python script works well for most test cases. However, I found an error, which I will explain below:
I have found some webhooks URL in our github enterprise account, that have the following struncture:
EXAMPLE:
http://10.10.10.10:8080/pr/XXXX-pr/v1/teamcity?buildType=BeCd_XXCDServer_Pr
I mean, that type of webhook URL has been around for a few years on our github, but since we will migrate some servers to the cloud, over time we will need to replace the host name in that URL. So, we created a python script to do it, which seems to work well.
The specific problem appears when the script attempts to replace the hostname in the URL that begins with "http".
If the URL starts with "https" the replacement process runs smoothly.
The error code I receive, is "422 Unprocessable Entity".
The URL that I send to the "requests.patch" command in JSON format, is something like this:
my_url = "http://MY_GITHUB_DOMAIN/repos/MY_ORG/MY_REPO/hooks/001"
And the python code is:
data_json='{"config":{"url: http://NEW_GITHUB_DOMAIN/repos/MY_ORG/MY_REPO/hooks/001"}}'
data_json_load=json.loads(data_json)
TOKEN="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
my_url = "http://MY_GITHUB_DOMAIN/repos/MY_ORG/MY_REPO/hooks/001"
requests.patch(url=my_url, data=json.dumps(data_json_load), auth=(username, TOKEN))
If I manually replace "http" with "https" in the original webhook in github (before executing the script) in this case the python code runs correctly.
What could be a root cause of this problem (The 422 error, when the script attempts to change a hostname in an URL that begin with http instead of https) ?
Could it be a bug of the requests python library ?
thanks in advance!
I'm having some trouble using urllib to fetch some web content on my Debian server. I use the following code to get the contents of most websites without problems:
import urllib.request as request
url = 'https://www.metal-archives.com/'
req = request.Request(url, headers={'User-Agent': "foobar"})
response = request.urlopen(req)
response.read()
However, if the website is using an older encryption protocol, the urlopen function will throw the following error:
ssl.SSLError: [SSL: VERSION_TOO_LOW] version too low (_ssl.c:748)
I have found a way to work around this problem, consisting in using an SSL context and passing it as an argument to the urlopen function, so the previous code would have to be modified:
...
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
response = request.urlopen(req, context=context)
...
Which will work, provided the protocol specified matches the website I'm trying to access. However, this does not seem like the best solution since:
If the site owners ever update their cryptography methods, the code will stop working
The code above will only work for this site, and I would have to create special cases for every website I visit in the entire program, since everyone could be using a different version of the protocol. That would lead to pretty messy code
The first solution I posted (the one without the ssl context) oddly seems to work on my ArchLinux machine, even though they both have the same versions of everything
Does anyone know about a generic solution that would work for every TLS version? Am I missing something here?
PS: For completeness, I will add that I'm using Debian 9, python v3.6.2, openssl v1.1.0f and urllib3 v1.22
In the end, I've opted to wrap the method call inside a try-except, so I can use the older SSL version as fallback. The final code is this:
url = 'https://www.metal-archives.com'
req = request.Request(url, headers={"User-Agent": "foobar"})
try:
response = request.urlopen(req)
except (ssl.SSLError, URLError):
# Try to use the older TLSv1 to see if we can fix the problem
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
response = request.urlopen(req, context=context)
I have only tested this code on a dozen websites and it seems to work so far, but I'm not sure it will work every time. Also, this solution seems inefficient, since it needs two http requests, which can be very slow.
Improvements are still welcome :)
I am trying to use requests to pull information from the NPI API but it is taking on average over 20 seconds to pull the information. If I try and access it via my web browser it takes less than a second. I'm rather new to this and any help would be greatly appreciated. Here is my code.
import json
import sys
import requests
url = "https://npiregistry.cms.hhs.gov/api/?number=&enumeration_type=&taxonomy_description=&first_name=&last_name=&organization_name=&address_purpose=&city=&state=&postal_code=10017&country_code=&limit=&skip="
htmlfile=requests.get(url)
data = htmlfile.json()
for i in data["results"]:
print(i)
This might be due to the response being incorrectly formatted, or due to requests taking longer than necessary to set up the request. To solve these issues, read on:
Server response formatted incorrectly
A possible issue might be that the response parsing is actually the offending line. You can check this by not reading the response you receive from the server. If the code is still slow, this is not your problem, but if this fixed it, the problem might lie with parsing the response.
In case some headers are set incorrectly, this can lead to parsing errors which prevents chunked transfer (source).
In other cases, setting the encoding manually might resolve parsing problems (source).
To fix those, try:
r = requests.get(url)
r.raw.chunked = True # Fix issue 1
r.encoding = 'utf-8' # Fix issue 2
print(response.text)
Setting up the request takes long
This is mainly applicable if you're sending multiple requests in a row. To prevent requests having to set up the connection each time, you can utilize a requests.Session. This makes sure the connection to the server stays open and configured and also persists cookies as a nice benefit. Try this (source):
import requests
session = requests.Session()
for _ in range(10):
session.get(url)
Didn't solve your issue?
If that did not solve your issue, I have collected some other possible solutions here.