I am trying to connect to the api as explained in http://api.instatfootball.com/ , It is supposed to be something like the following get /[lang]/data/[action].[format]?login=[login]&pass=[pass]. I know the [lang], [action] and [format] I need to use and I also have a login and password but don´t know how to access to the information inside the API.
If I write the following code:
import requests
r = requests.get('http://api.instatfootball.com/en/data/stat_params_players.json', auth=('login', 'pass'))
r.text
with the actual login and pass, I get the following output:
{"status":"error"}
This API requires authentication as parameters over an insecure connection, so be aware that this is highly lacking on the API part.
import requests
username = 'login'
password = 'password'
base_url = 'http://api.instatfootball.com/en/data/{endpoint}.json'
r = requests.get(base_url.format(endpoint='stat_params_players'), params={'login': username, 'pass': password})
data = r.json()
print(r.status_code)
print(r.text)
You will need to make a http-request using the URL. This will return the requested data in the response body. Depending on the [format] parameter, you will need to decode the data from xml / json to a native Python object.
As rdas already commented, you can use the request library for python (https://requests.readthedocs.io/en/master/). You will also find some code samples there. It will also do proper decoding of JSON data.
If you want to play around with the API a bit, you can use a tool like Postman for testing and debugging your requests. (https://www.postman.com/)
Related
I am writing a script that downloads Sentinel 2 products (satellite imagery) using sentinelsat Python API.
A product's description is structured as JSON and contains the parameter quicklook_url.
Example:
https://apihub.copernicus.eu/apihub/odata/v1/Products('862619d6-9b82-4fe0-b2bf-4e1c78296990')/Products('Quicklook')/$value
Any Sentinel API calls require credentials. So does retrieving a product and also opening the link stored inside quicklook_url. When I call the example in my browser I get asked to enter username and password in order to get
with the name S2A_MSIL2A_20210625T065621_N0300_R063_T39NTJ_20210625T093748-ql.jpg.
Needless to say I am just starting with the API so I am probably missing something but
requests.post(product_description['quicklook_url'], verify=False, auth=HTTPBasicAuth(username, password)).content
yields 0KB damaged file and
requests.get(product_description['quicklook_url']).content
yields 1KB damaged file.
I have looked into requests.Session
session = requests.Session()
session.auth = (username, password)
auth = session.post('URL_FOR_LOGING_IN')
img = session.get(product_description['quicklook_url']).content
The problem is I am unable to find the URL I need to post my session authentification. I am somewhat sure that the sentinelsat API does that but my looks have not yielded any successful result.
I am currently looking into the SentinelAPI class. It has the download_quicklook() function, which I am using right now but I am still curious how to do this without the function.
I guess you don't need to sent a post request. Basic authentication works by sending a header along with each request. The following should work
session = requests.Session()
session.auth = (username, password)
img = session.get(product_description['quicklook_url']).content
Your first attempt is failed because of using POST I think.
requests.gett(product_description['quicklook_url'], verify=False, auth=HTTPBasicAuth(username, password)).content
should also work.
I have to send a POST request to the /batch endpoint of : 'https://www.google-analytics.com'.
As mentioned in the Documentation I have to send the request to /batch endpoint and specify each payload on its own line.
I was able to achieve this using POSTMAN as follows:
My query is to make a POST request using Python's requests library
I tried something like this :
import requests
text = '''v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=65
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=15
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=95'''
response = requests.post('https://www.google-analytics.com/batch', data=text)
but it doesn't works.
UPDATE
I Tried this and it works !
import http.client
conn = http.client.HTTPSConnection("www.google-analytics.com")
payload = "v=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=63\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=11\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=23"
headers = {
'Content-Type': 'text/plain'
}
conn.request("POST", "/batch", payload, headers)
res = conn.getresponse()
But the question remains open, what's the issue with requests here.
You don't need to double-escape the newline symbol.
Moreover, you don't need the newline symbol at all for the multi-line string.
And also the indentations you put in your multi-line string are counted:
test = '''abc
def
ghi'''
print(test)
Here's an SO answer that explains this with some additional ways to make long stings: https://stackoverflow.com/a/10660443/4570170
Now the request body.
The documentation says
payload_data – The BODY of the post request. The body must include exactly 1 URI encoded payload and must be no longer than 8192 bytes.
So try uri-encoding your payload:
text = '''v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=65
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=15
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=95'''
text_final = requests.utils.quote(text)
response = requests.post('https://www.google-analytics.com/batch', data=text_final)
Finally , I figured out the solution myself.
Updating for others help.
The problem was I was working on AWS Cloud9 and as mentioned in the documentation
Some environments are not able to send hits to Google Analytics directly. Examples of this are older mobile phones that can't run JavaScript or corporate intranets behind a firewall.
So we just need to include the User Agent parameter
ua=Opera/9.80
in each of our payloads
It works !
I am using am attempting to do a bulk download of a series of PDFs from a site that requires login authentication. I am able to successfully log in, however, when I attempt a GET request for '/transcripts/transcript.pdf?user_id=3007' but, the request returns the content for '/transcripts/transcript.pdf'.
Does anyone have any idea why the URL param is not sending? Or why it would be rerouted?
I have tried passing the parameter 'user_id' as data, params, and hardcoded in the URL.
I have removed the actual domain from the strings below just for privacy
with requests.Session() as s:
login = s.get('<domain>/login/canvas')
# print the html returned or something more intelligent to see if it's a successful login page.
print(login.text)
login_html = lxml.html.fromstring(login.text)
hidden_inputs = login_html.xpath(r'//form//input[#type="hidden"]')
form = {x.attrib["name"]: x.attrib["value"] for x in hidden_inputs}
print("form: ",form)
form['pseudonym_session[unique_id]']= username
form['pseudonym_session[password]']= password
response = s.post('<domain>/login/canvas',data=form)
print(response.url, response.status_code) # gets <domain>?login_success=1 200
# An authorised request.
data = { 'user_id':'3007'}
r = s.get('<domain>/transcripts/transcript.pdf?user_id=3007', data=data)
print(r.url) # gets <domain>/transcripts/transcript.pdf
print(r.status_code) # gets 200
with open('test.pdf', 'wb') as f:
f.write(r.content)
GET response returns /transcripts/transcript.pdf and not /transcripts/transcript.pdf?user_id=3007
From the looks of it, you are trying to use canvas. I'm pretty sure in canvas, you can bulk download all test attachments.
If that's not the case, There are a few things to try:
after logging in, try typing the url with user_id into a browser. Does that take you directly to the PDF file or links to one?
if so, look at the url, it may simply not display the parameters; some websites do this, don't worry about it
If not, GET may not be enough; perhaps the site uses javascript, etc.
after looking through the '.history' of the request I found a series of 302 redirects.
The first was to '/login?force_login=0&target_uri=%2Ftranscripts%2Ftranscript.pdf'
In a desperate attempt, I tried: s.get('/login?force_login=0&target_uri=%2Ftranscripts%2Ftranscript.pdf%3Fuser_id%3D3007') and this still rerouted me a few times but ultimately got me the file I wanted!
If anyone has a more elegant solution to this or any resources that I can read I would greatly appreciate it!
i quite new to pyhton. I just try a simple way to get an HTTP response with python to a simple get from the sonar Web API
i use the request library and try a simple use :
project = requests.get(url=Sonar_Api_Projects_Search, params=param_Projects, verify=False, headers={'Authorization': 'token {}'.format(token)})
the request is well formatted and work fine when i use it in e web browser.
but as a response i get this strange output :
{"err_code":500,"err_msg":"undefined method empty?' for
nil:NilClass\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:132:in
login_from_basic_auth'\n\torg/jruby/RubyProc.java:290:in
call'\n\torg/jruby/RubyProc.java:224:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/http_authentication.rb:126:in
authenticate'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/http_authentication.rb:116:in
authenticate_with_http_basic'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:129:in
login_from_basic_auth'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:11:in
current_user'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/app/controllers/application_controller.rb:102:in set_user_session'\n\torg/jruby/RubyKernel.java:2223:in
send'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/callbacks.rb:178:in
evaluate_method'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/callbacks.rb:166:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:225:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:629:in
run_before_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:615:in
call_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:610:in
perform_action_with_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/benchmarking.rb:68:in
perform_action_with_benchmark'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/core_ext/benchmark.rb:17:in
ms'\n\tjar:file:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-complete-1.7.9.jar!/META-INF/jruby.home/lib/ruby/1.8/benchmark.rb:308:in
realtime'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/core_ext/benchmark.rb:17:in
ms'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/benchmarking.rb:68:in
perform_action_with_benchmark'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/rescue.rb:160:in
perform_action_with_rescue'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/flash.rb:151:in perform_action_with_flash'\n\torg/jruby/RubyKernel.java:2223:in
send'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:532:in
process'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:606:in
process_with_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:391:in
process'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:386:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/routing/route_set.rb:450:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:87:in
dispatch'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:85:in
dispatch'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:121:in
_call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:130:in
build_middleware_stack'\n\torg/jruby/RubyProc.java:290:in
call'\n\torg/jruby/RubyProc.java:224:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:29:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/query_cache.rb:34:in
cache'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:9:in
cache'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:28:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:361:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/config/environment.rb:67:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/string_coercion.rb:25:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/head.rb:9:in call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/methodoverride.rb:24:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/params_parser.rb:15:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/jruby/rack/session_store.rb:70:in
context'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/session/abstract/id.rb:58:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/failsafe.rb:26:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:106:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/adapter/rails.rb:34:in
serve_rails'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/adapter/rails.rb:39:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/handler/servlet.rb:22:in
call'\n"}
Can someone help me ?
Thanks a lot
Best regards
Arnaud
Direct use of requests never worked for me.
I do the following and it is working fine:
(below code is to list projects in Sonar)
import json , requests, pprint
url = 'http://sonar_url:9000/api/projects/search'
myToken = 'fa2377941a95125443f4efade615512jjkd221211a48'
session = requests.Session()
session.auth = myToken, ''
call = getattr(session, 'get')
res = call(url)
print(res.status_code)
binary = res.content
output = json.loads(binary)
pprint.pprint(output)
...
#Parse json result
In Sonarqube 8.9, requests is working for me.
First, you should should create an API token. Per the docs:
This is the recommended way. Benefits are described in the page User Token. The token is sent via the login field of HTTP basic authentication, without any password.
The docs go on to provide a weird curl usage example:
# note that the colon after the token is required in curl to set an empty password
curl -u THIS_IS_MY_TOKEN: https://sonarqube.com/api/user_tokens/search
In requests, this looks something like this:
response = requests.get(
"http://your-sonar-instance.com/api/blah",
auth=HTTPBasicAuth("Some Sonarqube API token", "")
)
return json.loads(response.text)
See https://docs.sonarqube.org/latest/extend/web-api/ for API details.
Also note that auth=HTTPBasicAuth("token", "") seems to behave differently from auth=HTTPBasicAuth("token", None).
I know its an old question. Thankfully there is a wrapper library available now - https://github.com/shijl0925/python-sonarqube-api. It works quite well and is easy to setup.
If possible people from Sonarsource could make it the official one so that more people start using it and it gets maintained in the future too.
I am trying to add a lead to a Zoho CRM module with Python. I keep getting:
< response>< error>< code>4600< /code>< message>Unable to process your request. Please verify if the name and value is appropriate for the "xmlData" parameter.< /message>< /error>< /response>
from the server. I have no idea if I am posting correctly or if it is a problem with our Xml Data. I am using urllib and urllib2 to format the post request.
The post request looks like this.
url = ("https://crm.zoho.com/crm/private/xml/Leads/insertRecords?authtoken="
""+str(self.authToken)+"&scope=crmapi")
params = {"xmlData":self.xml}
data = urllib.urlencode(params)
request = urllib2.Request(url = url, data =data)
request.add_header("Content-Type",'application/xml')
response = urllib2.urlopen(request)
You cannot combine HTTP GET query parameters (ones in URL) and HTTP POST parameters.
This is limitation on the HTTP protocol level, not in Python or Zoho.
Most likely you are doing it wrong. Revisit Zoho documentation how it should be done.
Here is another old library doing Zoho + CRM, written in Python. You might want to check it for inspiration: https://github.com/miohtama/mfabrik.zoho