I want to write a python script that retrieves data from a website using it's API. Using the code below (which comes from this [tutorial][1] - click [here for more context][2]), I am able to successfully submit a query to the website but not retrieve the data I need.
import requests, sys, json
WEBSITE_API = "https://rest.uniprot.org/beta"
def get_url(url, **kwargs):
response = requests.get(url, **kwargs);
if not response.ok:
print(response.text)
response.raise_for_status()
sys.exit()
return(response)
r = get_url(f"{WEBSITE_API}/uniprotkb/search?query=parkin AND (taxonomy_id:9606)&fields=id,accession,length,ft_site", headers={"Accept": "text/tsv"})
print(r.text)
Instead of printing the data I've queried, I get an error message:
{"url":"http://rest.uniprot.org/uniprotkb/search","messages":["Invalid request received. Requested media type/format not accepted: 'text/tsv'."]}
I figured out that I can print the data as a JSON-formatted string if I change the second-to-last line of my code to this:
r = get_url(f"{WEBSITE_API}/uniprotkb/search?query=parkin AND (taxonomy_id:9606)&fields=id,accession,length,ft_site")
In other words, I simply remove the following code snippet:
headers={"Accept": "text/tsv"}
I highly doubt this is an issue with the website because the tutorial the non-working code comes from was from a November 2021 webinar put on by an affiliated institution. I think a more likely explanation is that my python environment lacks some library and would appreciate any advice about how to investigate this possibility. Another possible solution I would appreciate is code and/or library suggestions that would put me on the path to simply parsing the JSON-string into a tab-delimited file.
[1]: https://colab.research.google.com/drive/1SU3j4VmXHYrYxOlrdK_NCBhajXb0iGes#scrollTo=EI91gRfPYfbc
[2]: https://drive.google.com/file/d/1qZXLl19apibjszCXroC1Jg2BGjcychqX/view
Related
I have almost the question already posted and answered here:
Perform Download via download button in Python
I also followed the instructions, in the answer of the above link.
In my case I want to download the data from the following page:
https://www.smard.de/home/downloadcenter/download-marktdaten#!?downloadAttributes=%7B%22selectedCategory%22:1,%22selectedSubCategory%22:1,%22selectedRegion%22:%22DE%22,%22from%22:1658872800000,%22to%22:1659563999999,%22selectedFileType%22:%22CSV%22%7D
The download-button is the one called "Datei herunterladen" in the lower right corner.
I went into the inspect mode downloaded the file and got the following output in the inspection
But the resulting
Request URL: 'https://www.smard.de/nip-download-manager/nip/download/market-data'
does not help getting the csv-file. Opened in the browser I get: 'The requested URL was rejected.'
On the other hand, it does not even contain the parameters anymore, so it can't be the right download url.
May anyone help to automate this download?
edit
Now I also tried
url = 'https://www.smard.de/nip-download-manager/nip/download/market-data'
json_body = {'format': "CSV",
'language': "de",
'moduleIds': [1001224, 1004066, 1004067, 1004068, 1001223, 1004069, 1004071, 1004070, 1001226, 1001228, 1001227,1001225],
'region': "DE",
'timestamp_from': 1659304800000,
'timestamp_to': 1659391199999,
'type': "discrete"}
x = requests.post(url, json = json_body)
x.content
> b'Exception when parsing incoming request to JSON object.'
So how do I get the csv-file based on this method?
the reason that you're getting "URL Rejected" is because the link specified in the request is not file-specific. The link that the POST request is being made to, contains several files, which are given out based on the body that the request is sent with. If you look into the developer tools, you can see that there is a request payload attached to the request (image attached) . If you want to learn more about sending JSON body data with requests, please take a look at w3 and this thread. I hope that helps!
The json_body of the request-payload was not correct.
And I found an easy way to get the correct one with less effort and less probability to make it wrong.
In the inspect mode after the file is downloaded:
Click on the "Raw"-Button in the upper right corner of the "request"-tab, than you can get the correct payload per copy and paste.
The final solution is:
url = 'https://www.smard.de/nip-download-manager/nip/download/market-data'
payload = {"request_form":[{"format":"CSV","moduleIds":[1001224,1004066,1004067,1004068,1001223,1004069,1004071,1004070,1001226,1001228,1001227,1001225],"region":"DE","timestamp_from":1658872800000,"timestamp_to":1659563999999,"type":"discrete","language":"de"}]}
x = requests.post(url, json = payload)
with open(f'energy_ger.csv', 'wb') as f:
f.write(x.content)
Im trying to see if I'm able to get the response data as I'm trying to learn how to use regex on Locust. I'm trying to reproduce my test script from JMeter using Locust.
This is the part of the code that I'm having problem with.
import time,csv,json
from locust import HttpUser, task,between,tag
class ResponseGet(HttpUser):
response_data= ""
wait_time= between (1,1.5)
host= "https://portal.com"
username= "NA"
password= "NA"
#task
def portal(self):
print("Portal Task")
response = self.client.post('/login', json={'username':'user','password':'123'})
print(response)
self.response_data = json.loads(response.text)
print(response_data)
I've tried this suggestion and I somehow can't make it work.
My idea is get response data > use regex to extract string > pass the string for the next task to use
For example:
Get login response data > use regex to extract token > use the token for the next task.
Is there any better way to do this?
The way you're doing it should work, but Locust's HttpUser's client is based on Requests so if you want to access the response data as a JSON you should be able to do that with just self.response_data = response.json(). But that will only work if the response body is valid JSON. Your code will also fail if the response body is not JSON.
If your problem is in parsing the response text as JSON, it's likely that the response just isn't JSON, possibly because you're getting an error or something. You could print the response body before your attempt to load it as JSON. But your current print(response) won't do that because it will just be printing the Response object returned by Requests. You'd need to print(response.text()) instead.
As far as whether a regex would be the right solution for getting at the token returned in the response, that will depend on how exactly the response is formatted.
I am trying to run my csv data thru "https://rxnav.nlm.nih.gov/REST/interaction" to identify any drug interactions using python. What else do I need in order to have the program be ready?
I got 200 when print status_code is that mean my code is up and ready?
import requests
response = requests.get("https://rxnav.nlm.nih.gov/REST/interaction")
print(response.status_code)
Here's how you'd hit this API, using requests and the details in their example:
import requests
uri = "https://rxnav.nlm.nih.gov/REST/interaction/interaction.json"
params = {'rxcui': 341248}
r = requests.get(uri, params)
Now you can check that r.status_code is 200, and get at the result of the request. For example:
r.json()
As you may realize, this returns a Python dictionary.
The general idea is that requsts.get() takes the base URL, followed by the query parameters, given as a dictionary. What you get back depends on the API endpoint you're querying, and/or on the parameters. In this, it's giving you JSON. Others might give you text (see r.text for this representation), or bytes (r.content).
i quite new to pyhton. I just try a simple way to get an HTTP response with python to a simple get from the sonar Web API
i use the request library and try a simple use :
project = requests.get(url=Sonar_Api_Projects_Search, params=param_Projects, verify=False, headers={'Authorization': 'token {}'.format(token)})
the request is well formatted and work fine when i use it in e web browser.
but as a response i get this strange output :
{"err_code":500,"err_msg":"undefined method empty?' for
nil:NilClass\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:132:in
login_from_basic_auth'\n\torg/jruby/RubyProc.java:290:in
call'\n\torg/jruby/RubyProc.java:224:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/http_authentication.rb:126:in
authenticate'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/http_authentication.rb:116:in
authenticate_with_http_basic'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:129:in
login_from_basic_auth'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:11:in
current_user'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/app/controllers/application_controller.rb:102:in set_user_session'\n\torg/jruby/RubyKernel.java:2223:in
send'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/callbacks.rb:178:in
evaluate_method'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/callbacks.rb:166:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:225:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:629:in
run_before_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:615:in
call_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:610:in
perform_action_with_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/benchmarking.rb:68:in
perform_action_with_benchmark'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/core_ext/benchmark.rb:17:in
ms'\n\tjar:file:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-complete-1.7.9.jar!/META-INF/jruby.home/lib/ruby/1.8/benchmark.rb:308:in
realtime'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/core_ext/benchmark.rb:17:in
ms'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/benchmarking.rb:68:in
perform_action_with_benchmark'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/rescue.rb:160:in
perform_action_with_rescue'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/flash.rb:151:in perform_action_with_flash'\n\torg/jruby/RubyKernel.java:2223:in
send'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:532:in
process'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:606:in
process_with_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:391:in
process'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:386:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/routing/route_set.rb:450:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:87:in
dispatch'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:85:in
dispatch'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:121:in
_call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:130:in
build_middleware_stack'\n\torg/jruby/RubyProc.java:290:in
call'\n\torg/jruby/RubyProc.java:224:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:29:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/query_cache.rb:34:in
cache'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:9:in
cache'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:28:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:361:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/config/environment.rb:67:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/string_coercion.rb:25:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/head.rb:9:in call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/methodoverride.rb:24:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/params_parser.rb:15:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/jruby/rack/session_store.rb:70:in
context'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/session/abstract/id.rb:58:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/failsafe.rb:26:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:106:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/adapter/rails.rb:34:in
serve_rails'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/adapter/rails.rb:39:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/handler/servlet.rb:22:in
call'\n"}
Can someone help me ?
Thanks a lot
Best regards
Arnaud
Direct use of requests never worked for me.
I do the following and it is working fine:
(below code is to list projects in Sonar)
import json , requests, pprint
url = 'http://sonar_url:9000/api/projects/search'
myToken = 'fa2377941a95125443f4efade615512jjkd221211a48'
session = requests.Session()
session.auth = myToken, ''
call = getattr(session, 'get')
res = call(url)
print(res.status_code)
binary = res.content
output = json.loads(binary)
pprint.pprint(output)
...
#Parse json result
In Sonarqube 8.9, requests is working for me.
First, you should should create an API token. Per the docs:
This is the recommended way. Benefits are described in the page User Token. The token is sent via the login field of HTTP basic authentication, without any password.
The docs go on to provide a weird curl usage example:
# note that the colon after the token is required in curl to set an empty password
curl -u THIS_IS_MY_TOKEN: https://sonarqube.com/api/user_tokens/search
In requests, this looks something like this:
response = requests.get(
"http://your-sonar-instance.com/api/blah",
auth=HTTPBasicAuth("Some Sonarqube API token", "")
)
return json.loads(response.text)
See https://docs.sonarqube.org/latest/extend/web-api/ for API details.
Also note that auth=HTTPBasicAuth("token", "") seems to behave differently from auth=HTTPBasicAuth("token", None).
I know its an old question. Thankfully there is a wrapper library available now - https://github.com/shijl0925/python-sonarqube-api. It works quite well and is easy to setup.
If possible people from Sonarsource could make it the official one so that more people start using it and it gets maintained in the future too.
I'm using the URLLib2 method to download a file from another server via a rest api (the url can't be exposed to the user--that's why it needs to be done on the backend).
It gives me the following response:
(<addinfourl at 4365818480 whose fp = <google.appengine.dist27.socket._fileobject object at 0x1043883d0>>
I'm now trying to find a way to serve this file to the end user (a download). I did quite a bit of research tonight but had no luck. I tried using print .read() and that didn't help either.
Here's some additional information:
The Platform is Google Appengine. And below is the relevant code:
In calltrunk.get_recording:
req = urllib2.Request(url, None, forward_headers)
print response[0].read()
stream = urllib2.urlopen(req)
In my main.py
response = calltrunk.get_recording(ConversationId=cId)
print response[0].read()
Could really use a hand here!