Following the example here: https://learn.microsoft.com/en-us/rest/api/azure/devops/git/items/get?view=azure-devops-rest-6.1
I can query a dev ops organization and get a response like so:
{
"count": 1,
"value": [
{
"objectId": "61a86fdaa79e5c6f5fb6e4026508489feb6ed92c",
"gitObjectType": "blob",
"commitId": "23d0bc5b128a10056dc68afece360d8a0fabb014",
"path": "/MyWebSite/MyWebSite/Views/Home/_Home.cshtml",
"url": "https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/items/MyWebSite/MyWebSite/Views/Home/_Home.cshtml?versionType=Branch&versionOptions=None"
}
]
}
How can I use Python to, I guess, download that url? The file should be an XML file. I want to read (download) it directly from Python.
If I take the url returned above and insert it into yet another GET request, I get sent in loops basically.
Edit:
I figured out that if I paste the URL that it gives me, I can download the file with my web browser. However, when I insert that same URL into a new request, I get the same meta-data over and over, trying:
response = requests.get(url=(url), headers=headers, stream=True)
response.text
response.content
response = requests.get(url=(url), headers=headers, stream=False)
response.text
response.content
I have solved the problem, as both comments and previous answer are incorrect and will not lead you to the right answer for future visitors.
Following the example here as originally posted, it presumes that download=True: https://learn.microsoft.com/en-us/rest/api/azure/devops/git/items/get?view=azure-devops-rest-6.1
However, you then need to query the file of interest and set this flag: &includeContent=true to actually get the file content from the devops GIT.
Like so:
https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/items?scopePath=/MyWebSite/MyWebSite/Views/Home/_Home.cshtml&includeContent=true&api-version=6.1-preview.1
or you will send as many GET requests as you like and get nothing of value in return.
Related
I am making a post request and it has been done successfully on reqbin but when I use the code in my script, even if I copy and paste exactly, it doesn't work. I'm getting a 500 status on these and if I try other languages i get a 301. But I don't know what exactly is going on or what I'm missing.
expected response:
```{
"data": {
"txGasUnits": 132500,
"returnMessage": "",
"smartContractResults": null
},
"error": "",
"code": "successful"
}```
response from python:
b'{"data":null,"error":"transaction not found","code":"internal_issue"}
response from php and bash:
```<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.14.0 (Ubuntu)</center>
</body>
</html>```
the code provided for Python:
import requests
from requests.structures import CaseInsensitiveDict
url = "http://testnet-gateway.elrond.com/transaction/cost"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer mt0dgHmLJMVQhvjpNXDyA83vA_PxH23Y"
headers["Content-Type"] = "application/json"
data = """
{"value": "0",
"sender": "erd1qgjj3t38fgv0pscvr3xk0xq2gkm974mx0wz8afnqfewkevpn7v8s70np82",
"receiver": "erd14edl4fvr8wc2sczrz5af6tfmfucgqzsh4u73hnxw96l07cekdg6schwtsh",
"data": "RVNEVFRyYW5zZmVyQDUyNDk0MzQ1MmQ2MjM0MzA2MjMyNjJAMDRlNzNlZjNjNjdlMTUwMDAwCg==",
"chainID": "T",
"version": 1}
"""
resp = requests.post(url, headers=headers, data=data)
print(resp.status_code)
I added the php and bash just to show that other code snippets also failed, and in a different way, incase that may be relevent. I tried my own code and had the same result. Then when I used reqbin i got it to succeed, and then after struggling with it a bit I simply copied the code they generated and tried that precisely but it still didn't work. What could be the error?
The mistake is in the URL you're using.
Instead of http://, use https:// and it'll work.
As a rule of thumb, try to use https to interface with any service Elrond exposes, as that's the secure and supported way to go. The http port might be dead or straight up closed.
This is sort of a follow up question to my previous post here for reference: Webscraping Blockchain data seemingly embedded in Javascript through Python, is this even the right approach?
Basically, I would receive an output and want to scrape some more features from it. In this case, the final link would be located at https://tracker.icon.foundation/address/hx4ae18d8f72200dc564673a0ae7206d862992753c
where I'm trying to retrieve the balance of 3,570.5434 ICX in the middle of the page. I'm clearly not calling the correct methods (couldn't find any documentation for this) and was wondering where in the source code I could find it. Attempt at trying it in Python:
import requests
url = "https://tracker.icon.foundation/v3/block/txList"
params = {
"from": 'hx4ae18d8f72200dc564673a0ae7206d862992753c',
}
response = requests.get(url, params=params)
response.json()["data"]
The value you're trying to scrape - the total ICX balance - appears to be the sum of the "available" ICX, and the "staked" ICX:
I've added the red lines for emphasis. The sum of these two values is the total ICX balance. Again, if you log your browser's network traffic, you'll find that these values come from requests made to different REST APIs. One is a HTTP GET request, the other an HTTP POST request. Again, you can find out how the POST payload is supposed to look by looking at the network traffic logs. If you need a little guidance for how to approach these kinds of network-traffic-sniffing solutions, I recommend you read this answer I posted for a different question, where someone needed help scraping information from a page about different wines and vineyards, and that page also happened to make XHR requests to a REST API. In it, I go more in-depth about each step of logging your network traffic, and finding the information you're looking for.
def get_available_icx(address):
import requests
url = "https://tracker.icon.foundation/v3/address/info"
params = {
"address": address
}
response = requests.get(url, params=params)
response.raise_for_status()
return float(response.json()["data"]["balance"])
def get_staked_icx(address):
import requests
url = "https://wallet.icon.foundation/api/v3"
form_data = {
"jsonrpc": "2.0",
"id": 0,
"method": "icx_call",
"params": {
"from": "hx0000000000000000000000000000000000000000",
"to": "cx0000000000000000000000000000000000000000",
"dataType": "call",
"data": {
"method": "getDelegation",
"params": {
"address": address
}
}
}
}
response = requests.post(url, json=form_data)
response.raise_for_status()
return int(response.json()["result"]["totalDelegated"], 16) / (10 ** 18)
def main():
address = "hx4ae18d8f72200dc564673a0ae7206d862992753c"
total_icx = get_available_icx(address) + get_staked_icx(address)
print(total_icx)
return 0
if __name__ == "__main__":
import sys
sys.exit(main())
I have a problem with a job in the Cloud Scheduler for my cloud function. I created the job with next parameters:
Target: HTTP
URL: my trigger url for cloud function
HTTP method: POST
Body:
{
"expertsender": {
"apiKey": "ExprtSender API key",
"apiAddress": "ExpertSender APIv2 address",
"date": "YYYY-MM-DD",
"entities": [
{
"entity": "Messages"
},
{
"entity": "Activities",
"types":[
"Subscriptions"
]
}
]
},
"bq": {
"project_id": "YOUR GCP PROJECT",
"dataset_id": "YOUR DATASET NAME",
"location": "US"
}
}
The real values has been changed in this body.
When I run this job I got an error. The reason is caused by processing body from POST request.
However, when I take this body and use it as Triggering event in Testing I don't get any errors. So I think, that problem in body representation for my job but I havn't any idea how fix it. I'll be very happy for any idea.
Disclaimer:
I have tried to solve the same issue using NodeJS and I'm able to get a solution
I understand that this is an old question. But I felt like its worth to answer this question as I have spent almost 2 hours figuring out the answer for this issue.
Scenario - 1: Trigger the Cloud Function via Cloud Scheduler
Function fails to read the message in request body.
Scenario - 2: Trigger the Cloud Function via Test tab in Cloud Function interface
Function call always executes fine with no errors.
What did I find?
When the GCF routine is executed via Cloud Scheduler, it sends the header content-type as application/octet-stream. This makes express js unable to parse the data in request body when Cloud scheduler POSTs the data.
But when the exact same request body is used to test the function via the Cloud Function interface, everything works fine because the Testing feature on the interface sends the header content-type as application/json and express js is able to read the request body and parses the data as a JSON object.
Solution
I had to manually parse the request body as JSON (explicitly using if condition based on the content-type header) to get hold of data in the request body.
/**
* Responds to any HTTP request.
*
* #param {!express:Request} req HTTP request context.
* #param {!express:Response} res HTTP response context.
*/
exports.helloWorld = (req, res) => {
let message = req.query.message || req.body.message || 'Hello World!';
console.log('Headers from request: ' + JSON.stringify(req.headers));
let parsedBody;
if(req.header('content-type') === 'application/json') {
console.log('request header content-type is application/json and auto parsing the req body as json');
parsedBody = req.body;
} else {
console.log('request header content-type is NOT application/json and MANUALLY parsing the req body as json');
parsedBody = JSON.parse(req.body);
}
console.log('Message from parsed json body is:' + parsedBody.message);
res.status(200).send(message);
};
It is truly a feature issue which Google has to address and hopefully Google fixes it soon.
Cloud Scheduler - Content Type header issue
Another way to solve the problem is this:
request.get_json(force=True)
It forces the parser to treat the payload as json, ingoring the Mimetype.
Reference to the flask documentation is here
I think this is a bit more concise then the other solutions proposed.
Thank you #Dinesh for pointing towards the request headers as a solution! For all those who still wander and are lost, the code in python 3.7.4:
import json
raw_request_data = request.data
# Luckily it's at least UTF-8 encoded...
string_request_data = raw_request_data.decode("utf-8")
request_json: dict = json.loads(string_request_data)
Totally agree, this is sub-par from a usability perspective. Having the testing utility pass a JSON and the cloud scheduler posting an "application/octet-stream" is incredibly irresponsibly designed.
You should, however, create a request handler, if you want to invoke the function in a different way:
def request_handler(request):
# This works if the request comes in from
# requests.post("cloud-function-etc", json={"key":"value"})
# or if the Cloud Function test was used
request_json = request.get_json()
if request_json:
return request_json
# That's the hard way, i.e. Google Cloud Scheduler sending its JSON payload as octet-stream
if not request_json and request.headers.get("Content-Type") == "application/octet-stream":
raw_request_data = request.data
string_request_data = raw_request_data.decode("utf-8")
request_json: dict = json.loads(string_request_data)
if request_json:
return request_json
# Error code is obviously up to you
else:
return "500"
One of the workarounds that you can use is to provide a header "Content-Type" set to "application/json". You can see a setup here.
I want to access details from Github using Github GraphQl v4 API. I found Graphene library, but I'm not sure how to authenticate with a personal access token in Python.
I tried to search on Google but couldn't found any example. It's Python library that can create graphical schema's and are not for consuming them, I tried with `requests' but failed. How can i authenticate and can find list of repositories?
I have used Github GraphQl explorer to find list of repositories via this code:
viewer {
repositories(first: 30) {
totalCount
pageInfo {
hasNextPage
endCursor
}
edges {
node {
name
}
}
}
Unlike rest, graphql has only one end point. You just need to do a POST with your query as a json object. You should provide your api_token you get from github as part of the headers.
import requests
url = 'https://api.github.com/graphql'
json = { 'query' : '{ viewer { repositories(first: 30) { totalCount pageInfo { hasNextPage endCursor } edges { node { name } } } } }' }
api_token = "your api token here..."
headers = {'Authorization': 'token %s' % api_token}
r = requests.post(url=url, json=json, headers=headers)
print (r.text)
Graphene is for building GraphQL APIs not for consuming them.
Did you see that: https://github.com/graphql-python/gql ?
It's a GraphQL client for Python.
Hope that's helpful.
As previous answers mentioned, calling GraphQL is as simple has making a POST request with the query string.
However, if you're on Python3 want something more advanced that'll also verify your queries during build and generate typed data-class response classes for you check out the new GQL library:
https://github.com/ekampf/gql
Exactly for GitHub, there is an example on using the Github GraphQL API with Python 3
https://gist.github.com/gbaman/b3137e18c739e0cf98539bf4ec4366ad
(check link as it has a lot of comments including better code for authentication)
# An example to get the remaining rate limit using the Github GraphQL API.
import requests
headers = {"Authorization": "Bearer YOUR API KEY"}
def run_query(query): # A simple function to use requests.post to make the API call. Note the json= section.
request = requests.post('https://api.github.com/graphql', json={'query': query}, headers=headers)
if request.status_code == 200:
return request.json()
else:
raise Exception("Query failed to run by returning code of {}. {}".format(request.status_code, query))
# The GraphQL query (with a few aditional bits included) itself defined as a multi-line string.
query = """
{
viewer {
login
}
rateLimit {
limit
cost
remaining
resetAt
}
}
"""
result = run_query(query) # Execute the query
remaining_rate_limit = result["data"]["rateLimit"]["remaining"] # Drill down the dictionary
print("Remaining rate limit - {}".format(remaining_rate_limit))
And there are many Python GraphQL client libraries:
https://github.com/graphql-python/gql (aka https://github.com/ekampf/gql)
https://github.com/graphql-python/gql-next
https://github.com/prodigyeducation/python-graphql-client
Official list is at https://graphql.org/code/#python
(just scroll down, client libraries are after server libraries)
I'm struggled with Google Drive REST API interface.
I need to create a folder programmatically. Reading api documents (https://developers.google.com/drive/v3/reference/files/create) it's possible to create a folder with a POST method to https://www.googleapis.com/drive/v3/files, a request body with folder name and mime type as 'application/vnd.google-apps.folder'
so I write this python function:
def createFolder(self,folderName):
if not self.authorization:
self.get_authorization()
url = 'https://www.googleapis.com/drive/v3/files'
headers = { 'Authorization':'Bearer {}'.format(self.access_token)}
metadata = {
"name": folderName,
"mimeType": 'application/vnd.google-apps.folder'
}
response = requests.post( url, headers = headers, params = metadata)
return response.json()
that outputs a response object like this:
{
u'mimeType': u'application/json',
u'kind': u'drive#file',
u'id': u'0B350e2U7rvyvR0k3NjJmTTVuWUE',
u'name': u'Untitled'
}
A file is created, but the folder metadata are not applied.
When I do the same with "Try it!" APIs Explorer I get a correct behaviour, so I can't understand where my code is wrong.
I'm writing a portable plugin and I don't want to deal with google library so I would prefer a simple Http approach.
I'll appreciate if you can give me any suggestions.
Thanks. I finally got it: (SOLVED)
def createFolder(self,folderName):
if not self.authorization:
self.get_authorization()
url = 'https://www.googleapis.com/drive/v3/files'
headers = {
'Authorization':'Bearer {}'.format(self.access_token),
'Content-Type': 'application/json'
}
metadata = {
'name': folderName,
'mimeType': 'application/vnd.google-apps.folder'
}
response = requests.post( url, headers = headers, data = json.dumps(metadata))
return response.json()
Google Drive API wants the needed parameters in {request body}, so metadata must be passed as json string and header "content-type" carefully set to "application/json", otherwise the API will not like very much python requests default to 'application/x-www-form-urlencoded'
There is a problem with your URL. Lose the ?uploadType=multipart as this isn't appropriate for creating a folder - "upload" is a reference to a file's content and a folder has no content.
Have you tried using postman to send rest API POST call? I work with rest API and python all day long. I first test it with postman. If that works, just have postman convert it to Python code. From there, you can create your variables, create your function. Run your python script and verify the folder was created.