So I have been having some issues solving how I can read my repo file, which is in JSON format, with requests. (Python)
Basically I have created something simple like:
r = requests.get('https://raw.githubusercontent.com/Test/testrepo/master/token.json?token=ADAJKFAHFAKNQ3RKVSUQ5T12333777777')
which works, however, every time I make a new commit/changes on that file, it gives me a new token and then I need to recode all over again.
So my question is, is it possible to access the JSON file without the token? (I do need to keep the repo in private as well), but the point is that I want to be able to do changes on the file without the URL being changed.
The easiest solution is probably to use the GitHub API, rather than trying to use the "raw" link you see in the browser.
First, acquire a personal access token
Now issue an API request to /repos using that access token:
import requests
token = "MY_SECRET_TOKEN"
owner = 'Test'
repo = 'testrepo'
path = 'token.json'
r = requests.get(
'https://api.github.com/repos/{owner}/{repo}/contents/{path}'.format(
owner=owner, repo=repo, path=path),
headers={
'accept': 'application/vnd.github.v3.raw',
'authorization': 'token {}'.format(token),
}
)
print(r.text)
You can use the Github python library to get any file in your repository. Since you mentioned keeping the repo in private, you have to login to github using one of the methods described here. Here is an example of getting the file using the github username and password
from github import Github
user_name = <YOUR_USERNAME>
password = <YOUR_PASSWORD>
g = Github(user_name, password)
file_name='test.json' #Choose your required file name location
repo_name = 'repo_name'
repo_location = '{}/{}'.format(user_name, repo_name)
repo = g.get_repo(repo_location)
file = repo.get_contents(file_name)
#if you want the download url for the file (this comes along with the token that you talked about earlier)
download_url = file.download_url
#if you simply want the content inside the file
content = file.decoded_content
#larsks provides solution are great, and I want to supplement.
I choose a public repositoryawesome-python as an example
suppose you want to access master/docs/CNAME contents
import requests
token = "MY_SECRET_TOKEN"
owner = 'vinta'
repo = 'awesome-python'
path = 'docs/CNAME'
branch = 'master' # or sha1, for example: 6831740
url = f'https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={branch}'
print(url)
r = requests.get(url,
headers={
'accept': 'application/vnd.github.v3.raw',
# 'authorization': f'token {token}', # If you are want to read "public" only, then you can ignore this line.
}
)
print(r.text)
"""
Type
r.text: str
r.content: bytes
"""
# If you want to save it as a file, then you can try as below.
# f=open('temp.ico','wb')
# f.write(r.content)
But I think many people may want to access a private repository.
then go to
github.com/settings/tokens
Generate a new token
click repo (Full control of private repositories)
add header of authorizationcancel comment
Related
I want to list/search all the issues in Jira. I have a code like :
url = 'https://company.com/rest/api/2/search'
auth = HTTPBasicAuth("username", "password") // I tries token as well
headers = {
'Accept': 'application/json'
}
query = {
'jql': 'project=PRKJECTKEY',
'startAt': 0
}
response = requests.request(
"GET",
url,
headers=headers,
auth=auth,
params=query
)
I am not sure if the password should be token or the actual password and the url should be should be starting from companyname.com. This gives me <Response [401]> but i have all the permissions with the account.
Can someone help me with the authentication is supposed to be used this way.
I can only describe my way of accessing the JIRA-API:
1. I am using an API-key for this which one can easily create online, if one has the necessary permissions
(https://developer.atlassian.com/cloud/jira/platform/basic-auth-for-rest-apis/)
2. You need to set up the Jira-object to query issues first
user = 'firstname.familyname#account.xx'
apikey = 'xxxxxxxxxxxxxxxxx'
server = 'https://companypage.atlassian.net'
options = {
'server': server
}
jira = JIRA(options, basic_auth=(user, apikey))
3. Now, you can use the Jira-object to query with
tickets = jira.search_issues('text ~ "my search text" ORDER BY updated DESC')
Now, you can look what you got back and play with the results
for ticket in tickets:
print(ticket)
For Packages, you only need to import JIRA at the top (obviously, need to install jira first):
from jira import JIRA
Well, why don't you use atlassian-python-api?
https://community.atlassian.com/t5/Jira-articles/Atlassian-Python-API-s/ba-p/2091355
It's much easier to work with Jira via their own library. I have been working with Confluence via this and it's pretty simple. Take a look it may solve your problem.
EDIT: here is documentation. https://atlassian-python-api.readthedocs.io/
I am writing a script that downloads Sentinel 2 products (satellite imagery) using sentinelsat Python API.
A product's description is structured as JSON and contains the parameter quicklook_url.
Example:
https://apihub.copernicus.eu/apihub/odata/v1/Products('862619d6-9b82-4fe0-b2bf-4e1c78296990')/Products('Quicklook')/$value
Any Sentinel API calls require credentials. So does retrieving a product and also opening the link stored inside quicklook_url. When I call the example in my browser I get asked to enter username and password in order to get
with the name S2A_MSIL2A_20210625T065621_N0300_R063_T39NTJ_20210625T093748-ql.jpg.
Needless to say I am just starting with the API so I am probably missing something but
requests.post(product_description['quicklook_url'], verify=False, auth=HTTPBasicAuth(username, password)).content
yields 0KB damaged file and
requests.get(product_description['quicklook_url']).content
yields 1KB damaged file.
I have looked into requests.Session
session = requests.Session()
session.auth = (username, password)
auth = session.post('URL_FOR_LOGING_IN')
img = session.get(product_description['quicklook_url']).content
The problem is I am unable to find the URL I need to post my session authentification. I am somewhat sure that the sentinelsat API does that but my looks have not yielded any successful result.
I am currently looking into the SentinelAPI class. It has the download_quicklook() function, which I am using right now but I am still curious how to do this without the function.
I guess you don't need to sent a post request. Basic authentication works by sending a header along with each request. The following should work
session = requests.Session()
session.auth = (username, password)
img = session.get(product_description['quicklook_url']).content
Your first attempt is failed because of using POST I think.
requests.gett(product_description['quicklook_url'], verify=False, auth=HTTPBasicAuth(username, password)).content
should also work.
This method of getting Google Drive file thumbnails has been working for me but seems to have stopped recently.
All answers I can find online indicate that this is because thumbnailLink requires authorization (eg). However, I'm am accessing the thumbnails with authorized access tokens. I can get the file info using the Drive API "Files: get" with these access tokens but the thumbnailLink returns 404.
print(http)
# <google_auth_httplib2.AuthorizedHttp object at 0x11561d0f0>
# An instance of google_auth_httplib2.AuthorizedHttp
url = 'https://www.googleapis.com/drive/v3/files/%s?fields=thumbnailLink' % file_id
response, content = http.request(url)
data = json.loads(content)
print(data['thumbnailLink'])
# https://docs.google.com/u//feeds/vt?gd=true&id=***fileID***&v=203&s=***&sz=s220
# Works ✓
response, content = http.request(data['thumbnailLink'])
print(response['status'])
# 404
# :(
Also giving a 404 error:
thumbnailLink + "&access_token=" + YOURTOKEN; as suggested here.
Opening thumbnailLink in a browser (logged in to Google as the file owner).
Opening a modified thumbnailLink in a browser - replacing /u// with /u/0/, /u/1/ , /u/2/ (When I open drive as this user the URL is https://drive.google.com/drive/u/1/my-drive)
Does anyone know a reliable way to get Google Drive thumbnail image files?
I believe your goal as follows.
You want to retrieve the thumbnail from the thumbnail link retrieved by the method of "files.get" in Drive API.
From your sample thumbnail link, you want to retrieve the thumbnail from Google Docs (Document, Spreadsheet, and so on).
Issue and workaround:
In the current stage, it seems that the situation of 404 from the thumbnail is the bug. This has already been reported to the Google issue tracker. Ref And it seems that Google side has already been known. Unfortunately, I think that this is the current direct answer. And also, I believe that this issue will be resolved by the future update.
Here, as the current workaround, how about converting it to PDF file and retrieve the thumbnail? In this case, the thumbnail link can be used. The flow of this workaround is as follows.
Convert Google Docs to a PDF file.
The PDF file is created to the same folder of the Google Docs.
Retrieve the thumbnail link from the created PDF file.
When above flow is converted to the python script, it becomes as follows.
Sample script:
Before you use this script, please set the access token and file ID. In this case, in order to request multipart/form-data with the simple script, I used requests library.
import json
import httplib2
import requests
import time
http = httplib2.Http()
access_token = '###' # Please set the access token.
file_id = '###' # Please set the file ID.
headers = {"Authorization": "Bearer " + access_token}
# 1. Retrieve filename and parent ID.
url1 = "https://www.googleapis.com/drive/v3/files/" + file_id + "?fields=*"
res, res1 = http.request(url1, 'GET', headers=headers)
d = json.loads(res1.decode('utf-8'))
# 2. Retrieve PDF data by converting from the Google Docs.
url2 = "https://www.googleapis.com/drive/v3/files/" + file_id + "/export?mimeType=application%2Fpdf"
res, res2 = http.request(url2, 'GET', headers=headers)
# 3. Upload PDF data as a file to the same folder of Google Docs.
para = {'name': d['name'] + '.pdf', 'parents': d['parents']}
files = {
'data': ('metadata', json.dumps(para), 'application/json; charset=UTF-8'),
'file': res2
}
res3 = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers=headers,
files=files
)
obj = res3.json()
# It seems that this is required to use by creating the thumbnail link from the uploaded file.
time.sleep(5)
# 4. Retrieve thumbnail link of the uploaded PDF file.
url3 = "https://www.googleapis.com/drive/v3/files/" + obj['id'] + "?fields=thumbnailLink"
res, res4 = http.request(url3, 'GET', headers=headers)
data = json.loads(res4.decode('utf-8')) # or data = json.loads(res4)
print(data['thumbnailLink'])
# 5. Retrieve thumbnail.
response, content = http.request(data['thumbnailLink'])
print(response['status'])
print(content)
When you run this script, the Google Docs file is exported as the PDF data, and the PDF data is uploaded to Google Drive and retrieve the thumbnail link.
Note:
In this case, please include the scope of https://www.googleapis.com/auth/drive to the scopes of your access token. Because the file is uploaded.
In order to retrieve the file metadata and export the PDF file and upload the data, the access token is required to be used. But when the thumbnail is retrieved from the thumbnail link, the access token is not required to be used.
After January, 2020, the access token cannot be used with the query parameter of access_token=###.So please use the access token to the request header. Ref
When above issue was resolved, I think that you can use your script.
References:
Files: get
Files: export
Files: create
I can't create new repositoies in bitbucket cloud with code below.
I'm able to delete repositiries (with change form 'post' to 'delete' requests method). When I'm using code below I've got http 400 which means according to api docs - If the input document was invalid, or if the caller lacks the privilege to create repositories under the targeted account.
import requests
username = 'user#mail.com'
password = 'password'
headers = {"Content-Type": 'application/json'}
auth = (username, password)
bb_base_url = f"https://api.bitbucket.org/2.0/repositories/username/reponame"
res = requests.post(bb_base_url, headers=headers, auth=auth)
print(res)
So would like to ask for help to refactor code in way I will be able to do two things
I am communicating with an API using HTTP.client in Python 3.6.2.
In order to upload a file it requires a three stage process.
I have managed to talk successfully using POST methods and the server returns data as I expect.
However, the stage that requires the actual file to be uploaded is a PUT method - and I cannot figure out how to syntax the code to include a pointer to the actual file on my storage - the file is an mp4 video file.
Here is a snippet of the code with my noob annotations :)
#define connection as HTTPS and define URL
uploadstep2 = http.client.HTTPSConnection("grabyo-prod.s3-accelerate.amazonaws.com")
#define headers
headers = {
'accept': "application/json",
'content-type': "application/x-www-form-urlencoded"
}
#define the structure of the request and send it.
#Here it is a PUT request to the unique URL as defined above with the correct file and headers.
uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
#get the response from the server
uploadstep2response = uploadstep2.getresponse()
#read the data from the response and put to a usable variable
step2responsedata = uploadstep2response.read()
The response I am getting back at this stage is an
"Error 400 Bad Request - Could not obtain the file information."
I am certain this relates to the body="C:\Test.mp4" section of the code.
Can you please advise how I can correctly reference a file within the PUT method?
Thanks in advance
uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
will put the actual string "C:\Test.mp4" in the body of your request, not the content of the file named "C:\Test.mp4" as you expect.
You need to open the file, read it's content then pass it as body. Or to stream it, but AFAIK http.client does not support that, and since your file seems to be a video, it is potentially huge and will use plenty of RAM for no good reason.
My suggestion would be to use requests, which is a way better lib to do this kind of things:
import requests
with open(r'C:\Test.mp4'), 'rb') as finput:
response = requests.put('https://grabyo-prod.s3-accelerate.amazonaws.com/youruploadpath', data=finput)
print(response.json())
I do not know if it is useful for you, but you can try to send a POST request with requests module :
import requests
url = ""
data = {'title':'metadata','timeDuration':120}
mp3_f = open('/path/your_file.mp3', 'rb')
files = {'messageFile': mp3_f}
req = requests.post(url, files=files, json=data)
print (req.status_code)
print (req.content)
Hope it helps .