I'm trying to write a script to port my existing database into Firebase. My data is stored in JSON and I thought I could just pull the JSON and then send that as data into a POST to my Firebase.
def Post_And_Recieve_JSON(url, data, headers):
print("Compiling query...")
Post_And_Recieve_JSON.url = url
Post_And_Recieve_JSON.headers = headers
Post_And_Recieve_JSON.data = (data)
print("Sending request...")
request = urllib.request.Request(url=url, data=data,headers=headers)
print("Recieving response...")
response = urllib.request.urlopen(request)
print("Reading response...")
response_data = response.read()
print("Converting into usable format...")
response_data_JSON = json.loads(response_data.decode(encoding='UTF-8'))
return response_data_JSON
for all_users in existing_database:
full_data.append(Post_And_Recieve_JSON(...)
for item in full_data:
url = 'firebaseurlhere ' + item['profileId'] + '.json'
data = json.dumps(item).encode('ascii')
Post_And_Recieve_JSON(url, data, headers)
Where full_data is a list of JSON objects I've properly pulled from teh existing database.
I'm getting "http.client.BadStatusLine: ''"
I've solved this using the firebase python lib found here: http://ozgur.github.io/python-firebase/
I used pip3 to install it. I just wish I could have done it the same way I do other REST calls instead of requiring a new lib.
full_data = []
from firebase import *
firebase = firebase.FirebaseApplication('https://secret_url.firebaseio.com/', None)
for id in id_list:
print(str(id))
url = 'from_url'
try:
result = firebase.put('/users/ ' + str(id) + '/', data=Post_And_Recieve_JSON(url, None, headers)["payload"], name='Data')
except:
print('Skipping a user')
Related
i have a list of google drive file links about 300 pdf files which i have to download
so what i am trying to do is using pythons request library i am requesting to google server and getting the files.
after 30 to 36 files download google blocks my requests and return
We're sorry...... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
i am using the following code
import requests
def download_file_from_google_drive(id, destination):
URL = "https://docs.google.com/uc?export=download"
session = requests.Session()
response = session.get(URL, params = { 'id' : id }, stream = True)
if response.status_code!=200:
print(response.status_code)
return response.status_code
print('downloading '+ destination)
token = get_confirm_token(response)
if token:
params = { 'id' : id, 'confirm' : token }
response = session.get(URL, params = params, stream = True)
save_response_content(response, destination)
def get_confirm_token(response):
for key, value in response.cookies.items():
if key.startswith('download_warning'):
return value
return None
def save_response_content(response, destination):
CHUNK_SIZE = 32768
with open(destination, "wb") as f:
i = 0
for chunk in response.iter_content(CHUNK_SIZE):
print(str(i)+'%')
i = i+1
if chunk: # filter out keep-alive new chunks
f.write(chunk)
print('downloaded '+ destination)
if __name__ == "__main__":
file_id = 'file id'
destination = file_id+'.pdf'
download_file_from_google_drive(file_id, destination)
i am iterating download_file_from_google_drive this function using my list
so can i bypass the security check
i tried using vpn which changes my ip address but nothing works.
after about 1hour downloading restart
I am writing a python script to fetch mail attachments through Graph API.
In the Graph Explorer, I can perfectly download file attachments by manually pressing the download button after calling:
https://graph.microsoft.com/v1.0/me/messages/{message-id}/attachments/{attachment-id}/$value
However, when trying to make the same request in my Python script, all I get returned is 'Response [200]' (so the request works, but the file is not reachable).
I try to make the request like this:
def get_mails_json():
requestHeaders = {'Authorization': 'Bearer ' +result["access_token"],'Content-Type': 'application/json'}
queryResults = msgraph_request(graphURI + "/v1.0/me/messages?$filter=isRead ne true",requestHeaders)
return json.dumps(queryResults)
try:
data = json.loads(mails)
values = data['value']
for i in values:
mail_id = i['id']
mail_subj = i['subject']
if i['hasAttachments'] != False:
attachments = o365.get_attachments(mail_id)
attachments = json.loads(attachments)
attachments = attachments['value']
for i in attachments:
details = o365.get_attachment_details(mail_id,i["id"])
except Exception as e:
print(e)
def get_attachment_details(mail,attachment):
requestHeaders = {'Authorization': 'Bearer ' + result["access_token"],'Content-Type': 'application/json'}
queryResults = msgraph_request(graphURI + "/v1.0/me/messages/"+mail+"/attachments/"+attachment+'/$value',requestHeaders)
return json.dumps(queryResults)
Is there a way for me to download the file to AT ALL through my python script ?
I found a simple solution to downloading a file through a python script!
I used chip's answer, found on this thread:
thread containing chip's answer
I make the request for the attachment like so:
def get_attachment_details(mail,attachment):
requestHeaders = {'Authorization': 'Bearer ' + result["access_token"],'Content-Type': 'application/file'}
resource= graphURI + "/v1.0/me/messages/"+mail+"/attachments/"+attachment+'/$value'
payload = {}
results = requests.request("GET", resource,headers=requestHeaders,data=payload, allow_redirects=False)
return results.content
This gets me the encoded bytes of the file, which I then decode and write to a file like so:
for i in attachments:
details = o365.get_attachment_details(mail_id,i["id"])
toread = io.BytesIO()
toread.write(details)
with open(i['name'], 'wb') as f:
f.write(toread.getbuffer())
I am trying to executing rest api get using python script.Below the code I am using.But api response doesn't have proper Json format I am getting error.
import requests
# api-endpoint
URL = "end point url"
# sending get request and saving the response as response object
r = requests.get(url = URL)
# extracting data in json format
data = r.json()
print(data)
My api response below
[{:mac_address=>"10:55", :Parameter=>"Device.Info", :status=>"Success", :response_code=>"200", :value=>"2.4Ghz"}]
I need to change response as below
[{"mac_address":"10:55","Parameter":"Device.Info","status":"Success","response_code":"200","value":"2.4Ghz"}]
How to achieve this in python? I am new to python.
test = '[{:mac_address=>"10:55", :Parameter=>"Device.Info", :status=>"Success", :response_code=>"200", :value=>"2.4Ghz"}]'
def to_json(val: str):
val = val.replace("{:", '{"').replace(" :", '"').replace("=>", '":')
return val
res = to_json(test)
print(res)
result:
[{"mac_address":"10:55","Parameter":"Device.Info","status":"Success","response_code":"200","value":"2.4Ghz"}]
in your case:
import requests
import json
def to_json(val: str):
val = val.replace("{:", '{"').replace(" :", '"').replace("=>", '":')
return val
# api-endpoint
URL = "end point url"
# sending get request and saving the response as response object
r = requests.get(url = URL)
# extracting data in string format
data = r.text
# converting string to json object
res = json.loads(to_json(data))
print(res)
Assuming you get a plain text response, you can manually re-format the response using String.replace():
response = '[{:mac_address=>"10:55", :Parameter=>"Device.Info", :status=>"Success", :response_code=>"200", :value=>"2.4Ghz"}]'
desired = {}
response = response[2:-3].replace('"', '').split(', ')
for r in response:
key, value = r.split('=>')
key = key.replace(':','')
desired[key] = value
I'm using a python script to send an API request to get the attachments of an email. The email I'm using has 4 attachments (plus pictures in the signature in the email). The python request only retrieves 1 attachment along with the pics from the signature. When using Postman with the exact same information, it retrieves all attachments along with the pics.
Any ideas on how I can get the other attachments?
import requests
url = 'https://graph.microsoft.com/v1.0/users/{{users email}}/messages/{{messageID}}/attachments'
body = None
head = {"Content-Type": "application/json;charset=UFT-8", "Authorization": "Bearer " + accessToken}
response1 = requests.get(url, data=body, headers=head)
response = response1.text
Below shows the response from the python script, with only 7 items, and the Postman response with 10 items.
Below code retrieves multiple attachments
(files being an array of attachment names)
def execute(accessToken, messageID, files, noAttachments):
import os
from os import path
import requests
import base64
import json
if noAttachments == "False":
url = 'https://graph.microsoft.com/v1.0/users/{{users email}}/messages/{{messageID}}/attachments'
body = {}
head = {"Authorization": "Bearer " + accessToken}
responseCode = requests.request("GET", url, headers=head, data = body)
response = responseCode.text
test = json.loads(responseCode.text.encode('utf8'))
x, contentBytes = response.split('"contentBytes":"',1)
if len(files) == 1:
imgdata = base64.b64decode(str(contentBytes))
filename = "C:/Temp/SAPCareAttachments/" + files[0]
with open(filename, 'wb') as f:
f.write(imgdata)
else:
for file in test["value"]:
imgdata = base64.b64decode(file["contentBytes"])
if file["name"] in files:
filename = "C:/Temp/" + file["name"]
with open(filename, 'wb') as f:
f.write(imgdata)
print(responseCode)
I am downloading files from One Drive with the following piece of code:
if response.status_code == 200:
print('\n> Response Success')
with open('Book2.xlsx', 'wb') as File:
File.write(response.content)
print('\n> File Downloaded')
else:
print('\n> Failed:', response.status_code)
print(response.content)
The code is from:This post here
The "File" is coming from One Drive with the use of the following code:
import sys, os, time, requests
import pandas as pd
import urllib.parse
OneDrive_FilePath = 'Book2.xlsx'
OneDrive_FileURL = 'https://graph.microsoft.com/v1.0/me/drive/root:/' + OneDrive_FilePath + ':/content'
OneDrive_FileURL = urllib.parse.quote(OneDrive_FileURL, safe=':/')
print(OneDrive_FileURL)
Client_Id = 'XXXX'
Tenant_Id = 'YYYYY'
Refresh_Token_First = 'ZZZZZ'
PostStr = {'grant_type': 'refresh_token', 'client_id': Client_Id, 'refresh_token': Refresh_Token_First}
Token_Response = requests.post('https://login.microsoftonline.com/' + Tenant_Id + '/oauth2/v2.0/token', data=PostStr)
Access_Token = Token_Response.json()['access_token']
New_Refresh_Token = Token_Response.json()['refresh_token']
if Access_Token is None or New_Refresh_Token is None:
print('\n> Failed: Access_Token NOT Retrieved')
sys.exit()
Response = requests.get(OneDrive_FileURL, headers={'Authorization': 'Bearer ' + Access_Token})
The "File" which is getting downloaded is in the form of "io.BufferedWriter". I want to actually load the "File" as a data frame so that I can do certain operations on that and upload it to AWS.
How can I do that, please help.
Thanks
As explained in the comments, the type of File is mostly irrelevant. All you need to do is read the excel directly from the response, as in :
url = "https://go.microsoft.com/fwlink/?LinkID=521962"
res = requests.get(url)
pd.read_excel(res.content)
The last statement produces a regular pandas dataframe. You can use that however you want.