I want to upload my file into my drive. However in Pydrive Documentation I found only upload() function that uploads a file created by drive.CreateFile() function and update it, and not the file in my hard drive (my own file).
file1 = drive.CreateFile({'title': 'Hello.txt'}) # Create GoogleDriveFile
instance with title 'Hello.txt'.
file1.SetContentString('Hello World!') # Set content of the file from given
string.
file1.Upload()
I've tried the ansewers of my question here in stackoverflow, but an error accured . here is my code :
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
#1st authentification
gauth = GoogleAuth()
gauth.LocalWebserverAuth() # Creates local webserver and auto handles
#authentication.
drive = GoogleDrive(gauth)
file1 = drive.CreateFile(metadata={"title": "big.txt"})
file1.SetContentFile('big.txt')
file1.Upload()
The file "big.txt" is in the same folder of my code file.
When I run it, I got this traceback:
Traceback (most recent call last):
File "C:\Users\**\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\pydrive\files.py", line 369, in _FilesInsert
http=self.http)
File "C:\Users\**\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\oauth2client\_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Users\**\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\googleapiclient\http.py", line 813, in execute
_, body = self.next_chunk(http=http, num_retries=num_retries)
File "C:\Users\**\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\oauth2client\_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Users\**\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\googleapiclient\http.py", line 981, in next_chunk
return self._process_response(resp, content)
File "C:\Users\**\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\googleapiclient\http.py", line 1012, in _process_response
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting
https://www.googleapis.com/upload/drive/v2/files?
alt=json&uploadType=resumable returned "Bad Request">
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/**/AppData/Local/Programs/Python/Python36-
32/quickstart.py", line 13, in <module>
file1.Upload()
File "C:\Users\**\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\pydrive\files.py", line 285, in Upload
self._FilesInsert(param=param)
File "C:\Users\**\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\pydrive\auth.py", line 75, in _decorated
return decoratee(self, *args, **kwargs)
File "C:\Users\**\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\pydrive\files.py", line 371, in _FilesInsert
raise ApiRequestError(error)
pydrive.files.ApiRequestError: <HttpError 400 when requesting
https://www.googleapis.com/upload/drive/v2/files?
alt=json&uploadType=resumable returned "Bad Request">
You have to set the content with SetContentFile() instead of SetContentString():
file1 = drive.CreateFile({'title': 'Hello.txt'})
file1.SetContentFile(path_to_your_file)
file1.Upload()
As the documentation states, if you haven't set the title and mimeType they will be set automatically from the name and type of the file your give. Therefore if you want to upload the file with the same name it already has on your computer you can do:
file1 = drive.CreateFile()
file1.SetContentFile(path_to_your_file)
file1.Upload()
Regarding your second point, as far as I'm aware GDrive can not convert a file to a different format.
Based on the documentation of PyDrive, I would say, you need to do the following:
file_path = "path/to/your/file.txt"
file1 = drive.CreateFile()
file1.SetContentFile(file_path)
file1.Upload()
Title and content type metadata are set automatically based on the provided file path. If you want to provide a different filename, pass it to CreateFile() like this:
file1 = drive.CreateFile(metadata={"title": "CustomFileName.txt"})
Related
I have the following code snippet:
import firebase_admin
from firebase_admin import credentials
from firebase_admin import storage
from google.cloud import storage
class firebase_storage():
def __init__(self, path_to_sak, root_bucket):
try:
self.cred = credentials.Certificate(path_to_sak)
firebase_admin.initialize_app(self.cred)
except Exception as e:
print("Firebase App may have already been initialized")
self.bucket = firebase_admin.storage.bucket(root_bucket)
def upload(self, key, file_path):
blob = storage.Blob(key, self.bucket)
blob.upload_from_filename(file_path)
def download(self, key, file_path):
blob = storage.Blob(key, self.bucket)
blob.download_to_filename(file_path)
def upload_string(self, key, string, mime_type):
blob = storage.Blob(key, self.bucket)
blob.upload_from_string(string, content_type=mime_type)
I'm using Firebase Emulators for Storage, I have verified that downloads work using the method call firebase_storage.download().
However, when I try to call upload() the following exception is thrown:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 2348, in upload_from_file
created_json = self._do_upload(
File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 2170, in _do_upload
response = self._do_multipart_upload(
File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 1732, in _do_multipart_upload
response = upload.transmit(
File "/usr/local/lib/python3.8/dist-packages/google/resumable_media/requests/upload.py", line 149, in transmit
self._process_response(response)
File "/usr/local/lib/python3.8/dist-packages/google/resumable_media/_upload.py", line 116, in _process_response
_helpers.require_status_code(response, (http_client.OK,), self._get_status_code)
File "/usr/local/lib/python3.8/dist-packages/google/resumable_media/_helpers.py", line 99, in require_status_code
raise common.InvalidResponse(
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "boot.py", line 55, in <module>
run()
File "boot.py", line 35, in run
fb_storage.upload(key, file)
File "/root/python_db_client/src/firebase_storage.py", line 20, in upload
blob.upload_from_filename(file_path)
File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 2475, in upload_from_filename
self.upload_from_file(
File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 2364, in upload_from_file
_raise_from_invalid_response(exc)
File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 3933, in _raise_from_invalid_response
raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.BadRequest: 400 POST http://myserver.com:9194/upload/storage/v1/b/xxxxxx.appspot.com/o?uploadType=multipart: Bad Request: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>)
My storage.rules look like this:
rules_version = '2';
service firebase.storage {
match /b/{bucket}/o {
match /{allPaths=**} {
allow write, read: if true;
}
}
}
And so, it would appear that public read/write access is allowed.
Everything is working, I have other emulators (Firestore, Auth) that is working fine, but Storage uploads refuse to work :(
Any help would be greatly appreciated thank you!
Maybe there is a problem initializing your app. I see your are taking granted that the app is initialized if there is an error while initializing. Try checking connection first! It may help...
Python Admin SDK does not currently support the Storage emulator according to the documentation
https://firebase.google.com/docs/emulator-suite/install_and_configure#admin_sdk_availability
I'm trying to upload a file into GCS, but I'm running into a permission issue which I'm not sure how to resolve. Reading a file from a bucket in GCS doesn't seem to be an issue. However, I'm getting issues for upload.
client = storage.Client()
bucket = client.get_bucket('fda-drug-label-data')
blob = bucket.get_blob(f'fda-label-doc-links.csv')
bt = blob.download_as_string()
s = str(bt, 'utf-8')
s = StringIO(s)
df = pd.read_csv(s)
df_doc_links = list(df['Link'])
a = pd.DataFrame([len(df_doc_links)])
a.to_csv('test.csv', index=False)
client = storage.Client()
bucket = client.get_bucket('fda-drug-label-data')
blob = bucket.blob('test.csv')
blob.upload_from_filename('test.csv')
This is the message I'm getting:
Traceback (most recent call last): File "/home/.../.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1567, in upload_from_file
if_metageneration_not_match, File "/home/.../.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1420, in _do_upload
if_metageneration_not_match, File "/home/.../.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1098, in _do_multipart_upload
response = upload.transmit(transport, data, object_metadata, content_type) File "/home/.../.local/lib/python3.7/site-packages/google/resumable_media/requests/upload.py", line 108, in transmit
self._process_response(response) File "/home/.../.local/lib/python3.7/site-packages/google/resumable_media/_upload.py", line 109, in _process_response
_helpers.require_status_code(response, (http_client.OK,), self._get_status_code) File "/home/.../.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 96, in require_status_code
*status_codes google.resumable_media.common.InvalidResponse: ('Request failed with status code', 403, 'Expected one of', <HTTPSta tus.OK: 200>) During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "scrape.py", line 134, in <module>
blob.upload_from_filename('test.csv') File "/home/.../.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1655, in upload_from_filename
if_metageneration_not_match=if_metageneration_not_match, File "/home/.../.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1571, in upload_from_file
_raise_from_invalid_response(exc) File "/home/.../.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 2620, in _raise_from_invalid_response
raise exceptions.from_http_status(response.status_code, message, response=response) google.api_core.exceptions.Forbidden: 403 POST https://storage.googleapis.com/upload/storage/v1/b/fda-drug-label-da ta/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>)
You don't have permission to upload to the data in your service account.Go to IAM and Admin section and under service accounts assign permission role to your account.After that generate the KEY again.
While downloading azure blobs to local file system, I'm getting the following exception:
Client-Request-ID=99bdb0e4-2d1c-11e8-8fe6-00155dbf7128 Retry policy did not allow for a retry: Server-Timestamp=Wed, 21 Mar 2018 15:29:09 GMT, Server-Request-ID=1e7ab8f5-101e-0076-5329-c16a24000000, HTTP status code=404, Exception=The specified blob does not exist.
ErrorCode: BlobNotFound<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:1e7ab8f5-101e-0076-5329-c16a24000000Time:2018-03-21T15:29:09.6565984Z</Message></Error>.
Traceback (most recent call last):
File "C:\Program Files\Commvault\ContentStore\Automation\CloudApps\CloudAppsUtils\cahelper.py", line 483, in download_contents_azure
session.get_blob_to_path(container_name,fl,fl)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 1817, in get_blob_to_path
timeout)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 2003, in get_blob_to_stream
raise ex
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 1971, in get_blob_to_stream
_context=operation_context)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 1695, in _get_blob
operation_context=_context)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\storageclient.py", line 354, in _perform_request
raise ex
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\storageclient.py", line 289, in _perform_request
raise ex
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\storageclient.py", line 275, in _perform_request
HTTPError(response.status, response.message, response.headers, response.body))
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\_error.py", line 111, in _http_error_handler
raise AzureHttpError(message, http_error.status)
azure.common.AzureMissingResourceHttpError: The specified blob does not exist.ErrorCode: BlobNotFound
<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:1e7ab8f5-101e-0076-5329-c16a24000000
Time:2018-03-21T15:29:09.6565984Z</Message></Error>
I'm using the following code to download blobs:
def download_contents_azure(self, account_name, account_key, content):
session=self.create_session_azure(account_name,account_key)
os.mkdir('in place')
os.chdir('in place')
for item in content:
# s = os.path.basename(item)
path_to_file = ("/".join(item.strip("/").split('/')[1:]))
container_name = Path(item).parts[1]
gen = session.list_blobs(container_name)
li = []
for i in gen:
li.append(i.name)
if path_to_file in li:
fl = os.path.basename(path_to_file)
print(fl)
c = self.splitall(item)
for i in range(1,len(c)-1):
if path.exists(c[i]) is False:
os.mkdir(c[i])
os.chdir(c[i])
session.get_blob_to_path(container_name,fl,fl)
for i in range(1,len(c)-1):
os.chdir("..")
else:
c = self.splitall(item)
for i in range(1,len(c)):
os.mkdir(c[i])
os.chdir(c[i])
generator = session.list_blobs(container_name,path_to_file+'/',delimiter='/')
for blob in generator:
bl = os.path.basename(blob.name)
session.get_blob_to_path(container_name,bl,bl)
I've a path in azure (/container/folder/subfolder).
I'm trying to download the structure and all the files under subfolder. the first file under the subfolder gets downloaded and then I've the above exception. Due to this, I'm unable to loop through and print the next items.
Thoughts?
Double check your line
session.get_blob_to_path(container_name,fl,fl)
You have fl for blob_name.
documentation here indicates second argument is blob_name while third is file path on file system where to download.
I think your blob name is wrong. Hence it does not exist.
NOTE: consider installing fiddler so you can look at the network traces. It allows you to see the raw request.
I am trying to get a .csv file stored in an S3 bucket. The CSV is being uploaded by a Mac compiler to the S3 bucket and my code (python 2.7) is running in a Unix environment. CSV looks like this (I have included carriage return character):
Order,Item,Date,Quantity\r
1,34975,8/4/15,10\r
2,921644,3/10/15,2\r
3,N18DAJ,1/7/15,10\r
4,20816,12/12/15,9\r
My code to get the file from the s3 bucket:
import boto3
def readcsvFromS3(bucket_name, key):
s3 = boto3.resource('s3')
obj = s3.Object(bucket_name=bucket_name, key=key)
response = obj.get()
data = response['Body'].read()
Error is happening on the response = obj.get() line. And the error I'm getting is:
Traceback (most recent call last):
File "slot.py", line 163, in <module>
columnNames, rowArray = neo.readcsvFromS3(bucket_name=config.s3bucket, key=config.orde
File "/home/jcgarciaram/WMSight/wmsight-api/api/utilities/pythonScripts/slotting/neo4jUt
response = obj.get()
File "/usr/local/lib/python2.7/dist-packages/boto3/resources/factory.py", line 481, in d
response = action(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/boto3/resources/action.py", line 83, in __c
response = getattr(parent.meta.client, operation_name)(**params)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 228, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 481, in _make_api
operation_model, request_dict)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 117, in make_re
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 144, in _send_r
request, operation_model, attempts)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 203, in _get_re
parser.parse(response_dict, operation_model.output_shape)),
File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 208, in parse
parsed = self._do_parse(response, shape)
File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 570, in _do_pars
member_shapes, final_parsed)
File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 626, in _parse_n
member_shape, headers[header_name])
File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 226, in _parse_s
return handler(shape, node)
File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 149, in _get_tex
return func(self, shape, text)
File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 380, in _handle_
return self._timestamp_parser(text)
File "/usr/local/lib/python2.7/dist-packages/botocore/utils.py", line 344, in parse_time
raise ValueError('Invalid timestamp "%s": %s' % (value, e))
ValueError: Invalid timestamp "Wed, 16 Jan 48199 20:37:02 GMT": year is out of range
I have been researching all over but can't seem to figure out the issue. Any ideas?
After days of searching and debugging we were able to finally determine the cause of the issue. We tried uploading the files in a JSON format rather than CSV format and imagine our surprise when we saw the same error when trying to download the file using boto3 in Python.
We started looking then at the properties of the files themselves in S3 (right-click on file and click on Properties) rather than the content.
We found a section called Metadata and found the following entry:
Key: Expires / Value: Tue, 15 Jan 48199 02:16:52 GMT.
After changing the year of the value to a date such as 2200, everything worked fine! We are now looking into our upload process in Node.js to see how we can make sure that this value is set correctly.
I am trying to check if a certain dataset exists in bigquery using the Google Api Client in Python. It always worked untill the last update where I got this strange error I don't know how to fix:
Traceback (most recent call last):
File "/root/miniconda/lib/python2.7/site-packages/dsUtils/bq_utils.py", line 106, in _get
resp = bq_service.datasets().get(projectId=self.project_id, datasetId=self.id).execute(num_retries=2)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "/root/miniconda/lib/python2.7/site-packages/googleapiclient/http.py", line 755, in execute
method=str(self.method), body=self.body, headers=self.headers)
File "/root/miniconda/lib/python2.7/site-packages/googleapiclient/http.py", line 93, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 598, in new_request
self._refresh(request_orig)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 864, in _refresh
self._do_refresh_request(http_request)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 891, in _do_refresh_request
body = self._generate_refresh_request_body()
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 1597, in _generate_refresh_req
uest_body
assertion = self._generate_assertion()
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/service_account.py", line 263, in _generate_ass
ertion
key_id=self._private_key_id)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/crypt.py", line 97, in make_signed_jwt
signature = signer.sign(signing_input)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/_pycrypto_crypt.py", line 101, in sign
return PKCS1_v1_5.new(self._key).sign(SHA256.new(message))
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Signature/PKCS1_v1_5.py", line 112, in sign
m = self._key.decrypt(em)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 174, in decrypt
return pubkey.pubkey.decrypt(self, ciphertext)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/pubkey.py", line 93, in decrypt
plaintext=self._decrypt(ciphertext)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 235, in _decrypt
r = getRandomRange(1, self.key.n-1, randfunc=self._randfunc)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Util/number.py", line 123, in getRandomRange
value = getRandomInteger(bits, randfunc)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Util/number.py", line 104, in getRandomInteger
S = randfunc(N>>3)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 202, in read
return self._singleton.read(bytes)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 178, in read
return _UserFriendlyRNG.read(self, bytes)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 137, in read
self._check_pid()
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 153, in _check_pid
raise AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
AssertionError: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()
Is someone understanding what is hapening?
Note that I also get this error with other bricks like GCStorage.
Note also that I use the following command to load my Google credentials:
from oauth2client.client import GoogleCredentials
def get_credentials(credentials_path): #my json credentials path
logger.info('Getting credentials...')
try:
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_path
credentials = GoogleCredentials.get_application_default()
return credentials
except Exception as e:
raise e
So if anyone know a better way to load my google credentials using my json service account file, and which would avoid the error, please tell me.
It looks like the error is in the PyCrypto module, which appears to be used under the hood by Google's OAuth2 implementation. If your code is calling os.fork() at some point, you may need to call Crypto.Random.atfork() afterward in both the parent and child process in order to update the module's internal state.
See here for PyCrypto docs; search for "atfork" for more info:
https://github.com/dlitz/pycrypto
This question and answer might also be relevant:
PyCrypto : AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")