We Keep Coding

Create a temp tar.gz file in azure function and write ftp server file content on that temp file in python

I have a task requirement of reading a tar.gz file from a ftp server and store it on a blob storage.
How I think I can accomplish is that I must first create a temp file in azure function temp directory, write all content on it, close it and then upload it on the blob storage.
What I have done so far is:
fp = tempfile.NamedTemporaryFile()
filesDirListInTemp = listdir(tempFilePath)
logging.info(filesDirListInTemp)
try:
with open('/tmp/fp', 'w+') as fp:
data = BytesIO()
save_file = ftp.retrbinary('RETR '+ filesDirListInTemp, data.write, 1024)
data.seek(0)
blobservice=BlobClient.from_connection_string(conn_str=connection_string,container_name=container_name,blob_name=filename,max_block_size=4*1024*1024,max_single_put_size=16*1024*1024)
blobservice.upload_blob(gzip.decompress(data.read()))
print("File Uploaded!")
except Exception as X:logging.info(X)
But I am getting error as: expected str, bytes or os.PathLike object, not list.
Please tell me what I am doing wrong here?

Opening a Django InMemoryUploadedFile leads to an I/O operation error?

I am attempting to upload a file to a bucket using Django. The file goes in from the view, is cleaned, and is then uploaded to s3 using a utility function in the model.
heres the upload function:
def upload_file(file_obj, object_name=None):
"""Upload a file to an S3 bucket
:param file_obj: File to upload
:param object_name: S3 object name. If not specified then file_obj is used
:return: True if file was uploaded, else False
"""
bucket = settings.BUCKET
# If S3 object_name was not specified, use file_obj
if object_name is None:
object_name = file_obj
s3 = boto3.client('s3')
try:
s3.upload_fileobj(file_obj.open(), bucket, object_name)
except ClientError as e:
logging.error(e)
return False
return True
the error returned is:
File "/opt/profiles/profiles/core/utils.py", line 217, in upload_file
s3.upload_fileobj(file_obj.open(), bucket, object_name)
File "/usr/local/lib/python3.8/site-packages/django/core/files/uploadedfile.py", line 90, in open
self.file.seek(0)
ValueError: I/O operation on closed file.
Now whats weird to me is that the error comes from file_obj.open(), whats the point of an open method that cant open a closed file? I don't want to pass the request.file straight from the view as I have in a property setter, and would like it to fire off whenever the property is changed in the model but I simply cannot figure out what the correct process should be.

Well, part of the error at least comes from setting two file fields. As soon as you set one, the file object is done, essentially.

Python FileNotFoundError when reading a public json file

I have a public viewable JSON file that is hosted on s3. I can access the file directly by clicking on the public url to the s3 object and view the JSON in my browser fine. Anyone on the internet can view the file easily.
Yet with the below code is ran in Python (using Lambda connected to an API trigger) I get [Errno 2] No such file or directory: as the errorMessage, and FileNotFoundError as the errorType.
def readLeetDictionary(self):
jsonfile = 'https://s3.amazonaws.com/url/to/public/facing/file/data.json'
with open(jsonfile, 'r') as data_file:
self.data = json.load(data_file)
What am I missing here? Since the file is a publicly viewable JSON file I would assume I wouldn't be forced to use boto3 library and formally handshake to the file in order to read the file (with object = s3.get_object(Bucket=bucket_names,Key=object_name) for example) - would I?

The conde you need should be something like:
import urllib, json
def readLeetDictionary():
jsonfile = 'https://s3.amazonaws.com/url/to/public/facing/file/data.json'
response = urllib.urlopen(jsonfile)
data = json.loads(response.read())
print data
Please feel free to ask further or explain if this does not suit you.

Writing a file to S3 using Lambda in Python with AWS

In AWS, I'm trying to save a file to S3 in Python using a Lambda function. While this works on my local computer, I am unable to get it to work in Lambda. I've been working on this problem for most of the day and would appreciate help. Thank you.
def pdfToTable(PDFfilename, apiKey, fileExt, bucket, key):
# parsing a PDF using an API
fileData = (PDFfilename, open(PDFfilename, "rb"))
files = {"f": fileData}
postUrl = "https://pdftables.com/api?key={0}&format={1}".format(apiKey, fileExt)
response = requests.post(postUrl, files=files)
response.raise_for_status()
# this code is probably the problem!
s3 = boto3.resource('s3')
bucket = s3.Bucket('transportation.manifests.parsed')
with open('/tmp/output2.csv', 'rb') as data:
data.write(response.content)
key = 'csv/' + key
bucket.upload_fileobj(data, key)
# FYI, on my own computer, this saves the file
with open('output.csv', "wb") as f:
f.write(response.content)
In S3, there is a bucket transportation.manifests.parsed containing the folder csv where the file should be saved.
The type of response.content is bytes.
From AWS, the error from the current set-up above is [Errno 2] No such file or directory: '/tmp/output2.csv': FileNotFoundError. In fact, my goal is to save the file to the csv folder under a unique name, so tmp/output2.csv might not be the best approach. Any guidance?
In addition, I've tried to use wb and w instead of rb also to no avail. The error with wb is Input <_io.BufferedWriter name='/tmp/output2.csv'> of type: <class '_io.BufferedWriter'> is not supported. The documentation suggests that using 'rb' is the recommended usage, but I do not understand why that would be the case.
Also, I've tried s3_client.put_object(Key=key, Body=response.content, Bucket=bucket) but receive An error occurred (404) when calling the HeadObject operation: Not Found.

Assuming Python 3.6. The way I usually do this is to wrap the bytes content in a BytesIO wrapper to create a file like object. And, per the boto3 docs you can use the-transfer-manager for a managed transfer:
from io import BytesIO
import boto3
s3 = boto3.client('s3')
fileobj = BytesIO(response.content)
s3.upload_fileobj(fileobj, 'mybucket', 'mykey')
If that doesn't work I'd double check all IAM permissions are correct.

You have a writable stream that you're asking boto3 to use as a readable stream which won't work.
Write the file, and then simply use bucket.upload_file() afterwards, like so:
s3 = boto3.resource('s3')
bucket = s3.Bucket('transportation.manifests.parsed')
with open('/tmp/output2.csv', 'w') as data:
data.write(response.content)
key = 'csv/' + key
bucket.upload_file('/tmp/output2.csv', key)

Upload a file using boto

import boto
conn = boto.connect_s3('', '')
mybucket = conn.get_bucket('data_report_321')
I can download the file from a bucket using the following code.
for b in mybucket:
print b.name
b.get_contents_to_filename('0000_part_00', headers=None, cb=None, num_cb=10, torrent=False, version_id=None, res_download_handler=None, response_headers=None)
But I am not able to upload a file. I get an error:
AttributeError: 'str' object has no attribute 'tell'
send_file nor set_contents functions are working as expected.
for b in mybucket:
b.send_file('mytest.txt', headers=None, cb=None, num_cb=10, query_args=None, chunked_transfer=False, size=None)
b.set_contents_from_file('mytest.txt', headers=None, replace=True, cb=None, num_cb=10, policy=None, md5=None, reduced_redundancy=False, query_args=None, encrypt_key=False, size=None, rewind=False)
How do I upload a file from current directory of local server to S3 bucket using boto?
Update:
I need to declare the key (filename) of the uploaded file first before calling the set_contents_from_filename function.
k = boto.s3.key.Key(mybucket)
k.key = 'uploaded_file.txt'
k.set_contents_from_filename("myteswt.txt")

Both send_file and set_contents_from_file take a File object for first argument. If you would like to pass a string you should see set_contents_from_filename.

You could use this utility script: https://github.com/TimelyToga/upload_s3
python upload_s3.py <filename> [key_to_upload_as]
That will upload any file to a s3 bucket that you specify.
Although, when you are uploading set_contents_from_file takes a python File object.