I have a Django Web application and i deployed it to Elastic Beanstalk environment. I also have the numpy-stl package. I'm trying to get a .stl file from Amazon S3 bucket and use this file with a stl package's function but i'm getting an error such as 'bytes' object has no attribute 'get_mass_properties'.
My code is;
s3 = boto3.client('s3')
obj = s3.get_object(Bucket=bucket_name, Key=key)
body = obj['Body'].read()
volume, cog, inertia = body.get_mass_properties()
How can i get the .stl file and use it?
Assuming that you are talking about this stl file format, once you read it in into python from S3, you need some python library to open it.
Quick search returns numpy-stl:
Simple library to make working with STL files (and 3D objects in general) fast and easy.
Thus you can install that library and attempt to use it on the file you are downloading.
In case you run your code on lambda (not written in your question?) then you would have to bundle the library with your deployment package or construct custom lambda layer for that.
I have fixed such as below.
import stl
import boto3
import tempfile
s3 = boto3.resource('s3', region_name=region)
bucket = s3.Bucket(bucket)
obj = bucket.Object(uploadedVolume)
tmp = tempfile.NamedTemporaryFile()
with open(tmp.name, 'wb') as f:
obj.download_fileobj(f)
stlMesh = stl.mesh.Mesh.from_file(tmp.name)
volume, cog, inertia = stlMesh.get_mass_properties()
Related
I have a really large 7z file in s3 bucket say s3://tempbucket1/Test_For7zip.7z that runs into several tens of GB. I do not want to download it, unzip it and re upload it back to s3. I want to use Boto3 to unzip it on the fly and save it into S3.
I tried to solve this using lzma package based on Previous SO answer which dealt with on the fly unzipping of *.zip files using the fileobj option present in gzip.GzipFile.
from io import BytesIO
import gzip
import lzma
import boto3
# setup constants
bucket = 'tempbucket1'
gzipped_key = 'Test_For7zip.7z'
uncompressed_key = 'Test_Unzip7zip'
# initialize s3 client, this is dependent upon your aws config being done
s3 = boto3.client('s3', use_ssl=False)
s3.upload_fileobj( # upload a new obj to s3
Fileobj=lzma.LZMAFile(
BytesIO(s3.get_object(Bucket=bucket,
Key=gzipped_key)['Body'].read()),
'rb'), # read binary
Bucket=bucket, # target bucket, writing to
Key=uncompressed_key) # target key, writing to
However, this thows the following error
LZMAError: Input format not supported by decoder
Is there a python package that provides can decode 7z files based on BytesIO, or is there a better way of achieving this?
I never tried this, but Googling gave me this as a possible solution. Please reach out through this post if this solves your problem.
Python has the ability to create a tempfile as a context manager. Rather than having to make a directory myself, along with the path, and then cleaning it up when done, it would be better to use this tempfile.
Is there any support in the boto3 client to download, from s3, to a tempfile?
Try the download_fileobj method which accepts a file-like object and requires binary mode, for example:
import boto3
import tempfile
s3 = boto3.client('s3')
with tempfile.TemporaryFile(mode='w+b') as f:
s3.download_fileobj('mybucket', 'mykey', f)
Using AWS glue and AWS s3 and s3fs, I've come up with the following (among other attempts). I even see the examples at https://s3fs.readthedocs.io/en/latest/ but they're not getting me there.
key = 'https://s3.console.aws.amazon.com/s3/buckets/datalake-sbox-landing-55555-us-weast/cloudwatch_logs/prod-coarsair/dt%267D2019-09-11/144221-3837-462d-a2e6-ba300/coarsair-coarsair-144221-3837-462d-a2e6-ba300/000000.gz'
s3 = s3fs.S3FileSystem()
with s3.open(key, 'r') as f:
args_gz_file = f
Then it should be as easy as typing "args_gz_file" now, right? Nope. Where am I going wrong?!
The samples show:
with fs.open('my-bucket/my-file.txt', 'rb') as f:
This suggests that the first parameter is BUCKET-NAME/KEY.
However, your code sample shows a URL (https://s3.console.aws.amazon.com/s3/buckets/) instead of a bucket name.
Perhaps try:
key = 'datalake-sbox-landing-55555-us-weast/cloudwatch_logs/prod-coarsair/dt%267D2019-09-11/144221-3837-462d-a2e6-ba300/coarsair-coarsair-144221-3837-462d-a2e6-ba300/000000.gz'
Side-note: It is recommended to use the official AWS SDK or the AWS CLI to access Amazon S3, rather than using s3fs. Amazon S3 is an object storage service, not a filesystem.
I have a python function that prints some stuff. I am gonig to put that to AWS Lambda and would like to print these stuff directly to a file in S3 bucket.
So something to redirect stdout in S3 bucket file.
This is how I call the fuction:
recurse_for_values(top_vault_prefix, top_level_keys)
Almost the same as Thomas L., but with io so that you don't actually write any file locally (everything is in memory).
import io
import tinys3
f=io.StringIO(u"some initial text data")
conn = tinys3.Connection('yourAccessKey', 'yourSecretKey', tls=True)
conn.upload('yourS3Key', f, 'yourBucketName')
f.close()
OR
you could use something like S3FS (https://github.com/s3fs-fuse/s3fs-fuse) to mount your S3 bucket as a disk and then simply redirect the output as you would do for a local disk, but I heavily discourage this option. S3 is definitely not a filesystem and shouldn't be used as such. For example it generates loads of requests to S3 (and thus costs) and may not be that reliable...
You can write your logs to a local file and then synchronise it with AWS S3 at the end of your script.
import tinys3
log_file_object = open(“logfile”, “w”)
log_file_object.write("Some logs...")
log_file_object.write("Some other logs...")
log_file_object.close()
conn = tinys3.Connection('S3_ACCESS_KEY','S3_SECRET_KEY',tls=True)
conn.upload('log_file_DDMMYYYY',f,'my_bucket')
You can also use boto3 to update your file but tinys3 is easier to use for tiny usage.
Hope it can help.
I have a zip file loaded into memory (do not have it persisted on disk). The zip file contains jpg images. I am trying to upload each jpg into s3 but am getting an error.
# already have an opened zipfile stored in zip_file
# already connected to s3
files = zip_file.namelist()
for f in files:
im = io.BytesIO(zip_file.read(f))
s3_key.key = f
s3_key.set_contents_from_stream(im)
I get the following error:
BotoClientError: BotoClientError: s3 does not support chunked transfer
What am I doing wrong?
Here is the solution. I was over thinking the problem.
files = zip_file.namelist()
for f in files:
data = zip_file.read(f)
s3_key._key.key = f
s3_key._key.set_contents_from_string(data)
That's all it took.
Boto supports other storage services, such as Google Cloud Storage, in addition to S3. The set_contents_from_stream method only works for services that support chunked transfer (see https://codereview.appspot.com/4515170). S3 does not support that (See their Technical FAQs at http://aws.amazon.com/articles/1109.)
It's unfortunate, but you can't upload from a stream to S3.