We Keep Coding

Why doesn't the password open my zip file in s3 when passed as a bytes object in python?

I have a small but mysterious and unsolvable problem using python to open a password protected file in an AWS S3 bucket.
The password I have been given is definitely correct and I can download the zip to Windows and extract it to reveal the csv data I need.
However I need to code up a process to load this data into a database regularly.
The password has a pattern like this (includes mixed case letters, numbers and a single "#"):-
ABCD#Efghi12324567890
The code below works with other zip files I place in the location with the same password:-
import boto3
import pyzipper
from io import BytesIO
s3_resource = boto3.resource('s3', aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
zip_obj = s3_resource.Object(bucket_name=my_bucket, key=my_folder + my_zip)
buffer = BytesIO(zip_obj.get()["Body"].read())
z = pyzipper.ZipFile(buffer)
my_newfile=z.namelist()[0]
s3_resource.meta.client.upload_fileobj(
z.open(my_newfile, pwd=b"ABCD#Efghi12324567890"), #HERE IS THE OPEN COMMAND
Bucket=my_bucket,
Key=my_folder + my_newfile)
I am told the password is incorrect:-
RuntimeError: Bad password for file 'ThisIsTheFileName.csv'
I resorted to using pyzipper rather than zipfile, since zipfile didn't support the compression method of the file in question:-
That compression method is not supported
In 7-zip I can see the following for the zip file:-
Method: AES-256 Deflate
Characteristics: WzAES: Encrypt
Host OS: FAT
So to confirm:-
-The password is definitely correct (can open it manually)
-The code seems ok - it opens my zip files with the same password
What is the issue here please and how do I fix it?
You would have my sincere thanks!
Phil

With some help from a colleague and a useful article, I now have this working.
Firstly as per the compression type, I have found it necessary to use the AESZipFile() method of pyzipper (although this method also seemed to work on other compression types).
Secondly the AESZipFile() method apparently accepts a BytesIO object as well as a file path, presumably because this is what it sees when it opens the file.
Therefore the zip file can be extracted in situ without having to download it first.
This method creates the pyzipper object which you can then read by specifying the file name and the password.
The final code looks like this:-
import pyzipper
import boto3
from io import BytesIO
my_bucket = ''
my_folder = ''
my_zip = ''
my_password = b''
aws_access_key_id=''
aws_secret_access_key=''
s3 = boto3.client('s3', aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
s3_file = s3.get_object(Bucket=my_bucket, Key=my_folder + my_zip)
s3_iodata = BytesIO(s3_file['Body'].read())
f = pyzipper.AESZipFile(s3_iodata)
my_file = f.namelist()[0]
file_content = f.read(my_file, pwd = my_password)
response = s3.put_object(Body=file_content, Bucket=my_bucket, Key=my_folder + my_file)
Here is an article that was useful:-
https://www.linkedin.com/pulse/extract-files-from-zip-archives-in-situ-aws-s3-using-python-tom-reid
I hope this is helpful to someone,
Phil

Import JSON Files from an entire directory into a MongoDB via a Python script

I want to import a directory with multiple sub directories and a lot of JSON-files into a MongoDB via a python script. However I can only import multiple JSON via GUI in Compass or one file at a time using a script using the following code I gathered from another question at stackoverflow(How to import JSON file to MongoDB using Python):
import json
from pymongo import MongoClient
client = MongoClient('localhost', 27017)
db=client['acme']
collection_posts = db ['posts']
with open('9995-f0763044.json') as f:
file_data = json.load(f)
collection_posts.insert_one(file_data)
client.close()
How can I change this so I can loop through an entire directory and import all of the JSON files? I have seen the insert_many() method but as far I understood it the specific filenames still have to be written into the code. In my perfect scenario I would just enter a directory in the script and it will scan and upload all the JSON-files in that directory. Is this even possible? Thanks for your help

something like this?
import glob
filelist = glob.glob('your/path/*.json')
for filename in filelist:
with open(filename) as f:
file_data = json.load(f)
collection_posts.insert_one(file_data)
client.close()

How to use Popen to read a json file in subprocess as there is a 33766 limit in terminal

I am using this code to read a json file in subprocess. It does work for only small jsons, If it exceeds over 33766 count. it will show a error showing
FileNotFoundError: [WinError 206] The filename or extension is too long.
this is beccause of exceeding 33766 count. so how to read the json file using popen .Read that this can solve the problem. Help me with suggestions. I am new here :\
import subprocess
import json
import os
from pprint import pprint
auth = "authorization: token 1234
file = "jsoninput11.json"
fd=open("jsoninput11.json")
json_content = fd.read()
fd.close()
subprocess.run(["grpcurl", "-plaintext","-H", auth,"-d","#",json_content,"-format","json","100.20.20.1:5000","api.Service/Method"])

I am not sure but maybe the problem is related to the bufsize (check this:
Very large input and piping using subprocess.Popen )
Does it works with capture_output=False?
subprocess.run(["grpcurl", "-plaintext","-H", auth,"-d","#",json_content,"-format","json","100.20.20.1:5000","api.Service/Method"], capture_output=False)
On the other side, if you need the output you may need to deal with the PIPE of Popen.

Query bson files using python

I have a bson file: xyz.bson full of useful data and I'd like to query/process the data using python. Is there a simple example/tutorial out there I can get started with?
I don't understand this one.

You could use the mongorestore command to import the data into a mongoDB server and then query it by connecting to that server.

If you want to stream the data as though it were a flat JSON file on disk rather than loading it into a mongod, you can use this small python-bson-streaming library:
https://github.com/bauman/python-bson-streaming
from bsonstream import KeyValueBSONInput
from sys import argv
for file in argv[1:]:
f = open(file, 'rb')
stream = KeyValueBSONInput(fh=f, fast_string_prematch="somthing") #remove fast string match if not needed
for id, dict_data in stream:
if id:
...process dict_data...

You may use sonq to query .bson file directly from bash, or you can import and use the lib in Python.
A few examples:
Query a .bson file
sonq -f '{"name": "Stark"}' source.bson
Convert query results to a newline separated .json file
sonq -f '{"name": {"$ne": "Stark"}}' -o target.json source.bson
Query a .bson file in python
from sonq.operation import query_son
record_list = list(query_son('source.bson', filters={"name": {"$in": ["Stark"]}}))

How to get the content of a remote file without a local temporary file with fabric

I want to get the content of a remote file with fabric, without creating a temporary file.

from StringIO import StringIO
from fabric.api import get
fd = StringIO()
get(remote_path, fd)
content=fd.getvalue()

With Python 3 (and fabric3), I get this fatal error when using io.StringIO: string argument expected, got 'bytes', apparently because Paramiko writes to the file-like object with bytes. So I switched to using io.BytesIO and it works:
from io import BytesIO
def _read_file(file_path, encoding='utf-8'):
io_obj = BytesIO()
get(file_path, io_obj)
return io_obj.getvalue().decode(encoding)

import tempfile
from fabric.api import get
with tempfile.TemporaryFile() as fd:
get(remote_path, fd)
fd.seek(0)
content=fd.read()
See: http://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile
and: http://docs.fabfile.org/en/latest/api/core/operations.html#fabric.operations.get