We Keep Coding

Lambda can't import the module pyminizip even though its already in the directory

So I have this similar problem with this person.
How to create password encrypted zip file in python through AWS lambda
We have the exact same problem but i already did everything from the answers in that thread but to no avail.
I have a lambda script that runs on python3.9 I need to compress the files in my s3 as a zip file that is password protected and i need to put it in another s3.
This is how it goes
import pyminizip
def zip_to_client():
# reportTitles = os.listdir(tempDir)
dateGenerated = datetime.now(tz=atz).strftime("%Y-%m-%d")
pyminizip.compress("Daily_Booking_Report.csv", subfolder + str(dateGenerated) +'/'+str(id)+'/'
, "/tmp/test.zip", "awesomepassword", 9)
s3 = boto3.resource('s3')
s3.meta.client.upload_file(Filename = '/tmp/test.zip', Bucket = bucket, Key = subfolder + 'test.zip', ExtraArgs={'Tagging':'archive=90days'})
print("SUCCESS: Transferred report into S3")
i'm not sure if it works but i can't debug it because lambda shows me the error:
Response
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'pyminizip'",
"errorType": "Runtime.ImportModuleError",
"requestId": "0000111000",
"stackTrace": []
}
I made sure that i put import pyminizip as well as pip installing it in the directory.
pip install pyminizip -t .
so far this is what the lambda directory looks like
https://ibb.co/ZGmLBbv
i've tried everything from putting it in a lambda layer to pip installing different versions from python 3.7 to 3.9

This is a common case when you create a lambda layer and get import error. And this occurs when you don't have created python files in a defined directory like python/python38/site-packages...
or
second reason might be a dependency is missing. In that use use docker and follow steps from here : https://www.geeksforgeeks.org/how-to-install-python-packages-for-aws-lambda-layers/.

Problem Creating One File exe with pyinstaller and pysqlcipher

I'm trying to create a one file .exe to run on any windows machine but I've hit a problem with pysqlcipher. I've gone back to some basic code that just creates a simple database with a key, on my dev machine all works fine whether I use the python file or the compiled exe. I seem to be missing a library, path or both? I've tried adding vaious items using --add-data but have spent hours and made no progress. Here is the basic bit of python:-
from pysqlcipher3 import dbapi2 as sqlite
import os
import sys
def resource_path(relative_path):
""" Get absolute path to resource, works for dev and for PyInstaller """
try:
base_path = sys._MEIPASS
except Exception:
base_path = os.path.abspath(".")
return os.path.join(base_path, relative_path)
conn = sqlite.connect('test.db')
c = conn.cursor()
c.execute("PRAGMA key='password'")
c.execute('''create table stocks (date text, trans text, symbol text, qty real, price real)''')
c.execute("""insert into stocks values ('2006-01-05','BUY','RHAT',100,35.14)""")
conn.commit()
c.close()
When I run the exe on a different windows 10 PC I get this error
Traceback (most recent call last):
File "testdb.py", line 1, in
File "c:\users\xxx\appdata\local\programs\python\python38\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
File "site-packages\pysqlcipher3-1.0.3-py3.8-win-amd64.egg\pysqlcipher3\dbapi2.py", line 33, in
ModuleNotFoundError: No module named 'pysqlcipher3._sqlite3'
[9248] Failed to execute script testdb
The error references the path on my dev PC and also refers to line 33 in dbapi2.py which is:-
from pysqlcipher3._sqlite3 import *
I have tried adding various files when running pyinstaller but I am making no progress, I'm sure its nothing simple but need help please.

I encountered a similar problem a while back and one thing that worked for me is adding the sqlite3 dll in the spec file of pyinstaller. It is also possible to add it from command line. I had tried many other things before such as re-creating a conda environment and rebuilding my python package but that did not work. In your spec file where you add the pyinstaller binaries please try something like.
binaries=[('C:\\Users\\myname\\newfolder\\sqlite3.dll','.')]
You can download the DLL from https://www.sqlite.org/download.html depending on windows or linux platform . Once you add the dll and recompile it should be fine. I think this error is primarily happening as dbapi2 requests the sqlite3 dll which in my case atlease was missing from the DLL folder. I just downloaded and added it in my spec file, but you can also try to add it to the hookspath or ENV folder.

pyhive, sqlalchemy can not connect to hadoop sandbox

I have installed,
pip install thrift
pip install PyHive
pip install thrift-sasl
and
since pip install sasl failed I downloaded sasl‑0.2.1‑cp27‑cp27m‑win_amd64.whl file and installed it in my Windows 8.1 PC.
Then I wrote this code,
from pyhive import hive
cursor = hive.connect('192.168.1.232', port=10000, auth='NONE')
cursor.execute('SELECT * from sample_07 LIMIT 5',async=True)
print cursor.fetchall()
this gives the error:
Traceback (most recent call last):
File "C:/DigInEngine/scripts/UserManagementService/fd.py", line 37, in <module>
cursor = hive.connect('192.168.1.232', port=10000, auth = 'NONE')
File "C:\Python27\lib\site-packages\pyhive\hive.py", line 63, in connect
return Connection(*args, **kwargs)
File "C:\Python27\lib\site-packages\pyhive\hive.py", line 104, in __init__
self._transport.open()
File "C:\Python27\lib\site-packages\thrift_sasl\__init__.py", line 72, in open
message=("Could not start SASL: %s" % self.sasl.getError()))
thrift.transport.TTransport.TTransportException: Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2
and this code gives,
from sqlalchemy import create_engine
engine = create_engine('hive://192.168.1.232:10000/default')
try:
connection = engine.connect()
except Exception, err:
print err
result = connection.execute('select * from sample_07;')
engine.dispose()
this error,
Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no
mechanism available: Unable to find a callback: 2
I have downloaded Hortonworks sandbox from here and use it in a separate server.
NOTE: I went through this as well but the accepted answer is not working for me, because importing ThriftHive from hive gives Import error although I have pip installed hive. So I decided to use pyhive or sqlalchemy
How can I connect to hive and execute a query easily?

Here are steps to build SASL on Windows, but your mileage may vary: A lot of this depends on your particular system's paths and available libraries.
Please also note that these instructions are specific to Python 2.7 (which I see you are using from the paths in your question).
The high-level overview is that you're installing this project: https://github.com/cyrusimap/cyrus-sasl. In order to do that, you have to use the legacy C++ compiler that was used to build Python 2.7. There are a couple of other steps to getting this to work.
Pre-build Steps:
Install Microsoft Visual C++ Compiler for Python 2.7. Use the default installation paths - take note of where it got installed for the next 2 steps (2 options are included in the list below)
Copy this file to whichever of the include locations is appropriate for your install
Make a unistd.h file from this answer in the same include directory
Build steps:
git clone https://github.com/cyrusimap/cyrus-sasl
Open the "VS2013 x64 Native Tools Command Prompt" that's installed with the Compiler from step 1
Change directory to the directory created by step 4, then the lib sub-directory
nmake /f ntmakefile STATIC=no prefix=C:\sasl64
nmake /f ntmakefile prefix=C:\sasl64 STATIC=no install see note below
copy /B C:\sasl64\lib\libsasl.lib /B C:\sasl64\lib\sasl2.lib
pip install thrift_sasl --global-option=build_ext \
--global-option=-IC:\\sasl64\\include \
--global-option=-LC:\\sasl64\\lib
'Include' locations:
"C:\Program Files (x86)\Common Files\Microsoft\Visual C++ for Python\9.0\VC\include\stdint.h"
"%USERPROFILE%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\include"
Here's a reference to these same steps, with some additional annotations and explanations: http://java2developer.blogspot.co.uk/2016/08/making-impala-connection-from-python-on.html.
Note
The referenced instructions also executed step (8) in the include and win32\include sub-directories, you may have to do that as well.

While using pyhive no authentication can be passed as auth="NOSASL", instead of "None", so your code should look like this:
from pyhive import hive
cursor = hive.connect('192.168.1.232', port=10000, auth='NOSASL')
cursor.execute('SELECT * from sample_07 LIMIT 5',async=True)
print cursor.fetchall()

Error deploying Python package to AWS Lambda

AWS Successfully creates the lambda function when I upload the zip file. But it's giving this error when I test it out.
{
"errorMessage": "Unable to import module 'amazonSNS'"
}
Following are the contents of the zip file that I created. I tried changing the name of the zip file to "amazonSNS" to match the amazonSNS.py file, but no help, same issue.
The Lambda handler in the Configuration of the Lambda function is set to "amazonSNS.handler" where amazonSNS is the filename and handler is the function name that needs to be called, as they have instructed in the documentation.
Here are the contents of the python file
import boto3
import MySQLdb
client = boto3.client("sns")
rds = boto3.client("rds")
def handler(event, context):
conn = MySQLdb.connect("host", "username", "password", "database")
cur = conn.cursor(MySQLdb.cursors.DictCursor)
query = "select * from login.login limit 10"
cur.execute(query)
print cur.fetchall()
print conn
What might be the issue here?
Here is the Log output
START RequestId: 76a61551-052a-11e6-b466-8fa0769ac309 Version: $LATEST
Unable to import module 'amazonSNS': No module named _mysql
END RequestId: 76a61551-052a-11e6-b466-8fa0769ac309 REPORT RequestId:
76a61551-052a-11e6-b466-8fa0769ac309 Duration: 0.33 ms Billed
Duration: 100 ms
UPDATE
I added a few more files from "site-package" folder that I thought was part of the MySQLdb package, Here are the current contents of the zip file.
And after this the new Log of the error is.
START RequestId: c0715d9a-0531-11e6-9409-a3b194fd4afd Version: $LATEST
Unable to import module 'amazonSNS': libmysqlclient.so.18: cannot open
shared object file: No such file or directory
END RequestId: c0715d9a-0531-11e6-9409-a3b194fd4afd REPORT RequestId:
c0715d9a-0531-11e6-9409-a3b194fd4afd Duration: 0.35 ms Billed
Duration: 100 ms Memory Size: 128 MB Max Memory Used: 22 MB

To solve this issue:
I searched for libmysqlclient.so.20 (the version number at the end may differ)
find /. -name "libmysqlclient.so.20"
My ouput was
/./usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
I then copied that file into the root directory of my package
cp /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 <your package path>

I failed to compile MySQL-python in way to make it work in Lambda. Instead I switched to pymysql. I am not sure about performance, but at least this works.
P.S. I wonder why there is no official recommendations about suggested MySQL driver on amazon. At least I haven't found it.

I had this issue when using mysqlclient (the MySQLd fork which works on Python3).
Since I use Zappa for easy deployment, the solution was simple: just switch to the original MySQLd package (which does not support Python 3, though): pip install mysql-python
Zappa comes with a pre-compiled version of it.

How did you install MySQLdb? http://mysql-python.sourceforge.net/FAQ.html says:
ImportError: No module named _mysql
If you see this, it's likely you did some wrong when installing MySQLdb; re-read (or read) README. _mysql is the low-level C module that interfaces with the MySQL client library.
Install MySQLdb with pip if you didn't already.

cx_oracle and python 2.7 [duplicate]

This question already has answers here:
Python module "cx_Oracle" module could not be found
(4 answers)
Closed 4 years ago.
Im using python 2.7 and cx_oracle ( Windows x86 Installer (Oracle 10g, Python 2.7) ) and 'm having a bad time to set this simple example bellow to work:
import cx_Oracle
connection = cx_Oracle.connect('user/pass#someserver:port')
cursor = connection.cursor()
cursor.execute('select sysdate from dual')
for row in cursor:
print row
connection.close()
Error Message:
Traceback (most recent call last):
File "C:\ORACON.py", line 1, in <module>
import cx_Oracle
ImportError: DLL load failed: The specified module could not be found.
For now, what i have done was:
1) installed the cx_oracle binary;
2) downloaded instantclient_10_2 from oracle website and exported the path to environment;
Anyone know what im missing?
Thank you for your time on reading this.

I was able to solve this problem with the following steps:
Download instantclient-basic-win32-10.2.0.5 from Oracle Website
unzipped the into my c:\ with the name oraclient
Created the directory structure C:\oraclient\network\admin to add the TNSNAMES.ORA
Added the TNS_ADMIN env var pointing to C:\oraclient\network\admin
Added the ORACLE_HOME env var pointing to C:\oraclient\
After that i used the following code:
import cx_Oracle
con = cx_Oracle.connect('theuser', 'thepass', 'your DB alias on your TNSNAMES.ORA file ')
cur = con.cursor()
if cur.execute('select * from dual'):
print "finally, it works!!!"
else:
print "facepalm"
con.close()
I hope it helps someone.