We Keep Coding

No module named 'gensim' in AWS Glue

I am trying to use Gensim in AWS Glue ETL job. I have created and tested the gensim wheel in sagemaker and it appears to be working correctly. I have added the wheel file in S3 and added the path in the "Python library path" in glue job details. Still the glue script fails with gensim not found error.
Gensim setup.py file
from setuptools import setup
setup(
name="gensim",
version="4.1.0",
packages=['gensim'],
install_requires=['Cython', 'numpy', 'scipy', 'smart-open']
)
After creating the wheel file, I tested it and it installed gensim properly.
sh-4.2$ pip install dist/gensim-4.1.0-py3-none-any.whl
Processing ./dist/gensim-4.1.0-py3-none-any.whl
Requirement already satisfied: Cython in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/Cython-3.0.0a11-py3.7.egg (from gensim==4.1.0) (3.0.0a11)
Requirement already satisfied: smart-open in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/smart_open-6.2.0-py3.7.egg (from gensim==4.1.0) (6.2.0)
Requirement already satisfied: numpy in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages (from gensim==4.1.0) (1.21.6)
Requirement already satisfied: scipy in /home/ec2-user/.local/lib/python3.7/site-packages (from gensim==4.1.0) (1.7.3)
Installing collected packages: gensim
Successfully installed gensim-4.1.0
Glue job output logs show that it is installed properly
2022-09-28T18:33:40.657+05:30 Processing ./glue-python-libs-rutqf0ex/gensim-4.1.0-py3-none-any.whl
2022-09-28T18:33:40.730+05:30 Collecting smart-open
2022-09-28T18:33:40.749+05:30 Downloading smart_open-6.2.0-py3-none-any.whl (58 kB)
2022-09-28T18:33:41.500+05:30 Collecting Cython
2022-09-28T18:33:41.508+05:30 Downloading Cython-0.29.32-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.0 MB)
2022-09-28T18:33:42.216+05:30 Collecting numpy
2022-09-28T18:33:42.221+05:30 Downloading numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)
2022-09-28T18:33:43.126+05:30 Collecting scipy
2022-09-28T18:33:43.143+05:30 Downloading scipy-1.5.4-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
2022-09-28T18:33:44.258+05:30 Installing collected packages: smart-open, Cython, numpy, scipy, gensim
2022-09-28T18:33:49.744+05:30
Copy
Successfully installed Cython-0.29.32 gensim-4.1.0 numpy-1.19.5 scipy-1.5.4 smart-open-6.2.0
Successfully installed Cython-0.29.32 gensim-4.1.0 numpy-1.19.5 scipy-1.5.4 smart-open-6.2.0
2022-09-28T18:33:53.981+05:30 Processing ./glue-python-libs-rutqf0ex/bio-1.3.9-py3-none-any.whl
Glue job error
ModuleNotFoundError: No module named 'gensim'
I came across these two errors in the glue error log.
ERROR: botocore 1.12.232 has requirement urllib3<1.26,>=1.20; python_version >= "3.4", but you'll have urllib3 1.26.12 which is incompatible.
Traceback (most recent call last):
File "/tmp/runscript.py", line 211, in <module>
runpy.run_path(temp_file_path, run_name='__main__')
File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/glue-python-scripts-cr7714l4/train-lda-model.py", line 11, in <module>
ModuleNotFoundError: No module named 'gensim'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 230, in <module>
raise e_type(e_value).with_traceback(new_stack)
File "/tmp/glue-python-scripts-cr7714l4/train-lda-model.py", line 11, in <module>
ModuleNotFoundError: No module named 'gensim'

Since "gensim" is a custom library you need to pack the dependency as a zip file, uploading it to s3 and indicate the s3 path in the Glue job param called "Python library path". AWS Documentation

pip3 install mysql ERROR

I get this error
pi#teambaboy:~ $ pip3 install mysql
Collecting mysql
Downloading https://www.piwheels.org/simple/mysql/mysql-0.0.1-py3-none-any.whl
Collecting MySQL-python (from mysql)
Using cached MySQL-python-1.2.5.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-k38v3y9c/MySQL-python/setup.py", line 13, in <module>
from setup_posix import get_config
File "/tmp/pip-build-k38v3y9c/MySQL-python/setup_posix.py", line 2, in <module>
from ConfigParser import SafeConfigParser
ImportError: No module named 'ConfigParser'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-k38v3y9c/MySQL-python/

Use mysqlclient instead. It's the modern drop-in replacement.
$ pip install mysqlclient

Errors when trying to import sklearn

Traceback (most recent call last):
File "project.py", line 2, in <module>
import sklearn
File "/usr/local/lib/python3.6/site-packages/sklearn/__init__.py", line 134, in <module>
from .base import clone
File "/usr/local/lib/python3.6/site-packages/sklearn/base.py", line 11, in <module>
from scipy import sparse
ModuleNotFoundError: No module named 'scipy'
I've installed the sklearn package, but whenever I try to run the script with import sklearn I keep getting these errors.
The terminal is telling me it is installed.
Requirement already satisfied: sklearn in /usr/local/lib/python3.6/site-packages
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/site-packages (from sklearn)

You're just missing another requirement which is scipy. You just need to run pip3 install scipy.
As noted by #sascha you may want to follow the system install instructions recommended by the scipy project.

Python pip conflict for robotframework-selenium2library

I've tried to install robotframework-selenium2library but taken a pkg_resources.VersionConflict error:
Downloading/unpacking robotframework-selenium2library
Downloading robotframework-selenium2library-1.8.0.tar.gz (118kB): 118kB downloaded
Running setup.py (path:/tmp/pip_build_root/robotframework-selenium2library/setup.py) egg_info for package robotframework-selenium2library
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/robotframework-selenium2library/setup.py", line 7, in <module>
from ez_setup import use_setuptools
File "/tmp/pip_build_root/robotframework-selenium2library/src/ez_setup.py", line 106
except pkg_resources.VersionConflict, e:
^
SyntaxError: invalid syntax
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/robotframework-selenium2library/setup.py", line 7, in <module>
from ez_setup import use_setuptools
File "/tmp/pip_build_root/robotframework-selenium2library/src/ez_setup.py", line 106
except pkg_resources.VersionConflict, e:
^
SyntaxError: invalid syntax
----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_root/robotframework-selenium2library
Storing debug log for failure in /root/.pip/pip.log
Not so many packages have been already installed:
pip list
> apparmor (2.9.4)
> LibAppArmor (2.9.4)
> pip (1.5.6)
> pygobject (3.14.0)
> robotframework (3.0.2)
> robotframework-databaselibrary (1.0.1)
> selenium (3.4.3)
> setuptools (3.6)
python --version
> Python 2.7.12
Does anybody know how to resolve that issue?

The version of robotframework-selenium2library at PyPI (1.8.0) is rather old and supports only Python 2 while you're trying to install it with Python 3. Either use Python 2.7 or get the new code for the framework from GitHub.

I review the github Seleniumlibrary, and you can use:
pip install --pre --upgrade robotframework-seleniumlibrary

Exception while importing pymorphy2

I've installed pymorphy2 using pip:
Requirement already satisfied (use --upgrade to upgrade): pymorphy2 in /usr/local/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): docopt>=0.6 in /usr/local/lib/python2.7/site-packages (from pymorphy2)
Requirement already satisfied (use --upgrade to upgrade): pymorphy2-dicts<3.0,>=2.4 in /usr/local/lib/python2.7/site-packages (from pymorphy2)
But when I'm trying to import this library in Python interactive shell I get this:
>>> import pymorphy2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/pymorphy2/__init__.py", line 3, in <module>
from .analyzer import MorphAnalyzer
File "/usr/local/lib/python2.7/site-packages/pymorphy2/analyzer.py", line 10, in <module>
from pymorphy2 import opencorpora_dict
File "/usr/local/lib/python2.7/site-packages/pymorphy2/opencorpora_dict/__init__.py", line 4, in <module>
from .storage import load_dict as load
File "/usr/local/lib/python2.7/site-packages/pymorphy2/opencorpora_dict/storage.py", line 24, in <module>
from pymorphy2.utils import json_write, json_read
File "/usr/local/lib/python2.7/site-packages/pymorphy2/utils.py", line 5, in <module>
import bz2
ImportError: No module named bz2
Ok, no problem:
sudo apt-get install libbz2-dev
And finally:
maestro#UIServer:~$ sudo apt-get install libbz2-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libbz2-dev is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 64 not upgraded.
1 not fully installed or removed.
After this operation, 0B of additional disk space will be used.
Setting up python-pip (0.3.1-1ubuntu2) ...
Traceback (most recent call last):
File "/usr/bin/pycentral", line 2196, in <module>
main()
File "/usr/bin/pycentral", line 2190, in main
rv = action.run(global_options)
File "/usr/bin/pycentral", line 1478, in run
runtimes = get_installed_runtimes()
File "/usr/bin/pycentral", line 279, in get_installed_runtimes
default_version = pyversions.default_version(version_only=True)
File "/usr/share/pycentral-data/pyversions.py", line 172, in default_version
raise ValueError, "/usr/bin/python does not match the python default version. It must be reset to point to %s" % debian_default
ValueError: /usr/bin/python does not match the python default version. It must be reset to point to python2.6
dpkg: error processing python-pip (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
python-pip
E: Sub-process /usr/bin/dpkg returned an error code (1)
How to resolve this conflict with versions of python in my case?

Looks like you need to create a new symbolic link for python 2.6.
ln -sf /usr/bin/python2.6 /usr/bin/python