We Keep Coding

No module named 'gensim' in AWS Glue

I am trying to use Gensim in AWS Glue ETL job. I have created and tested the gensim wheel in sagemaker and it appears to be working correctly. I have added the wheel file in S3 and added the path in the "Python library path" in glue job details. Still the glue script fails with gensim not found error.
Gensim setup.py file
from setuptools import setup
setup(
name="gensim",
version="4.1.0",
packages=['gensim'],
install_requires=['Cython', 'numpy', 'scipy', 'smart-open']
)
After creating the wheel file, I tested it and it installed gensim properly.
sh-4.2$ pip install dist/gensim-4.1.0-py3-none-any.whl
Processing ./dist/gensim-4.1.0-py3-none-any.whl
Requirement already satisfied: Cython in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/Cython-3.0.0a11-py3.7.egg (from gensim==4.1.0) (3.0.0a11)
Requirement already satisfied: smart-open in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/smart_open-6.2.0-py3.7.egg (from gensim==4.1.0) (6.2.0)
Requirement already satisfied: numpy in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages (from gensim==4.1.0) (1.21.6)
Requirement already satisfied: scipy in /home/ec2-user/.local/lib/python3.7/site-packages (from gensim==4.1.0) (1.7.3)
Installing collected packages: gensim
Successfully installed gensim-4.1.0
Glue job output logs show that it is installed properly
2022-09-28T18:33:40.657+05:30 Processing ./glue-python-libs-rutqf0ex/gensim-4.1.0-py3-none-any.whl
2022-09-28T18:33:40.730+05:30 Collecting smart-open
2022-09-28T18:33:40.749+05:30 Downloading smart_open-6.2.0-py3-none-any.whl (58 kB)
2022-09-28T18:33:41.500+05:30 Collecting Cython
2022-09-28T18:33:41.508+05:30 Downloading Cython-0.29.32-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.0 MB)
2022-09-28T18:33:42.216+05:30 Collecting numpy
2022-09-28T18:33:42.221+05:30 Downloading numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)
2022-09-28T18:33:43.126+05:30 Collecting scipy
2022-09-28T18:33:43.143+05:30 Downloading scipy-1.5.4-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
2022-09-28T18:33:44.258+05:30 Installing collected packages: smart-open, Cython, numpy, scipy, gensim
2022-09-28T18:33:49.744+05:30
Copy
Successfully installed Cython-0.29.32 gensim-4.1.0 numpy-1.19.5 scipy-1.5.4 smart-open-6.2.0
Successfully installed Cython-0.29.32 gensim-4.1.0 numpy-1.19.5 scipy-1.5.4 smart-open-6.2.0
2022-09-28T18:33:53.981+05:30 Processing ./glue-python-libs-rutqf0ex/bio-1.3.9-py3-none-any.whl
Glue job error
ModuleNotFoundError: No module named 'gensim'
I came across these two errors in the glue error log.
ERROR: botocore 1.12.232 has requirement urllib3<1.26,>=1.20; python_version >= "3.4", but you'll have urllib3 1.26.12 which is incompatible.
Traceback (most recent call last):
File "/tmp/runscript.py", line 211, in <module>
runpy.run_path(temp_file_path, run_name='__main__')
File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/glue-python-scripts-cr7714l4/train-lda-model.py", line 11, in <module>
ModuleNotFoundError: No module named 'gensim'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 230, in <module>
raise e_type(e_value).with_traceback(new_stack)
File "/tmp/glue-python-scripts-cr7714l4/train-lda-model.py", line 11, in <module>
ModuleNotFoundError: No module named 'gensim'

Since "gensim" is a custom library you need to pack the dependency as a zip file, uploading it to s3 and indicate the s3 path in the Glue job param called "Python library path". AWS Documentation

unable to install scikit-learn from git using jupyter notebook in my windows

i'm trying to import sklearn.metrics.multilabel_confusion_matrix, but since it is not in the normal package what I'm trying to do is install it using !pip install git+http://github.com/scikit-learn/scikit-learn.git in my windows using jupyter notebook. I used command prompt as well but the same problem occurred. but when I'm doing this an error is popping up which is shown bellow
it's saying something related to "Failed building wheel for scikit-learn
Could not install packages due to an EnvironmentError:"
have shown the complete dialogue below.
Please Help!!!
!pip install git+http://github.com/scikit-learn/scikit-learn.git
Collecting git+http://github.com/scikit-learn/scikit-learn.git
Cloning http://github.com/scikit-learn/scikit-learn.git to c:\users\intel\appdata\local\temp\pip-req-build-bg_zxcrm
Requirement already satisfied: numpy>=1.11.0 in c:\programdata\anaconda3\lib\site-packages (from scikit-learn==0.22.dev0) (1.15.4)
Requirement already satisfied: scipy>=0.17.0 in c:\programdata\anaconda3\lib\site-packages (from scikit-learn==0.22.dev0) (1.1.0)
Requirement already satisfied: joblib>=0.11 in c:\programdata\anaconda3\lib\site-packages (from scikit-learn==0.22.dev0) (0.14.0)
Building wheels for collected packages: scikit-learn
Running setup.py bdist_wheel for scikit-learn: started
Running setup.py bdist_wheel for scikit-learn: finished with status 'error'
Complete output from command C:\PROGRAMDATA\ANACONDA3\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\INTEL\\AppData\\Local\\Temp\\pip-req-build-bg_zxcrm\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\INTEL\AppData\Local\Temp\pip-wheel-vp_d4jnx --python-tag cp37:
Partial import of sklearn during the build process.
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
Traceback (most recent call last):
File "C:\PROGRAMDATA\ANACONDA3\lib\site-packages\setuptools\msvc.py", line 489, in _find_latest_available_vc_ver
return self.find_available_vc_vers()[-1]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\INTEL\AppData\Local\Temp\pip-req-build-bg_zxcrm\setup.py", line 290, in <module>
setup_package()
File "C:\Users\INTEL\AppData\Local\Temp\pip-req-build-bg_zxcrm\setup.py", line 286, in setup_package
setup(**metadata)
File "C:\PROGRAMDATA\ANACONDA3\lib\site-packages\numpy\distutils\core.py", line 135, in setup
config = configuration()
File "C:\Users\INTEL\AppData\Local\Temp\pip-req-build-bg_zxcrm\setup.py", line 174, in configuration
config.add_subpackage('sklearn')
File "C:\PROGRAMDATA\ANACONDA3\lib\site-packages\numpy\distutils\misc_util.py", line 1037, in add_subpackage
caller_level = 2)
File "C:\PROGRAMDATA\ANACONDA3\lib\site-packages\numpy\distutils\misc_util.py", line 1006, in get_subpackage
caller_level = caller_level + 1)
File "C:\PROGRAMDATA\ANACONDA3\lib\site-packages\numpy\distutils\misc_util.py", line 943, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "sklearn\setup.py", line 81, in configuration
maybe_cythonize_extensions(top_path, config)
File "C:\Users\INTEL\AppData\Local\Temp\pip-req-build-bg_zxcrm\sklearn\_build_utils\__init__.py", line 43, in maybe_cythonize_extensions
with_openmp = check_openmp_support()
File "C:\Users\INTEL\AppData\Local\Temp\pip-req-build-bg_zxcrm\sklearn\_build_utils\openmp_helpers.py", line 84, in check_openmp_support
extra_postargs=openmp_flags)
File "C:\PROGRAMDATA\ANACONDA3\lib\distutils\_msvccompiler.py", line 345, in compile
self.initialize()
File "C:\PROGRAMDATA\ANACONDA3\lib\distutils\_msvccompiler.py", line 238, in initialize
vc_env = _get_vc_env(plat_spec)
File "C:\PROGRAMDATA\ANACONDA3\lib\site-packages\setuptools\msvc.py", line 185, in msvc14_get_vc_env
return EnvironmentInfo(plat_spec, vc_min_ver=14.0).return_env()
File "C:\PROGRAMDATA\ANACONDA3\lib\site-packages\setuptools\msvc.py", line 843, in __init__
self.si = SystemInfo(self.ri, vc_ver)
File "C:\PROGRAMDATA\ANACONDA3\lib\site-packages\setuptools\msvc.py", line 485, in __init__
self.vc_ver = vc_ver or self._find_latest_available_vc_ver()
File "C:\PROGRAMDATA\ANACONDA3\lib\site-packages\setuptools\msvc.py", line 492, in _find_latest_available_vc_ver
raise distutils.errors.DistutilsPlatformError(err)
distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/
----------------------------------------
Running setup.py clean for scikit-learn
Failed to build scikit-learn
Installing collected packages: scikit-learn
Found existing installation: scikit-learn 0.20.1
Uninstalling scikit-learn-0.20.1:
Failed building wheel for scikit-learn
Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'c:\\programdata\\anaconda3\\lib\\site-packages\\scikit_learn-0.20.1-py3.7.egg-info\\dependency_links.txt'
Consider using the `--user` option or check the permissions.

After getting a similar error I was able to install scikit-learn by following the instructions from the official site https://scikit-learn.org/stable/install.html.
Error caused by file path length limit on Windows
It can happen that pip fails to install packages when reaching the default path size limit of Windows if Python is installed in a nested location such as the AppData folder structure under the user home directory, for instance:
Collecting scikit-learn
...
Installing collected packages: scikit-learn
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: 'C:\\Users\\username\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python37\\site-packages\\sklearn\\datasets\\tests\\data\\openml\\292\\api-v1-json-data-list-data_name-australian-limit-2-data_version-1-status-deactivated.json.gz'
In this case it is possible to lift that limit in the Windows registry by using the regedit tool:
Type “regedit” in the Windows start menu to launch regedit.
Go to the Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem key.
Edit the value of the LongPathsEnabled property of that key and set it to 1.
Reinstall scikit-learn (ignoring the previous broken installation):
pip install --exists-action=i scikit-learn

error in installing request module in python 3.6

I am facing some errors and exceptions while installing requests module in python 3.6 version.
I am attaching screenshot of the errors that i am facing.click here for error screenshot
here is the error that i got on cmd windows
C:\Program Files (x86)\Python36-32\Scripts>pip3 install requests
Collecting requests
Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
Using cached idna-2.6-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: certifi, urllib3, idna, chardet, requests
Exception:
Traceback (most recent call last):
File "c:\program files (x86)\python36-32\lib\site-packages\pip\basecommand.py"
, line 215, in main
status = self.run(options, args)
File "c:\program files (x86)\python36-32\lib\site-packages\pip\commands\instal
l.py", line 342, in run
prefix=options.prefix_path,
File "c:\program files (x86)\python36-32\lib\site-packages\pip\req\req_set.py"
, line 784, in install
**kwargs
File "c:\program files (x86)\python36-32\lib\site-packages\pip\req\req_install
.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "c:\program files (x86)\python36-32\lib\site-packages\pip\req\req_install
.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "c:\program files (x86)\python36-32\lib\site-packages\pip\wheel.py", line
345, in move_wheel_files
clobber(source, lib_dir, True)
File "c:\program files (x86)\python36-32\lib\site-packages\pip\wheel.py", line
316, in clobber
ensure_dir(destdir)
File "c:\program files (x86)\python36-32\lib\site-packages\pip\utils\__init__.
py", line 83, in ensure_dir
os.makedirs(path)
File "c:\program files (x86)\python36-32\lib\os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [WinError 5] Access is denied: 'c:\\program files (x86)\\python
36-32\\Lib\\site-packages\\certifi'
Thanks in advance

The issue that you are facing is because of a permission error. This is likely because you have not run the command line prompt as an administrator. So, the best thing to do is right click on the command line icon, and run as administrator. Once you've done that, continue normally and install requests using pip install requests. Normally, when you try to install python packages in the main python installation, this error occurs, because you need admin privileges in order to write to anything in Program Files.
That should solve the problem. However, what would be a better option would be to install the virtualenv package, and then have localized installation of packages, so that you can avoid these messy installations that require special rights.

Can't install packages in Windows 10

I just reinstalled Python 3.6, but when I try to install packages, Windows returns errors. First I tried installing package BeautifulSoup from the Windows command prompt:
python -m pip install BeautifulSoup
but this returns the following error:
Collecting BeautifulSoup
Using cached BeautifulSoup-3.2.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Robert\AppData\Local\Temp\pip-build-iiyyxs2o\BeautifulSoup\setup.py", line 22
print "Unit tests have failed!"
^
SyntaxError: Missing parentheses in call to 'print'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\Robert\AppData\Local\Temp\pip-build-iiyyxs2o\BeautifulSoup\
Then I tried installing package requests:
python -m pip install requests
but this returns
C:\Program Files\Python36>python -m pip install requests
Collecting requests
Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
Using cached idna-2.6-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
Using cached urllib3-1.22-py2.py3-none-any.whl
Installing collected packages: certifi, idna, chardet, urllib3, requests
Exception:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\pip\basecommand.py", line 215, in main
status = self.run(options, args)
File "C:\Program Files\Python36\lib\site-packages\pip\commands\install.py", line 342, in run
prefix=options.prefix_path,
File "C:\Program Files\Python36\lib\site-packages\pip\req\req_set.py", line 784, in install
**kwargs
File "C:\Program Files\Python36\lib\site-packages\pip\req\req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "C:\Program Files\Python36\lib\site-packages\pip\req\req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "C:\Program Files\Python36\lib\site-packages\pip\wheel.py", line 345, in move_wheel_files
clobber(source, lib_dir, True)
File "C:\Program Files\Python36\lib\site-packages\pip\wheel.py", line 316, in clobber
ensure_dir(destdir)
File "C:\Program Files\Python36\lib\site-packages\pip\utils\__init__.py", line 83, in ensure_dir
os.makedirs(path)
File "C:\Program Files\Python36\lib\os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [WinError 5] Access is denied: 'C:\\Program Files\\Python36\\Lib\\site-packages\\certifi'
What do I need to do so that pip successfully installs packages?

BeautifulSoup is for Python 2 only. For Python 3 install beautifulsoup4:
pip install beautifulsoup4
The second problem seems obvious: you don't have write-access to C:\\Program Files\\Python36\\Lib\\site-packages and subdirectories.

IOError while doing virtualenv venv in Python Mac OSX

I am getting the following error, whenever I do virtualenv venv. I have installed virtualenv via sudo pip install --user virtualenv
New python executable in /Users/rajrohit/Desktop/Work/LambdaAnsible/TestLambda/venv/bin/python
Please make sure you remove any previous custom paths from your /Users/rajrohit/.pydistutils.cfg file.
Installing setuptools, pip, wheel...
Complete output from command /Users/rajrohit/Desk...mbda/venv/bin/python - setuptools pip wheel:
Collecting setuptools
Using cached setuptools-34.2.0-py2.py3-none-any.whl
Collecting pip
Collecting wheel
Collecting packaging>=16.8 (from setuptools)
Using cached packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools)
Using cached appdirs-1.4.0-py2.py3-none-any.whl
Collecting six>=1.6.0 (from setuptools)
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools)
Using cached pyparsing-2.1.10-py2.py3-none-any.whl
Installing collected packages: six, pyparsing, packaging, appdirs, setuptools, pip, wheel
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 784, in install
**kwargs
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line 478, in move_wheel_files
generated.extend(maker.make(spec))
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/distlib/scripts.py", line 372, in make
self._make_script(entry, filenames, options=options)
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/distlib/scripts.py", line 276, in _make_script
self._write_script(scriptnames, shebang, script, filenames, ext)
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/distlib/scripts.py", line 250, in _write_script
self._fileop.write_binary_file(outname, script_bytes)
File "/Library/Python/2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/distlib/util.py", line 407, in write_binary_file
with open(path, 'wb') as f:
IOError: [Errno 1] Operation not permitted: '/bin/easy_install'
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
File "/usr/local/bin/virtualenv", line 11, in <module>
sys.exit(main())
File "/Library/Python/2.7/site-packages/virtualenv.py", line 713, in main
symlink=options.symlink)
File "/Library/Python/2.7/site-packages/virtualenv.py", line 945, in create_environment
download=download,
File "/Library/Python/2.7/site-packages/virtualenv.py", line 901, in install_wheel
call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
File "/Library/Python/2.7/site-packages/virtualenv.py", line 797, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /Users/rajrohit/Desk...mbda/venv/bin/python - setuptools pip wheel failed with error code 2
How do I work with this permissions error, and make virtualenv work properly?

I've always had permission issues when running/modifying the default OSX python interpreter. Currently I use PyEnv (https://github.com/yyuu/pyenv) which installs separate user modifiable python interpreters and together with pyenv-virtualenv (https://github.com/yyuu/pyenv-virtualenv) it works like a charm. I've never had issues since the change.

I recently had similar issues setting up a new machine, I found this solution worked best.
https://apple.stackexchange.com/a/209583/226131
A more detailed explanation is contained within the same question.
https://apple.stackexchange.com/a/223163/226131
TLDR: install a user-modifiable version of python brew install python then you can just pip install.