mysql docker container to run a parallelized optimization with optuna - python

I'm trying to use mysql docker container to run a parallelized optimization with optuna. The mysql docker container was run using:
docker run --name=user_mysql_1 --env="MYSQL_ROOT_PASSWORD=root_password" -p 3306:3306 -d mysql:latest
Then i followd the optuna documentation example and tryed to run the python script in directly on my machine and not in the docker host:
import optuna
def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 2) ** 2
if __name__ == "__main__":
study = optuna.load_study(
study_name="distributed-example", storage="mysql://root#localhost:3306/example"
)
study.optimize(objective, n_trials=100)
However, the code throw the following error:
Traceback (most recent call last):
File "/home/invitado/anaconda3/envs/alex_oa/lib/python3.7/site-packages/optuna/storages/_rdb/storage.py", line 175, in __init__
self.engine = create_engine(self.url, **self.engine_kwargs)
File "/home/invitado/anaconda3/envs/alex_oa/lib/python3.7/site-packages/sqlalchemy/engine/__init__.py", line 520, in create_engine
return strategy.create(*args, **kwargs)
File "/home/invitado/anaconda3/envs/alex_oa/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/home/invitado/anaconda3/envs/alex_oa/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 118, in dbapi
return __import__("MySQLdb")
ModuleNotFoundError: No module named 'MySQLdb'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "parralel.py", line 11, in <module>
study_name="distributed-example", storage="mysql://root#localhost:3306/example"
File "/home/invitado/anaconda3/envs/alex_oa/lib/python3.7/site-packages/optuna/study/study.py", line 1236, in load_study
return Study(study_name=study_name, storage=storage, sampler=sampler, pruner=pruner)
File "/home/invitado/anaconda3/envs/alex_oa/lib/python3.7/site-packages/optuna/study/study.py", line 230, in __init__
storage = storages.get_storage(storage)
File "/home/invitado/anaconda3/envs/alex_oa/lib/python3.7/site-packages/optuna/storages/__init__.py", line 31, in get_storage
return _CachedStorage(RDBStorage(storage))
File "/home/invitado/anaconda3/envs/alex_oa/lib/python3.7/site-packages/optuna/storages/_rdb/storage.py", line 180, in __init__
) from e
ImportError: Failed to import DB access module for the specified storage URL. Please install appropriate one.
I think thar for i needed to install the module named 'MySQLdb'. Following other stackoverflow answers, I ran pip install mysqlclient in the terminal, but the following error was raised:
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/6b/ba/4729d99e85a0a35bb46d55500570de05b4af10431cef174b6da9f58a0e50/mysqlclient-1.3.1.tar.gz#sha256=3549e8a61f10c8cd8eac6581d3f44d0594f535fb7b29e6090db3a0bc547b25ad (from https://pypi.org/simple/mysqlclient/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Using cached mysqlclient-1.3.0.tar.gz (76 kB)
ERROR: Command errored out with exit status 1:
command: /home/invitado/anaconda3/envs/alex_oa/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7onlhnwm/mysqlclient_c70ddd5d2f30483080fe78636206454b/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7onlhnwm/mysqlclient_c70ddd5d2f30483080fe78636206454b/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-i44ydm17
cwd: /tmp/pip-install-7onlhnwm/mysqlclient_c70ddd5d2f30483080fe78636206454b/
Complete output (10 lines):
/bin/sh: 1: mysql_config: not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-7onlhnwm/mysqlclient_c70ddd5d2f30483080fe78636206454b/setup.py", line 17, in <module>
metadata, options = get_config()
File "/tmp/pip-install-7onlhnwm/mysqlclient_c70ddd5d2f30483080fe78636206454b/setup_posix.py", line 47, in get_config
libs = mysql_config("libs_r")
File "/tmp/pip-install-7onlhnwm/mysqlclient_c70ddd5d2f30483080fe78636206454b/setup_posix.py", line 29, in mysql_config
raise EnvironmentError("%s not found" % (mysql_config.path,))
OSError: mysql_config not found
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/6a/91/bdfe808fb5dc99a5f65833b370818161b77ef6d1e19b488e4c146ab615aa/mysqlclient-1.3.0.tar.gz#sha256=06eb5664e3738b283ea2262ee60ed83192e898f019cc7ff251f4d05a564ab3b7 (from https://pypi.org/simple/mysqlclient/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement mysqlclient (from versions: 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.3.8, 1.3.9, 1.3.10, 1.3.11rc1, 1.3.11, 1.3.12, 1.3.13, 1.3.14, 1.4.0rc1, 1.4.0rc2, 1.4.0rc3, 1.4.0, 1.4.1, 1.4.2, 1.4.2.post1, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.1.0rc1, 2.1.0)
ERROR: No matching distribution found for mysqlclient
I think that the problem is that I need to run the optuna parallelization problem and to install the pip install mysqlclient in the same container. But I'm not sure if it is the best solution or the only one. I would like to use the mysql separated from my code, so is there a way or I'm doing something wrong?

Related

Problems installing django-observer package

Hello I am trying to install django-observer for python3 on my Win10 PC but I keep getting following error message:
Collecting django-observer
Using cached django-observer-0.4.3.tar.gz (7.4 kB)
Requirement already satisfied: django>=1.2 in c:\users\luca dieling\appdata\local\packages\pythonsoftwarefoundation.python.3.8_qbz5n2kfra8p0\localcache\local-packages\python38\site-packages (from django-observer) (3.1.4)
Collecting distribute
Using cached distribute-0.7.3.zip (145 kB)
ERROR: Command errored out with exit status 1:
command: 'C:\Users\Luca Dieling\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Luca Dieling\\AppData\\Local\\Temp\\pip-install-k757dx1_\\distribute_c11e8be23eda49c7809d38d8e88b222e\\setup.py'"'"'; __file__='"'"'C:\\Users\\Luca Dieling\\AppData\\Local\\Temp\\pip-install-k757dx1_\\distribute_c11e8be23eda49c7809d38d8e88b222e\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\Luca Dieling\AppData\Local\Temp\pip-pip-egg-info-00os3tdp'
cwd: C:\Users\Luca Dieling\AppData\Local\Temp\pip-install-k757dx1_\distribute_c11e8be23eda49c7809d38d8e88b222e\
Complete output (15 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Luca Dieling\AppData\Local\Temp\pip-install-k757dx1_\distribute_c11e8be23eda49c7809d38d8e88b222e\setuptools\__init__.py", line 2, in <module>
from setuptools.extension import Extension, Library
File "C:\Users\Luca Dieling\AppData\Local\Temp\pip-install-k757dx1_\distribute_c11e8be23eda49c7809d38d8e88b222e\setuptools\extension.py", line 5, in <module>
from setuptools.dist import _get_unpatched
File "C:\Users\Luca Dieling\AppData\Local\Temp\pip-install-k757dx1_\distribute_c11e8be23eda49c7809d38d8e88b222e\setuptools\dist.py", line 7, in <module>
from setuptools.command.install import install
File "C:\Users\Luca Dieling\AppData\Local\Temp\pip-install-k757dx1_\distribute_c11e8be23eda49c7809d38d8e88b222e\setuptools\command\__init__.py", line 8, in <module>
from setuptools.command import install_scripts
File "C:\Users\Luca Dieling\AppData\Local\Temp\pip-install-k757dx1_\distribute_c11e8be23eda49c7809d38d8e88b222e\setuptools\command\install_scripts.py", line 3, in <module>
from pkg_resources import Distribution, PathMetadata, ensure_directory
File "C:\Users\Luca Dieling\AppData\Local\Temp\pip-install-k757dx1_\distribute_c11e8be23eda49c7809d38d8e88b222e\pkg_resources.py", line 1518, in <module>
register_loader_type(importlib_bootstrap.SourceFileLoader, DefaultProvider)
AttributeError: module 'importlib._bootstrap' has no attribute 'SourceFileLoader'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
I am running the command pip install django-observer
Thank you for your help
This django-observer package hasn't been maintained for a long time. The github repository states it works on Python 3 version 3.2 and 3.3 though. So you might have luck setting up a virtualenv with python 3.2 or 3.3 and working from there.
The package appears to have done work similar to that of Django Signals. You can read up on them here. Signals are an implementation of the Observer pattern.
Just FYI, I tried installing the package too and got the same error on my MacOS machine. This is why I suggest switching to Signals.

Issue with installing causalml package python

I am trying to install the package causalml in Jupyter using pip, and I get the following error:
ERROR: Command errored out with exit status 1:
command: /opt/conda/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h3qf0rqq/causalml_92ba9279cfbd4f77ac1f50f04998c187/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h3qf0rqq/causalml_92ba9279cfbd4f77ac1f50f04998c187/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-m0xr_f2v
cwd: /tmp/pip-install-h3qf0rqq/causalml_92ba9279cfbd4f77ac1f50f04998c187/
Complete output (20 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-h3qf0rqq/causalml_92ba9279cfbd4f77ac1f50f04998c187/setup.py", line 56, in <module>
ext_modules=cythonize(extensions),
File "/opt/conda/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 962, in cythonize
ctx = c_options.create_context()
File "/opt/conda/lib/python3.6/site-packages/Cython/Compiler/Main.py", line 597, in create_context
self.cplus, self.language_level, options=self)
File "/opt/conda/lib/python3.6/site-packages/Cython/Compiler/Main.py", line 80, in __init__
from . import Builtin, CythonScope
File "/opt/conda/lib/python3.6/site-packages/Cython/Compiler/CythonScope.py", line 5, in <module>
from .UtilityCode import CythonUtilityCode
File "/opt/conda/lib/python3.6/site-packages/Cython/Compiler/UtilityCode.py", line 3, in <module>
from .TreeFragment import parse_from_strings, StringParseContext
File "/opt/conda/lib/python3.6/site-packages/Cython/Compiler/TreeFragment.py", line 17, in <module>
from .Visitor import VisitorTransform
File "Cython/Compiler/Visitor.py", line 17, in init Cython.Compiler.Visitor
File "/opt/conda/lib/python3.6/site-packages/Cython/Compiler/ExprNodes.py", line 46, in <module>
from .Pythran import (to_pythran, is_pythran_supported_type, is_pythran_supported_operation_type,
ImportError: cannot import name 'pythran_is_numpy_func_supported'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
In another forum, I read that this is a cython issue, I have uninstalled and reinstalled it, and it did not work out. I am working with Python 3.6.10, and here are the versions of the dependent packages on my system.
setuptools= 50.3.0
pip=20.3
numpy=1.18.5
scipy=1.4.1
matplotlib
pandas=1.1.2
scikit-learn=0.23.2
statsmodels=0.12.0
seaborn
Cython=0.29.21
xgboost
pydotplus
tqdm
shap
dill
lightgbm
pygam
packaging
keras
tensorflow=2.3.0
I checked the dependency requirements, and I believe all the above packages are acceptable for installing causalml. I also tried cloning the git repo and installing it, and I got the same error. Your feedback and suggestion are greatly appreciated.

Google App Engine rtree Python Module Installation

Google App Engine, does not include libspatialindex, and hence I am unable to install rtree python module, even if it is in requirements.txt,
it gives following error, while deploying app. ( app works fine locally. ) How do I install specific lib in python standard google app engine environment ? or How do I request them to do so ?
Collecting rtree
Downloading Rtree-0.9.4.tar.gz (62 kB)
ERROR: Command errored out with exit status 1:
command: /opt/python3.8/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-hl9513g_/rtree/setup.py'"'"'; __file__='"'"'/tmp/pip-install-hl9513g_/rtree/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-0tyv4fij
cwd: /tmp/pip-install-hl9513g_/rtree/
Complete output (15 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-hl9513g_/rtree/setup.py", line 3, in <module>
import rtree
File "/tmp/pip-install-hl9513g_/rtree/rtree/__init__.py", line 1, in <module>
from .index import Rtree
File "/tmp/pip-install-hl9513g_/rtree/rtree/index.py", line 6, in <module>
from . import core
File "/tmp/pip-install-hl9513g_/rtree/rtree/core.py", line 143, in <module>
rt.Error_GetLastErrorNum.restype = ctypes.c_int
File "/opt/python3.8/lib/python3.8/ctypes/__init__.py", line 386, in __getattr__
func = self.__getitem__(name)
File "/opt/python3.8/lib/python3.8/ctypes/__init__.py", line 391, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /opt/python3.8/bin/python3: undefined symbol: Error_GetLastErrorNum
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
You will need to use flexible environment with custom runtime to utilize packages not preinstalled in standard environment.
To install libspatialindex, add this to the Dockerfile:
sudo apt-get update -y
sudo apt-get install -y libspatialindex-dev

Install mysqlclient for Django database via pip

I have installed MySQL-server, python3-dev, libmysqlclient-dev. But whem I try to install mysqlclient via pip, I get some errors that I don't understand. Like this:
(Env) randomparatololer#randomparatololer:~/Documents/DJANGO/tigabelas$ pip install mysqlclient
Collecting mysqlclient
Using cached mysqlclient-1.4.6.tar.gz (85 kB)
ERROR: Command errored out with exit status 1:
command: /home/randomparatololer/Documents/DJANGO/Env/bin/python3.8 -C 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r_ybnapl/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-install-r_ybnapl/mysqlclient/setup.py'"'"';f=getattr (tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace ('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec (compile (code, __file__, '"'"'exec'"'"')' egg_info – egg-base /tmp/pip-install-r_ybnapl/mysqlclient/pip-egg-info
cwd: /tmp/pip-install-r_ybnapl/mysqlclient/
Complete output (11 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/randomparatololer/Documents/DJANGO/Env/lib/python3.8/site-packages/setuptools/__init__.py", line 20, in <module>
from setuptools.dist import Distribution, Feature
File "/home/randomparatololer/Documents/DJANGO/Env/lib/python3.8/site-packages/setuptools/dist.py", line 35, in <module>
from setuptools import windows_support
File "/home/randomparatololer/Documents/DJANGO/Env/lib/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module>
import ctypes
File "/usr/local/lib/python3.8/ctypes/__init__.py", line 7, in <module>
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
----------------------------------------
ERROR: Command errored out with exit status 1: Python setup.py egg_info Check the logs for full command output.

Virtualenv doesn't work after cloning repository

I clone my Script from Github. Then I install the requirements.txt with Gitbash.
1.) source ../venv/Scripts/activate
2.) pip install -r requirements.txt
If I try now to run the script some packages are missing everytime.
Then I need to manually install them and it is just unconvenient.
python script.py # ModuleNotFoundError: No modul named 'dateutil'
And I really dont know what I am making wrong. My Goal is after Cloning and installing to the venv it should work but it doesnt work.
$ python --version
Python 3.8.0
$ pip --version
pip 19.3.1 from c:\users\user\desktop\dokumente\venv\lib\site-packages\pip (python 3.8)
$ pip show --files python-dateutil
WARNING: Package(s) not found: python-dateutil
$ pip install -r requirements.txt
Collecting aniso8601==7.0.0
Using cached https://files.pythonhosted.org/packages/45/a4/b4fcadbdab46c2ec2d2f6f8b4ab3f64fd0040789ac7f065eba82119cd602/aniso8601-7.0.0-py2.py3-none-any.whl
Collecting asn1crypto==0.24.0
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting astroid==2.2.5
Using cached https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl
Collecting attrs==19.1.0
Using cached https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
Collecting Automat==0.7.0
Using cached https://files.pythonhosted.org/packages/a3/86/14c16bb98a5a3542ed8fed5d74fb064a902de3bdd98d6584b34553353c45/Automat-0.7.0-py2.py3-none-any.whl
Processing c:\users\user\appdata\local\pip\cache\wheels\b1\fe\08\84d2143069bc44c20127c38cc1bf202332319b3da7315ca766\camelcase-0.2-cp38-none-any.whl
Collecting certifi==2019.9.11
Using cached https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl
Collecting cffi==1.12.3
Using cached https://files.pythonhosted.org/packages/93/1a/ab8c62b5838722f29f3daffcc8d4bd61844aa9b5f437341cc890ceee483b/cffi-1.12.3.tar.gz
ERROR: Command errored out with exit status 1:
command: 'c:\users\user\desktop\dokumente\agent_venv\venv\scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\User\\AppData\\Local\\Temp\\pip-install-onw899ll\\cffi\\setup.py'"'"'; __file__='"'"'C:\\Users\\User\\AppData\\Local\\Temp\\pip-install-onw899ll\\cffi\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\User\AppData\Local\Temp\pip-install-onw899ll\cffi\pip-egg-info'
cwd: C:\Users\User\AppData\Local\Temp\pip-install-onw899ll\cffi\
Complete output (23 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\User\AppData\Local\Temp\pip-install-onw899ll\cffi\setup.py", line 127, in <module>
if sys.platform == 'win32' and uses_msvc():
File "C:\Users\User\AppData\Local\Temp\pip-install-onw899ll\cffi\setup.py", line 105, in uses_msvc
return config.try_compile('#ifndef _MSC_VER\n#error "not MSVC"\n#endif')
File "c:\users\user\appdata\local\programs\python\python38-32\Lib\distutils\command\config.py", line 225, in try_compile
self._compile(body, headers, include_dirs, lang)
File "c:\users\user\appdata\local\programs\python\python38-32\Lib\distutils\command\config.py", line 132, in _compile
self.compiler.compile([src], include_dirs=include_dirs)
File "c:\users\user\appdata\local\programs\python\python38-32\Lib\distutils\_msvccompiler.py", line 360, in compile
self.initialize()
File "c:\users\user\appdata\local\programs\python\python38-32\Lib\distutils\_msvccompiler.py", line 253, in initialize
vc_env = _get_vc_env(plat_spec)
File "c:\users\user\desktop\dokumente\agent_venv\venv\lib\site-packages\setuptools\msvc.py", line 171, in msvc14_get_vc_env
return EnvironmentInfo(plat_spec, vc_min_ver=14.0).return_env()
File "c:\users\user\desktop\dokumente\agent_venv\venv\lib\site-packages\setuptools\msvc.py", line 1075, in __init__
self.si = SystemInfo(self.ri, vc_ver)
File "c:\users\user\desktop\dokumente\agent_venv\venv\lib\site-packages\setuptools\msvc.py", line 547, in __init__
vc_ver or self._find_latest_available_vs_ver())
File "c:\users\user\desktop\dokumente\agent_venv\venv\lib\site-packages\setuptools\msvc.py", line 561, in _find_latest_available_vs_ver
raise distutils.errors.DistutilsPlatformError(
distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
I hope you can help me. If you need some extra information to solve the problem then please tell it to me.
Best Regards
Cren

Categories