How do I set different keras backends in different conda environments? Because in a specific environment, if I change the backend to tensorflow in keras.json, then in another python environment, keras backend will be tensorflow too. There is only one keras.json in my documents.
For using different keras backends in different environments in Anaconda - 'env1' and 'env2'
Activate the first environment 'env1'
Import keras from python with the default backend (if it fails to load tensorflow for example, install tensorflow in that environment) successfully
In the ~ folder, a '.keras' folder will be created which will contain keras.json file
For the other environment create a copy of the '.keras' folder as '.keras1'
Change the keras.json file in that folder as per requirements (the 'backend' field)
For using that config in 'env2' go to '~/anaconda3/envs/env2/lib/pythonx.x/site-packages/keras/backend' and edit the __init__.py file
Make the changes marked with ##
You will be able to import keras with different backends in env1 and env2
from __future__ import absolute_import
from __future__ import print_function
import os
import json
import sys
import importlib
from .common import epsilon
from .common import floatx
from .common import set_epsilon
from .common import set_floatx
from .common import cast_to_floatx
from .common import image_data_format
from .common import set_image_data_format
# Set Keras base dir path given KERAS_HOME env variable, if applicable.
# Otherwise either ~/.keras or /tmp.
if 'KERAS_HOME' in os.environ:
_keras_dir = os.environ.get('KERAS_HOME')
else:
_keras_base_dir = os.path.expanduser('~')
if not os.access(_keras_base_dir, os.W_OK):
_keras_base_dir = '/tmp'
_keras_dir = os.path.join(_keras_base_dir, '.keras1')##
# Default backend: TensorFlow.
_BACKEND = 'tensorflow'
# Attempt to read Keras config file.
_config_path = os.path.expanduser(os.path.join(_keras_dir, 'keras.json'))
if os.path.exists(_config_path):
try:
with open(_config_path) as f:
_config = json.load(f)
except ValueError:
_config = {}
_floatx = _config.get('floatx', floatx())
assert _floatx in {'float16', 'float32', 'float64'}
_epsilon = _config.get('epsilon', epsilon())
assert isinstance(_epsilon, float)
_backend = _config.get('backend', _BACKEND)
_image_data_format = _config.get('image_data_format',
image_data_format())
assert _image_data_format in {'channels_last', 'channels_first'}
set_floatx(_floatx)
set_epsilon(_epsilon)
set_image_data_format(_image_data_format)
_BACKEND = _backend
# Save config file, if possible.
if not os.path.exists(_keras_dir):
try:
os.makedirs(_keras_dir)
except OSError:
# Except permission denied and potential race conditions
# in multi-threaded environments.
pass
if not os.path.exists(_config_path):
_config = {
'floatx': floatx(),
'epsilon': epsilon(),
'backend': _BACKEND,
'image_data_format': image_data_format()
}
try:
with open(_config_path, 'w') as f:
f.write(json.dumps(_config, indent=4))
except IOError:
# Except permission denied.
pass
# Set backend based on KERAS_BACKEND flag, if applicable.
if 'KERAS_BACKEND' in os.environ:
_backend = os.environ['KERAS_BACKEND']
_BACKEND = _backend
# Import backend functions.
if _BACKEND == 'cntk':
sys.stderr.write('Using CNTK backend\n')
from .cntk_backend import *
elif _BACKEND == 'theano':
sys.stderr.write('Using Theano backend.\n')
from .theano_backend import *
elif _BACKEND == 'tensorflow':
sys.stderr.write('Using TensorFlow backend.\n')
from .tensorflow_backend import *
else:
# Try and load external backend.
try:
backend_module = importlib.import_module(_BACKEND)
entries = backend_module.__dict__
# Check if valid backend.
# Module is a valid backend if it has the required entries.
required_entries = ['placeholder', 'variable', 'function']
for e in required_entries:
if e not in entries:
raise ValueError('Invalid backend. Missing required entry : ' + e)
namespace = globals()
for k, v in entries.items():
# Make sure we don't override any entries from common, such as epsilon.
if k not in namespace:
namespace[k] = v
sys.stderr.write('Using ' + _BACKEND + ' backend.\n')
except ImportError:
raise ValueError('Unable to import backend : ' + str(_BACKEND))
def backend():
"""Publicly accessible method
for determining the current backend.
# Returns
String, the name of the backend Keras is currently using.
# Example
```python
>>> keras.backend.backend()
'tensorflow'
```
"""
return _BACKEND
Here is what I did for my own purposes, same logic as Kedar's answer, but on a Windows install (and Keras version) for which locations and file names may differ :
1/ Set a specific keras.json file, in a folder of your targeted Anaconda environment. Modify the "backend" value.
2/ Then force the 'load_backend.py' (the one specific to your anaconda env.) to load this specific keras.json.
Also, force de "default backend" to the one you want in that very same file.
=======================================================
IN DETAILS :
1.1 Open the Anaconda environment folder for which you want a specific backend. In my case it's C:\ProgramData\Anaconda3\envs\[MyAnacondaEnvironment]\
1.2 Here create a folder .keras, and in that folder copy or create a file keras.json (I copied mine from C:\Users\[MyWindowsUserProfile]\.keras\keras.json).
Now in that file, change the backend for the one you want, I've chosen 'cntk' for some tests. The file's content should now look like that :
{
"floatx": "float32",
"epsilon": 1e-07,
"backend": "cntk",
"image_data_format": "channels_last"
}
And the file's name and location looks like C:\ProgramData\Anaconda3\envs\[MyAnacondaEnvironment]\.keras\keras.json
2.1 Now open the file 'load_backend.py' specific to the environment you are customizing, located here (in my case) C:\ProgramData\Anaconda3\envs\[MyAnacondaEnvironment]\Lib\site-packages\keras\backend
2.2 Here line 17 to 25 in my Keras version (2.3.1), the file usually loads the backend from the configuration file it locates with the help of your environment variables or of your current Windows user for instance. That's why currently your backend is cross environment.
Get rid of this by forcing 'load_backend.py' to look which backend to load directly in your environment specific configuration file (the one you created at step 1.2)
For instance, line 26 of that 'load_backend.py' file (line 26 in my case, anyway right after the attempt to load the configuration file automatically) add that line (and customize it for your own location) :
_keras_dir = 'C:\ProgramData\Anaconda3\envs\[MyAnacondaEnvironment]\.keras' ##Force script to get configuration from a specific file
3.1 Then replace (line 28 in my case, anyway right after you forced _keras_dir path) the default backend _BACKEND = 'tensorflow' by _BACKEND = 'cntk'.
You should be done
One solution is to create different users for different environments and put different keras.json files for both:
$HOME/.keras/keras.json
This way you'll be able to change any keras parameter independently.
If you only need to change the backend, it is easier to use KERAS_BACKEND env variable. The following command will use tensorflow, not matter what's in keras.json:
$ KERAS_BACKEND=tensorflow python -c "from keras import backend"
Using TensorFlow backend.
So can you start a new shell terminal, run export KERAS_BACKEND=tensorflow in it and all subsequent commands will use tensorflow. You can go further and set this variable per conda env activation as discussed in this question (if you need it permanently):
$PREFIX/etc/conda/activate.d
Related
Good day everyone. I got a module from the internet which is a module about NMT. In the module I have an import for tensorflow, but unfortunately even after the installation of tensorflow in my system using pip, I still get the error. Here is the error
from tensorflow.keras.models import load_model
ModuleNotFoundError: No module named 'tensorflow'
The module hello_app.py is below:
from flask import Flask
from flask import request
from flask import jsonify
import uuid
import os
from tensorflow.keras.models import load_model
import numpy as np
EXPECTED = {
"cylinders":{"min":3,"max":8},
"displacement":{"min":68.0,"max":455.0},
"horsepower":{"min":46.0,"max":230.0},
"weight":{"min":1613,"max":5140},
"acceleration":{"min":8.0,"max":24.8},
"year":{"min":70,"max":82},
"origin":{"min":1,"max":3}
}
# Load neural network when Flask boots up
model = load_model(os.path.join("../dnn/","mpg_model.h5"))
#app.route('/api/mpg', methods=['POST'])
def calc_mpg():
content = request.json
errors = []
for name in content:
if name in EXPECTED:
expected_min = EXPECTED[name]['min']
expected_max = EXPECTED[name]['max']
value = content[name]
if value < expected_min or value > expected_max:
errors.append(f"Out of bounds: {name}, has value of: {value}, but should be between {expected_min} and {expected_max}.")
else:
errors.append(f"Unexpected field: {name}.")
# Check for missing input fields
for name in EXPECTED:
if name not in content:
errors.append(f"Missing value: {name}.")
if len(errors) <1:
x = np.zeros( (1,7) )
# Predict
x[0,0] = content['cylinders']
x[0,1] = content['displacement']
x[0,2] = content['horsepower']
x[0,3] = content['weight']
x[0,4] = content['acceleration']
x[0,5] = content['year']
x[0,6] = content['origin']
pred = model.predict(x)
mpg = float(pred[0])
response = {"id":str(uuid.uuid4()),"mpg":mpg,"errors":errors}
else:
response = {"id":str(uuid.uuid4()),"errors":errors}
print(content['displacement'])
return jsonify(response)
if __name__ == '__main__':
app.run(host= '0.0.0.0',debug=True)
Please I would really appreciate your answers. Thank you.
This is the github repo where I got the code
https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_13_01_flask.ipynb
To avoid package or version conflict, one can use virtual environment.
pip install virtualenv
virtualenv -p /usr/bin/python3 tf
source tf/bin/activate
tf$ pip install tensorflow
If you have Anaconda or conda
#Set Up Anaconda Environments
conda create --name tf python=3
#Activate the new Environment
source activate tf
tf$pip install tensorflow
Pardon my relative inexperience in Python, I am trying to run this code (taken from GitHub) but interpreter is unable to resolve the reference for ini_file_io and model (I have seen a similar post, but I am seeing the same issue both on PyCharm and MS Visual Studio Code). Directory Structure is as follows:
Here is the main.py: (both ini_file_io.py and model.py are available in same directory)
import os
import tensorflow as tf
from ini_file_io import load_train_ini #Unresolved Reference
from model import cgan_unet_xy #Unresolved Reference
def main(_):
# load training parameter #
ini_file = '../outcome/model/ini/tr_param.ini'
param_sets = load_train_ini(ini_file)
param_set = param_sets[0]
print('====== Phase >>> %s <<< ======' % param_set['phase'])
if not os.path.exists(param_set['chkpoint_dir']):
os.makedirs(param_set['chkpoint_dir'])
if not os.path.exists(param_set['labeling_dir']):
os.makedirs(param_set['labeling_dir'])
with tf.Session() as sess:
model = cgan_unet_xy(sess, param_set)
if param_set['phase'] == 'train':
model.train()
elif param_set['phase'] == 'test':
model.test()
elif param_set['phase'] == 'crsv':
model.test4crsv()
if __name__ == '__main__':
tf.app.run()
Any help will be really appreciated.
Try adding a empty file named __init__.py that is the same directory level as the 3 files in question: main, ini.., and model
Python uses these files as markers for directory level imports
Note that this should not be a concern for same dir level files. I believe you may be running the code from the wrong place. Try to cd into the directory that these files exist it and run the main.py there instead of some other directory.
If you can't do that then you'd have to add that dir to your python path.
You can also try a relative import — try from .ini_file_io import load_train_ini instead.
My traceback from running pandas takes me to:
site-packages\pandas\io\excel.py line 58, in get_writer
AttributeError: 'module' object has no attribute '__version__'
I found this link to a git issue in the PyInstaller repo
https://github.com/pyinstaller/pyinstaller/issues/1890 and found my openpyxl version, manually added it into the get_writer method like so:
def get_writer(engine_name):
if engine_name == 'openpyxl':
try:
import openpyxl
#remove when conda update available
openpyxl.__version__ = '2.3.2'
# with version-less openpyxl engine
# make sure we make the intelligent choice for the user
if LooseVersion(openpyxl.__version__) < '2.0.0':
return _writers['openpyxl1']
elif LooseVersion(openpyxl.__version__) < '2.2.0':
return _writers['openpyxl20']
else:
return _writers['openpyxl22']
except ImportError:
# fall through to normal exception handling below
pass
try:
return _writers[engine_name]
except KeyError:
raise ValueError("No Excel writer '%s'" % engine_name)
Still no dice. The line number given in the error traceback doesn't even change. I then updated the openpyxl version to 2.3.5, still receiving the error. The openpyxl init file has a version variable in it:
try:
here = os.path.abspath(os.path.dirname(__file__))
src_file = os.path.join(here, ".constants.json")
with open(src_file) as src:
constants = json.load(src)
__author__ = constants['__author__']
__author_email__ = constants["__author_email__"]
__license__ = constants["__license__"]
__maintainer_email__ = constants["__maintainer_email__"]
__url__ = constants["__url__"]
__version__ = constants["__version__"]
except IOError:
# packaged
pass
Any known or potential fixes or workarounds?
Edits were not making an impact because the process was compiled into an exe that these modules were running through. Exported the sections I needed outside of my anaconda environment and now the process works without a hitch.
I will add my workaround to this discussion as I was having the same problem using openpyxl 2.4.0 and maybe a few others are stuck too.
I found that to create a .exe file you have to revert to an older version of openpyxl. To do so:
Open the command prompt and uninstall openpyxl with 'pip uninstall openpyxl'
Reinstall openpyxl using an older version 'pip install openpyxl==2.3.5'
Now you should be able to create your .exe file using py2exe, cx_freeze, etc.
This is my fix: go to your openpyxl site-package folder (for me it's: C:\Python27\Lib\site-packages\openpyxl). Copy all variables in your .constant.json file directly into the _ init _.py file, so it looks like:
import json
import os
__author__= "See AUTHORS"
__author_email__= "eric.gazoni#gmail.com"
__license__= "MIT/Expat",
__maintainer_email__= "openpyxl-users#googlegroups.com"
__url__= "http://openpyxl.readthedocs.org"
__version__= "2.4.0"
try:
here = os.path.abspath(os.path.dirname(__file__))
I tried loading a shapefile on Python using PyQgis API but to no avail. I double-checked the path to shapefile and found it to be correct. QGIS module also seems to be imported just fine. When I checked on the provider list in QgsRegistry, it returns nothing. May I know what I'm missing or how I should troubleshoot?
I am using Ubuntu 12.04, QGIS 2.4.0 Chugiak and Python 2.7.3.
Thank you in advance!
Following are my output and code:
"
/usr/bin/python2.7 /home/victorzhiyulee/IdeaProjects/Delineation/select_dun_calculate_print.py
Application state:
QGIS_PREFIX_PATH env var:
Prefix: /usr/bin/qgis
Plugin Path: /usr/bin/qgis/lib/qgis/plugins
Package Data Path: /usr/bin/qgis/share/qgis
Active Theme Name:
Active Theme Path: :/images/themes//
Default Theme Path: :/images/themes/default/
SVG Search Paths: /usr/bin/qgis/share/qgis/svg/
User DB Path: /usr/bin/qgis/share/qgis/resources/qgis.db
Provider List
Could not find OGR provider!
File exists; Path is correct
('/home/victorzhiyulee/Desktop/dun.shp', 'dun', 'ogr')
Layer failed to load!
Process finished with exit code 0
"
__author__ = 'victorzhiyulee'
# Importing QGis API
# Importing OGR & OSR
import os
import sys
import PyQt4.QtCore
import PyQt4.QtGui
import qgis.core
import qgis.gui
from qgis.core import *
from qgis.gui import *
from osgeo import ogr, osr
from PyQt4.QtCore import *
# Supply path to the QGis resources on your PC
# noinspection PyTypeChecker
QgsApplication.setPrefixPath("/usr/bin/qgis", True)
# Load providers
QgsApplication.initQgis()
# Show setting of parameters
print QgsApplication.showSettings()
# Load vector layer
data_source = "/home/victorzhiyulee/Desktop/dun.shp"
layer_name = "dun"
provider_name = "ogr"
fileInfo = QFileInfo(data_source)
print('Provider List')
print(QgsProviderRegistry.instance().providerList())
r = QgsProviderRegistry.instance()
if not 'ogr' in r.providerList():
print 'Could not find OGR provider!'
else:
print 'Providers found ok!'
# Add layer to the registry
layer = QgsVectorLayer(data_source, fileInfo.fileName(), provider_name)
QgsMapLayerRegistry.instance().addMapLayer(layer)
if fileInfo.exists():
print("File exists; Path is correct")
print(data_source, layer_name, provider_name)
layer = QgsVectorLayer(data_source, fileInfo.fileName(), provider_name)
if not layer.isValid():
print("Layer failed to load!")
else:
print("Yes, layer loads successfully")
features = layer.getFeatures()
else:
print("Check if your path is correct")
QgsApplication.exitQgis()
iteration = layer.getFeatures()
for features in iteration:
# Fetch attributes
attris = features.attributes()
print(attris)
QgsApplication.exitQgis()
I think that the prefix path isn't correct, the path shuld be "/usr/share/qgis", so the prefix for me is only "/usr".
I have check the paths in the output of print QgsApplication.showSettings() to discover this.
I am making an app using python 2.7 on windows and keyring-3.2.1 . In my python code on eclipse, I used
import keyring
keyring.set_password("service","jsonkey",json_res)
json_res= keyring.get_password("service","jsonkey")
is working fine as I am storing json response in keyring. But, when I converted python code into exe by using py2exe, it shows import error keyring while making dist. Please suggest how to include keyring in py2exe.
Traceback (most recent call last):
File "APP.py", line 8, in <module>
File "keyring\__init__.pyc", line 12, in <module>
File "keyring\core.pyc", line 15, in <module>
File "keyring\util\platform_.pyc", line 4, in <module>
File "keyring\util\platform.pyc", line 29, in <module>
AttributeError: 'module' object has no attribute 'system'
platform_.py code is :
from __future__ import absolute_import
import os
import platform
def _data_root_Windows():
try:
root = os.environ['LOCALAPPDATA']
except KeyError:
# Windows XP
root = os.path.join(os.environ['USERPROFILE'], 'Local Settings')
return os.path.join(root, 'Python Keyring')
def _data_root_Linux():
"""
Use freedesktop.org Base Dir Specfication to determine storage
location.
"""
fallback = os.path.expanduser('~/.local/share')
root = os.environ.get('XDG_DATA_HOME', None) or fallback
return os.path.join(root, 'python_keyring')
# by default, use Unix convention
data_root = globals().get('_data_root_' + platform.system(), _data_root_Linux)
platform.py code is:
import os
import sys
# While we support Python 2.4, use a convoluted technique to import
# platform from the stdlib.
# With Python 2.5 or later, just do "from __future__ import absolute_import"
# and "import platform"
exec('__import__("platform", globals=dict())')
platform = sys.modules['platform']
def _data_root_Windows():
try:
root = os.environ['LOCALAPPDATA']
except KeyError:
# Windows XP
root = os.path.join(os.environ['USERPROFILE'], 'Local Settings')
return os.path.join(root, 'Python Keyring')
def _data_root_Linux():
"""
Use freedesktop.org Base Dir Specfication to determine storage
location.
"""
fallback = os.path.expanduser('~/.local/share')
root = os.environ.get('XDG_DATA_HOME', None) or fallback
return os.path.join(root, 'python_keyring')
# by default, use Unix convention
data_root = globals().get('_data_root_' + platform.system(), _data_root_Linux)
The issue you're reporting is due to an environment that contains invalid modules, perhaps from an improper installation of one version of keyring over another. You will want to ensure that you've removed remnants of the older version of keyring. In particular, make sure there's no file called keyring\util\platform.* in your site-packages.
After doing that, however, you'll encounter another problem. Keyring loads its backend modules programmatically, so py2exe won't detect them.
To work around that, you'll want to add a 'packages' declaration to your py2exe options to specifically include the keyring.backends package. I invoked the following setup.py script with Python 2.7 to convert 'app.py' (which imports keyring) to an exe:
from distutils.core import setup
import py2exe
setup(
console=['app.py'],
options=dict(py2exe=dict(
packages='keyring.backends',
)),
)
The resulting app.exe will import and invoke keyring.