I'm doing the building Neo4j applications with Python course on Neo4j's GraphAcademy and am stuck early in the process with Setting Environment Variables. I've installed the dependencies (FLASK etc.) but don't seem to have an .env file for the next part...
Setting Environment Variables
This project will read environment variables from the .env file located in the project root.
The project contains an example file at .env.example. You can run the following command in your terminal window to copy the example file to .env.
cp .env.example .env
But when I try to run this in the shell I get the following error:
cp: .env.example: No such file or directory
I don't seem to have a .env file in any of the newly created folders in the sandbox. Can anyone help with this?
For me, this worked:
Clone the git repository that provides the scaffolding
git clone https://github.com/neo4j-graphacademy/app-python.git
Change directory to be in the newly checked project root folder.
This step is not explicitly mentioned in the graph academy course.
cd app-python
Copy the template env file
cp .env.example .env
Inspect the file to make sure it looks right
cat .env
which printed
FLASK_APP=api
FLASK_ENV=development
FLASK_RUN_PORT=3000
NEO4J_URI=neo4j://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=neo
JWT_SECRET=secret
SALT_ROUNDS=10
If you still can't get it to work, let me know which step fails and with which error.
Python version: 3.6.9
I've used pickle to dump a machine learning model into a file, and when I try to run a prediction on it using Flask, it fails with ModuleNotFoundError: No module named 'predictors'. How can I fix this error so that it recognizes my model, whether I try to run a prediction via Flask or via the Python command (e.g. python predict_edu.py)?
Here is my file structure:
- video_discovery
__init__.py
- data_science
- model
- __init__.py
- predict_edu.py
- predictors.py
- train_model.py
Here's my predict_edu.py file:
import pickle
with open('model', 'rb') as f:
bow_model = pickle.load(f)
Here's my predictors.py file:
from sklearn.base import TransformerMixin
# Basic function to clean the text
def clean_text(text):
# Removing spaces and converting text into lowercase
return text.strip().lower()
# Custom transformer using spaCy
class predictor_transformer(TransformerMixin):
def transform(self, X, **transform_params):
# Cleaning Text
return [clean_text(text) for text in X]
def fit(self, X, y=None, **fit_params):
return self
def get_params(self, deep=True):
return {}
Here's how I train my model:
python data_science/train_model.py
Here's my train_model.py file:
from predictors import predictor_transformer
# pipeline = Pipeline([("cleaner", predictor_transformer()), ('vectorizer', bow_vector), ('classifier', classifier_18p)])
pipeline = Pipeline([("cleaner", predictor_transformer())])
with open('model', 'wb') as f:
pickle.dump(pipeline, f)
My Flask app is in: video_discovery/__init__.py
Here's how I run my Flask app:
FLASK_ENV=development FLASK_APP=video_discovery flask run
I believe the issue may be occurring because I'm training the model by running the Python script directly instead of using Flask, so there might be some namespace issues, but I'm not sure how to fix this. It takes a while to train my model, so I can't exactly wait on an HTTP request.
What am I missing that might fix this issue?
It seems a bit strange that you get that error when executing predict_edu.py, as it is in the same directory as predictors.py, and thus, using absolute import such as from predictors import predictor_transformer (without the dot . operator) should normally work as expected. However, below are a few options that you could try out, if the error persists.
Option 1
You could add the parent directory of the predictors file to the system PATH variable, before attempting to import the module, as described here. This should work fine for smaller projects.
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from predictors import predictor_transformer
Option 2
Use relative imports, e.g., from .predictors import..., and make sure you run the script from the parent directory of your package, as shown below. The -m option "searches the sys.path for the named module and execute its contents as the __main__ module", and not as the top-level script. Read more about the -m option in the following references: [1], [2], [3], [4], [5], [6]. Read more about "relative imports" here: [1], [2], [3], [4].
python -m video_discovery.data_science.predict_edu
However, the PEP 8 style guide recommends using absolute imports in general.
Absolute imports are recommended, as they are usually more readable
and tend to be better behaved (or at least give better error messages)
if the import system is incorrectly configured (such as when a
directory inside a package ends up on sys.path)
In certain cases, however, absolute imports can get quite verbose, depending on the complexity of the directory structure, as shown below. On the other hand, "relative imports can be messy, particularly for shared projects where directory structure is likely to change". They are also "not as readable as absolute ones, and it is hard to tell the location of the imported resources". Read more about Python Import and Absolute vs Relative Imports.
from package1.subpackage2.subpackage3.subpackage4.module5 import function6
Option 3
Include the directory containing your package directory in PYTHONPATH and use absolute imports instead. PYTHONPATH is used to set the path for user-defined modules, so that they can be directly imported into a Python script. The PYTHONPATH variable is a string with a list of directories that need to be added to the sys.path directory list by Python. The primary use of this variable is to allow users to import modules that have not yet made into an installable Python package. Read more about it here and here.
For instance, let’s say you have a package named video_discovery (under /Users/my_user/code/video_discovery) and wanted to add the directory /Users/my_user/code to the PYTHONPATH:
On Mac
Open Terminal.app
Open the file ~/.bash_profile in your text editor – e.g. atom ~/.bash_profile
Add the following line to the end: export PYTHONPATH="/Users/my_user/code"
Save the file.
Close Terminal.app
Start Terminal.app again, to read in the new settings, and type
echo $PYTHONPATH. It should show something like /Users/my_user/code.
On Linux
Open your favorite terminal program
Open the file ~/.bashrc in your text editor – e.g. atom ~/.bashrc
Add the following line to the end: export PYTHONPATH=/home/my_user/code
Save the file.
Close your terminal application.
Start your terminal application again, to read in the new settings,
and type echo $PYTHONPATH. It should show something like /home/my_user/code.
On Windows
Open This PC (or Computer), right-click inside and select
Properties.
From the computer properties dialog, select Advanced system settings on the left.
From the advanced system settings dialog, choose the Environment variables button.
In the Environment variables dialog, click the New button in the
top half of the dialog, to make a new user variable:
Give the variable name as PYTHONPATH and in value add the path to
your module directory. Choose OK and OK again to save this variable.
Now open a cmd window and type echo %PYTHONPATH% to confirm the environment variable is correctly set. Remember to open a new cmd window to run your Python program, so that it picks up the new settings in PYTHONPATH.
Option 4
Another solution would be to install the package in an editable state (all edits made to the .py files will be automatically included in the installed package), as described here and here. However, the amount of work required to get this to work might make Option 3 a better choice for you.
The contents for the setup.py should be as shown below, and the command for installing the package should be pip install -e . (-e flag stands for "editable" and . stands for "current directory").
from setuptools import setup, find_packages
setup(name='myproject', version='1.0', packages=find_packages())
From https://docs.python.org/3/library/pickle.html:
pickle can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was stored.
When you run python data_science/train_model.py and import from predictors, Python imports predictors as a top-level module and predictor_transformer is in that module.
However, when you run a prediction via Flask from the parent folder of video_discovery, predictor_transformer is in the video_discovery.data_science.predictors module.
Use relative imports and run from a consistent path
train_model.py: Use relative import
# from predictors import predictor_transformer # -
from .predictors import predictor_transformer # +
Train model: Run train_model with video_discovery as top-level module
# python data_science/train_model.py # -
python -m video_discovery.data_science.train_model # +
Run a prediction via a Python command: Run predict_edu with video_discovery as top-level module
# python predict_edu.py # -
python -m video_discovery.data_science.predict_edu # +
Run a prediction via Flask: (no change, already run with video_discovery as top-level module)
FLASK_ENV=development FLASK_APP=video_discovery flask run
I have a problem similar to that described here How to fix "ImportError: No module named ..." error in Python? but I cannot fix it with the suggestion to set PYTHONPATH.
my directory looks like:
- project
- python
- src
- ml
- __init__.py
- classifier_dnn.py
- util.py
- vectorizer
- fv_davison.py
- __init__.py
And I am running classifier_dnn.py at the project folder path:
~project&PYTHONPATH=/home/project/
~project$python3 /home/project/python/src/ml/classifier_dnn.py /home/project/data/labeled_data_all.csv /home/project/output
But an error is generated when classifier_dn imports ml.util:
Traceback (most recent call last):
File "/home/project/chase/python/src/ml/classifier_dnn.py", line 5, in <module>
from ml import util
ImportError: No module named 'ml'
I have also tried setting PYTHONPATH=/home/project/python or PYTHONPATH=/home/project/src but the same error happens.
When I test this in PyCharm, it works if set python/src to become source root, regardless what working directory is. But I cannot figure out how to set this properly when I run this from command line.
Any help please
Thanks
I have a writeup on this but I'll copy the relevant text inline.
You have two problems:
You need to export PYTHONPATH (export PYTHONPATH=/full/path/to/src)
You should run with python -m module.path instead of python path/to/file.py
The core problem is python path/to/file.py puts path/to on the beginning of the PYTHONPATH (sys.path)
This causes imports to start from (in your case) src/ml instead of the expected src.
By using -m, you avoid this path munging and it will correctly keep src as the beginning of your sys.path.
When I test this in PyCharm, it works if set python/src to become source root, regardless what working directory is. But I cannot figure out how to set this properly when I run this from command line.
You can do this by cding into the src directory
You should not put executable scripts into a library structure.
Only library modules should go into the package structure. Executables should stay outside it. This is important for installation and day-to-day use. Once you have installed your package (using distutils for example) using it would be complicated, if you leave the executable in the library. You would have to give the complete installation path into the python library (and don't forget about -m!) every time or create shortcuts or other workarounds.
Seperate executables and library. Executables are installed to something like usr/local/bin and are callable without path, -m or other problems, and your library is installed into the python library.
I am facing a small problem in executing custom made python code from Hiero. As everybody knows, the first place to look for details is the offical documentation "Nukepedia", the Python Dev Guide and the Hiero User Guide. Well according to the Python Dev Guide, Hiero creates a ".hiero" directory by default which lets people add plugin paths to use them in the software. Well, I can't find that directory and I deduced after several tests that Hiero is using the init.py saved in the ".nuke" directory.
So I thought that maybe I could add the plugin paths there but kept getting the famous Import Error for the _fnpython module (before creating Python/Startup directory).
I added Python/Startup folders in .nuke and added the plugins in Startup, I got the same error, I even tried it by adding the path to the plugins in init.py and got the same error too.
Then I created a ".hiero" folder and did the same thing as before but Hiero never took that folder into consideration, I deduced that by printing some strings in the console, Hiero always took the init.py saved in the ".nuke" folder and kept showing the same error.
Finally, I tried to look into the installation process and try to seperate Nuke and Hiero's folders maybe that would create the ".hiero" directory but everything was automated.
The code that I want to run is given by Nuke and Hiero (in the examples directory), I just can't figure out what to do in order to run it from the program.
What should I do in order to fix this problem ?
Thank you in advance.
The setup for The Foundry HIERO is a little different than for NUKE.
HIERO has a core module. You'll see it in __init__.py file:
import FnRedirect
import FnPythonFixes
import core
import ui
try:
if hasattr(core, "taskRegistry"):
import importers
import exporters
except ImportError as e:
pass
I'm running HIERO and NUKE STUDIO on a Mac, so there's a full path to HIERO's __init__.py file inside package contents:
/Applications/Nuke10.5v5/Contents/MacOs/pythonextensions/site-packages/hiero/__init__.py
You need to import this module using:
import hiero.core
or using a reference to the core package:
from core import *
To find HIERO's current paths you have to run this line in its Script Editor:
print hiero.core.pluginPath()
Click this link for further details: Hiero's Environment Setup
All these instructions are suitable for macOS 10.9 and above. Here are two blocks of steps: first for Terminal Mode and second for UI Mode.
BLOCK 1: setup for Terminal Sessions
You need to manually create .hiero directory in your Home area.
The recommended default location for running Python on startup is:
~/.hiero/Python/Startup
~/.hiero/Python/StartupUI
Type in your bash Terminal (when you're inside your Home user directory) the following line:
mkdir .hiero/
then:
mkdir .hiero/Python/
and then:
mkdir .hiero/Python/StartupUI/
then navigate to Home directory with:
cd ~
and check it with:
ls -a
Also you can specify any number of user-defined paths using the environment variable HIERO_PLUGIN_PATH, just like the standard Unix PATH environment variable.
For that you need to set up an environment variable in .bash_profile. To run in Terminal PICO editor just type (remember you need an administrator's password for sudo command):
sudo pico .bash_profile
and paste these three lines in it (change swift for <yourName> and save this file):
echo HIERO environment var is set...
export HIERO_PLUGIN_PATH=/Users/swift/.hiero/Python/StartupUI/
export PATH=$PATH:$HIERO_PLUGIN_PATH
Write out a file with ctrl o
Exit pico editor with ctrl x
Restart Terminal
In Terminal you could print this environment variable typing:
printenv HIERO_PLUGIN_PATH
You should put inside that StartupUI directory menu.py, any set of xxxx.py or xxxx.pyc files, as well as __init__.py file.
Now you can use /Users/swift/.hiero/Python/StartupUI/ path in Terminal Mode.
BLOCK 2: setup for UI Sessions
To assign the current paths that HIERO searches when loading plug-ins, you need to create __init__.py file with the following lines:
import hiero.core
path='/Users/swift/.hiero/Python/Startup/'
hiero.core.addPluginPath(path)
After that make Python/Startup/ subdirectories under ~/.nuke/ folder.
It's not a mistake: I typed .nuke.
Then place this __init__.py file into /Users/swift/.nuke/Python/Startup/ directory.
Restart HIERO (or NUKE STUDIO) if it works.
After that launch HIERO or NUKE STUDIO and run
print hiero.core.pluginPath()
command in the HIERO's Script Editor or in NUKE STUDIO's Script Editor and you'll see this result:
After that you'll find new __init__.pyc file in /Users/swift/.nuke/Python/Startup/ directory.
Each package or module discovered when you launch HIERO is imported and added to the built-in package hiero.plugins.
TEST 1: custom_guides.py
I do not have a commercial version of HIERO so I tested custom_guides.py script ( found here ) using NUKE STUDIO NC.
I placed custom_guides.py in ~/.nuke/Python/Startup directory and then added two lines to NUKE's init.py file located in ~/.nuke directory.
import nuke
nuke.pluginAddPath("./Python/Startup")
The only thing I could say: "it works" Do the same actions as I did and it'll work for HIERO.
Look at safe_zone and masking_ratio dropdown menus. They are different: before and after.
Before uploading custom_guides.py script:
After uploading custom_guides.py script:
# file custom_guides.py contains these lines:
viewer_masks = [
hiero.ui.guides.MaskGuide("NTSC", 0.91),
hiero.ui.guides.MaskGuide("PAL", 1.09),
hiero.ui.guides.MaskGuide("NTSC_16:9", 1.21),
hiero.ui.guides.MaskGuide("PAL_16:9", 1.46),
hiero.ui.guides.MaskGuide("Cinemascope 2:1", 2.0)
]
TEST 2: web_browser.py
I placed web_browser.py file in ~/.nuke/Python/Startup directory. This Python script creates dockable panel with web browser written with PySide Qt.
I do not have a commercial version of HIERO so I tested web_browser.py script ( found here ) using NUKE STUDIO NC.
Using the ConfigParser module I am attempting to read in a configuration file so I can safely allow users to edit settings for an application and share these configurations between scripts.
This has been working perfectly for the main script. However there is a secondary script that is called and it reads the same configuration file in the same location but returns an error that the location cannot be found.
Both scripts are located within the same directory, application/bin/
The configuration file is located in application/conf/ To reference the config file successfully in the main script I use the following code which works perfectly.
config = ConfigParser.ConfigParser()
config.readfp(open('../conf/settings.conf'))
When the secondary script executes with the same code it reports that the location does not exist? I used the logger module and got it to log sys.path[0] which correctly returned the same bin folder as the main script. Is there possibly something simple I am missing here?
Also any troubleshooting tips for problems like these are welcome.
You can prefer to use dirname and __file__:
from os.path import dirname, join
config = ConfigParser.ConfigParser()
config_fn = join(dirname(__file__), '..', 'conf', 'settings.conf')
config.read(config_fn)
Depending how you're launching the app (python bin/app.py, or python app.py), the '..' will be incorrect. By starting from the directory of the .py file, you'll always be able to construct the path from the .py to the .conf using that method.