We Keep Coding

Python3 ModuleNotFoundError when running from command line but works if I enter the shell

I think I'm missing something obvious here. I cloned this repo, and now have this directory structure on my computer:
When I try to run python baby_cry_detection/pc_main/train_set.py, I get a ModuleNotFoundError.
Traceback (most recent call last):
File "baby_cry_detection/pc_main/train_set.py", line 10, in <module>
from baby_cry_detection.pc_methods import Reader
ModuleNotFoundError: No module named 'baby_cry_detection'
However, if I type python and enter the interactive shell and then type the command
from baby_cry_detection.pc_methods import Reader
it imports just fine with no error. I'm completely baffled. I'm using a virtualenv and both instances are using the same python installation, and I haven't changed directories at all.

I think sys.path could be the reason that the module is not found when python command is executed. Here is how we can check if that is indeed the case:
In the train_set.py file, add import sys; print(sys.path). Looking at the error, the path may contain /path/to/baby_cry_detection/baby_cry_detection/pc_main. If that is the case, then we have found the issue which is that baby_cry_detection.pc_methods will not be found in the directory that sys.path is looking into. We'll need to append the parent baby_cry_detection directory to sys.path or use relative imports. See this answer.
The reason that python prompt successfully imports the module could be because the prompt is started in the correct parent directory. Try changing the directory to baby_cry_detection/pc_main/ and try importing the module.

Python external project script execution

After many researches on the web, on many different topics, I wasn't able to find the solution answering my problem.
I am developing a tool (with graphic interface) allowing to process python scripts from different projects.
To not have any problem of dependences, I ask the user to target the project directory as well as the script to process.
Thus to develop this tool I have in my IDE a project "benchmark_tool" which is the project integrating the GUI, and the project "test_project" which groups various testing scripts. Obviously, the final goal is the tool to be compiled and thus that the project "benchmark_tool" disappears from IDE.
Here is the IDE architecture:
benchmark_tool (python project)
__init__.py
main.py
test_project (python project)
__init__.py
module1
__init__.py
test_script.py
module2
__init__.py
imports.py
How it works: The main.py shall call test_script.py.
test_script.py calls imports.py at the first beggining of the script.
UPDATE
I tried some modifications in my main code:
import sys
import subprocess
subprocess.check_call([sys.executable, '-m', 'test_project.module1.test_script'], cwd='D:/project/python')
I got this error
Traceback(most recent call last):
File "C:/Python31/lib/runpy.py", line 110, in run module as main
mod_name, loader, code, fname = _get_module_detail(mod_name)
File "C:/Python31/lib/runpy.py", line 91, in get module details
code = loader.get_code(mod_name)
File "C:/Python31/lib/pkgutil.py", line 272, in get code
self.code = compile(source, self.filename, 'exec')
File "D:/project/python/test_project/module1/test_script.py", line 474
SyntaxError: invalid syntax
Traceback (most recent call last):
File "D:/other_project/benchmark_tool/main.py", line 187, in read
subprocess.check_call([sys.executable, '-m', 'module1.test_script.py'], cwd='D:/project/python/test_project')
File "C:/Python31/lib/subprocess.py", line 446, in check call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:/Python31/python.exe', '-m', 'module1.test_script.py']' returned non-zero exit status 1
Note
It works with subprocess.Popen(['python','D:/project/python/test_project/module1/test_script.py])
What's the main difference between both methods ? I'll also have to pass arguments to test_scripts.py, which one is the best to use to communicate with a python script (input and output datas are exchanged) ?
Thanks by advance

There are 3 problems I see that you have to fix:
In order to import from module2, you need to turn that directory into a package by placing an empty *__init__.py file in it
When executing test_script.py, you need the full path to the file, not just file name
Fixing up sys.path only work for your script. If you want to propagate that to test_script.py, set the PYTHONPATH environment variable
Therefore, my solution is:
import os
import sys
import subprocess
# Here is one way to determine test_project
# You might want to do it differently
script_dir = os.path.abspath(os.path.dirname(__file__))
test_project = os.path.normpath(os.path.join(script_dir, '..', 'test_project'))
python_path = '{}:{}'.format(os.environ.get('PYTHONPATH', ''), test_project)
python_path = python_path.lstrip(':') # In case PYTHONPATH was not set
# Need to determine the full path to the script
script = os.path.join(test_project, 'module1', 'test_script.py')
proc = subprocess.Popen(['python', script], env=dict(PYTHONPATH=python_path))

To run an "external project script", install it e.g.: pip install project (run it in a virtualenv if you'd like) and then run the script as you would with any other executable:
#!/usr/bin/env python
import subprocess
subprocess.check_call(['executable', 'arg 1', '2'])
If you don't want to install the project and it has no dependencies (or you assume they are already installed) then to run a script from a given path as a subprocess (assuming there is module1/__init__.py file):
#!/usr/bin/env python
import subprocess
import sys
subprocess.check_call([sys.executable, '-m', 'module1.test_script'],
cwd='/abs/path/to/test_project')
Using -m to run the script, avoids many import issues.
The above assumes that test_project is not a Python package (no test_project/__init__.py). If test_project is itself a Python package then include it in the module name and start from the parent directory instead:
subprocess.check_call([sys.executable, '-m', 'test_project.module1.test_script'],
cwd='/abs/path/to')
You could pass the path to test_project directory as a command-line parameter (sys.argv) or read it from a config file (configparser,json). Avoid calculating it relative to the installation path of your parent script (if you have to then use pkgutil, setuptools' pkg_resources, to get the data). To find a place to store user data, you could use appdirs Python package.

How do I replicate the way PyCharm is running my Python 3.4 project at the command line?

My project looks like this:
running-pycharm-project-at-cmd
- main.py
- c
- run_project.py
- z
- __init__.py
- the_module.py
- y
- __init__.py
- template.md
- the_module_module.py
- the_support_function.py
The contents of the .py files are shown below:
main.py
from c.run_project import run
print('running main.py...')
run()
c/run_project.py
from c.z.the_module import the_module_function, the_module_write_function
def run():
print('Running run_project.py!')
the_module_function()
# write a file:
the_module_write_function(read_file='./z/y/template.md', write_file='../README.md')
if __name__ == '__main__':
run()
c/z/the_module.py
from c.z.y.the_module_module import the_module_module_function
def the_module_function():
print('the_module_function is running!')
the_module_module_function()
pass
def the_module_write_function(read_file, write_file):
with open(read_file, 'r') as fid:
with open(write_file, 'w') as fid_out:
contents = fid.read()
contents.replace('{}', 'THE-FINAL-PRODUCT!')
fid_out.write(contents)
c/z/y/the_module_module.py
from .the_support_function import this_support_data
def the_module_module_function():
print('The module module function is running!')
print("Here is support data: {}".format(this_support_data))
pass
c/z/y/the_support_function.py
this_support_data = "I am the support data!"
GitHub repo to make replication easier: running-pycharm-project-at-cmd
The problem: in Pycharm load up the project with running-pycharm-project-at-cmd as the root. Then I right click and run the run_project.py file. Everything works fine. I cannot figure out how to run run_project.py from the command line in the same way. Creating main.py was a workaround suggested elsewhere, but it fails due to the relative references to the template file (in actual project, the y folder is a git submodule and therefore cannot have absolute references in it).

If you copy your main.py file into the c folder and rename it __init__.py, then you could run:
$ python -m c
The -m argument tells python to run a module or package (in this case c). Python will look in the c's folder for an __init__.py file and run that. You just need to ensure that the folder c is either in your PYTHONPATH or is a subfolder of the current working directory.

I circled back to this and was able to get it working. Note that I am on Windows 7, and some of this might be platform dependent. In order to get this working without changing any code, the key is setting PYTHONPATH environment variable before running. PyCharm does this automatically (by default, but it is configurable in the Run Configuration options).
Steps:
Recreating the issue:
Clone the github repo to G:\TEST.
Navigate to G:\TEST\running-pycharm-project-at-cmd-master\c
python run_project.py
Traceback (most recent call last):
File "run_project.py", line 1, in <module>
from c.z.the_module import the_module_function, the_module_write_function
ImportError: No module named 'c'
Simplest Solution (if able to run Python from the script location):
set the PYTHONPATH environment variable before running. e.g.
SET PYTHONPATH=G:\TEST\running-pycharm-project-at-cmd-master
Now python run_project.py
Running run_project.py!
the_module_function is running!
The module module function is running!
Here is support data: I am the support data!
To run from a remote location (this is definitely Windows specific):
Create a .bat file that navigates to correct working directory before running Python. i.e.:
START cmd /K "cd G:\TEST\running-pycharm-project-at-cmd-master\c & python run_project.py"
G:\TEST\myotherlocation
run_it.bat
Running run_project.py!
the_module_function is running!
The module module function is running!
Here is support data: I am the support data!

how to provide 'make directory as source root' from pyCharm to terminal

I have some problems with migrating to production:
cabox#box-codeanywhere:~/workspace/PEP$ python ./dev_scrapers/jordan.py
Traceback (most recent call last):
File "./dev_scrapers/jordan.py", line 3, in <module>
from utils import create_entity, create_id, custom_opener
ImportError: No module named utils
i have used pyCharm with button 'make directory as source root'
how to execute such command in terminal?

You should add your source root directory to PYTHONPATH:
export PYTHONPATH="${PYTHONPATH}:/your/source/root"

You can set the environment variable PYTHONPATH within the terminal as suggested by the accepted answer. This has to be done every time you start a terminal. In the case you use a virtual environment, you can place the variable assignment, like export PYTHONPATH="/your/source/root", in the file venv/bin/activate. Where venv stands for the name of the virtual environment directory.

I was able to set directory as root (and fix the unresolved import issue) doing the following:
from pycharm>Settings>Project>Project Structure select your project and from the file tree, select your django project directory then click the blue folder Source button to define the directory as your source.

Why is PYTHONPATH being ignored?

I am setting PYTHONPATH to have a directory that includes a few .py files.
When I go into python and type "import file", the file cannot be find (it says "No module named wsj10").
If, however, I cd to the directory, and repeat the same process, then the file is found.
I am just not sure why PYTHONPATH is being ignored. I followed exact instructions from installation instructions of some software, so I know I am doing the right thing.
Any circumstances under which PYTHONPATH will be ignored, or import won't work?
Thanks.
Following a comment below, here is a transcript:
untar file1.tgz to file1/. file1.tgz contains a library/file called file1.py.
type in the shell:
export PYTHONPATH=`pwd`/file1/:./
echo $PYTHONPATH shows the variable was set.
run python and type "import file1"
I get the error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named file1
If I do first "cd file1" and then to import file1 it identifies the file.

Any circumstances under which PYTHONPATH will be ignored, or import won't work?
Yes. I've set PYTHONPATH in my /home/me/.bashrc and all worked ok from terminal, but when Apache w/ mod_wsgi starts my python scripts, it acts under sysem or dedicated user, which knows nothing of my .bashrc.
For this particular situation, I just used apache config to set python path for apache (WSGIPythonPath option).