A project (Python 3.6) consists of some folders. In some files there is scope extension like
sys.path.append('../foo')
due to import not from the same directory where the file is.
It works from PyCharm and when the application starts from a command line as python.exe app.py
But this scope extension for import doesn't work when projects starts as service. It is checked when starting project with pm2 utilite and just starting as service. Scope extension for import just doesn't work and import fails.
The way to avoid the error is to change the relative paths in sys.path.append with absolute like
from pathlib import Path
sys.path.append(str(Path(__file__).parent.parent)+'/foo')
After this changes the code starts as service without errors.
I suspect that relative paths don't work when the application running as service. But why?
sys.pqth.append() method simply append it's argument as it is. Thus, if you run it in Python console, you can see the result like this:
>>> sys.path.append("../foo")
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/yunbo/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', '../foo']
When you run your code in PyCharm, it may works fine. But if you run your code as a service, CWD is not the same as your project folder(or some sub path). Your code will look for the CWD/../foo and that won't be there. That's why your code fails to run as a service.
Using sys.path or pathlib is the correct way to do it.
When you start your project as a service using pm2, what is current working directory your python interpreter is in? os.getcwd().
It's likely not in your project root.
According to https://pm2.keymetrics.io/docs/usage/application-declaration/#general
pm2 can be configured to set cwd - so you need to set it to your project root.
Related
I have the following folder structure:
app
__init__.py
utils
__init__.py
transform.py
products
__init__.py
fish.py
In fish.py I'm importing transform as following: import utils.transform.
When I'm running fish.py from Pycharm, it works perfectly fine. However when I am running fish.py from the Terminal, I am getting error ModuleNotFoundError: No module named 'utils'.
Command I use in Terminal: from app folder python products/fish.py.
I've already looked into the solutions suggested here: Importing files from different folder, adding a path to the application folder into the sys.path helps. However I am wondering if there is any other way of making it work without adding two lines of code into the fish.py. It's because I have many scripts in the /products directory, and do not want to add 2 lines of code into each of them.
I looked into some open source projects, and I saw many examples of importing modules from a parallel folder without adding anything into sys.path, e.g. here:
https://github.com/jakubroztocil/httpie/blob/master/httpie/plugins/builtin.py#L5
How to make it work for my project in the same way?
You probably want to run python -m products.fish. The difference between that and python products/fish.py is that the former is roughly equivalent to doing import products.fish in the shell (but with __name__ set to __main__), while the latter does not have awareness of its place in a package hierarchy.
This expands on #Mad Physicist's answer.
First, assuming app is itself a package (since you added __init__.py to it) and utils and products are its subpackages, you should change the import to import app.utils.transform, and run Python from the root directory (the parent of app). The rest of this answer assumes you've done this. (If it wasn't your intention making app the root package, tell me in a comment.)
The problem is that you're running app.products.fish as if it were a script, i.e. by giving the full path of the file to the python command:
python app/products/fish.py
This makes Python think this fish.py file is a standalone script that isn't part of any package. As defined in the docs (see here, under <script>), this means that Python will search for modules in the same directory as the script, i.e. app/products/:
If the script name refers directly to a Python file, the directory
containing that file is added to the start of sys.path, and the file
is executed as the __main__ module.
But of course, the app folder is not in app/products/, so it will throw an error if you try to import app or any subpackage (e.g. app.utils).
The correct way to start a script that is part of a package is to use the -m (module) switch (reference), which takes a module path as an argument and executes that module as a script (but keeping the current working directory as a module search path):
If this option is given, [...] the current directory
will be added to the start of sys.path.
So you should use the following to start your program:
python -m app.products.fish
Now when app.products.fish tries to import the app.utils.transform module, it will search for app in your current working directory (which contains the app/... tree) and succeed.
As a personal recommendation: don't put runnable scripts inside packages. Use packages only to store all the logic and functionality (functions, classes, constants, etc.) and write a separate script to run your application as you wish, putting it outside the package. This will save you from this kind of problems (including the double import trap), and has also the advantage that you can write several run configurations for the same package by just making a separate startup script for each.
Trying to figure this out, because there is an inconsistency between when I run the code from Pycharm and from terminal.
Pycharm add automatically the current working directory; so if I add a module that is contained in my CWD, that is not in Pythonpath, it works just fine.
But when running from terminal, Python does complain, because my import statements refer to modules that are not reachable, because the CWD is not added to PYTHONPATH (I did verify this printing out the content of the variable, while running from Pycharm and from Terminal).
So at this point I am assuming that in my startup code, I need to add to Pythonpath the current directory, or this is not correct? I have no way to tell where the final user may put my code in; I just assume that the whole directory containing all my different modules, is located in a specific place.
To be more specific, this is where I am at:
my CWD when I run from Pycharm is /apps/myapp/logic/, I run the script after cd in that directory, and I call the script with ./myscript.py
The script has the #!/usr/bin/python3 line as first line, instead of running it with python3 -m myscript.py
The error I get, is when loading a module that is either in the same directory of my script (/apps/myapp/logic/) or one level above (/apps/myapp/); sadly the module load happen before my __main__ is running; so I can't add to sys.path the current directory from which the script run.
All these issues are not happening if I just run the script from Pycharm
After various trial and error, and thanks to the info that I did get from the comments; I did find 2 ways to solve the issue.
1) Create a shell script or another python script, which is adding the current directory (where all the files lives), and have no import in this file. Then the script call the script that has the main function.
2) On the top of the module, right after import sys, add the statement to add the path, in this way the current directory will be added to the PATH and it will be accessible, when the import try to access the module.
Neither look very nice, but this is all that I was able to find, to solve the issue. Pretty sure there is a more elegant way to do so
I've written my own mail.py module in spider (anaconda). I want to import this py file in other python (spider) files just by 'import mail'
I searched on the internet and couldn't find a clearly solution.
To import any python script, it should exist in the PYTHONPATH. You can check this with the following code:
import sys
print sys.path
To import your Python script:
Put both the scripts (main and the imported python script) in the
same directory.
Add the location of the file to be imported to the
sys.path.
For example, if the script is located as '/location/to/file/script.py':
import sys
sys.path.append('/location/to/file/')
import script
I had the same problem, my files were in same folder, yet it was throwing an error while importing the "to_be_imported_file.py".
I had to run the "to_be_imported_file.py" seperately before importing it to another file.
I hope it works for you too.
Searched for an answer for this question to. To use a .py file as a import module from your main folder, you need to place both files in one folder or append a path to the location. If you storage both files in one folder, then check the working directory in the upper right corner of the spyder interface. Because of the wrong working directory you will see a ModuleNotFoundError.
There are many options, e.g.
Place the mail.py file alongside the other python files (this works because the current working dir is on the PYTHONPATH.
Save a copy of mail.py in the anaconda python environment's "/lib/site-packages" folder so it will be available for any python script using that environment.
I did a slightly different solution approach that is less sophisticated. When I start my anaconda terminal it is at a C prompt. I just did a cd d:\mypython\lib in the beginning window before starting python. once I did that I could simply just import my own classes that I put in that library with "import MyClass as my" then I was off and running. It is interesting, I did 2 days of internet searching in my part time and could not find the answer either, until I asked a friend.
cd d:\mypython\lib
python
>>> import MyClass as my
>>> my1=my.MyClass()
>>> my1.doSomething()
worked for me on my anaconda / windows 10 environment python 3.6.6.
when calling any function from another file, it should be noted to not import any library inside the function
I believe the easiest solution is to place the directory containing your Python files into the Anaconda site-packages folder on your machine. I wrote an article outlining the whole process but, in short, you'll need to create a folder containing your python script and an __init__.py file. Then place that folder inside the site-packages folder in the Anaconda directory.
On Windows the site-packages directory is typically located at:
C:\Users\[your_username]\Anaconda3\Lib\site-packages\
On Mac the site-packages directory is typically located located at:
Users/[your_username]/opt/anaconda3/lib/[python3.8]/site-packages/
Notice, on Mac the Python version matters. You'll need to look for the directory that corresponds to the (base) Python version used by Anaconda. Also, anything I've placed inside of square brackets in the file paths above need to be changed according to your particular machine and Python version.
The file structure should look something like this:
~/
|__site-packages/
|__your_folder/
script.py
__init__.py
After you have the folder containing your script.py and __init__.py file moved into the site-packages subdirectory of Anaconda, you'll be able to import it from any script you run on your machine.
I am trying to deploy my project to a server and run it there.
When I try to start a script from command line it shows errors
when importing scripts that are in parrent directories.
I made the project (python 2.7.10) using PyCharm and it is spread out into multiple directories.
The folders look something like this:
project/dir/subdir/main_dir/script1.py
from dir.subdir.other_dir.script2 import * //gives error here
project/dir/subdir/other_dir/script2.py
def my_function():
//do something
I run the script by going to the main_dir and running: python script1.py
If you are running your script from the main_dir, that means when running your Python command, your relative reference is main_dir. So your imports are with respect to main_dir being your root.
This means if we take your script1 for example, your import should look like this:
from other_dir.script2 import *
Chances are your PyCharm project root is actually set to run from
project/
Which is why your references work within PyCharm.
What I suggest you do is, if your server is supposed to run within main_dir then you should re-configure PyCharm so that its execution root is the same in order to remove this confusion.
An alternative solution to this problem in my case was to add a main.py script in the root of the python project which triggers the program.
project/__main__.py:
from dir.subdir.other_dir.script2 import * //doesn't give errors
This means that when calling the program from the terminal the workspace would be correct and every inclusion of script would have the folders mapped correctly (from the root).
project/dir/subdir/main_dir/script1.py:
from dir.subdir.other_dir.script2 import * //also doesn't give errors
Another solution where you can skip the parent directories while importing (and don't have have to change anything in your script going from a Pycharm execution to a manual execution):
from script2 import *
works when you set the PYTHONPATH variable before running your script, e.g. like this in Windows:
set PYTHONPATH=../other_dir && python script1.py
for Linux (bash) it is:
PYTHONPATH=../other_dir python script1.py
I believe this is also what PyCharm does upon execution: adding the according folders to the PYTHONPATH.
Im developing an installer for a GNU/Linux distribution in Python using Eclipse+PyDev. For some tasks on it there is needed that the program runs with root priviledges, but I run Eclipse as a common user.
I had searched a lot of stuff on the Internet about how to run an app as root without having to run Eclipse with priviledges, but no a single clue of how to accomplish this in a "nice way". So I tried with the "gksu2" python module, with has the gksu2.sudo() functions in the same way as gksu in bash.
I created a new module, imported gksu2 and executed the main.py module of the app, but I got a "ImportError: No module named ui.regular_ui.wizard". It runs ok without gksu2 in eclipse, but it doesn't if I use it. I thought it was an environment variables problem, but the sys.path is ok.
The same error happens if I run the app from a terminal, outside of Eclipse. What do you think?
It seems like your PYTHONPATH is different outside/inside Eclipse. Try just removing the Python interpreter and adding it again to gather new paths -- if that's not enough, do: import sys;print('\n'.join(sorted(sys.path))) outside/inside Eclipse to know what's different and fix your paths inside Eclipse.