When I am adding new functions to a file I can't import them neither if I just run the script in terminal or if I launch ipython and try importing a function there. I have no .pyc files. It looks as if there is some kind of caching going on. I never actually faced such an issue even though have been working with various projects for a while. Why could it happen?
What I see is the following: I launch ipython and the functions that were written long time ago by other programmers can be imported fine. If I comment them out and save the file, they still can be imported without any issues. If I write new functions they can't be imported.
The directory is git directory, I cloned the repo. Then the new branch was created and I switched to it. Python version is 3.7.5, and I am working with virtual environment that I created some time ago which I activated with source activate py37.
I don't know whether its important but I have an empty __init__.py in the folder where script is located.
The code (I don't think its relevant, but still):
import hail as hl
import os
class SeqrDataValidationError(Exception):
pass
# Get public properties of a class
def public_class_props(cls):
return {k: v for k, v in cls.__dict__.items() if k[:1] != '_'}
def hello():
print('hello')
public_class_props is an old function and can be imported, but hello - can't.
assuming that you've successfully cloned the project into your machine, If it is a problem with importing functions or methods from a different .py file, please check the points below.
check your working directory, whether you're in the same directory where the .py file/module with functions/methods exist.
once you import any function/method from a module, even you comment out the function/method and save the .py file, it won't affect the already imported functions as long as you re-import it.
as long as it is a problem of importing from your own .py files, the virtual environment has nothing to do with that.
EDITED:
check this link, it may give you some information about how caching works while importing python modules.
The issue was that PYTHONPATH was set to a wrong folder. We had two folders with the project: old and new, similarly named, with identical project structure (but different file contents) and PYTHONPATH was set to the old project.
Related
I'm currently trying to import one of my modules from a different folder.
Like this...
from Assets.resources.libs.pout import Printer, ForeColor, BackColor
This import method works completely fine in PyCharm, however, when i try to launch the file in cmd or IDLE, i get this error.
ModuleNotFoundError: No module named 'Assets'
This is my file structure from main.py to pout.py:
- Assets
- main.py
- resources
- libs
- pout.py
Any clue about how i could fix this ?
Any help is appreciated !
Edit: The original answer was based on the assumption that the script you're running is within the folder structure given, which a re-read tells me may not be true. The general solution is to do
sys.path.append('path_to_Assets')
but read below for more detail.
Original answer
The paths that modules can be loaded from will differ between the two methods of running the script.
If you add
import sys
print(sys.path)
to the top of your script before the imports you should be able to see the difference.
When you run it yourself the first entry will be the location of the script itself, followed by various system/environment paths. When you run it in PyCharm you will see the same first entry, followed by an entry for the top level of the project. This is how it finds the modules when run from PyCharm. This behaviour is controlled by the "Add content roots to PYTHONPATH" option in the run configuration.
Adding this path programmatically in a way that will work in all situations isn't trivial because, unlike PyCharm, your script doesn't have a concept of where the top level should be. BUT if you know you'll be running the script from the top level, i.e. your working directory will be the folder containing Assets and you're running something like python Assets/main.py then you can do
sys.path.append(os.path.abspath('.'))
and that will add the correct folder to the path.
Appending sys path didn't work for me on windows, hence here is the solution that worked for me:
Add an empty __init__.py file to each directory
i.e. in Assets, resources, libs.
Then try importing with only the base package names.
Worked for me!
I need to pack an application written in Python.
I can't use PyInstaller because I can't have a .exe or anything like that, I need pure python scripts being executed (I have Popen in my code calling some of my scripts and passing parameters);
I need it to work cross-plataform (I KNOW that to do so, I need to generate the package in each system because a package generated on Linux won't work in a Mac and vice-versa);
Since I can't really pack it, my idea is to have a folder called modules and have all the dependencies inside it, then, in my code I would just point the imports to this folder. I would zip the entire project and ship it, the user would just unpack and run it.
The problem is:
How to not only download the package in a specific folder but also install it there? (I can't alter my user's environment, this is a MUST);
How to direct my imports to this local folder? I think I can do something like import modules.numpy for example, but then, numpy have dependencies of its own... how to make sure it will look into my custom folder?
My scenario:
I have a requirements.txt for my project; I have several local files that are used inside the project; I have a Popen that calls one of my file.py; I am using Python3;
I can't use virtualenv because I am using wxPython and there's a conflict between it and virtualenv (no idea why - something related with main thread...)
I have the following folder structure:
app
__init__.py
utils
__init__.py
transform.py
products
__init__.py
fish.py
In fish.py I'm importing transform as following: import utils.transform.
When I'm running fish.py from Pycharm, it works perfectly fine. However when I am running fish.py from the Terminal, I am getting error ModuleNotFoundError: No module named 'utils'.
Command I use in Terminal: from app folder python products/fish.py.
I've already looked into the solutions suggested here: Importing files from different folder, adding a path to the application folder into the sys.path helps. However I am wondering if there is any other way of making it work without adding two lines of code into the fish.py. It's because I have many scripts in the /products directory, and do not want to add 2 lines of code into each of them.
I looked into some open source projects, and I saw many examples of importing modules from a parallel folder without adding anything into sys.path, e.g. here:
https://github.com/jakubroztocil/httpie/blob/master/httpie/plugins/builtin.py#L5
How to make it work for my project in the same way?
You probably want to run python -m products.fish. The difference between that and python products/fish.py is that the former is roughly equivalent to doing import products.fish in the shell (but with __name__ set to __main__), while the latter does not have awareness of its place in a package hierarchy.
This expands on #Mad Physicist's answer.
First, assuming app is itself a package (since you added __init__.py to it) and utils and products are its subpackages, you should change the import to import app.utils.transform, and run Python from the root directory (the parent of app). The rest of this answer assumes you've done this. (If it wasn't your intention making app the root package, tell me in a comment.)
The problem is that you're running app.products.fish as if it were a script, i.e. by giving the full path of the file to the python command:
python app/products/fish.py
This makes Python think this fish.py file is a standalone script that isn't part of any package. As defined in the docs (see here, under <script>), this means that Python will search for modules in the same directory as the script, i.e. app/products/:
If the script name refers directly to a Python file, the directory
containing that file is added to the start of sys.path, and the file
is executed as the __main__ module.
But of course, the app folder is not in app/products/, so it will throw an error if you try to import app or any subpackage (e.g. app.utils).
The correct way to start a script that is part of a package is to use the -m (module) switch (reference), which takes a module path as an argument and executes that module as a script (but keeping the current working directory as a module search path):
If this option is given, [...] the current directory
will be added to the start of sys.path.
So you should use the following to start your program:
python -m app.products.fish
Now when app.products.fish tries to import the app.utils.transform module, it will search for app in your current working directory (which contains the app/... tree) and succeed.
As a personal recommendation: don't put runnable scripts inside packages. Use packages only to store all the logic and functionality (functions, classes, constants, etc.) and write a separate script to run your application as you wish, putting it outside the package. This will save you from this kind of problems (including the double import trap), and has also the advantage that you can write several run configurations for the same package by just making a separate startup script for each.
I wrote some Python programs about 7 or 8 years ago and I haven't touched them since, although they are still in heavy use by my company.
We have moved to a new server, and I'm trying to set that environment up. I have created all of the code, but apparently I have some sort of path problem that I don't remember how to solve. I haven't written any Python at all in at least 6 years.
At the top of this program I'm trying to execute, I have this:
from mycompany.initializer import Configurator
I'm running this program from a directory called:
/usr/code/myapp/migrate
The "mycompany" python module is located in:
/usr/code/mycompany
What path or other variable do I need to export in my user account so that above program can find the appropriate Python module?
You can manually add the path (like Kasapo shows). Though I think it would be pythonically preferable if you managed to properly install the module so that it becomes part of the python search path. One way to do that is with PYTHONPATH (ie: exporting PYTHONPATH to include '/usr/code' upon shell start up).
http://docs.python.org/using/cmdline.html
You should be able to add the directory to the system path:
import sys
sys.path.append('/usr/code/')
Also, not sure if this changed in python at any point, but you should have __init__.py in the 'mycompany' directory (and any module components/submodules in subdirectories...) to properly make it an importable module.
Currently, when trying to reference some library code, I'm doing this at the top of my python file:
import sys
sys.path.append('''C:\code\my-library''')
from my-library import my-library
Then, my-library will be part of sys.path for as long as the session is active. If I start a new file, I have to remember to include sys.path.append again.
I feel like there must be a much better way of doing this. How can I make my-library available to every python script on my windows machine without having to use sys.path.append each time?
Simply add this path to your PYTHONPATH environment variable. To do this, go to Control Panel / System / Advanced / Environment variable, and in the "User variables" sections, check if you already have PYTHONPATH. If yes, select it and click "Edit", if not, click "New" to add it.
Paths in PYTHONPATH should be separated with ";".
You should use
os.path.join
to make your code more reliable.
You have already used __my-library__ in the path. So don't use it the second time in import.
If you have a directory structure like this
C:\code\my-library\lib.py and a function in there, e.g.:
def main():
print("Hello, world")
then your resulting code should be
import sys
sys.path.append(os.path.join('C:/', 'code', 'my-library'))
from lib import main
If this is a library that you use throughout your code, you should install it as such. Package it up properly, and either install it in your site-packages directory - or, if it's specific to certain projects, use virtualenv and install it just within the relevant virtualenvs.
To do such a thing, you'll have to use a sitecustomize.py (or usercustomize.py) file where you'll do your sys.path modifications (source python docs).
Create the sitecustomize.py file into the \Lib\site-packages directory of your python installation, and it will be imported each time a python interpreter is launched.
If you are doing this interactively, the best thing to do would be to install ipython and configure your startup settings to include that code. If you intend to have it be part of a script you run from the interpreter, the same thing applies, since it will have access to your namespace.
On the other hand, a stand alone script should not include that automatically. In the future, you or some other maintainer will come along, and all the code should be obvious, and not dependent upon a specific machine setup. The best thing to do would be to set up a skeleton file for new projects that includes all of the basic functionality you need. That, along with oft-used snippets will handle the problem.
All of your code to run the script, will be in the script, and you won't have to think about adding that code every time.
Using jupyter with multiple environments, adding the path to .bashrc didn't work. I had to edit the kernel.json file for that particular kernel and append it to the PYTHONPATH in env section.
This only worked in that kernel but maybe this can help someone else.