The structure of my package is the following:
package
|___package
| |_subpackage
| | |_ __init__.py
| | |_ module_Y.py
| |_ __init__.py
| |_ module_X.py
|_ main.py
|_ setup.py
My __init__.py files are all empty and module_Y.py has the line from package import module_X.
I have not yet installed the package since it's not even remotely close to be working, but I want Pylint to be able to understand that the import statement in module_Y.py is going to be correct. I know that this must be possible because cloning the repo of TF-Agents and opening it in VS code, pylint understand the references inside the files1 even if I have not yet installed the TF-agents repo.
I know that I could use relative imports like from .. import module_X, and I know that I could just disable these pylint warnings, but these two me are half solutions. The first is not as clean and clear as the statement from package import module_X and the second solution possibly doesn't tell me of something being actually wrong.
What am I missing?
1Take for example tf_agents/agents/dqn/dqn_agent.py which is able to resolve the imports to tf_agents.policies
According to your description, I reproduced this problem, the following is my solution and you could refer to it:
Way 1:
Please add the following code at the beginning of the file "module_Y.py", which adds the file path to the upper level directory "package":
import sys
sys.path.append("..")
Please set "cwd": "${fileDirname}", in "launch.json";
Click F5 to debug the code: (Since this warning does not affect the use of the code, we can close it after the code is available: use "python.analysis.disabled": [ "unresolved-import" ], in settings.json )
Way 2:
Or you could also set in "launch.json": (It adds the folder path of the current workspace to the system path.)
"env": {
"PYTHONPATH": "${workspaceFolder}"
},
Related
Problem
I cannot get VSCode/unittest to recognize submodules (i.e., common.nested_module in the example in background) that I wrote unittests for.
VSCode discovers the tests cases and executes those in test_my_module.py but complains that there is no module common.nested_module on the import statement line in test_nested_module.py (in Output: ModuleNotFoundError: No module named 'common.nested_module').
I added print(sys.path) to the executed test case and in Python Test log I even get the following output when adding .env file as described below in What I tried 2.:
test_get_version (test_my_module.TestCase1) ... ['/workspaces/test_project/project/tests', '/workspaces/test_project/project/src', '/workspaces/test_project/project/common', '/usr/local/lib/python39.zip', '/usr/local/lib/python3.9', '/usr/local/lib/python3.9/lib-dynload', '/home/vscode/.local/lib/python3.9/site-packages', '/usr/local/lib/python3.9/site-packages']
Background
Following the PyPA tutorial for packaging and distribution I created a project structure, leaving out other details (e.g., README.md), as follows:
project
|--src
|--common
|-- __init__.py
|-- nested_module.py
|-- __init__.py
|-- my_module.py
|--tests
|--__init__.py
|--test_my_module.py
|--test_nested_module.py
Settings related to testing in settings.json:
{
"python.testing.unittestArgs": [
"-v",
"-s",
"${workspaceFolder}/project/tests",
"-p",
"test_*.py"
],
"python.testing.cwd": "project/src",
"python.testing.pytestEnabled": false,
"python.testing.unittestEnabled": true,
}
What I tried
Adding the following to my __init__.py in project/tests as well as to test_nested_module.py as suggested here.
import os
import sys
PROJECT_PATH = os.getcwd()
SOURCE_PATH = os.path.join(
PROJECT_PATH,"project/src/common"
)
sys.path.append(SOURCE_PATH)
Adding an .env to project containing the following while adding "python.envFile": "${workspaceFolder}/afinad/.env" to settings.json similar to what was suggested here.
PYTHONPATH = ./common
Adding -t option with ${workspaceFolder}/project/src and . to settings.json for python.testing.unittestArgs as sggested here:
"python.testing.unittestArgs": [
"-v",
"-s",
"${workspaceFolder}/project/tests",
"-p",
"test_*.py",
"-t",
"${workspaceFolder}/project/src"
],
Running python3 -m unittest discover after changing into ./tests in terminal gives me:
ModuleNotFoundError: No module named 'my_module'
Thanks for taking time to read. Any help is much appreciated!
After nearly two days of trying different combinations of various suggested solutions I found my mistake. It seems, newbie as I am, that I felt for the name shadowing trap.
Originally, I intended to have the same folder structure in tests as I had in src. However, when I figured that the test_my_module.py worked, I moved the test_nested_module.py from tests/common to tests but I did not remove the folder tests/common. This caused, as it seems, the ModuleNotFoundError, because after removing the folder, it works as expected.
I hope this will save others some time.
Forgive me for another "relative imports" post but I've read every stack overflow post about them and I'm still confused.
I have a file structure that looks something like this:
|---MyProject
| |---constants.py
| |---MyScripts
| | |---script.py
Inside of script.py I have from .. import constants.py
And of course when I run python3 script.py, I get the error ImportError: attempted relative import with no known parent package.
From what I understand, this doesn't work because when you run a python file directly you can't use relative imports. After hours of searching it seems that the most popular solution is to add "../" to sys.path and then do import constants.py but this feels a little hacky. I've got to believe that there is a better solution than this right? Importing constants has got to be a fairly common task.
I've also seen some people recommend adding __init__.py to make the project into a package but I don't really understand how this solves anything?
Are there other solutions out there that I've either missed or just don't understand?
Your library code should live in a package. For your folder to be a package, it needs to have __init__.py.
Your scripts can live in the top directory in development. Here is how you would structure this layout:
|---MyProject
| |---myproject
| | |---__init__.py
| | |---constants.py
| |---script.py
If you were to productionize this, you could place scripts into a special folder, but then they would rely on the package being installed on your system. The layout could look like this:
|---MyProject
| |---myproject
| | |---__init__.py
| | |---constants.py
| |---scripts
| | |---script.py
In both cases your imports would be normal absolute (non-relative) imports like
from myproject.constants import ...
OR
from myproject import constants
You are correct that attempting a relative import or a path modification in a standalone script is hacky. If you want to use the more flexible second layout, make a setup.py install script in the root folder, and run python setup.py develop.
|---MyProject
| |---__init__.py # File1
| |---constants.py
| |---MyScripts
| | |---__init__.py # File 2
| | |---script.py
And then the content of File1 __init__.py is:
from . import constants
from . import MyScripts
And then the content of File2 __init__.py is:
from . import script
By converting MyProject to a package, you would be able to import constants like:
from MyProject import constants
# Rest of your code
After that, you need to add the root of your MyProject to your python path. If you are using PyCharm, you do not need to do anything. By default the following line is added to your Python Console or Debugging sessions:
import sys
sys.path.extend(['path/to/MyProject'])
If you are not using PyCharm, one is is to add the above script before codes you run or for debugging, add that as a code to run before every session, or you define a setup.py for your MyProject package, install it in your environment and nothing else need to be changed.
The syntax from .. import x is only intended to be used inside packages (thus your error message). If you don't want to make a package, the other way to accomplish what you want is to manipulate the path.
In absolute terms, you can do:
import sys
sys.path.append(r'c:\here\is\MyProject')
import constants
where "MyProject" is the folder you described.
If you want to use relative paths, you need to do a little more work:
import inspect
import os
import sys
parent_dir = os.path.split(
os.path.dirname(inspect.getfile(inspect.currentframe())))[0]
sys.path.append(parent_dir)
import constants
To put it concisely, the error message is telling you exactly what's happening; t's just not explaining why. .. isn't a package. It resolves to MyProject but MyProject isn't a package. To make it a package, you must include an “__init__.py” file directly in the “MyProject” directory. It can even be an empty file if you're feeling lazy.
But given what you've shown us, it doesn't seem like your project is supposed to be a package. Try changing the import in script.py to import constants and running python3 MyScripts/script.py from your project directory.
I've read quite a few links but I can't find out a single link that has accurately lists out the steps that need to be followed.
PROBLEM
> I have the following directory structure (a sample).
|MAINDIR
|script1.py
> |script2.py
|COMMONSCRIPTS-------|script3.py
> | |script4.py
| |script5.py
> |TASK1---------------|script6.py
| |script7.py
I'm trying to import a function in script3.py from script6.py.
But I'm seeing an issue with "Unresolved References" in the imports section.
WHAT I'VE TRIED
Here's what I did.
I set "MAIN DIR" as the "Sources Root" so it's marked blue instead of beige.
Then I invalidated caches and restarted PyCharm but it still hasn't solved my problem.
Can someone please provide a list of steps that I can try to resolve this issue ?
BTW, I'm running PyCharm Community Edition 5.0.4
TL;DR - your dirs are not Python modules and your import namespaces may be wrong. Dirs can be easily made Python modules too by adding __init__.py like so:
mypkg/
__init__.py
s1.py
common/
__init__.py
s3.py
task1/
__init__.py
s6.py
(mypgk/, common/ and task1/ are directories obviously)
In s6.py:
from mypkg.common.s3 import myfun
Note the namespace spec: mypkg.common.s3.
I have an app structured like so:
api/
|_app/
|_ __init__.py
|_conf/
|_resources/
|_ __init__.py
|_controller.py
|_dao/
|_ __init__.py
|_thing_dao.py
|_etc...
I want to use the function get_thing_by_id in thing_dao.py inside of controller.py. In PyCharm I start to type get_thing_by_id and it prompts me to auto-import the function. The problem is it simply does from thing_dao import get_thing_by_id but when I attempt to run the app (Flask) I get ImportError: No module named 'thing_dao'. What I end up having to do is a relative import or start at the module level from app.dao.thing_dao import get_thing_by_id.
I suspect this issue is related to my project structure and less of a PyCharm bug. Is there a way I could structure my project to better hint to PyCharm how to do imports? I've looked through the configuration option for auto-imports and PyCharm and they're quite slim so I suspect that I'm doing something wrong.
I discovered I had my directories marked as a "Source Root". Unmarking them made the imports work properly.
I've seen all sorts of examples and other similar questions, but I can't seem to find an example that exactly matches my scenario. I feel like a total goon asking this because there are so many similar questions, but I just can't seem to get this working "correctly." Here is my project:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
|------- Scripts/
| |
| |----- __init__.py
| |----- CreateUser.py
| |----- FindUser.py
If I move "CreateUser.py" to the main user_management directory, I can easily use: "import Modules.LDAPManager" to import LDAPManager.py --- this works. What I can't do (which I want to do), is keep CreateUser.py in the Scripts subfolder, and import LDAPManager.py. I was hoping to accomplish this by using "import user_management.Modules.LDAPManager.py". This doesn't work. In short, I can get Python files to easily look deeper in the hierarchy, but I can't get a Python script to reference up one directory and down into another.
Note that I am able to solve my problem using:
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
import Modules.LDAPManager as LDAPManager
I've heard that this is bad practice and discouraged.
The files in Scripts are intended to be executed directly (is the init.py in Scripts even necessary?). I've read that in this case, I should be executing CreateUser.py with the -m flag. I've tried some variations on this and just can't seem to get CreateUser.py to recognize LDAPManager.py.
If I move CreateUser.py to the main user_management directory, I can
easily use: import Modules.LDAPManager to import LDAPManager.py
--- this works.
Please, don't. In this way the LDAPManager module used by CreateUser will not be the same as the one imported via other imports. This can create problems when you have some global state in the module or during pickling/unpickling. Avoid imports that work only because the module happens to be in the same directory.
When you have a package structure you should either:
Use relative imports, i.e if the CreateUser.py is in Scripts/:
from ..Modules import LDAPManager
Note that this was (note the past tense) discouraged by PEP 8 only because old versions of python didn't support them very well, but this problem was solved years ago. The current version of PEP 8 does suggest them as an acceptable alternative to absolute imports. I actually like them inside packages.
Use absolute imports using the whole package name(CreateUser.py in Scripts/):
from user_management.Modules import LDAPManager
In order for the second one to work the package user_management should be installed inside the PYTHONPATH. During development you can configure the IDE so that this happens, without having to manually add calls to sys.path.append anywhere.
Also I find it odd that Scripts/ is a subpackage. Because in a real installation the user_management module would be installed under the site-packages found in the lib/ directory (whichever directory is used to install libraries in your OS), while the scripts should be installed under a bin/ directory (whichever contains executables for your OS).
In fact I believe Script/ shouldn't even be under user_management. It should be at the same level of user_management.
In this way you do not have to use -m, but you simply have to make sure the package can be found (this again is a matter of configuring the IDE, installing the package correctly or using PYTHONPATH=. python Scripts/CreateUser.py to launch the scripts with the correct path).
In summary, the hierarchy I would use is:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
Scripts/ (*not* a package)
|
|----- CreateUser.py
|----- FindUser.py
Then the code of CreateUser.py and FindUser.py should use absolute imports to import the modules:
from user_management.Modules import LDAPManager
During installation you make sure that user_management ends up somewhere in the PYTHONPATH, and the scripts inside the directory for executables so that they are able to find the modules. During development you either rely on IDE configuration, or you launch CreateUser.py adding the Scripts/ parent directory to the PYTHONPATH (I mean the directory that contains both user_management and Scripts):
PYTHONPATH=/the/parent/directory python Scripts/CreateUser.py
Or you can modify the PYTHONPATH globally so that you don't have to specify this each time. On unix OSes (linux, Mac OS X etc.) you can modify one of the shell scripts to define the PYTHONPATH external variable, on Windows you have to change the environmental variables settings.
Addendum I believe, if you are using python2, it's better to make sure to avoid implicit relative imports by putting:
from __future__ import absolute_import
at the top of your modules. In this way import X always means to import the toplevel module X and will never try to import the X.py file that's in the same directory (if that directory isn't in the PYTHONPATH). In this way the only way to do a relative import is to use the explicit syntax (the from . import X), which is better (explicit is better than implicit).
This will make sure you never happen to use the "bogus" implicit relative imports, since these would raise an ImportError clearly signalling that something is wrong. Otherwise you could use a module that's not what you think it is.
From Python 2.5 onwards, you can use
from ..Modules import LDAPManager
The leading period takes you "up" a level in your heirarchy.
See the Python docs on intra-package references for imports.
In the "root" __init__.py you can also do a
import sys
sys.path.insert(1, '.')
which should make both modules importable.
I faced the same issues. To solve this, I used export PYTHONPATH="$PWD". However, in this case, you will need to modify imports in your Scripts dir depending on the below:
Case 1: If you are in the user_management dir, your scripts should use this style from Modules import LDAPManager to import module.
Case 2: If you are out of the user_management 1 level like main, your scripts should use this style from user_management.Modules import LDAPManager to import modules.