I have this git repo structure:
.gitignore
JSONs/subdirA/some.json
JSONs/subdirB/other.json
MyPackage/__init__.py
MyPackage/myModule.py
How do I properly pack the JSONs folder into MyPackage/JSONs, without moving it there permanently (mostly because customers use this git repo directly for non-python usage, and the folder at the top of the repo is easy/intuitive... But now I also want to release this same dir into my PyPi package)?
I've tried adding it to the MANIFEST.in and then playing with data_files in setup.py as well as package_data... But to no avail. Maybe some .pyc or cached build files got the best of me... But I haven't figured it out from all the other (not quite) duplicate questions as they don't specifically call out their directory structure and desired final location.
I've tried resorting to os.walk and shutil.copy before the call to setup and then deleting that directory after setup... While it seems to work locally, when pushing to our local devpi package server, something goes wrong. Is my goal totally off the radar for setuptools/pip ideals??? Or am I just missing some key understanding? Please enlighten me!
Something like the following could help:
First we need to make sure that the json files are added to the source distribution.
MANIFEST.in:
recursive-include JSONs *.json
Then in the actual setup script, the list of packages has to be modified on the fly to take into account the target package structure.
setup.py:
#!/usr/bin/env python3
import setuptools
PACKAGES = (
setuptools.find_packages(exclude=['JSONs*'])
+
[
f'MyPackage.{package}'
for package
in setuptools.find_namespace_packages(include=['JSONs*'])
]
)
setuptools.setup(
packages=PACKAGES,
package_dir={
'MyPackage.JSONs': 'JSONs',
},
include_package_data=True,
#
name='Something',
version='1.2.3',
)
JSONs/subdirA/some.json :
{"Marco": "Polo"}
Such package data can be read like so:
MyPackage/myModule.py:
import pkgutil
print(pkgutil.get_data('MyPackage', 'JSONs/subdirA/some.json').decode())
And use it like in the following:
$ python -m pip install .
$ # Move to another directory to prevent that the current working directory
$ # ... overshadows the installed project
$ cd ..
$ python -m MyPackage.myModule
{"Marco": "Polo"}
I created project and helper modules for it. But some of modules are using each other like worker 1 uses helper1 and helper2 also uses helper1. So I completle confused how I need to import all those modules so so can work standalone (for example I want to debug helper2 out of main script) and they still will be functional. Summarizing - how to correctly import modules so maint_script will work and other modules when using out of main_script. Sorry for my English.
main program dir/
main_script.py
-classes/
|
|--helper1.py
|--helper2.py
-worker_classes/
|
|--worker1.py
At the moment I'am using this constructions in the begging of each script, but I feel that this approach isn't appropriate for python
import os
import sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), 'shell_modules')))
The way I deal with imports inside a project is to install the project in editable mode. This way, all files will be able to locate each other, always starting from your project root directory.
In order to do this, follow these steps:
1) write a setup.py file and add it to your project root folder - it doesn't need much info at all:
# setup.py
from setuptools import setup, find_packages
setup(name='MyPackageName', version='1.0.0', packages=find_packages())
2) install your package in editable mode (ideally from a virtual environment). From a terminal in your project folder, write
$ pip install -e .
Note the dot - this means "install the package from the current directory in editable mode".
3) your files are now able to locate each other, always starting from the project root. To import helper1.py, for example, you write:
from classes import helper1
or alternatively:
from classes.helper1 import foo, bar
This will be true to import helper1.py for any file, no matter where it is located in the project structure.
Like I said, you should use a virtual environment for this, so that pip does not install your package to your main Python installation (which could be messy if your project has many dependencies).
Currently my favorite tool for this is pipenv. When using it, replace the terminal command with
$ pipenv install -e .
So that your project gets added to the Pipfile.
When I upload a (zipped) deployment package as a lambda function on AWS I get "no module named..." errors for both bs4 and google.
I created a virtual environment using venv and I installed the required dependencies
The app works fine when running from within the virtual environment. But, when I zip it up and upload it as a lambda function on AWS, I get "no module named..." errors for both "bs4" and (if I hash out the import of bs4 for debug reasons) also for "google". I checked the site-packages folder in the zip file and they seem to be there.
Why is AWS saying there is no module when there is?!
I am using python3.6 on Ubuntu.
Lambda needs ZIP with all the libraries and your main python code file in the same folder.
Here is what i do:
Create a new package folder with the following hierarchy
mkdir -p ./package/tmp/lib
Copy Project folder into the temp folder
cp -a ./$(PROJECT)/. ./package/tmp/
Copy python site-packages from virtual env to temp folder in package
cp -a $(VIRTUAL_ENV)/lib/python2.7/site-packages/. ./package/tmp/
Remove any unused libraries (that are not required for this particular lambda to run) from temp folder
rm -rf ./package/tmp/wheel*
Zip the temp package directory
cd ./package/tmp && zip -r ../$(PROJECT).zip .
This final zip so created is ready for upload on Lambda.
I have a script that is placed in a folder structure as such:
~/wofc/folder1/folder2/script.py
script.py uses the git module to go some tasks. However when I run the script from outside of folder2 i.e. when I have cd into folder1 i run python folder2/script.py arg1 arg2 I get the raise InvalidGitRepositoryError(epath) error. The script runs fine when I run it from inside folder2 i.e. cd into folder2 and run python script.py arg1 arg2. Below is the relevant code snippet. Can You please let me know what is the issue?
git = Repo('{}/..'.format(os.getcwd())).git
git.checkout('master')
git.pull()
Instead of Repo('{}/..'.format(os.getcwd())).git, use os.path.abspath:
git = Repo(os.path.abspath('{}/..'.format(os.getcwd())).git
git.checkout('master')
git.pull()
To run git commands, the current folder should be a git repo.
.git repo should be present to execute git commands.
That is the reason for the error.
The problem is that you use os.getcwd() which returns the current working directory. If you stand just outside folder2 this function will return ~/wofc/folder1.
You should swap it for something like:
import os
os.path.dirname(os.path.abspath(__file__))
For example like this:
import os
path = os.path.dirname(os.path.abspath(__file__))
git = Repo('{}/..'.format(path)).git
git.checkout('master')
git.pull()
As user1846747 said, gitPython requires a Repo object to run a git command.
This is a classic bootstrap issue (chicken and egg problem): "how can I run a git command running gitPython to find where the Repo root is, when I need to know where the root is to create a Repo object to run a git command?"
#MaxNoe solved this in Find the root of the git repository where the file lives with his python-gitpath project httpsgithub.com/MaxNoe/python-gitpath
I'm having a hard time understanding how module importing works in Python (I've never done it in any other language before either).
Let's say I have:
myapp/__init__.py
myapp/myapp/myapp.py
myapp/myapp/SomeObject.py
myapp/tests/TestCase.py
Now I'm trying to get something like this:
myapp.py
===================
from myapp import SomeObject
# stuff ...
TestCase.py
===================
from myapp import SomeObject
# some tests on SomeObject
However, I'm definitely doing something wrong as Python can't see that myapp is a module:
ImportError: No module named myapp
In your particular case it looks like you're trying to import SomeObject from the myapp.py and TestCase.py scripts. From myapp.py, do
import SomeObject
since it is in the same folder. For TestCase.py, do
from ..myapp import SomeObject
However, this will work only if you are importing TestCase from the package. If you want to directly run python TestCase.py, you would have to mess with your path. This can be done within Python:
import sys
sys.path.append("..")
from myapp import SomeObject
though that is generally not recommended.
In general, if you want other people to use your Python package, you should use distutils to create a setup script. That way, anyone can install your package easily using a command like python setup.py install and it will be available everywhere on their machine. If you're serious about the package, you could even add it to the Python Package Index, PyPI.
The function import looks for files into your PYTHONPATH env. variable and your local directory. So you can either put all your files in the same directory, or export the path typing into a terminal::
export PYTHONPATH="$PYTHONPATH:/path_to_myapp/myapp/myapp/"
exporting path is a good way. Another way is to add a .pth to your site-packages location.
On my mac my python keeps site-packages in /Library/Python shown below
/Library/Python/2.7/site-packages
I created a file called awesome.pth at /Library/Python/2.7/site-packages/awesome.pth and in the file put the following path that references my awesome modules
/opt/awesome/custom_python_modules
You can try
from myapp.myapp import SomeObject
because your project name is the same as the myapp.py which makes it search the project document first
You need to have
__init__.py
in all the folders that have code you need to interact with.
You also need to specify the top folder name of your project in every import even if the file you tried to import is at the same level.
In your first myapp directory ,u can add a setup.py file and add two python code in setup.py
from setuptools import setup
setup(name='myapp')
in your first myapp directory in commandline , use pip install -e . to install the package
pip install on Windows 10 defaults to installing in 'Program Files/PythonXX/Lib/site-packages' which is a directory that requires administrative privileges. So I fixed my issue by running pip install as Administrator (you have to open command prompt as administrator even if you are logged in with an admin account). Also, it is safer to call pip from python.
e.g.
python -m pip install <package-name>
instead of
pip install <package-name>
In my case it was Windows vs Python surprise, despite Windows filenames are not case sensitive, Python import is. So if you have Stuff.py file you need to import this name as-is.
let's say i write a module
import os
my_home_dir=os.environ['HOME'] // in windows 'HOMEPATH'
file_abs_path=os.path.join(my_home_dir,"my_module.py")
with open(file_abs_path,"w") as f:
f.write("print('I am loaded successfully')")
import importlib
importlib.util.find_spec('my_module') ==> cannot find
we have to tell python where to look for the module. we have to add our path to the sys.path
import sys
sys.path.append(file_abs_path)
now importlib.util.find_spec('my_module') returns:
ModuleSpec(name='my_module', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fa40143e8e0>, origin='/Users/name/my_module.py')
we created our module, we informed python its path, now we should be able to import it
import my_module
//I am loaded successfully
This worked for me:
from .myapp import SomeObject
The . signifies that it will search any local modules from the parent module.
Short Answer:
python -m ParentPackage.Submodule
Executing the required file via module flag worked for me. Lets say we got a typical directory structure as below:
my_project:
| Core
->myScript.py
| Utils
->helpers.py
configs.py
Now if you want to run a file inside a directory, that has imports from other modules, all you need to do is like below:
python -m Core.myscript
PS: You gotta use dot notation to refer the submodules(Files/scripts you want to execute). Also I used python3.9+. So I didnt require neither any init.py nor any sys path append statements.
Hope that helps! Happy Coding!
If you use Anaconda you can do:
conda develop /Path/To/Your/Modules
from the Shell and it will write your path into a conda.pth file into the standard directory for 3rd party modules (site-packages in my case).
If you are using the IPython Console, make sure your IDE (e.g., spyder) is pointing to the right working directory (i.e., your project folder)
Besides the suggested solutions like the accepted answer, I had the same problem in Pycharm, and I didn't want to modify imports like the relative addressing suggested above.
I finally found out that if I mark my src/ (root directory of my python codes) as the source in Interpreter settings, the issue will be resolved.