We Keep Coding

Install a local/personal library so it can be imported directly by a python project

I have a silly question, but I haven't found any mention of it so far. I created a .py file in Python containing all of my functions that I am using for a job. Whenever I need to use them in a script, I have to address the path to the folder where the .py file is located, like the script below.
import os
os.chdir('...\\path-to-my-file')
import my-file as mfl
My question is: is there any way I can save the .py file with my functions right at the root of Anaconda and call it the same way I call Numpy, for example? I know that the libraries are in the 'C:\Users\User\anaconda3\Lib' folder, so I could save directly to that folder and call it in a more simplified way, like:
import numpy as np
import my-file as mfl
If this is feasible, would there be a standardized way to write the code?

In order to be able to import mypackage the same way you do with any other module, the correct approach is to use pip locally:
python -m pip install -e /path_to_package/mypackage/
python -m ensures you are using the pip package from the same python installation you are currently using.
-e makes it editable, i/e import mypackage will reload after you make some changes, instead of using the cached one.
mypackage must be an installable package, i/e contain an __init__.py
file, and a basic setup.py (or pyproject.toml file for pipenv)
minimal setup.py
from setuptools import find_packages, setup
setup(
name='mypackage', # Required
version='0.0.1', # Required
packages=find_packages(), # Required
)
the package structure must be like this:
mypackage/
setup.py
mypackage/ <----- this is a folder inside the other `mypackage/` folder
__init__.py
or as a tree:
└── python_perso folder
└── mypackage folder
├── mypackage folder
│ └── __init__.py
└── setup.py
[edit] after installation, the directory will look like this:
(for a package named mypackage)
└── python_perso
└── mypackage
├── mypackage
│   ├── __init__.py
│   └── __pycache__
│   └── __init__.cpython-38.pyc
├── mypackage.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
└── setup.py
5 directories, 7 files

I would suggest you create an Environment Variable called PYTHONPATH which will point you to additional functions. It is better to leave your anaconda root as it is if you are unsure of what you are doing. More on this here.

ModuleNotFoundError when running individual Python files with absolute imports

ModuleNotFoundError running Python 3.8.x
I'm building a python package containing classes and functions each with verbose tests. Inside the package I'm trying to use these building blocks to provide end-to-end uses and examples by using absolute imports at the top of these files (like in my tests).
The project structure is as follows:
.
├── __init__.py
├── setup.py
├── examples
│   ├── __init__.py
│   └── end_to_end_1.py
├── tests
│ ├── __init__.py
│   └── utils
│ ├── __init__.py
│ ├── test_useful_one.py
│ └── test_useful_two.py
└── utils
├── __init__.py
├── useful_one.py
└── useful_two.py
I'm running all tests from the package root using python -m unittest tests/**/*.py, and both test files contain absolute package imports from utils like so,
from utils.useful_one import UsefulClass
this approach is succesful in running the tests and importing the UsefulClass class into the test files.
My issue arises when trying to use the same import statement inside the examples/end_to_end_1.py module and executing the file (again from the root of the package) using
python examples/end_to_end_1.py
Now I get a runtime ModuleNotFoundError: No module named 'utils'.
In trying to follow the python language guidelines I'm trying to use absolute imports where possible but to no avail.
I have definitely misunderstood how the __init__.py files are supposed to tell the runtime where to resolve packages from. I don't think this use case is abnormal, since I see this same pattern inside node packages and ruby gems all the time.
Makeshift (Temporary) Solution
At the moment to solve this I have applied the solution from Absolute import results in ModuleNotFoundError which, despite working, seems not too scalable when the repository is publically available and I feel like the Python ecosystem will have a solution for this issue. As Raymond Hettinger says, it feels like..
There must be a better way!

Package serverless python service with individually packaged functions but shared code

Files from the utils folder are not included in the package when individually packaging with serverless-python-requirements plugin.
In my Serverless.com AWS Python project I have the following folder structure:
.
├── serverless.yml
├── generate_features
│   ├── requirements.txt
│   └── generate_features.py
├── requirements.txt
├── utils
│   ├── utility.py
│   └── additional_code.py
│  
:
The relevant section of my serverless.yml looks as follows
package:
individually: true
functions:
generate-features:
handler: generate_features.handler
module: generate_features
timeout: 400
...
I would not include everything that is in the utils folder with each individually packaged function (there are more than one and they share some code).
Unfortunately when I use the serverless-python-requirements it appears that it won't let me do that. It only includes whatever is in the module directory. I would basically like to add additional modules though.
Any ideas? Am I not seeing some obvious way to include utils/? adding
package:
include:
- utils/**
at the function level unfortunately doesn't seem to work.
Thx

Python: include a third party library in a personal Python package

I would like to include a third party library into my Python script folder to distribute it all togehter (I am awary of the distribution license, and this library is fine to distribute). This is in order to avoid installing the library on another machine.
Saying I have a script (my_script.py), which calls this external library. I tried to copy this library from the site-packages subdirectory of Python directory into the directory where I have my files, but it seems not to be enough (I think th reason is in the __init__.py of this library which probably needs the folder to be in the PYTHONPATH).
Would it be reasonable to insert some lines of code in my_script.py to temporary append its folder to sys.path in order to make the all things working?
For instance, if I have a structure similar to this:
Main_folder
my_script.py
/external_lib_folder
__init__.py
external_lib.py
and external_lib_folder is the external library I copied from site-packages and inserted in my Main_folder, would it be fine if I write these lines (e.g.) in my_script.py?
import os,sys
main_dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(main_dir)
EDIT
I ended up choosing the sys.path.append solution. I added these lines to my my_script.py:
import os, sys
# temporarily appends the folder containing this file into sys.path
main_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)),'functions')
sys.path.append(main_dir)
Anyway, I chose to insert this as an edit in my question and accept the answer of Torxed because of the time he spent in helping me (and of course because his solution works as well).

Python3
import importlib.machinery, imp
namespace = 'external_lib'
loader = importlib.machinery.SourceFileLoader(namespace, '/home/user/external_lib_folder/external_lib.py')
external_lib = loader.load_module(namespace)
# How to use it:
external_lib.function(data_or_something)
This would be an ideal way to load custom paths in Python 3.
Not entirely sure this is what you wanted but It's relevant enough to post an alternative to adding to sys.path.
Python2
In python 2 you could just do (if i'm not mistaken, been a while since i used an older version of Python):
external_lib = __import__('external_lib_folder')
This does however require you to keep the __init__.py and a proper declaration of functions in sad script, otherwise it will fail.
**It's also important that the folder you're trying to import from is of the same name that the __init__.py script in sad folder is trying to import it's sub-libraries from, for instance geopy would be:
./myscript.py
./geopy/
./geopy/__init__.py
./geopy/compat.py
...
And the code of myscript.py would look like this:
handle = __import__('geopy')
print(handle)
Which would produce the following output:
[user#machine project]$ python2 myscript.py
<module 'geopy' from '/home/user/project/geopy/__init__.pyc'>
[user#machine project]$ tree -L 2
.
├── geopy
│   ├── compat.py
│   ├── compat.pyc
│   ├── distance.py
│   ├── distance.pyc
│   ├── exc.py
│   ├── exc.pyc
│   ├── format.py
│   ├── format.pyc
│   ├── geocoders
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── location.py
│   ├── location.pyc
│   ├── point.py
│   ├── point.pyc
│   ├── units.py
│   ├── units.pyc
│   ├── util.py
│   ├── util.pyc
│   └── version.pyc
└── myscript.py
2 directories, 20 files
Because in __init__.py of geopy, it's defined imports such as from geopy.point import Point which requires a namespace or a folder of geopy to be present.
There for you can't rename the folder to functions and place a folder called geopy in there because that won't work, nor will placing the contents of geopy in a folder called functions because that's not what geopy will look for.
Adding the path to sys.path (Py2 + 3)
As discussed in the comments, you can also add the folder to your sys.path variable prior to imports.
import sys
sys.path.insert(0, './functions')
import geopy
print(geopy)
>>> <module 'geopy' from './functions/geopy/__init__.pyc'>
Why this is a bad idea: It will work, and is used by many. The problems that can occur is that you might replace system functions or other modules might get loaded from other folders if you're not careful where you import stuff from. There for use .insert(0, ...) for most and be sure you actually want to risk replacing system built-ins with "shady" path names.

What you suggest is bad practice, it is a weak arrangement. The best solution (which is also easy to do) is to package it properly and add an explicit dependency, like this:
from setuptools import setup
setup(name='funniest',
version='0.1',
description='The funniest joke in the world',
url='http://github.com/storborg/funniest',
author='Flying Circus',
author_email='flyingcircus#example.com',
license='MIT',
packages=['funniest'],
install_requires=[
'markdown',
],
zip_safe=False)
This will work if the third party library is on pipy. If it's not, use this:
setup(
...
dependency_links=['http://github.com/user/repo/tarball/master#egg=package-1.0']
...
)
(See this explanation for packaging).

directory structure for a project that mixes C++ and Python

Say you want want to create a programming project that mixes C++ and Python. The Foo C++ project structure uses CMake, and a Python module is created by using Swig. The tree structure would look something like this:
├── CMakeLists.txt
├── FooConfig.cmake.in
├── FooConfigVersion.cmake.in
├── Makefile
├── README
├── foo
│   ├── CMakeLists.txt
│   ├── config.hpp.in
│   ├── foo.cpp
│   └── foo.hpp
└── swig
└── foo.i
Now you would like to make use of the Foo project within a Python project, say Bar:
├── AUTHORS.rst
├── CONTRIBUTING.rst
├── HISTORY.rst
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.rst
├── docs
│   ├── Makefile
│   ├── authors.rst
│   ├── conf.py
│   ├── contributing.rst
│   ├── history.rst
│   ├── index.rst
│   ├── installation.rst
│   ├── make.bat
│   ├── readme.rst
│   └── usage.rst
├── bar
│   ├── __init__.py
│   └── bar.py
├── requirements.txt
├── setup.cfg
├── setup.py
├── tests
│   ├── __init__.py
│   └── test_bar.py
└── tox.ini
This structure was crated by using cookiecutter's pypackage template. A BoilerplatePP template is also available to generate a CMake C++ project using cookiecutter (no Swig part).
So now that I have the structure of both projects, and considering that the development will take place mainly in Python and the the project will be run in different systems, I need to address the following questions:
What's the best way to mix them? Should I collapse both root directories? Should I have the Foo C++ project as a directory of the Bar project or the other way around? I may be inclined to put the entire C++ structure shown above in a folder at the root level of the Python project, but I would like to know a priori any pitfalls as the CMake system is quite powerful and it may be convenient to do it the other way around.
In case I decide to put the Foo project as a directory within Bar, is the Python setuptools package as powerful as the CMake build system? I ask this because when I take a look at the Bar project, at the top level it seems there's only a bunch of scripts, but I don't know if this is the equivalent to CMake as I'm new to Python.
The Bar project outlined above has a single bar directory, but I assume that whenever this project expands, instead of having many other directories at the root level, other directories containing Python code will be placed within bar. Is this correct (in the Pythonic sense)?
I assume that a single egg will be produced from the entire project, so that it can be installed and run in many different python systems. Is the integration of the module created by the Foo project easy? I assume that this module will be created in a different directory than bar.
In order for the Python code within the bar directory, the module created by Swig has to be available, so I guess the most straightforward way to do this is to modify the environmental variable PYTHONPATH using the CMake system. Is this fine or is there a better way?

If the C++ application has no use outside the Python package that will contain it:
You can pretty safely place the C++ code within the python package that owns it. Have the "foo" directory within the "bar" directory within your example. This will make packaging the final Python module a bit easier.
If the C++ application is reusable:
I would definitely try to think of things in terms of "packages", where independent parts are self-contained. All independent parts live on the same level. If one part depends on another, you import from its corresponding "package" from the same level. This is how dependencies typically work.
I would NOT include one within the other, because one does not strictly belong to the other. What if you started a third project that needed "foo", but did not need "bar"?
I would place both "foo" and "bar" packages into the same "project" directory (and I would probably give each package it's own code repository so each package can be easily maintained and installed).