We Keep Coding

Irerate over package_data files and copy them to current working directory

Background
I'm developing a python package with roughly the following directory structure:
mossutils/
setup.py
mossutils/
__init__.py
init.py
data/
style.css
script.js
...
My package's setup.py declares console_scripts and includes package_data files:
setup(
name='mossutils',
packages=['mossutils'],
package_data={"mossutils": ["data/*"]},
entry_points = {
"console_scripts": ['mu-init = mossutils.init:main']
},
...)
Installing the package via pip install works as expected: everything is installed in my Python's Lib\site-packages, including the data directory and all files in it, and script mu-init can be executed from the shell (or rather, command prompt, since I'm using Windows).
Goal
Script mu-init is supposed to do some kind of project scaffolding in the current working directory it is invoked from. In particular, it should copy all package_data files (data/style.css, data/script.js, ...) to the current directory.
Solution Attempt
Using module pkgutil, I can read the content of a file, e.g.
import pkgutil
...
data = pkgutil.get_data(__name__, "data/style.css")
Questions
Is there a way for my init.py script to iterate over the contents of the data directory, without hard-coding the file names (in init.py)?
Can the files from the data directory be copied to the current working directory, without opening the source file, reading the content, and writing it to a destination file?

You can get the list of files in the directory using pkg_resources library, which is distributed together with setuptools.
import pkg_resources
pkg_resources.resource_listdir("mossutils", "data")

Creating an egg file generates only EGG-INFO folder

I have a python file named file_processor.py. I would like to create an egg file out of this python file to use it in another projects. My setup.py file looks as following:
from setuptools import setup, find_packages
setup(
name = "file_processor",
version = "0.5",
packages = find_packages()
)
And I run this script with the following command:
python setup.py bdist_egg
This command generates 3 folders, namely: build, dist, file_processor.egg-info. My .egg file is located in the dist folder. However, if I change its extension from .egg to .zip to see the contents, I find only one folder which is EGG-INFO, and not the actual python file. An so, if I try to add that .egg file into my project path and import file_processor module, python throws an error that no module named file_processor found. What am I doing wrong here? Note: I got the information for generating egg files from this link

Wheel files are generally preferred over eggs these days.
Regardless, I would guess that you don't have the file_processor.py in a separate directory and you have it in the same directory as the setup.py, it needs to be in it's own directory.
You should also include a __init__.py in that directory, inside the file you can put
from .file_processor import *
This will import all the functions from your file into the package so you can use them.
This tutorial is quite good if you're looking for more information https://python-packaging.readthedocs.io/en/latest/minimal.html

Possible to create python sdist from different directory?

Want to create python source distribution by running python setup.py sdist from a directory outside of the one I want to package up. Can't seem to find a way to do this. I have a script that generates a setup.py and MANIFEST.in dynamically, and I'd like to tell python to use those files to create an sdist of the source in a different directory "over there".
What I'm doing is creating a script that lets a user create an sdist w/o any setup.py etc. They just say "package up this directory and everything under it". So I generate a setup.py and MANIFEST.in (with recursive-include * to grab all files) in a python tempfile.mkdtemp directory (in an unrelated file path like /tmp/whatever) that I can clean up afterwards...but I can't seem to use those to package their directory. I don't want to create those files in their source dir.

You can use setuptools's, --dist-dir=DIR / -d DIR option to specify where the otherwise default dist/-folder is made. In other words, this changes the output directory.
E.g.:
python setup.py sdist -d /tmp/whatever
If you are using distutils.core: Instead of using from distutils.core import setup you can use from setuptools import setup.
In order to define where the source directories come from, I think you can add the directory to sys.path and then setup() will discover the content files automatically:
import sys
from os import path
# ...
# Add other folders to sys.path
sys.path.append('/tmp/whatever')
sys.path.append(path.join(path.abspath('..'), 'some', 'folder'))

Sort of a hack, but this worked for me. Right before running setup(), use os.chdir() to change the directory to that of the base path where setup.py would normally run. To specify where the distribution packages go, I use the arguments to setup.py, specifically:
python setup.py sdist --formats=gztar -d 'directory_for_the_distribution' egg_info --egg-base 'directory_for_the_egg_info'
Thus you can run setuptools from a directory other than at the base of the package directories and the distribution and temp egg directories go wherever you want.

Importing from python modules inside parent directory into jupyter notebook files inside subdirectory

I have a file structure like this:
project_folder/
notebooks/
notebook01.ipynb
notebook02.ipynb
...
notebookXY.ipynb
module01.py
module02.py
module03.py
In .ipynb files inside notebook/ folder I want to import classes and functions from module01.py, module02.py and module03.py.
I have found answer in this question that it is possible using following lines of code inside every notebook and run those lines as first cell every time:
import os
import sys
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
sys.path.append(module_path)
Is there please a better way for this? What if I have A LOT of .ipynb files inside notebooks/ folder, do I have to paste those lines of code at the beginning of every single one? Is there a better, more minimalist or cleaner way?

Try adding the project_folder to your PYTHONPATH environment variable. This will allow you to tell python to search that directory for imports.
You would do this in your user profile settings, or in your startup script - not in python. It's something that has to be set before python ever gets run.

Another solution is to move all your Python modules (.py files) into a folder and make them an installable package. If you pip install it into your current environment, you can then import the package into any notebook in that environment, regardless of folder structure.
So in your situation you could have:
project_folder/
notebooks/
notebook01.ipynb
notebook02.ipynb
...
notebookXY.ipynb
my_package/
__init__.py
module01.py
module02.py
module03.py
setup.py
__init__.py can just be an empty file, and tells Python "everything in this folder is part of a package"
For an explanation of what goes in setup.py see here.
A basic setup.py can be as simple as this:
import setuptools
setuptools.setup(
name="my_package",
version="0.0.1",
description="A small example package",
packages=setuptools.find_packages(),
python_requires='>=3.7',
)
Install it:
cd project_folder
pip install [-e] .
Including the optional -e flag will install my_package in "editable" mode, meaning that instead of copying the files into your virtual environment, a symlink will be created to the files where they are.
Now in any notebook you can do:
import my_package
Or
from my_package.module01 import <some object>

Running a script from a package

I'm new to python coming from java. I created a folder called: 'Project'. In 'Project' I created many packages (with __init__.py files) like 'test1' and 'tests2'. 'test1' contains a python script file .py that uses scripts from 'test2' (import a module from test2). I want to run a script x.py in 'test1' from command line. How can I do that?
Edit: if you have better recommendations on how I can better organize my files I would be thankful. (notice my java mentality)
Edit: I need to run the script from a bash script, so I need to provide full paths.

There are probably several ways to achieve what you want.
One thing that I do when I need to make sure the module paths are correct in an executable scripts is to get the parent directory and insert in the module search path (sys.path):
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
import test1 # next imports go here...
from test2 import something
# any import what works from the parent dir will work here
This way you are safe to run your scripts without worrying how the script is called.

Python code is organized into modules and packages. A module is just a .py file that can contain class definitions, function definitions and variables. A package is a directory with a __init__.py file.
A standard Python project might look something like this:
thingsproject/
README
setup.py
doc/
...
things/
__init__.py
animals.py
vegetables.py
minerals.py
test/
test_animals.py
test_vegetables.py
test_minerals.py
The setup.py file describes the metadata about your project. See Writing the Setup Script and particularly the section on installing scripts.
Entry points exist to help distribute command line tools in Python. An entry point is defined in setup.py like this:
setup(
name='thingsproject',
....
entry_points = {
'console_scripts': ['dog = things.animals:dog_main_function']
},
...
)
The effect is that when the package is installed using python setup.py install a script is automatically created in some reasonable place according to your OS, such as /usr/local/bin. The script then calls the dog_main_function in the animals module of the things package.
Yet another Python convention to consider is have a __main__.py file. This signifies the "main" script within a directory or zip file full of python code. This is a good place to define a command line interface to your code using the argparse parser for command line arguments.
Good and up-to-date information on the somewhat muddled world of Python packaging can be found in the Python Packaging User Guide.