We Keep Coding

How to create a .exe similar to pip.exe, jupyter.exe, etc. from C:\Python37\Scripts?

In order to make pip or jupyter available from command-line on Windows no matter the current working directory with just pip ... or jupyter ..., Python on Windows seems to use this method:
put C:\Python37\Scripts in the PATH
create a small 100KB .exe file C:\Python37\Scripts\pip.exe or jupyter.exe that probably does not much more than calling a Python script with the interpreter (I guess?)
Then doing pip ... or jupyter ... in command-line works, no matter the current directory.
Question: how can I create a similar 100KB mycommand.exe that I could put in a directory which is in the PATH, so that I could do mycommand from anywhere in command-line?
Note: I'm not speaking about pyinstaller, cxfreeze, etc. (that I already know and have used before); I don't think these C:\Python37\Scripts\ exe files use this.

Assuming you are using setuptools to package this application, you need to create a console_scripts entry point as documented here:
https://packaging.python.org/guides/distributing-packages-using-setuptools/#console-scripts
For a single top-level module it could look like the following:
setup.py
#!/usr/bin/env python3
import setuptools
setuptools.setup(
py_modules=['my_top_level_module'],
entry_points={
'console_scripts': [
'mycommand = my_top_level_module:my_main_function',
],
},
# ...
name='MyProject',
version='1.0.0',
)
my_top_level_module.py
#!/usr/bin/env python3
def my_main_function():
print("Hi!")
if __name__ == '__main__':
my_main_function()
And then install it with a command such as:
path/to/pythonX.Y -m pip install path/to/MyProject

In NixOS, how can I install an environment with the Python packages SpaCy, pandas, and jenks-natural-breaks?

I'm very new to NixOS, so please forgive my ignorance. I'm just trying to set up a Python environment---any kind of environment---for developing with SpaCy, the SpaCy data, pandas, and jenks-natural-breaks. Here's what I've tried so far:
pypi2nix -V "3.6" -E gcc -E libffi -e spacy -e pandas -e numpy --default-overrides, followed by nix-build -r requirements.nix -A packages. I've managed to get the first command to work, but the second fails with Could not find a version that satisfies the requirement python-dateutil>=2.5.0 (from pandas==0.23.4)
Writing a default.nix that looks like this: with import <nixpkgs> {};
python36.withPackages (ps: with ps; [ spacy pandas scikitlearn ]). This fails with collision between /nix/store/9szpqlby9kvgif3mfm7fsw4y119an2kb-python3.6-msgpack-0.5.6/lib/python3.6/site-packages/msgpack/_packer.cpython-36m-x86_64-linux-gnu.so and /nix/store/d08bgskfbrp6dh70h3agv16s212zdn6w-python3.6-msgpack-python-0.5.6/lib/python3.6/site-packages/msgpack/_packer.cpython-36m-x86_64-linux-gnu.so
Making a new virtualenv, and then running pip install on all these packages. Scikit-learn fails to install, with fish: Unknown command 'ar rc build/temp.linux-x86_64-3.6/liblibsvm-skl.a build/temp.linux-x86_64-3.6/sklearn/svm/src/libsvm/libsvm_template.o'
I guess ideally I'd like to install this environment with nix, so that I could enter it with nix-shell, and so other environments could reuse the same python packages. How would I go about doing that? Especially since some of these packages exist in nixpkgs, and others are only on Pypi.

Caveat
I had trouble with jenks-natural-breaks to the tune of
nix-shell ❯ poetry run python -c 'import jenks_natural_breaks'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/matt/2022/12/28-2/.venv/lib/python3.10/site-packages/jenks_natural_breaks/__init__.py", line 5, in <module>
from ._jenks_matrices import ffi as _ffi
ModuleNotFoundError: No module named 'jenks_natural_breaks._jenks_matrices'
So I'm going to use jenkspy which appears to be a bit livelier. If that doesn't scratch your itch, I'd contact the maintainer of jenks-natural-breaks for guidance
Flakes
you said:
so other environments could reuse the same python packages
Which makes me think that a flake.nix is what you need. What's cool about flakes is that you can define an environment that has spacy, pandas, and jenkspy with one flake. And then you (or somebody else) might say:
I want an env like Jonathan's, except I also want sympy
and rather than copying your env and making tweaks, they can declare your env as a build input and write a flake.nix with their modifications--which can be further modified by others.
One could imagine a sort of family-tree of environments, so you just need to pick the one that suits your task. The python community has not yet converged on this vision.
Poetry
Poetry will treat you like you're trying to publish a library when all you asked for is an environment, but a library's dependencies are pretty much an environment so there's nothing wrong with having an empty package and just using poetry as an environment factory.
Bonus: if you decide to publish a library after all, you're ready.
The Setup
nix flakes thinks in terms of git repo's, so we'll start with one:
$ git init
Then create a file called flake.nix. Usually I end up with poetry handling 90% of the python stuff, but both pandas and spacy are in that 10% that has dependencies which link to system libraries. So we ask nix to install them so that when poetry tries to install them in the nix develop shell, it has what it needs.
{
description = "Jonathan's awesome env";
inputs = {
nixpkgs.url = "github:nixos/nixpkgs";
};
outputs = { self, nixpkgs, flake-utils }: (flake-utils.lib.eachSystem [
"x86_64-linux"
"x86_64-darwin"
"aarch64-linux"
"aarch64-darwin"
] (system:
let
pkgs = nixpkgs.legacyPackages.${system};
in
rec {
packages.jonathansenv = pkgs.poetry2nix.mkPoetryApplication {
projectDir = ./.;
};
defaultPackage = packages.jonathansenv;
devShell = pkgs.mkShell {
buildInputs = [
pkgs.poetry
pkgs.python310Packages.pandas
pkgs.python310Packages.spacy
];
};
}));
}
Now we let git know about the flake and enter the environment:
❯ git add flake.nix
❯ nix develop
$
Then we initialize the poetry project. I've found that poetry, installed by nix, is kind of odd about which python it uses by default, so we'll set it explicitly
$ poetry init # follow prompts
$ poetry env use $(which python)
$ poetry run python --version
Python 3.10.9 # declared in the flake.nix
At this point, we should have a pyproject.toml:
[tool.poetry]
name = "jonathansenv"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.10"
jenkspy = "^0.3.2"
spacy = "^3.4.4"
pandas = "^1.5.2"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Usage
Now we create the venv that poetry will use, and run a command that depends on these.
$ poetry install
$ poetry run python -c 'import jenkspy, spacy, pandas'
You can also have poetry put you in a shell:
$ poetry shell
(venv)$ python -c 'import jenkspy, spacy, pandas'
It's kind of awkward to do so though, because we're two subshells deep and any shell customizations that we have the grandparent shell are not available. So I recommend using direnv, to enter the dev shell whenever I navigate to that directory and then just use poetry run ... to run commands in the environment.
Publishing the env
In addition to running nix develop with the flake.nix in your current dir, you can also do nix develop /local/path/to/repo or develop nix develop github:/githubuser/githubproject to achieve the same result.
To demonstrate the github example, I have pushed the files referenced above here. So you ought to be able to run this from any linux shell with nix installed:
❯ nix develop github:/MatrixManAtYrService/nix-flake-pandas-spacy
$ poetry install
$ poetry run python -c 'import jenkspy, spacy, pandas'
I say "ought" because if I run that command on a mac it complains about linux-headers-5.19.16 being unsupported on x86_64-darwin.
Presumably there's a way to write the flake (or fix a package) so that it doesn't insist on building linux stuff on a mac, but until I figure it out I'm afraid that this is only a partial answer.

Custom logic in setup.py to change environment header

I have a Python package that I'm distributing on PyPI. I create a script called run_program1 that launches a GUI on the command line.
A snippet of my setup.py file:
setup(
name='my_package',
...
entry_points={
'gui_scripts': [
'run_program1 = program1:start_func',
]
}
)
Unfortunately, the run_program1 executable fails to when installed with Anaconda Python, with an error like this:
This program needs access to the screen. Please run with a Framework build of python, and only when you are logged in on the main display of your Mac.
This issue turns out to be a fundamental issue between Anaconda and setuptools:
https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/9kQreoBIj3A
I'm trying to create an ugly hack to change the environment in the executable that pip creates -- run_program1 -- from #!/Users/***/anaconda2/bin/python to #/usr/bin/env pythonw. I can do this manually after installing on my machine by opening ~/anaconda2/bin/run_program1 and simply replacing the first line. With that edit, the executable works as expected. However, I need to create a hack that will allow me to do this for all users who use pip to install my_package.
I am using this approach to insert custom logic into my setup.py file: https://blog.niteoweb.com/setuptools-run-custom-code-in-setup-py/
class CustomInstallCommand(install):
"""Customized setuptools install command - prints a friendly greeting."""
def run(self):
print "Hello, developer, how are you? :)"
install.run(self)
setup(
...
cmdclass={
'install': CustomInstallCommand,
}, ...)
What I can't figure out is, what should I put into the custom class to change the header in the run_program1 executable? Any ideas of how to approach this?

Python How to install packages to specific directory using setuptools

setup.py
from setuptools import setup
setup(
name = "Project",
version = "1.0",
packages = ['Project','Project.project','Project.LOG',\
'Project.reporting','Project.templates',\
],
install_requires = ['django-grappelli==2.3.8','pycairo==1.10.0','django-chart-tools==0.2.1','django-admin-tools==0.4.0'],
package_data = {
'': ['*.html','*.pyd','*.txt','*.gif','*.png','*.jpeg','*.jpg','*.css','*.js','*.py','*.html~','*.sh','*.wsgi'],
},
# metadata for upload to PyPI
author = "Me",
author_email = "me#example.com",
description = "This is an Example Package", )
Now I want all the packages to be installed in dev/workspace in case of windows and /var/www in case of ubuntu OS. And I want all the install_requires to be installed in python/Lib/site-packages.
How can I do this?

I think you really want to have a look at virtualenv. It lets you create your own Python environment where your code and the dependencies get installed to - wherever you want and totally independent from your existing Python installation. I don't see a good reason why you would want to install dependencies in your existing Python installation.

Optional dependencies in distutils / pip

When installing my python package, I want to be able to tell the user about various optional dependencies. Ideally I would also like to print out a message about these optional requirements and what each of them do.
I haven't seen anything yet in the docs of either pip or docutils. Do tools these support optional dependencies?

These are called extras, here is how to use them in your setup.py, setup.cfg, or pyproject.toml.
The base support is in pkg_resources. You need to enable distribute in your setup.py. pip will also understand them:
pip install 'package[extras]'

Yes, at stated by #Tobu and explained here. In your setup.py file you can add a line like this:
extras_require = {
'full': ['matplotlib', 'tensorflow', 'numpy', 'tikzplotlib']
}
I have an example of this line here.
Now you can either install via PIP basic/vanilla package like pip install package_name or the package with all the optional dependencies like pip install package_name[full]
Where package_name is the name of your package and full is because we put "full" in the extras_require dictionary but it depends on what you put as a name.
If someone is interested in how to code a library that can work with or without a package I recommend this answer

Since PEP-621, this information is better placed in the pyproject.toml rather than setup.py. Here's the relevant specification from PEP 621. Here's an example snippet from a pyproject.toml (credit to #GwynBleidD):
[project.optional-dependencies]
test = [
"pytest < 5.0.0",
"pytest-cov[all]"
]
lint = [
"black",
"flake8"
]
ci = [
"pytest < 5.0.0",
"pytest-cov[all]",
"black",
"flake8"
]
A more complete example is found in the PEP