We Keep Coding

How to get the location of installed Python package into the shell

I want my users to be able to reference a file in my python package (specifically a docker-compose.yml file) directly from the shell.
I couldnt find a way to get only the location from pip show (and grep-ing out "location" from its output feels ugly), so my current (somewhat verbose) solution is:
docker compose -f $(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")/timescale/docker-compose.yml up
Is there a better way?
Edit: I solved it by installing a wrapper command I call locust-compose as part of the package. Not perfect, but it gets the job done:
#!/bin/bash
module_location=$(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")
set -x
docker compose -f $module_location/timescale/docker-compose.yml "$#"

Most of the support you need for this is in the core setuptools suite.
First of all, you need to make sure the data file is included in your package. In a setup.cfg file you can write:
[options.package_data]
timescale = docker-compose.yml
Now if you pip install . or pip wheel, that will include the Compose file as part of the Python package.
Next, you can retrieve this in Python code using the ResourceManager API:
#!/usr/bin/env python3
# timescale/compose_path.py
import pkg_resources
if __name__ == '__main__':
print(pkg_resources.resource_filename('timescale', 'docker-compose.yml'))
And finally, you can take that script and make it a setuptools entry point script (as distinct from the similarly-named Docker concept), so that you can just run it as a single command.
[options.entry_points]
console_scripts=
timescale_compose_path = timescale:compose_path
Again, if you pip install . into a virtual environment, you should be able to run timescale_compose_path and get the path name out.
Having done all of those steps, you can finally run a simpler
docker-compose -f $(timescale_compose_path) up

Why can't the Python command "subprocess.Popen" find the jar file to run?

I'm trying to run code from this repo: https://github.com/tylin/coco-caption, specifically from https://github.com/tylin/coco-caption/blob/master/pycocoevalcap/tokenizer/ptbtokenizer.py, line 51-52:
p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, \
stdout=subprocess.PIPE)
The error I get running this is
OSError: [Errno 2] No such file or directory
I can't figure out why the file can't be found.
The jar I'm trying to run is:
stanford-corenlp-3.4.1.jar
You can see the structure of directory by going to https://github.com/tylin/coco-caption/tree/master/pycocoevalcap/tokenizer. For more specificity into what my actual arguments are when I run the line of code:
cmd= ['java', '-cp', 'stanford-corenlp-3.4.1.jar', 'edu.stanford.nlp.process.PTBTokenizer', '-preserveLines', '-lowerCase', 'tmpWS5p0Z'],
and
path_to_dirname =abs_path_to_folder/tokenizer
I can see the jar that needs to be run, and it looks to be in the right place, so why can't python find it. (Note: I'm using python2.7.) And the temporary File 'tmpWS5p0Z' is where it should be.
Edit: I'm using Ubuntu

try an absolute path ( meaning the path beginning from root / )
https://en.wikipedia.org/wiki/Path_(computing)#Absolute_and_relative_paths
for relative paths in python see i.e. Relative paths in Python , How to refer to relative paths of resources when working with a code repository in Python
UPDATE:
As a test try subprocess.Popen() with the shell=True option and give an absolute path for any involved file, including tmpWS5p0Z
in this subprocess.Popen() call are involved two paths :
1) the python path, python has to find the java executable and the stanford-corenlp-3.4.1.jar which is essentially a java program with its own path
2) the java path of stanford-corenlp-3.4.1.jar
as this is all too complicated try
p_tokenizer = subprocess.Popen(['/absolute_path_to/java -cp /absolute_path_to/stanford-corenlp-3.4.1.jar /absolute_path_to/edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase /absolute_path_to/tmpWS5p0Z' ], shell=True)
Python specify popen working directory via argument
Python subprocess.Popen() error (No such file or directory)

Just in case it might help someone:
I was struggling with the same problem (same https://github.com/tylin/coco-caption code). Might be relevant to say that I was running the code with python 3.7 on CentOS using qsub. So I changed
cmd = ['java', '-cp', 'stanford-corenlp-3.4.1.jar', 'edu.stanford.nlp.process.PTBTokenizer', '-preserveLines', '-lowerCase', 'tmpWS5p0Z']
to
cmd = ['/abs/path/to/java -cp /abs/path/to/stanford-corenlp-3.4.1.jar edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase ', ' /abs/path/to/temporary_file']
Using absolute paths fixed the OSError: [Errno 2] No such file or directory. Notice that I still put '/abs/path/to/temporary_file' as second element in the cmd list, because it got added later on. But then something went wrong in the tokenizer java subprocess, I don't know why or what, just observing because:
p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, stdout=subprocess.PIPE, shell=True)
token_lines = p_tokenizer.communicate(input=sentences.rstrip())[0]
Here token_lines was an empty list (which is not the wanted behavior). Executing this in IPython resulted in the following (just the subprocess.Popen(..., not the communicate).
Exception in thread "main" edu.stanford.nlp.io.RuntimeIOException: java.io.IOException: Input/output error
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:278)
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:163)
at edu.stanford.nlp.process.AbstractTokenizer.hasNext(AbstractTokenizer.java:55)
at edu.stanford.nlp.process.PTBTokenizer.tokReader(PTBTokenizer.java:444)
at edu.stanford.nlp.process.PTBTokenizer.tok(PTBTokenizer.java:416)
at edu.stanford.nlp.process.PTBTokenizer.main(PTBTokenizer.java:760)
Caused by: java.io.IOException: Input/output error
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:279)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:290)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
at java.base/java.io.BufferedReader.read1(BufferedReader.java:210)
at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
at edu.stanford.nlp.process.PTBLexer.zzRefill(PTBLexer.java:24511)
at edu.stanford.nlp.process.PTBLexer.next(PTBLexer.java:24718)
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:276)
... 5 more
Again, I don't know why or what, but I just wanted to share that doing this fixed it:
cmd = ['/abs/path/to/java -cp /abs/path/to/stanford-corenlp-3.4.1.jar edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase /abs/path/to/temporary_file']
And changing cmd.append(os.path.join(path_to_jar_dirname, os.path.basename(tmp_file.name))) into cmd[0] += os.path.join(path_to_jar_dirname, os.path.basename(tmp_file.name)).
So making cmd into a list with only 1 element, containing the entire command with absolute paths at once. Thanks for your help!

As #Lars mentioned above the issue I had was that I Java wasn't installed. Solved it with:
sudo apt update
sudo apt install default-jdk
sudo apt install default-jre
Making this post since I had this issue twice (due to reinstallation problems) and forgot about it.

While running doxygen, "sh: doxypy: command not found" error is coming

I have installed doxygen 1.8.8 on my linux machine, and trying to run doxypy but seeing issue with it
[nipatel#caspgval4 docs]$ doxygen --version
1.8.8
[nipatel#caspgval4 docs]$ whereis doxypy
doxypy: /usr/bin/doxypy.py
[nipatel#caspgval4 docs]$ which doxypy
doxypy: Command not found.
While running doxygen, I see following error:
Reading /mypath/myfile.py...
sh: doxypy: command not found
PATH on the machine:
echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/bin/doxypy.py
Can someone let me know if I am missing anything here? How can I make doxypy command work? Do I need to add any PATH or something?

I suggest to not rename doxypy.py to doxypy.
Create a symbolic link: ln -s /usr/bin/doxypy.py /usr/bin/doxypy

Maybe Your $PATH variable doesn't contain the default paths? echo $PATH should give something like:
[root#server ~]# echo $PATH
/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
If it's empty, You may set the PATH variable with this command
export PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin"

I renamed /usr/bin/doxypy.py to /usr/bin/doxypy and it resolved the issue.

How do I make a python script executable?

How can I run a python script with my own command line name like myscript without having to do python myscript.py in the terminal?

Add a shebang line to the top of the script:
#!/usr/bin/env python
Mark the script as executable:
chmod +x myscript.py
Add the dir containing it to your PATH variable. (If you want it to stick, you'll have to do this in .bashrc or .bash_profile in your home dir.)
export PATH=/path/to/script:$PATH

The best way, which is cross-platform, is to create setup.py, define an entry point in it and install with pip.
Say you have the following contents of myscript.py:
def run():
print('Hello world')
Then you add setup.py with the following:
from setuptools import setup
setup(
name='myscript',
version='0.0.1',
entry_points={
'console_scripts': [
'myscript=myscript:run'
]
}
)
Entry point format is terminal_command_name=python_script_name:main_method_name
Finally install with the following command.
pip install -e /path/to/script/folder
-e stands for editable, meaning you'll be able to work on the script and invoke the latest version without need to reinstall
After that you can run myscript from any directory.

I usually do in the script:
#!/usr/bin/python
... code ...
And in terminal:
$: chmod 755 yourfile.py
$: ./yourfile.py

Another related solution which some people may be interested in. One can also directly embed the contents of myscript.py into your .bashrc file on Linux (should also work for MacOS I think)
For example, I have the following function defined in my .bashrc for dumping Python pickles to the terminal, note that the ${1} is the first argument following the function name:
depickle() {
python << EOPYTHON
import pickle
f = open('${1}', 'rb')
while True:
try:
print(pickle.load(f))
except EOFError:
break
EOPYTHON
}
With this in place (and after reloading .bashrc), I can now run depickle a.pickle from any terminal or directory on my computer.

The simplest way that comes to my mind is to use "pyinstaller".
create an environment that contains all the lib you have used in your code.
activate the environment and in the command window write pip install pyinstaller
Use the command window to open the main directory that codes maincode.py is located.
remember to keep the environment active and write pyinstaller maincode.py
Check the folder named "build" and you will find the executable file.
I hope that this solution helps you.
GL

I've struggled for a few days with the problem of not finding the command py -3 or any other related to pylauncher command if script was running by service created using Nssm tool.
But same commands worked when run directly from cmd.
What was the solution? Just to re-run Python installer and at the very end click the option to disable path length limit.
I'll just leave it here, so that anyone can use this answer and find it helpful.

Issue with running sudo command with python on plumbum

I'm using Python 2.7 with the latest plumbum package from mac ports.
In general, plumbum works great. Though I'm having a heck of a time getting a sudo'd command to work. I've setup my /etc/sudoers for the commands I want to run without having to be prompted, so that's fine. I can run the commands manually without issue.
However, when I try the same from python using this:
sudo["/usr/local/some-magic-command here"]
sudo("-u " + sudoUser) # sudo user is userfoo
I receive the following error:
plumbum.commands.processes.ProcessExecutionError: Command line: ['/usr/bin/sudo', '-u userfoo']
Exit code: 1
Stderr: | sudo: unknown user: userfoo
The user does exist, so not exactly sure what the deal is here.
Comments?

There is no "-u userfoo" user. There is probably just "userfoo". Note: no -u prefix. Try:
from plumbum.cmd import sudo
as_userfoo = sudo["-u", sudo_user]
print(as_userfoo("whoami"))