We Keep Coding

Makefile on Windows 10 - file not found

After some time I finally managed to successfully install python and pip and run it on my machine using Visual Studio Code.
I am working in virtual environment in python and we have a Makefile with following statement:
test:
source .env && PYTHONPATH=. PY_ENV=testing py.test ${ARGS} --duration=20
File .env lives in the main directory next to Makefile. It contains some environmental variables needed for testing certain APIs.
When I take the line out of the file and run it in my terminal, everything works fine and all tests are running etc.
However if I call the following: make test I am getting this error:
$ make test
source .env && PYTHONPATH=. PY_ENV=testing py.test --duration=20
/usr/bin/sh: line 0: source: .env: file not found
make: *** [test] Error 1
(venv)
To me it looks like when running this command from within Makefile it can't see the .env file but have no idea how to solve it.

The source command isn't looking up the file in the current working directory. As mentioned in man source:
Read and execute commands from filename in the current shell
environment and return the exit status of the last command executed
from filename. If filename does not contain a slash, filenames in
PATH are used to find the directory containing filename.
Change the file path like so:
test:
source ./.env && PYTHONPATH=. PY_ENV=testing py.test ${ARGS} --duration=20
Note that this error does not occur in bash version < 4. This is due to an implementation bug when run under POSIX mode (what make uses, since its default shell is sh, which is usually bash --posix). The correct behaviour was first mentioned in the documentation of bash-2.05 (revision 28ef6c31, file doc/bashref.info):
When Bash is not in POSIX mode, the current directory is searched if
FILENAME is not found in `$PATH'.
These older versions searched the current directory regardless of POSIX mode. It was only in bash-4.0-rc1 (revision 3185942a, file general.c) that this was corrected. Running git diff 3185942a~ 3185942a general.c outputs this section:
## -69,6 +69,7 ## posix_initialize (on)
if (on != 0)
{
interactive_comments = source_uses_path = expand_aliases = 1;
+ source_searches_cwd = 0;
}

Run python module from command line

I don't really know how to ask this question but I can describe what I want to achieve. I would update any edits that would be suggested.
I have a python module that makes use of some command line arguments. Using the module requires some initial setup outside of the python interpreter. The python file that does the setup runs fine, but the problem is that I have to dig through the python installation to find where that file is located i.e. I have to do python full-path-to-setup-script.py -a argA -b argB etc.I would like to call the setup script like this
some-setup-command -a argA -b argB etc.
I want to achieve something like
workon environmnent_name as in the virtualenv module or
pipenv install as in the pipenv module.
I know both of the above commands call a script of some kind (whether bash or python). I've tried digging through the source codes of virtualenv and pipenv without any success.
I would really appreciate if someone could point me to any necessary resource for coding such programs.

If full-path-to-setup-script.py is executable and has a proper shebang line
#! /usr/bin/env python
then you can
ln -s full-path-to-setup-script.py ~/bin/some-command
considering ~/bin exists and is in your PATH,
and you'll be able to invoke
some-command -a argA -b argB

It's a bit difficult to understand what you're looking for, but python -m is my best guess.
For example, to make a new Jupyter kernel, we call
python -m ipykernel arg --option --option
Where arg is the CLI argument and option is a CLI option, and ipykernel is the module receiving the args and options.

Commands that are callable from the command prompt are located in one of the directories in your system's PATH variable. If you are on Windows, you see the locations via:
echo %PATH%
Or if you want a nicer readout:
powershell -c "$env:path -split(';')"
One solution is to create a folder, add it to your system's PATH, and then create a callable file that you can run. In this example we will create a folder in your user profile, add it to the path, then create a callable file in that folder.
mkdir %USERPROFILE%\path
set PATH=%PATH%%USERPROFILE%\path;
setx PATH %PATH%
In the folder %USERPROFILE%\path, we create a batch file with following content:
# file name:
# some-command.bat
#
python C:\full\path\to\setup-script.py %*
Now you should be able to call
some-command -a argA -b argB
And the batch file will call python with python script and pass the arguments you added.

Looking at the above answers, I see no one has mentioned this:
You can of course compile the python file and give executable permissions with
chmod +x filename.py
and then run it as
./filename.py -a argA -b argB ...
Moreover, you can also remove the extention .py (since it is an executable now) and then run it only as
./filename -a argA -b argB ...

Why this error when I try to create workspaces in ROS?

Whenever I try to create a workspace:
~/catkin_ws$ catkin_make
It shows like this:
ImportError: "from catkin_pkg.package import parse_package" failed: No module named 'catkin_pkg'
Make sure that you have installed "catkin_pkg", it is up to date and on the PYTHONPATH.
CMake Error at /opt/ros/kinetic/share/catkin/cmake/safe_execute_process.cmake:11 (message):
execute_process(/home/usuario/miniconda3/bin/python
"/opt/ros/kinetic/share/catkin/cmake/parse_package_xml.py"
"/opt/ros/kinetic/share/catkin/cmake/../package.xml"
"/home/usuario/catkin_ws/build/catkin/catkin_generated/version/package.cmake")
returned error code 1
Call Stack (most recent call first):
/opt/ros/kinetic/share/catkin/cmake/catkin_package_xml.cmake:63 (safe_execute_process)
/opt/ros/kinetic/share/catkin/cmake/all.cmake:151 (_catkin_package_xml)
/opt/ros/kinetic/share/catkin/cmake/catkinConfig.cmake:20 (include)
CMakeLists.txt:52 (find_package)
-- Configuring incomplete, errors occurred!
See also "/home/usuario/catkin_ws/build/CMakeFiles/CMakeOutput.log".
See also "/home/usuario/catkin_ws/build/CMakeFiles/CMakeError.log".
Invoking "cmake" failed
It seems like there is a problem with catkin_pkg but I dont find the solution

I just installed ROS on Ubuntu 16.04, had the same issue, and fixed it. The location for catkin_pkg is likely not on your PYTHONPATH and needs to be added.
From the error output:
Make sure that you have installed "catkin_pkg", it is up to date and on the PYTHONPATH.
Try locating catkin_pkg and check your PYTHONPATH. catkin_pkg wasn't on my PYTHONPATH (likely due to other program installs), so I added it and ran catkin_make again, this time successfully.
~/catkin_ws$ locate catkin_pkg
/usr/lib/python2.7/dist-packages/catkin_pkg
~/catkin_ws$ echo $PYTHONPATH
/opt/ros/kinetic/lib/python2.7/dist-packages
To append the catkin_pkg dir to PYTHONPATH (for this session):
~/catkin_ws$ export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/dist-packages
For permanency I appended the catkin_pkg dir to PYTHONPATH in my .bashrc (you might want to backup your .bashrc file first, e.g. cp -p ~/.bashrc ~/.bashrc-ros-catkin.bak).
To do this, edit your ~/.bashrc file (you might need to use sudo to edit this file) and add the following two lines to the end of the file:
# manually added for ROS catkin_make workspace setup
export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/dist-packages
Save the file and run source to update your session:
~/catkin_ws$ source ~/.bashrc
Check your PYTHONPATH again:
~/catkin_ws$ echo $PYTHONPATH
/opt/ros/kinetic/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages
Obviously the location of your catkin_pkg files might be different to mine, so use that path instead when appending to $PYTHONPATH above.
Now try running catkin_make again. If you get the same error, paste the output of your catkin_pkg location and PYTHONPATH here.
Cheers,
sb

Are you using Anaconda environment?
This issue is quite common with Anaconda's Python installation.
Try: python --version
If you see Anaconda in the output, go to your bashrc file with vi ~/.bashrc and then comment the line where anaconda is added to path.
It would be something like,
export PATH="username/anaconda2/bin:$PATH"
After that source your bashrc with source ~/.bashrc, open a new terminal and navigate to your catkin workspace. Delete the old build folder and try the catkin_make command again.
Should solve your issue.

The Error Output:
ImportError: "from catkin_pkg.package import parse_package" failed: No module named 'catkin_pkg'
Make sure that you have installed "catkin_pkg", it is up to date and on the PYTHONPATH.
As mentioned above you need have "catkin_pkg" in PYTHONPATH. The easiest way if you ask me is:
$ pip install catkin_pkg

try this : pip install -U rosdep rosinstall_generator wstool rosinstall six vcstools
if pip shows any errror, switch to root and install pip and then try

Actually when you want to use Anaconda and ROS simultaneously then generally this error comes. So firstly go in .bashrc file and comment the path of anaconda.
Export PATH="/home/gaurav/anaconda3/bin:$PATH".
After commenting go in catkin_ws folder and delete the existing folder within the folder and make new src folder.
Then follow three steps:
1 - $ mkdir src
2 - $ catkin_init_workspace src
3 - $ catkin_make
It works for me.

This means you havent sourced ROS into your environment.
Add this line to the last line of your .bashrc file
source /opt/ros/kinetic/setup.bash
Save the .bashrc file, close the current terminal and start your process in a new terminal.

for me, it fixed by
sudo apt-get remove python3-catkin-tools
sudo apt-get install python-catkin-tools

How do I make a python script executable?

How can I run a python script with my own command line name like myscript without having to do python myscript.py in the terminal?

Add a shebang line to the top of the script:
#!/usr/bin/env python
Mark the script as executable:
chmod +x myscript.py
Add the dir containing it to your PATH variable. (If you want it to stick, you'll have to do this in .bashrc or .bash_profile in your home dir.)
export PATH=/path/to/script:$PATH

The best way, which is cross-platform, is to create setup.py, define an entry point in it and install with pip.
Say you have the following contents of myscript.py:
def run():
print('Hello world')
Then you add setup.py with the following:
from setuptools import setup
setup(
name='myscript',
version='0.0.1',
entry_points={
'console_scripts': [
'myscript=myscript:run'
]
}
)
Entry point format is terminal_command_name=python_script_name:main_method_name
Finally install with the following command.
pip install -e /path/to/script/folder
-e stands for editable, meaning you'll be able to work on the script and invoke the latest version without need to reinstall
After that you can run myscript from any directory.

I usually do in the script:
#!/usr/bin/python
... code ...
And in terminal:
$: chmod 755 yourfile.py
$: ./yourfile.py

Another related solution which some people may be interested in. One can also directly embed the contents of myscript.py into your .bashrc file on Linux (should also work for MacOS I think)
For example, I have the following function defined in my .bashrc for dumping Python pickles to the terminal, note that the ${1} is the first argument following the function name:
depickle() {
python << EOPYTHON
import pickle
f = open('${1}', 'rb')
while True:
try:
print(pickle.load(f))
except EOFError:
break
EOPYTHON
}
With this in place (and after reloading .bashrc), I can now run depickle a.pickle from any terminal or directory on my computer.

The simplest way that comes to my mind is to use "pyinstaller".
create an environment that contains all the lib you have used in your code.
activate the environment and in the command window write pip install pyinstaller
Use the command window to open the main directory that codes maincode.py is located.
remember to keep the environment active and write pyinstaller maincode.py
Check the folder named "build" and you will find the executable file.
I hope that this solution helps you.
GL

I've struggled for a few days with the problem of not finding the command py -3 or any other related to pylauncher command if script was running by service created using Nssm tool.
But same commands worked when run directly from cmd.
What was the solution? Just to re-run Python installer and at the very end click the option to disable path length limit.
I'll just leave it here, so that anyone can use this answer and find it helpful.

Activating/deactivating virtualenv in a python script [duplicate]

I want to activate a virtualenv instance from a Python script.
I know it's quite easy to do, but all the examples I've seen use it to run commands within the env and then close the subprocess.
I simply want to activate the virtualenv and return to the shell, the same way that bin/activate does.
Something like this:
$me: my-script.py -d env-name
$(env-name)me:
Is this possible?
Relevant:
virtualenv › Invoking an env from a script

If you want to run a Python subprocess under the virtualenv, you can do that by running the script using the Python interpreter that lives inside virtualenv's /bin/ directory:
import subprocess
# Path to a Python interpreter that runs any Python script
# under the virtualenv /path/to/virtualenv/
python_bin = "/path/to/virtualenv/bin/python"
# Path to the script that must run under the virtualenv
script_file = "must/run/under/virtualenv/script.py"
subprocess.Popen([python_bin, script_file])
However, if you want to activate the virtualenv under the current Python interpreter instead of a subprocess, you can use the activate_this.py script:
# Doing execfile() on this file will alter the current interpreter's
# environment so you can import libraries in the virtualenv
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
execfile(activate_this_file, dict(__file__=activate_this_file))

The simplest solution to run your script under virtualenv's interpreter is to replace the default shebang line with path to your virtualenv's interpreter like so at the beginning of the script:
#!/path/to/project/venv/bin/python
Make the script executable:
chmod u+x script.py
Run the script:
./script.py
Voila!

It turns out that, yes, the problem is not simple, but the solution is.
First I had to create a shell script to wrap the "source" command. That said I used the "." instead, because I've read that it's better to use it than source for Bash scripts.
#!/bin/bash
. /path/to/env/bin/activate
Then from my Python script I can simply do this:
import os
os.system('/bin/bash --rcfile /path/to/myscript.sh')
The whole trick lies within the --rcfile argument.
When the Python interpreter exits it leaves the current shell in the activated environment.
Win!

To run another Python environment according to the official Virtualenv documentation, in the command line you can specify the full path to the executable Python binary, just that (no need to active the virtualenv before):
/path/to/virtualenv/bin/python
The same applies if you want to invoke a script from the command line with your virtualenv. You don't need to activate it before:
me$ /path/to/virtualenv/bin/python myscript.py
The same for a Windows environment (whether it is from the command line or from a script):
> \path\to\env\Scripts\python.exe myscript.py

Just a simple solution that works for me. I don't know why you need the Bash script which basically does a useless step (am I wrong ?)
import os
os.system('/bin/bash --rcfile flask/bin/activate')
Which basically does what you need:
[hellsing#silence Foundation]$ python2.7 pythonvenv.py
(flask)[hellsing#silence Foundation]$
Then instead of deactivating the virtual environment, just Ctrl + D or exit. Is that a possible solution or isn't that what you wanted?

The top answer only works for Python 2.x
For Python 3.x, use this:
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
exec(compile(open(activate_this_file, "rb").read(), activate_this_file, 'exec'), dict(__file__=activate_this_file))
Reference: What is an alternative to execfile in Python 3?

The child process environment is lost in the moment it ceases to exist, and moving the environment content from there to the parent is somewhat tricky.
You probably need to spawn a shell script (you can generate one dynamically to /tmp) which will output the virtualenv environment variables to a file, which you then read in the parent Python process and put in os.environ.
Or you simply parse the activate script in using for the line in open("bin/activate"), manually extract stuff, and put in os.environ. It is tricky, but not impossible.

For python2/3, Using below code snippet we can activate virtual env.
activate_this = "/home/<--path-->/<--virtual env name -->/bin/activate_this.py" #for ubuntu
activate_this = "D:\<-- path -->\<--virtual env name -->\Scripts\\activate_this.py" #for windows
with open(activate_this) as f:
code = compile(f.read(), activate_this, 'exec')
exec(code, dict(__file__=activate_this))

I had the same issue and there was no activate_this.py in the Scripts directory of my environment.
activate_this.py
"""By using execfile(this_file, dict(__file__=this_file)) you will
activate this virtualenv environment.
This can be used when you must use an existing Python interpreter, not
the virtualenv bin/python
"""
try:
__file__
except NameError:
raise AssertionError(
"You must run this like execfile('path/to/active_this.py', dict(__file__='path/to/activate_this.py'))")
import sys
import os
base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if(sys.platform=='win32'):
site_packages = os.path.join(base, 'Lib', 'site-packages')
else:
site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
sys.real_prefix = sys.prefix
sys.prefix = base
# Move the added items to the front of the path:
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
Copy the file to the Scripts directory of your environment and use it like this:
def activate_virtual_environment(environment_root):
"""Configures the virtual environment starting at ``environment_root``."""
activate_script = os.path.join(
environment_root, 'Scripts', 'activate_this.py')
execfile(activate_script, {'__file__': activate_script})
activate_virtual_environment('path/to/your/venv')
Refrence: https://github.com/dcreager/virtualenv/blob/master/virtualenv_support/activate_this.py

You should create all your virtualenvs in one folder, such as virt.
Assuming your virtualenv folder name is virt, if not change it
cd
mkdir custom
Copy the below lines...
#!/usr/bin/env bash
ENV_PATH="$HOME/virt/$1/bin/activate"
bash --rcfile $ENV_PATH -i
Create a shell script file and paste the above lines...
touch custom/vhelper
nano custom/vhelper
Grant executable permission to your file:
sudo chmod +x custom/vhelper
Now export that custom folder path so that you can find it on the command-line by clicking tab...
export PATH=$PATH:"$HOME/custom"
Now you can use it from anywhere by just typing the below command...
vhelper YOUR_VIRTUAL_ENV_FOLDER_NAME
Suppose it is abc then...
vhelper abc