We Keep Coding

Source Python Virtual Environment for Hive UDF

I'm using a Python virtual environment to load modules that aren't available on our cluster for use in a Hive UDF. I'm unable to source the venv, so when the Python UDF is called in the shell script, the script errors since the modules cannot be found.
When calling ls from the shell script, the venv appears in the list.
DELETE FILE /temp/venv;
ADD FILE /temp/venv;
DELETE FILE udf.sh;
ADD FILE udf.sh;
SOURCE venv/bin/activate;
SELECT TRANSFORM(1)
USING 'bash udf.sh'
AS (test_result)
Results in File: venv/bin/activate is not a file.
SOURCE ../venv/bin/activate;
Results in FAILED: ParseException line 1:2 cannot recognize input near 'This' 'file' 'must'
Within the shell script, If I try to use:
. venv/bin/activate
It returns an exit code 1.
Any thoughts?
Thanks,
Dave
Solved using this: https://stackoverflow.com/a/23069201/10542262
Within the shell script, instead of doing:
python [path]/[script].py
You can call Python from the venv and no longer need to activate the venv.
[path/to/venv/]/bin/python [path]/[script].py

How do i append to the Maya PYTHONPATH?

I am working through Practical Maya Programming, and trying to set a 'development root' on my PC, I have followed the instructions (below) exactly but it is not working - At the point where I type 'mayapy.exe' I get the warning "'mayapy.exe' is not recognized as an internal or external command, operable program or batch file."
From the book:
Let's decide where we will do our coding. We'll call this location the development root for the rest of the book. To be concise, I'll choose C:\mayapybook\pylib to house all of our Python code.
Create the development root folder, and inside of it create an empty file named minspect.py.
Now, we need to get C:\mayapybook\pylib onto Python's sys.path so it can be imported. The easiest way to do this is to use the PYTHONPATH environment variable. From a Windows command line you can run the following to add the path, and ensure it worked:
> set PYTHONPATH=%PYTHONPATH%;C:\mayapybook\pylib
> mayapy.exe
>>> import sys
>>> 'C:\\mayapybook\\pylib' in sys.path
True
>>> import minspect
>>> minspect
<module 'minspect' from '...\minspect.py'>
EDIT
This is how it is working for me at the moment:
PS C:\Users\Me> set PYTHONPATH=%PYTHONPATH%;C:\mayapybook\pylib
C:\mayapybook\pylib : The term 'C:\mayapybook\pylib' is not recognized as the name of a cmdlet, function, script file,
or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and
try again.
At line:1 char:29
+ set PYTHONPATH=%PYTHONPATH%;C:\mayapybook\pylib
+ ~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (C:\mayapybook\pylib:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
So the code from the book is not working but the code from the post by DrHaze seems to:
PS C:\Users\Me> setx PATH "%PATH%C:\mayapybook\pylib\"
SUCCESS: Specified value was saved.
But when I run the Maya Python Interpreter and check if C:\mayapybook\pylib\ is in sys path it returns false:
>>> 'C:\\mayapybook\\pylib' in sys.path
False

This error "'mayapy.exe' is not recognized as an internal or external command, operable program or batch file." means that the path where mayapy.exe is located is not included in the PATH environment variable. Your system tries to look in all the folders included in the PATH variable but can't find an executable called mayapy.exe.
The executable mayapy.exe is generally located here:
C:\Program Files\Autodesk\Maya(VERSION)\bin\mayapy.exe
on my computer it's located here: C:\Program Files\Autodesk\Maya2014\bin\mayapy.exe
To add the mayapy.exe location to your path, use one of the following commands:
setx PATH "%PATH%;C:\Program Files\Autodesk\Maya2014\bin\" if you
want to change it permanently
set PATH "%PATH%;C:\Program Files\Autodesk\Maya2014\bin\" only works for the current instance of the cmd session.
EDIT
The error you show in your edit is the source of the problem. Windows failed to set the environment variable PYTHONPATH. Hence, when you execute 'C:\\mayapybook\\pylib' in sys.path it returns False. sys.path is in fact containing the value of PYTHONPATH. That's why it returns False.
Now, why did it fail to set this environment variable?
First I can see that you are using Windows Powershell, keep this in mind.
The command I gave you is:
set PATH "%PATH%;C:\Program Files\Autodesk\Maya2014\bin\"
You wrote:
set PYTHONPATH=%PYTHONPATH%;C:\mayapybook\pylib
What it should be:
set PYTHONPATH "%PYTHONPATH%;C:\mayapybook\pylib\"
The syntax is a bit different and this last command should work.
As an explaination, your book gives you some commands to type in the vintage/old-style windows terminal: cmd.exe
As you are using Windows Powershell, some commands might have a different syntax.
Now what you can do is:
Use cmd.exe (Right click on the title bar -> Properties to custom it)
Use Powershell but keep in mind that the syntax might be a bit different than in your book

If you are using Powershell, there are different commands and strategies for managing environment variables.
You can set a variable permanently with SetEnvironmentVariable
You can set for the current shell session with: $env:VARNAME =
VARVALUE
You can put the commands to set variables in a powershell profile
file.
I would go with the third option. All three are detailed below:
Option 1. To append the directory "C:\mayapybook\pylib\" to the existing
PYTHONPATH permanently for your account:
[Environment]::SetEnvironmentVariable("PYTHONPATH", $env:PYTHONPATH +";C:\mayapybook\pylib\", "User")
Option 2. To append the Maya bin folder to your PATH for only the current
shell session:
$env:PATH += ";C:\Program Files\Autodesk\Maya2016\bin\"
Option 3. Create a powershell profile and set your env vars there.
First you'll need to make sure powershell scripts can run locally:
Hit the windows button, start typing powershell, right click and open as administrator. Enter:
Get-ExecutionPolicy
If it is says Restricted or AllSigned, set it to RemoteSigned like so:
Set-ExecutionPolicy RemoteSigned
Close that shell. Now in another powershell (not admin) type:
cd ~\Documents
md WindowsPowerShell
cd WindowsPowerShell
New-Item -path "profile.ps1" -type file
notepad.exe profile.ps1
Paste into the file any commands you want to run whenever a new powershell is opened:
Write-Host "Hello From Your Profile"
$env:PATH += ";C:\Program Files\Autodesk\Maya2016\bin\"
$env:PYTHONPATH += ";C:\mayapybook\pylib\"
Now whenever you open a powershell, you'll get a silly message and those paths will be set. You can test by typing:
Write-Host $env:PATH
or to list all env vars:
Get-ChildItem Env:
You should now be able to run commands from the maya bin directory. For example, type: maya to start maya.
Some other useful powershell env var commands here.

How do I make a python script executable?

How can I run a python script with my own command line name like myscript without having to do python myscript.py in the terminal?

Add a shebang line to the top of the script:
#!/usr/bin/env python
Mark the script as executable:
chmod +x myscript.py
Add the dir containing it to your PATH variable. (If you want it to stick, you'll have to do this in .bashrc or .bash_profile in your home dir.)
export PATH=/path/to/script:$PATH

The best way, which is cross-platform, is to create setup.py, define an entry point in it and install with pip.
Say you have the following contents of myscript.py:
def run():
print('Hello world')
Then you add setup.py with the following:
from setuptools import setup
setup(
name='myscript',
version='0.0.1',
entry_points={
'console_scripts': [
'myscript=myscript:run'
]
}
)
Entry point format is terminal_command_name=python_script_name:main_method_name
Finally install with the following command.
pip install -e /path/to/script/folder
-e stands for editable, meaning you'll be able to work on the script and invoke the latest version without need to reinstall
After that you can run myscript from any directory.

I usually do in the script:
#!/usr/bin/python
... code ...
And in terminal:
$: chmod 755 yourfile.py
$: ./yourfile.py

Another related solution which some people may be interested in. One can also directly embed the contents of myscript.py into your .bashrc file on Linux (should also work for MacOS I think)
For example, I have the following function defined in my .bashrc for dumping Python pickles to the terminal, note that the ${1} is the first argument following the function name:
depickle() {
python << EOPYTHON
import pickle
f = open('${1}', 'rb')
while True:
try:
print(pickle.load(f))
except EOFError:
break
EOPYTHON
}
With this in place (and after reloading .bashrc), I can now run depickle a.pickle from any terminal or directory on my computer.

The simplest way that comes to my mind is to use "pyinstaller".
create an environment that contains all the lib you have used in your code.
activate the environment and in the command window write pip install pyinstaller
Use the command window to open the main directory that codes maincode.py is located.
remember to keep the environment active and write pyinstaller maincode.py
Check the folder named "build" and you will find the executable file.
I hope that this solution helps you.
GL

I've struggled for a few days with the problem of not finding the command py -3 or any other related to pylauncher command if script was running by service created using Nssm tool.
But same commands worked when run directly from cmd.
What was the solution? Just to re-run Python installer and at the very end click the option to disable path length limit.
I'll just leave it here, so that anyone can use this answer and find it helpful.

Activating/deactivating virtualenv in a python script [duplicate]

I want to activate a virtualenv instance from a Python script.
I know it's quite easy to do, but all the examples I've seen use it to run commands within the env and then close the subprocess.
I simply want to activate the virtualenv and return to the shell, the same way that bin/activate does.
Something like this:
$me: my-script.py -d env-name
$(env-name)me:
Is this possible?
Relevant:
virtualenv › Invoking an env from a script

If you want to run a Python subprocess under the virtualenv, you can do that by running the script using the Python interpreter that lives inside virtualenv's /bin/ directory:
import subprocess
# Path to a Python interpreter that runs any Python script
# under the virtualenv /path/to/virtualenv/
python_bin = "/path/to/virtualenv/bin/python"
# Path to the script that must run under the virtualenv
script_file = "must/run/under/virtualenv/script.py"
subprocess.Popen([python_bin, script_file])
However, if you want to activate the virtualenv under the current Python interpreter instead of a subprocess, you can use the activate_this.py script:
# Doing execfile() on this file will alter the current interpreter's
# environment so you can import libraries in the virtualenv
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
execfile(activate_this_file, dict(__file__=activate_this_file))

The simplest solution to run your script under virtualenv's interpreter is to replace the default shebang line with path to your virtualenv's interpreter like so at the beginning of the script:
#!/path/to/project/venv/bin/python
Make the script executable:
chmod u+x script.py
Run the script:
./script.py
Voila!

It turns out that, yes, the problem is not simple, but the solution is.
First I had to create a shell script to wrap the "source" command. That said I used the "." instead, because I've read that it's better to use it than source for Bash scripts.
#!/bin/bash
. /path/to/env/bin/activate
Then from my Python script I can simply do this:
import os
os.system('/bin/bash --rcfile /path/to/myscript.sh')
The whole trick lies within the --rcfile argument.
When the Python interpreter exits it leaves the current shell in the activated environment.
Win!

To run another Python environment according to the official Virtualenv documentation, in the command line you can specify the full path to the executable Python binary, just that (no need to active the virtualenv before):
/path/to/virtualenv/bin/python
The same applies if you want to invoke a script from the command line with your virtualenv. You don't need to activate it before:
me$ /path/to/virtualenv/bin/python myscript.py
The same for a Windows environment (whether it is from the command line or from a script):
> \path\to\env\Scripts\python.exe myscript.py

Just a simple solution that works for me. I don't know why you need the Bash script which basically does a useless step (am I wrong ?)
import os
os.system('/bin/bash --rcfile flask/bin/activate')
Which basically does what you need:
[hellsing#silence Foundation]$ python2.7 pythonvenv.py
(flask)[hellsing#silence Foundation]$
Then instead of deactivating the virtual environment, just Ctrl + D or exit. Is that a possible solution or isn't that what you wanted?

The top answer only works for Python 2.x
For Python 3.x, use this:
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
exec(compile(open(activate_this_file, "rb").read(), activate_this_file, 'exec'), dict(__file__=activate_this_file))
Reference: What is an alternative to execfile in Python 3?

The child process environment is lost in the moment it ceases to exist, and moving the environment content from there to the parent is somewhat tricky.
You probably need to spawn a shell script (you can generate one dynamically to /tmp) which will output the virtualenv environment variables to a file, which you then read in the parent Python process and put in os.environ.
Or you simply parse the activate script in using for the line in open("bin/activate"), manually extract stuff, and put in os.environ. It is tricky, but not impossible.

For python2/3, Using below code snippet we can activate virtual env.
activate_this = "/home/<--path-->/<--virtual env name -->/bin/activate_this.py" #for ubuntu
activate_this = "D:\<-- path -->\<--virtual env name -->\Scripts\\activate_this.py" #for windows
with open(activate_this) as f:
code = compile(f.read(), activate_this, 'exec')
exec(code, dict(__file__=activate_this))

I had the same issue and there was no activate_this.py in the Scripts directory of my environment.
activate_this.py
"""By using execfile(this_file, dict(__file__=this_file)) you will
activate this virtualenv environment.
This can be used when you must use an existing Python interpreter, not
the virtualenv bin/python
"""
try:
__file__
except NameError:
raise AssertionError(
"You must run this like execfile('path/to/active_this.py', dict(__file__='path/to/activate_this.py'))")
import sys
import os
base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if(sys.platform=='win32'):
site_packages = os.path.join(base, 'Lib', 'site-packages')
else:
site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
sys.real_prefix = sys.prefix
sys.prefix = base
# Move the added items to the front of the path:
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
Copy the file to the Scripts directory of your environment and use it like this:
def activate_virtual_environment(environment_root):
"""Configures the virtual environment starting at ``environment_root``."""
activate_script = os.path.join(
environment_root, 'Scripts', 'activate_this.py')
execfile(activate_script, {'__file__': activate_script})
activate_virtual_environment('path/to/your/venv')
Refrence: https://github.com/dcreager/virtualenv/blob/master/virtualenv_support/activate_this.py

You should create all your virtualenvs in one folder, such as virt.
Assuming your virtualenv folder name is virt, if not change it
cd
mkdir custom
Copy the below lines...
#!/usr/bin/env bash
ENV_PATH="$HOME/virt/$1/bin/activate"
bash --rcfile $ENV_PATH -i
Create a shell script file and paste the above lines...
touch custom/vhelper
nano custom/vhelper
Grant executable permission to your file:
sudo chmod +x custom/vhelper
Now export that custom folder path so that you can find it on the command-line by clicking tab...
export PATH=$PATH:"$HOME/custom"
Now you can use it from anywhere by just typing the below command...
vhelper YOUR_VIRTUAL_ENV_FOLDER_NAME
Suppose it is abc then...
vhelper abc

PYTHONPATH: command line calls on Windows 7

There are python scripts with command line arguments that I'd like to call from any location on my PC.
The idea is to share the corresponding package with others so they can open up a CMD window and run
python thescript.py arg1 arg2
regardless of their location.
How do I setup the python path/ PATH environment variables?
I've setup a package in site-packages, added that path to $PATH and edited PYTHONPATH to include the module directory (which includes __init__.py), but CMD won't find the relevant scripts.
python: can't open file 'thescript.py': [Errno 2] No such file or directory
Thanks.

Python does not look up scripts on some sort of path.
You have 2 options:
Use the full path:
python /path/to/thescript.py
Place the script in a directory that is on your PATH, make it executable (chmod +x thescript.py) and give it a Shebang line:
#!/bin/env python
The second option is probably preferable. On Windows, you can install pylauncher to support shebang lines; if you use Python 3.3 or newer, it is already included with your Python installation.

Another option would be to create a batch file for each script you care about, and put the batch file somewhere in your PATH, e.g. create a file called thescript.bat containing...
#echo off
the\path\to\python.exe the\path\to\thescript.py %*
...then you can just run...
thescript arg1 arg2
...which is about as terse a syntax as possible.