I wrote a python script which is being used in githooks/post-merge, and its purpose is to install or update modules from a env.yml file that exists in the same repository. I used the following code:
for module in modules_to_update:
print("\n================================= Updating {} ==================================\n".format(module))
update_command = ['conda', 'install', '{}={}'.format(module.name, module.version)]
subprocess.run(update_command)
As can be seen, modules_to_update contains the modules that will be installed/updated. I didn't use conda update because I didn't want the updates to be automatic.
The problem is: I expected that conda will ask for permission to install/update every module individually, but it didn't work as planned. When I tried it to do so, every module did the same thing- proceeded without waiting for my input:
Proceed ([y]/n)?
Preparing transaction: done
Verifying transaction: done
...
I didn't use the -y flag so it should not work this way, and when I tried using the python script separately it worked just fine.
You're not attaching any stdin to the subprocess. Thus the subprocess likely does a read() call of some sort, gets an empty string (since nothing is attached) and defaults to that [y] yes default in the prompt.
You could do run(..., stdin=sys.stdin), but it might be easier to run an interactive command via os.system(). (Don't use plain string joining to form a command line, though, to avoid shell injection.)
import shlex
update_command = ['conda', 'install', '{}={}'.format(module.name, module.version)]
os.system(shlex.join(update_command))
Related
I'm trying to run a Ruby project from a Python interface, but in its most current form it only works when run from a terminal, not from a Subprocess in Python. The Ruby project consists of an executable file (let's call it exeFile) which runs a command-line interface tool (let's call it cli.rb), which requires and calls various structures from various other Ruby files in the project. The CLI file takes command-line arguments via Thor. This works in a terminal, but fails in a subprocess call when the cli.rb file has been modified.
I've made sure I'm passing all the right arguments to subprocess.Popen. For example:
popen = subprocess.Popen(['/home/daveshere/.rbenv/shims/ruby', '/media/daveshere/Data/exeFile', '--b', '1.1.1', '-p'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, universal_newlines=True, shell=False)
The p argument hasn't changed in the cli.rb file, and works in both the terminal and the Subprocess. But days ago, I changed the type of b from numeric to string, which allows the program to run properly. Again, this works in the terminal, but not when running from the Subprocess. Instead, printing the Subprocess output shows me the error:
Expected numeric value for '--b'; got "1.1.1"
This is despite the cli.rb file requiring a string type for b now, and the fact that the same call works in the terminal. It seems as if there is some locally-cached outdated version of the cli.rb file being called by ruby in the Subprocess, but not the terminal. I even tried adding this to the cli.rb file to see if it registers:
puts "TESTING MODIFICATION"
That string prints when running from the terminal, but not from the Subprocess (although other, older output does).
I also made sure ruby -v returns the same version in both cases, and it does. I'm really not sure what's causing the disconnect here. I'm also running the Python Subprocess via Pycharm on Ubuntu, if that has any relevance. Any ideas?
Fixed it thanks to engineersmnky's help. For whatever reason, the Subprocess, regardless of whether shell=True or shell=False was used, whether bash was or wasn't used, whether env=os.environ was used, etc., could not read the ruby environment, including the local changes to the gem. It could only read the git-published changes. I'm a ruby/gem newbie, so the reason wasn't obvious to me.
I didn't have ownership of the repo for the gem I was editing, so I created a private fork of the repo, moved my local changes to that repo, edited the *.gemspec file to reflect the changes, and built and installed a new gem locally with the same name. Now everything works. All I have to do now is remember to add my local changes to the private git and reinstall as needed when new changes are made. Technically, maybe I could have just added the local changes to the public repo without committing anything, but I'd prefer playing it safe with the private repo.
When installing libraries directly in Databricks notebook cells via %pip install, the Python interpreter gets restarted. I have the understanding that in order for the newly installed packages to become visible and accessible to the rest of the notebook cells, the interpreter must be restarted.
My question: How would it be possible to perform this interpreter restart programatically?
I am installing packages using a function call, based on requirements stored on a separate file, but I noticed that newly installed packages are not present, despite the installation seemingly taking place, either on notebook or cluster scope. I figured out that the reason might be the lack of interpreter restart from my installation code.
Is such a behavior possible?
Firstly, you can preinstall packages at cluster level or job level before you even start the notebook. I would recommend that before you try anything else. Trust me I've run thousands of libraries including custom built ones and have never needed to-do this.
On to your actual question. Yes but it will throw an exception.
This code will cause you to kill the python process on your DBX workspace
%sh
pid=$(ps aux | grep PythonShell | grep -v grep | awk '{print $2}')
kill -9 $pid
However, it poses a problem for you. If you run this as a bash notebook cell this will not be wrappable in try catch logic which makes automated workflows impossible.
I would have said you can call the shell commands from python but that wouldn't work as the exception would be thrown in that cell. You could perhaps use scala and sys_process library to achieve it but I'm no scala expert sadly.
Hope that helps!
you can use this:
dbutils.library.restartPython()
for more info look at https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-utils#dbutils-library-restartpython
I'm trying to run my virtualenv called env (created before) using call() from subprocess and it doesn't work. Command is fine, it's working when I type it directly in terminal.
python code:
import subprocess
subprocess.call("source env/bin/activate", shell=True)
I was trying also:
import os
os.system("source env/bin/activate")
any ideas why command is not performed or what should I use instead os.system() and subprocess.call()?
In both examples, your code launches a subprocess. That subprocess, which has its own environment, attempts to run the "source" command, and then exits. Since the subprocess has its own environment, this has no effect on the current process.
Assuming your end goal is to run some other command in the subprocess, you should just run it directly. You don't specifically need to activate the virtual environment.
subprocess.call(["./env/bin/pip", "list"])
Avoid using the shell=True option if at all possible; it can be quite dangerous if you're not extremely careful with it.
If you really need to set the environment variables that the activate script sets in this script, you need to manually set them in os.environ. You can read the activate script to see what they are. They usually won't matter, though.
I'm just getting started using imposm to help get openstreetmap data into a postgis database. All the docs point to making all commands via Terminal. This is fine for one off imports but I plan to have many many imports of varying bounding boxes and would like to script the loading of the data in the database.
Currently I use:
imposm --overwrite-cache --read --write -d postgis_test --user postgres -p "" /Users/Me/MapnikTest/osmXML.osm
Which works fine from the command line but as osmXML.osm is being created many times I would like somehow to import this at the point of creation.
Putting the same thing in a python script as:
os.system("imposm --overwrite-cache --read --write -d postgis_test --user postgres -p "" /Users/Ali\ Mac\ Pro/Desktop/MapnikTest/osmXML.osm")
just returns:
/bin/sh: imposm: command not found
Solving this would be the final step to automate the acquisition of data to render small maps on demand but I'm falling at the final hurdle!
** Edit full path to imposm solved the first problem but imputing the password for the postgres user happens when prompted. Is there a way to send the password in the same single line command? (maybe this needs to be a new post?, happy if someone points me in the right direction)**
This is probably because os.system() is calling /bin/sh which uses a different shell environment from the one you use when working on the command line.
To work around this, in your script, get the full path to the imposm script and then use that in your command. Use can use some code like this to find the executable.
Or you can fix your shell definitions so that /bin/sh has the proper PATH defined, but that depends greatly on your setup...
Solved with the help of further research and the comments from #Eli Rose (many thanks): find out what path to imposm (or whichever command you are trying to make) with
which <command>
Then include the path in the python shell command. Using a module from subprocess you can even see the full terminal output.
import subprocess
from subprocess import *
print Popen("/usr/local/bin/imposm --overwrite-cache --read --write --connection postgis://<database user>:<password>#<host>/<database> /path/to/data.osm", stdout=PIPE, shell=True).stdout.read()
The
--connection postgis://<database user>:<password>#<host>/<database>
means you can make the command in a single line and not have to worry about entering the database user password in a following command.
I'm new to the site, and not an experienced programmer. I am trying to install a text aligner to my mac (leopard version 10.5.8). I have used the program before on other computers without difficulty. It requires the installation of XCode (done), SOX (done) and HTK (done). However, after navigating to the appropriate folder, I enter the command:
$ ./align.py [audio data folder]
and this error pops up:
Unknown option: -B
usage: /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
So I type:
$ python -h
which gives me a list of options, including:
-B : don't write .py[co] files on import; also PYTHONDONTWRITEBYTECODE=x
Like I said, I'm pretty green to the world of programming, and so I'm not sure what to do with this information or how to fix the problem. I tried updating python, even though my previous version (2.5.1) should have worked. I now have 2.7.3 installed. Any advice would be much appreciated!!!
The first line of an executable script is called the "she-bang". It tells the shell which interpreter to use to run the following code.
The reason it was breaking by running ./align.py is because when you use that form, you tell it to execute using whatever is specified in that very first line:
#!/usr/local/bin/python
This says to use a python located specifically at this location. Usually you do not want an explicit path as it can cause problems (as it did for you) if that is not the appropriate python on your system. The better way, unless you have some specific needs, is to allow it to use whatever python your own environment resolves:
#!/usr/bin/env python
It worked when you ran: "python align.py", because now you are bypassing the shebang line, and calling your own python with the script as the first argument. This runs it directly.
I assume the first line of your script contains a shebang pointing at a wrong python and includes the -B flag.