I'm just getting started using imposm to help get openstreetmap data into a postgis database. All the docs point to making all commands via Terminal. This is fine for one off imports but I plan to have many many imports of varying bounding boxes and would like to script the loading of the data in the database.
Currently I use:
imposm --overwrite-cache --read --write -d postgis_test --user postgres -p "" /Users/Me/MapnikTest/osmXML.osm
Which works fine from the command line but as osmXML.osm is being created many times I would like somehow to import this at the point of creation.
Putting the same thing in a python script as:
os.system("imposm --overwrite-cache --read --write -d postgis_test --user postgres -p "" /Users/Ali\ Mac\ Pro/Desktop/MapnikTest/osmXML.osm")
just returns:
/bin/sh: imposm: command not found
Solving this would be the final step to automate the acquisition of data to render small maps on demand but I'm falling at the final hurdle!
** Edit full path to imposm solved the first problem but imputing the password for the postgres user happens when prompted. Is there a way to send the password in the same single line command? (maybe this needs to be a new post?, happy if someone points me in the right direction)**
This is probably because os.system() is calling /bin/sh which uses a different shell environment from the one you use when working on the command line.
To work around this, in your script, get the full path to the imposm script and then use that in your command. Use can use some code like this to find the executable.
Or you can fix your shell definitions so that /bin/sh has the proper PATH defined, but that depends greatly on your setup...
Solved with the help of further research and the comments from #Eli Rose (many thanks): find out what path to imposm (or whichever command you are trying to make) with
which <command>
Then include the path in the python shell command. Using a module from subprocess you can even see the full terminal output.
import subprocess
from subprocess import *
print Popen("/usr/local/bin/imposm --overwrite-cache --read --write --connection postgis://<database user>:<password>#<host>/<database> /path/to/data.osm", stdout=PIPE, shell=True).stdout.read()
The
--connection postgis://<database user>:<password>#<host>/<database>
means you can make the command in a single line and not have to worry about entering the database user password in a following command.
Related
EDIT: It turns out the shell that the subprocess module does not have the same directories assigned to the PATH variable as my system's shell. Thus the solution is simply to either call Maxima using the full path (found through which maxima) or changing env={'PATH':maxima_paths} in the subprocess.run() parameters. As #RobertDodier pointed out, this is not an issue with Maxima specifically.
I have a file called testMaxima.txt that contains the following Maxima code:
write_data([[1,2,3]], "test.txt");
This Maxima code creates the file test.txt in the working directory.
I am looking to run this file using a Python script. I do not want to open Maxima (a math engine) manually in order to do this.
What I've tried: I know that I can run this file in Command Line/Terminal using maxima -b testMaxima.txt and I have tried to use the python module subprocess to emulate this code with the following python code:
import subprocess
subprocess.run("maxima -b testMaxima.txt", shell=True)
but this does not create the new test.txt file like it should. If this helps, when I run
>>> subprocess.call("maxima -b testMaxima.txt", shell=True)
127
I get an output of 127. From what I understand, this means that the terminal can't find the given commands. However, there is no issues when I execute the code maxima -b testMaxima.txt through the terminal directly, so I'm not sure why it cannot find the given commands. Does anyone have any thoughts on what I should try?
Thank you in advance!
EDIT: From #mkrieger 's advice, I am using a string "maxima -b testMaxima.txt" instead of the sequence ["maxima", "-b", "testMaxima.txt"]. Unfortunately, this does not make a difference.
how about using os.system in python.
import os
os.system('maxima -b testMaxima.txt')
Also make sure that you are running this python script in the location where testMaxima.txt file is present
My python script generate a proper command that user need to run in the same console. My scenario is that user is running a script and then as a result see the command that must to run. Is there any way to exit python script and send that command to console, so user do not need to copy/paste?
A solution would be to have your python script (let's call it script.py) just print the command: print('ls -l') and use it in a terminal like so: $(python3 script.py). This makes bash run the output of your script as a command, and would basically run a ls -l in the terminal.
You can even go a step beyond and create an alias in ~/.bashrc so that you no longer need to call the whole line. You can write at the end of the file something like alias printls=$(python3 /path/to/script.py). After starting a new terminal, you can type printls and the script will run.
A drawback of this method is that you have no proper way of handling exceptions or errors in your code, since everything it prints will be run as a command. One way (though ugly) would be to print('echo "An error occured!"') so that the user who runs the command can see that something malfunctioned.
However, I'd suggest going for the "traditional" way and running the command directly from python. Here's a link to how you can achieve this: Calling an external command in Python.
Python can run system commands in new subshells. The proper way of doing this is via the subprocess module, but for simple tasks it's easier to just use os.system. Example Python script (assuming a Unix-like system):
import os
os.system('ls')
I am trying to write a python script that will execute a bash script I have on my Windows machine. Up until now I have been using the Cygwin terminal so executing the bash script RunModels.scr has been as easy as ./RunModels.scr. Now I want to be able to utilize subprocess of Python, but because Windows doesn't have the built in functionality to handle bash I'm not sure what to do.
I am trying to emulate ./RunModels.scr < validationInput > validationOutput
I originally wrote this:
os.chdir(atm)
vin = open("validationInput", 'r')
vout = open("validationOutput", 'w')
subprocess.call(['./RunModels.scr'], stdin=vin, stdout=vout, shell=True)
vin.close()
vout.close()
os.chdir(home)
But after spending a while trying to figure out why my access was denied, I realized my issue wasn't the file permissions but the fact that I was trying to execute a bash file on Windows in general. Can someone please explain how to execute a bash script with directed input/output on windows using a python script?
Edit (Follow up Question):
Thanks for the responses, I needed the full path to my bash.exe as the first param. Now however, command line calls from within RunModels.scr come back in the python output as command not found. For example, ls, cp, make. Any suggestions for this?
Follow up #2:
I updated my call to this:
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
The error I now get is /usr/bin/bash: RunModels.scr: No such file or directory.
Using cwd does not seem to have any effect on this error, either way the subprocess is looking in /usr/bin/bash for RunModels.scr.
SELF-ANSWERED
I needed to specify the path to RunModels.scr in the call as well as using cwd.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'C:\\path\\dir_where_RunModels\\RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
But another problem...
Regardless of specifying cwd, the commands executed by RunModels.scr are throwing errors as if RunModels.scr is in the wrong directory. The script executes, but cp and cd throw the error no such file or directory. If I navigate to where RunModels.scr is through the command line and execute it the old fashioned way I don't get these errors.
Python 3.4 and below
Just put bash.exe in first place in your list of subprocess.call arguments. You can remove shell=True, that's not necessary in this case.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Depending on how bash is installed (is it in the PATH or not), you might have to use the full path to the bash executable.
Python 3.5 and above
subprocess.call() has been effectively replaced by subprocess.run().
subprocess.run(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Edit:
With regard to the second question, you might need to add the -l option to the shell invocation to make sure it reads all the restart command files like /etc/profile. I presume these files contain settings for the $PATH in bash.
Edit 2:
Add something like pwd to the beginning of RunModels.scr so you can verify that you are really in the right directory. Check that there is no cd command in the rc-files!
Edit 3:
The error /usr/bin/bash: RunModels.scr: No such file or directory can also be generated if bash cannot find one of the commands that are called in the script. Try adding the -v option to bash to see if that gives more info.
A better solution than the accepted answer is to use the executable keyword argument to specify the path to your shell. Behind the curtain, Python does something like
exec([executable, '-c`, subprocess_arg_string])
So, concretely, in this case,
subprocess.call(
'./RunModels.scr',
stdin=vin, stdout=vout,
shell=True,
executable="C:/cygwin64/bin/bash.exe")
(Windows thankfully lets you use forward slashes instead of backslashes, so you can avoid the requirement to double the backslashes or use a raw string.)
I have a set of python scripts which I run as a daemon services. These all work great, but when all the scripts are running and I use top -u <USER>, I see all my scripts running as python.
I would really like to know which script is running under which process id. So is there any way to execute a python script as a different process name?
I'm stuck here, and I'm not ever sure what terms to Google. :-)
Note: I'm using Ubuntu Linux. Not sure if the OS matters or not.
Try using setproctitle. It should work fine on Linux.
Don't have a linux system here to test this on appropriately, but if the above doesn't work, you should be able to use the same trick they use for things like gzip etc.
The script has to tell what to run it at the top like this:
#!/usr/local/bin/python
Use a softlink like this:
ln -s /usr/local/bin/python ~/bin/myutil
Then just change your script to
#!~/bin/myutil
and it should show up that way instead. You may need to use a hard link instead of a soft link.
Launching a python script using the python script itself (and file associations and/or shell magic) is not very portable, but you can use similar methods on nearly any OS.
The easiest way to get this is using she bang. The first line of your python script should be:
#!/usr/bin/python
or
#!/usr/bin/python3
depending upon whether you use python or python3
and then assign executable permissions to the script as follows:
chmod +x <scriptname>
and then run the script as
./scriptname
this will show up as scriptname in top.
I am new to Ubuntu... I am trying to run my first simple python program "Hello World" ...
After running following commands in terminal
1. chmod +x filename.py
2. ./filename.py
terminal is showing following error "bash: ./filename.py: Permission denied"
what can I do for solve about problem?
Do you have the appropriate incantation at the top of your python file? e.g.,
#!/usr/bin/python (or alternatively #!/usr/bin/env python)
Just to clarify, chmod +x only makes a file executable, it doesn't run it.
And I'm assuming your script looks like nothing more complex than this:
#!/usr/bin/env python
print 'hello world'
Some possibilities:
What does it say if you type umask? chmod +x will only make a file executable for you if your umask doesn't block the user executable bit. A typical umask such as 0022 will not block the user execute bit, but a umask like 0122 can. (See the Description section of chmod(1) for more info.)
To execute a script such as a Python script, you also need read permission. Try chmod u+rx filename.py and execute the script again.
It's also remotely possible that whatever interpreter you've specified in the file with the "hashbang" line at the beginning of your file (e.g. #!/usr/bin/env python) isn't executable, although that in my experience yields a different error message.
I deal with the same problem on my new system.
It is the third time I tried to solve this, and your post is the first one appearing on google results. My post is late, but think that it will help another users with the same problem.
In my case, it was about partition table setup.
Check in your /etc/mtab file how python script is being stored. Check if there is a clause: noexec
noexec is a flag that forbid executing under the partition. By default, it is set with exec. But, sometimes, this kind of things happen.
Now, it is working fine here.