So I have been working on integrating the Geospatial Modeling Environment tools (formerly Hawth's) with ArcGIS 10.1 via Python. Below is the code I am using, which works great, to create a text file of code and then call up GME via Python to process the shapefiles I am using. As far as I can tell, I have been able to mimic verbatim what the creator states will work in Python (see his documentation here: http://www.spatialecology.com/gme/images/SpatialEcologyGME.pdf)
Code:
import arcpy, sys, os, subprocess
from arcpy import env
#Supply the following arguments:
#Workspace (full path)
#Catchment Polygons (full path)
#Raster Data (full path)
#Prefix for the output: 6 characters to denote the raster dataset.
#Thematic value: TRUE or FALSE
#An output txt file (full path -> eg. C:/Users/Alison/Desktop/file.txt)
########
#Each argument must be in double quotes, and they must be separated by a space.
#The polygon and raster datasets must be in same coordinate system.
env.workspace = sys.argv[1]
print env.workspace
inputPoly = sys.argv[2]
inputRast = sys.argv[3]
prefix = sys.argv[4]
thematic = sys.argv[5]
code = 'isectpolyrst(in="' + inputPoly + '", raster="' + inputRast + '", prefix="' + prefix + '", thematic="' + thematic +'");'
print code
newFile = sys.argv[6]
print newFile
newFileObj = open(newFile, 'w')
newFileObj.write(code)
newFileObj.close()
print newFile
os.system(r'C:\Program Files (x86)\SpatialEcology\GME\SEGME.exe')
print "subprocess.call(r'C:\Program Files (x86)\SpatialEcology\GME\SEGME.exe -c run(in=\\\"" + newFile + "\\\");');"
subprocess.call(r'C:\Program Files (x86)\SpatialEcology\GME\SEGME.exe -c run(in=\\\"" + newFile + "\\\");');
However, while this process works fine, I just end up hitting another wall...It opens GME, but alas, it doesn't actually do anything. It ultimately doesn't seem to run the created text file. The isectpolyrst tool works like Tabulate Area, so in theory, the values should all be appended to the polygon data, but through Python it doesn't seem to do it....(and I am using GME because Tabulate Area cannot handle the size of my datafiles and crashes both in Arc but also as a Python script).
I am wondering if anyone has successfully been able to run GME through Python for use in what will be an ArcPy script, so that the task can be automated, rather than having to go through GME and then into Arc. My searching suggests that this is a common problem for those trying to automate the process, but for all I know I am just missing a colon somewhere, or some other piece of code.
Thanks for the feedback!
Figured it out!
GME can use a text file to read the desired code, so I wrote the inputs in Python in as they should appear in GME, and wrote these to a text file. These were then read into a subprocess call that brings up GME and runs the text file. Works like a charm.
Took some tinkering, but worth it!
Related
I have been using OneDrive to store a large amount of images and now I need to process those, so I have sync'd my OneDrive folder to my computer, which takes relatively no space on disk. However, since I have to open() them in my code, they all get downloaded, which would take much more than the available memory on my computer. I can manually use the Free up space action in the right-click contextual menu, which keeps them sync'd without taking space.
I'm looking for a way to do the same thing but in my code instead, after every image I process.
Trying to find how to get the commands of contextual menu items led me to these two places in the registry:
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Directory\shell
HKEY_LOCAL_MACHINE\SOFTWARE\Classes*\shellex\ContextMenuHandlers
However I couldn't find anything related to it and those trees have way too many keys to check blindly. Also this forum post (outside link) shows a few ways to free up space automatically, but it seems to affect all files and is limited to full days intervals.
So is there any way to either access that command or to free up the space in python ?
According to this microsoft post it is possible to call Attrib.exe to do that sort of manipulation on files.
This little snippet does the job for a per-file usage. As shown in the linked post, it's also possible to do it on the full contents of a folder using the /s argument, and much more.
import subprocess
def process_image(path):
# Open the file, which downloads it automatically
with open(path, 'r') as img:
print(img)
# Free up space (OneDrive) after usage
subprocess.run('attrib +U -P "' + path + '"')
The download and freeing up space are fairly quick, but in the case of running this heavily in parallel, it is possible that some disk space will be consumed for a short amount of time. In general though, this is pretty instantaneous.
In addition to Mat's answer. If you are working on a Mac then you can replace Attrib.exe with "/Applications/OneDrive.App/Contents/MacOS/OneDrive /unpin" to make the file online only.
import subprocess
path = "/Users/OneDrive/file.png"
subprocess.run(["/Applications/OneDrive.App/Contents/MacOS/OneDrive", "/unpin", path])
Free up space for multiples files.
import os
import subprocess
path = r"C:\Users\yourUser\Folder"
diret = os.listdir(path)
for di in diret:
dir_atual = path + "\\" + di
for root, dirs, files in os.walk(dir_atual):
for file in files:
arquivos = (os.path.join(root, file))
print (arquivos)
subprocess.run('attrib +U -P "' + arquivos + '"')
I am trying to remove absolute path info from my playlist.m3u file(s) and convert this file(s) to a relative path, using Python. I can do this with an Excel script that concatenates and it works great, but I would think the Python route would be easier.
Here is a script I have been trying to get to work, without success.
import sys
import os
infile = sys.argv[1]
basepath = os.path.dirname(os.path.abspath(infile))
outlines = []
fp = open(infile)
for line in fp.readlines():
if line.startswith('#'): # m3u comments start with #
outlines.append(line)
else:
outlines.append(os.path.relpath(line, basepath))
fp.close()
fp = open(infile, "w")
for line in outlines:
fp.write(line)
fp.close()
Here is an example of the absolute path playlist file contents:
J:\NTFS_1\MP3_D\Dan Fogelberg - River of Souls - 08 - A Love Like This.mp3
J:\NTFS_1\MP3_H\Harry Chapin - Verities & Balderdash - 04 - 30,000 Pounds Of Bananas.mp3
Here is the relative path playlist contents from Excel:
\Dan Fogelberg - River of Souls - 08 - A Love Like This.mp3
\Harry Chapin - Verities & Balderdash - 04 - 30,000 Pounds Of Bananas.mp3
I execute the python code with the command line:
c:\temp>playlist.py playlist.m3u > playlistout.m3u
The program does generate an output file playlistout.m3u but it is blank or empty. I have really looked around and posted elsewhere about a solution, without success. I'm exhausted at this point. Anyone? Thanks.
You are calling this with
c:\temp>playlist.py playlist.m3u > playlistout.m3u
which redirects the stdout printed output of playlist.py to playlistout.m3u. However, in your program you aren't printing anything. Your program actually writes back into the original input file in these lines
fp = open(infile, "w")
for line in outlines:
fp.write(line)
fp.close()
The way you're calling this program you would actually want to replace those lines with
for line in outlines:
print(line)
You don't actually need to store the outlines and could rewrite the program to exclude that part but we'll save that for later.
If you want to make sure that you are producing anything in outlines at all you can do:
import sys
print("Number of lines to print: {}".format(len(outlines)), file=sys.stderr)
for line in outlines:
print(line)
This answer talks primarily about script code that sends output to a specific local computer hard drive and folder. There are 2 answers that I have posted here. I am answering my own questions, as the originating poster. I have put in a lot of personal effort researching the question in the original post. Just seems like a lot of things came together, here recently. So, I furnished 2 answers on the same general question. Thanks. Bill M.
My previous answer stands as is and serves the purpose described. The script output location, in the earlier case, was a function of a command line statement variable. I noted this constraint. My preference was to be able to specify, within the python script itself, where the output file should be placed. After additional research, I have cobbled together a couple lines of code that make my needs complete. The product is spartan. Not very elegant. But it works for me. If I could have found a script that performed this directly, I would have used it. Maybe others will find it useful directly or indirectly.
So, in this second answer, to my own question, I have reworked some Stackoverflow code from another topic to allow for stipulating where the converted playlist output is placed.
The original code for handling how to redirect the output of print to a TXT file was found at the link How to redirect the output of print to a TXT file.
One of the responses by Eli Courtwright was ideal for this situation. Here are the pertinent code pieces that I extracted:
log = open("c:\\goat.txt", "w")
print >>log, "test"
Here is how I adapted them and (later) incorporated them for my situation
outfile = open("c:\\temp\goat.m3u", "w")
print >> outfile,('\%s' % (filename))
I changed from:log to:outfile. I modified my earlier print command to now point to outfile, which previously was associated with my preferred local hard drive location. Note the use of the double rear facing slashes.
Finally, here is the complete 9 lines of code required to read an absolute path playlist, then convert and generate a new playlist file for my Android mp3 player application.
# Python Script to extract playlist relative folder path and names
# Script removes local storage absolute path info from playlist records.
import sys
import os
outfile = open("c:\\temp\goat.m3u", "w")
f = open('output.txt','w')
with open('C:/temp/playlist.m3u') as f:
for line in f:
drive, path = os.path.splitdrive(line)
path, filename = os.path.split(path)
# Note the '\' in the print command. Need this because the
# previous operation removes any/all path info. This 1 '\'
# is required to be restored
print >> outfile,('\%s' % (filename))
That's all I have. I'm thinking I can still write a batch file like "convert.bat" where I can drop my original playlist.m3u file and have it execute the python script directly, instead of running it from PyCharm or the Command Line. But that's another day. Merry Christmas.
This answer talks about the script code required to extract the relative path information from the absolute path information that is generally found in a playlist created by a stand alone media center file and database manager. Playlists can then be generated and exported, along with media files, to portable tablets and cellphones within a household.
I am answering my own question, as the originating poster. I have put in a lot of personal effort researching the question in the original post. Just seems like a lot of things came together, here recently. So, I furnished 2 answers on the same general question. Thanks. Bill M.
The below script is a very compact way to convert a playlist with full absolute path information to a playlist with minimal relative path information. I use a popular Android music player, except. It cannot decipher a standard .m3u playlist, when that playlist is created externally And when the playlist includes the full Drive and Folder path information that is generated by my home media center application. Note, other Android mp3 players apps didn't suffer this shortcoming, but they failed to impress me for other reasons. So, I needed a means to remove the absolute path information. Here are the initial conditions and the solution.
I have a file named playlist.m3u. It is plain text. The file contents are:
J:\NTFS_1\MP3_D\Dan Fogelberg - River of Souls - 08 - A Love Like This.mp3
J:\NTFS_1\MP3_H\Harry Chapin - Verities & Balderdash - 04 - 30,000 Pounds Of Bananas.mp3
The above file reords are not recognized by my Android music player, because each line or record contains absolute path information, such as J:\NTFS_1\MP3_D and J:\NTFS_1\MP3_H. The above is only an example and there are additional absolute path folders for MP3_A, MP_B, etc., etc.
If I edit, or otherwise process the above lines either manually or with an Excel formula conversion, this needs to be the opening format of each record:
\Dan Fogelberg - River of Souls - 08 - A Love Like This.mp3
\Harry Chapin - Verities & Balderdash - 04 - 30,000 Pounds Of Bananas.mp3
Here is a verbatim code snippet that I found here on Stackoverflow at the link: (Splitting path strings into drive, path and file name parts):
with open('C:/Users/visc/scratch/scratch_child/test.txt') as f:
for line in f:
drive, path = os.path.splitdrive(line)
path, filename = os.path.split(path)
print('Drive is %s Path is %s and file is %s' % (drive, path, filename))
Here is a reworked version that meets my requirements. I execute this from the DOS command line, so that I can direct the results to an output file, at the command line.
I named this script playlist.py. On the command line I enter:
C:\temp\playlist.py > output.m3u.
Here is playlist.py:
import sys
import os
with open('C:/temp/playlist.m3u') as f:
for line in f:
drive, path = os.path.splitdrive(line)
path, filename = os.path.split(path)
# print('Drive is %s Path is %s and file is %s' % (drive, path, filename))
# The original print adds spaces and such. I removed the not necessary
# Drive and Path fields. Note, I Added a "\"
print('\%s' % (filename))
I tested the above at the command line. It produces an output file populated as required and shown above. I also tested the script in PyCharm Projects Community. PyCharm will produce the print output on screen. This agrees, of course, with the redirect output for the command line > output.m3u
I'm happy to have satisfied my curiosity and have a functional solution to the issue of the original post.
I could still use some help. What additional code will allow me to specify in the python script, where and to what named file I want the output printed to?
1. Introduction
I have a bunch of files in netcdf format.
Each file contain the meteorology condition of somewhere in different period(hourly data).
I need to extract the first 12 h data for each file. So I select to use NCO(netcdf operator) to deal with.
NCO works with terminal environment. With >ncks -d Time 0,11 input.nc output.nc, I can get one datafile called out.ncwhich contain the first 12h data of in.nc.
2. My attempt
I want to keep all the process inside my ipython notebook. But I stuck on two aspects.
How to execute terminal code in python loop
How to transfer the string in python into terminal code.
Here is my fake code for example.
files = os.listdir('.')
for file in files:
filename,extname = os.path.splitext(file)
if extname == '.nc':
output = filename + "_0-12_" + extname
## The code below was my attempt
!ncks -d Time 0,11 file output`
3. Conclusion
Basically, my target was letting the fake code !ncks -d Time 0,11 file output coming true. That means:
execute netcdf operator directly in python loop...
...using filename which is an string in python environment.
Sorry for my unclear question. Any advice would be appreciated!
You can use subprocess.check_output to execute external program:
import glob
import subprocess
for fn in glob.iglob('*.nc'):
filename, extname = os.path.splitext(fn)
output_fn = filename + "_0-12_" + extname
output = subprocess.call(['ncks', '-d', 'Time', '0,11', fn, output_fn])
print(output)
NOTE: updated the code to use glob.iglob; you don't need to check extension manually.
You may also check out pynco which wraps the NCO with subprocess calls, similar to #falsetru's answer. Your application may look something like
nco = Nco()
for fn in glob.iglob('*.nc'):
filename, extname = os.path.splitext(fn)
output_fn = filename + "_0-12_" + extname
nco.ncks(input=filename, output=output_fn, dimension='Time 0,11')
I am currently able to visualize correctly in ParaView a .vtp file for each time step of a simulation, and to print a screenshot for each. I want to do that in batch, but I want to keep the same state for each one (view point, filters applied, etc). I have already saved the state into a .psvm file , and I tried to write a python script which, after being run by pvbatch, will (hopefully) print the screenshots. But, unfortunately, it is not working. I tried to change the filename in the state by processing the state file and doing a search and replace, but still it is not working. For instance, it keeps plotting the first data input only, even if the current file is different (altough GetSources() shows an always increasing list of sources). I use ParaView 3.14.0 in Snow Leopard. I am sure this is easy, but I am overwhelmed with the large amount of info about python and paraview with no reference to this particularissue. Please, please, any advice is greatly welcome, and I am sorry if this has been answered previously (I looked at google, the paraview mailing list, and here). Below is my script, which can also be found at http://pastebin.com/4xiLNrS0 . Furthermore, you can find some example files and state in http://goo.gl/XjPpE .
#!/bin/python
import glob, string, os, commands
from paraview.simple import *
#help(servermanager)
# vtp files are inside the local subdir DISPLAY
files = (commands.getoutput("ls DISPLAY/data-*.vtp | grep -v contacts")).split()
# process each file
for filename in files:
fullfn = commands.getoutput("ls $PWD/" + filename)
fn = filename.replace('DISPLAY/', '')
#os.system("cp ../dem_git/addons/paraview_state.pvsm tmp.pvsm")
os.system("cp ~/Desktop/state.pvsm tmp.pvsm")
os.system("sed -i.bck 's/DATA.vtp/" + fullfn.replace('/','\/') + "/1' tmp.pvsm") # replace first intance with full path
os.system("sed -i.bck 's/DATA.vtp/" + fullfn.replace('/','\/') + "/1' tmp.pvsm") # replace second intance with full path
os.system("sed -i.bck 's/DATA.vtp/" + fn + "/1' tmp.pvsm") # replace third with just the filename path
servermanager.LoadState("tmp.pvsm")
pm = servermanager.ProxyManager()
reader = pm.GetProxy('sources', fullfn)
reader.FileName = fullfn
reader.FileNameChanged()
reader.UpdatePipeline()
view = servermanager.GetRenderView()
SetActiveView(view)
view.StillRender()
WriteImage(filename.replace(".vtp", ".png"))
os.system("rm -f tmp.pvsm")
os.system("rm -f tmp.pvsm.bck")
Delete(reader)
I realise this is an old question, but I had exactly the same problem recently and couldn't find any answers either. All you need to do is add Delete(view) after Delete(reader) for your script to work.
I'm working on a python program that will automatically combine sets of files based on their names.
Being a newbie, I wasn't quite sure how to go about it, so I decided to just brute force it with the win32api.
So I'm attempting to do everything with virtual keys. So I run the script, it selects the top file (after arranging the by name), then sends a right click command,selects 'combine as adobe PDF', and then have it push enter. This launched the Acrobat combine window, where I send another 'enter' command. The here's where I hit the problem.
The folder where I'm converting these things loses focus and I'm unsure how to get it back. Sending alt+tab commands seems somewhat unreliable. It sometimes switches to the wrong thing.
A much bigger issue for me.. Different combination of files take different times to combine. though I haven't gotten this far in my code, my plan was to set some arbitrarily long time.sleep() command before it finally sent the last "enter" command to finish and confirm the file name completing the combination process. Is there a way to monitor another programs progress? Is there a way to have python not execute anymore code until something else has finished?
I would suggest using a command-line tool like pdftk http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/ - it does exactly what you want, it's cross-platform, it's free, and it's a small download.
You can easily call it from python with (for example) subprocess.Popen
Edit: sample code as below:
import subprocess
import os
def combine_pdfs(infiles, outfile, basedir=''):
"""
Accept a list of pdf filenames,
merge the files,
save the result as outfile
#param infiles: list of string, names of PDF files to combine
#param outfile: string, name of merged PDF file to create
#param basedir: string, base directory for PDFs (if filenames are not absolute)
"""
# From the pdftk documentation:
# Merge Two or More PDFs into a New Document:
# pdftk 1.pdf 2.pdf 3.pdf cat output 123.pdf
if basedir:
infiles = [os.path.join(basedir,i) for i in infiles]
outfile = [os.path.join(basedir,outfile)]
pdftk = [r'C:\Program Files (x86)\Pdftk\pdftk.exe'] # or wherever you installed it
op = ['cat']
outcmd = ['output']
args = pdftk + infiles + op + outcmd + outfile
res = subprocess.call(args)
combine_pdfs(
['p1.pdf', 'p2.pdf'],
'p_total.pdf',
'C:\\Users\\Me\\Downloads'
)