We Keep Coding

How to to calculate unnested watersheds in GRASS GIS?

I am running into a few issues using the GRASS GIS module r.accumulate while running it in Python. I use the module to calculate sub watersheds for over 7000 measurement points. Unfortunately, the output of the algorithm is nested. So all sub watersheds are overlapping each other. Running the r.accumulate sub watershed module takes roughly 2 minutes for either one or multiple points, I assume the bottleneck is loading the direction raster.
I was wondering if there is an unnested variant in GRASS GIS available and if not, how to overcome the bottleneck of loading the direction raster every time you call the module accumulate. Below is a code snippet of what I have tried so far (resulting in a nested variant):
locations = VectorTopo('locations',mapset='PERMANENT')
locations.open('r')
points=[]
for i in range(len(locations)):
points.append(locations.read(i+1).coords())
for j in range(0,len(points),255):
output = "watershed_batch_{}#Watersheds".format(j)
gs.run_command("r.accumulate", direction='direction#PERMANENT', subwatershed=output,overwrite=True, flags = "r", coordinates = points[j:j+255])
gs.run_command('r.stats', flags="ac", input=output, output="stat_batch_{}.csv".format(j),overwrite=True)
Any thoughts or ideas are very welcome.

I already replied to your email, but now I see your Python code and better understand your "overlapping" issue. In this case, you don't want to feed individual outlet points one at a time. You can just run
r.accumulate direction=direction#PERMANENT subwatershed=output outlet=locations
r.accumulate's outlet option can handle multiple outlets and will generate non-overlapping subwatersheds.

The answer provided via email was very usefull. To share the answer I have provided the code below to do an unnested basin subwatershed calculation. A small remark: I had to feed the coordinates in batches as the list of coordinates exceeded the max length of characters windows could handle.
Thanks to #Huidae Cho, the call to R.accumulate to calculate subwatersheds and longest flow path can now be done in one call instead of two seperate calls.
The output are unnested basins. Where the largers subwatersheds are seperated from the smaller subbasins instead of being clipped up into the smaller basins. This had to with the fact that the output is the raster format, where each cell can only represent one basin.
gs.run_command('g.mapset',mapset='Watersheds')
gs.run_command('g.region', rast='direction#PERMANENT')
StationIds = list(gs.vector.vector_db_select('locations_snapped_new', columns = 'StationId')["values"].values())
XY = list(gs.vector.vector_db_select('locations_snapped_new', columns = 'x_real,y_real')["values"].values())
for j in range(0,len(XY),255):
output_ws = "watershed_batch_{}#Watersheds".format(j)
output_lfp = "lfp_batch_{}#Watersheds".format(j)
output_lfp_unique = "lfp_unique_batch_{}#Watersheds".format(j)
gs.run_command("r.accumulate", direction='direction#PERMANENT', subwatershed=output_ws, flags = "ar", coordinates = XY[j:j+255],lfp=output_lfp, id=StationIds[j:j+255], id_column="id",overwrite=True)
gs.run_command("r.to.vect", input=output_ws, output=output_ws, type="area", overwrite=True)
gs.run_command("v.extract", input=output_lfp, where="1 order by id", output=output_lfp_unique,overwrite=True)
To export the unique watersheds I used the following code. I had to transform the longest_flow_path to point as some of the longest_flow_paths intersected with the corner boundary of the watershed next to it. Some longest flow paths were thus not fully within the subwatershed. See image below where the red line (longest flow path) touches the subwatershed boundary:
enter image description here
gs.run_command('g.mapset',mapset='Watersheds')
lfps= gs.list_grouped('vect', pattern='lfp_unique_*')['Watersheds']
ws= gs.list_grouped('vect', pattern='watershed_batch*')['Watersheds']
files=np.stack((lfps,ws)).T
#print(files)
for file in files:
print(file)
ids = list(gs.vector.vector_db_select(file[0],columns="id")["values"].values())
for idx in ids:
idx=int(idx[0])
expr = f'id="{idx}"'
gs.run_command('v.extract',input=file[0], where=expr, output="tmp_lfp",overwrite=True)
gs.run_command("v.to.points", input="tmp_lfp", output="tmp_lfp_points", use="vertex", overwrite=True)
gs.run_command('v.select', ainput= file[1], binput = "tmp_lfp_points", output="tmp_subwatersheds", overwrite=True)
gs.run_command('v.db.update', map = "tmp_subwatersheds",col= "value", value=idx)
gs.run_command('g.mapset',mapset='vector_out')
gs.run_command('v.dissolve',input= "tmp_subwatersheds#Watersheds", output="subwatersheds_{}".format(idx),col="value",overwrite=True)
gs.run_command('g.mapset',mapset='Watersheds')
gs.run_command("g.remove", flags="f", type="vector",name="tmp_lfp,tmp_subwatersheds")
I ended up with a vector for each subwatershed

Colour Difference in The Foundry NUKE

So I am trying to make a painterly node group with Python code straight so I can use it for future projects but I can't seem to get the power part of the formula in nuke to work from this colour difference formula( I'm also new to Nuke so if there is a better way of writing this please let me know it would be awesome thank you, or if I'm doing this wrong completely also let me know)
The following formula for color difference is used to create the
difference image: |(r1,g1,b1) – (r2,g2,b2)| = ((r1 – r2)^2 + (g1
–g2)^2 + (b1 – b2)^2)^1/2.
nRedShuffle = nuke.nodes.Shuffle()
nRedShuffle['red'].setValue('red')
nRedShuffle['green'].setValue('red')
nRedShuffle['blue'].setValue('red')
nRedShuffle['alpha'].setValue('red')
nGreenShuffle = nuke.nodes.Shuffle()
nGreenShuffle['red'].setValue('green')
nGreenShuffle['green'].setValue('green')
nGreenShuffle['blue'].setValue('green')
nGreenShuffle['alpha'].setValue('green')
#...(so on for the rest of rgba1 and rgba2)
nGreenShuffle2 = nuke.nodes.Shuffle()
nGreenShuffle2['red'].setValue('green')
nGreenShuffle2['green'].setValue('green')
nGreenShuffle2['blue'].setValue('green')
nGreenShuffle2['alpha'].setValue('green')
nBlueShuffle2 = nuke.nodes.Shuffle()
nBlueShuffle2['red'].setValue('blue')
nBlueShuffle2['green'].setValue('blue')
nBlueShuffle2['blue'].setValue('blue')
nBlueShuffle2['alpha'].setValue('blue')
#I am having troubles with the powers below
redDiff = nuke.nodes.Merge2(operation='minus', inputs=[nRedShuffle2, nRedShuffle])
redDiffMuli = nuke.nodes.Merge2(operation='multiply', inputs=[redDiff, redDiff])
greenDiff = nuke.nodes.Merge2(operation='minus', inputs=[nGreenShuffle2, nGreenShuffle])
greenDiffMuli = nuke.nodes.Merge2(operation='multiply', inputs=[greenDiff, greenDiff])
blueDiff = nuke.nodes.Merge2(operation='minus', inputs=[nBlueShuffle2, nBlueShuffle])
blueDiffMuli = nuke.nodes.Merge2(operation='multiply', inputs=[blueDiff, blueDiff])
redGreenAdd = nuke.nodes.Merge2(operation='plus', inputs=[redDiffMuli, greenDiffMuli])
redGreenBlueAdd = nuke.nodes.Merge2(operation='plus', inputs=[redGreenAdd, blueDiffMuli])

Here are at least two ways to implement Color Difference formula for two images. You can use difference op in Merge node or you can write a formula in field for each channel inside MergeExpression node:
Expression for each channel is as simple as this:
abs(Ar-Br)
abs(Ag-Bg)
abs(Ab-Bb)
Python commands
You can use .nodes.MergeExpression methodology:
import nuke
merge = nuke.nodes.MergeExpression(expr0='abs(Ar-Br)',
expr1='abs(Ag-Bg)',
expr2='abs(Ab-Bb)')
or regular .createNode syntax:
merge = nuke.createNode('MergeExpression')
merge['expr0'].setValue('abs(Ar-Br)')
merge['expr1'].setValue('abs(Ag-Bg)')
merge['expr2'].setValue('abs(Ab-Bb)')
Full code version
import nuke
import nukescripts
red = nuke.createNode("Constant")
red['color'].setValue([1,0,0,1])
merge = nuke.createNode('MergeExpression')
merge['expr0'].setValue('abs(Ar-Br)')
merge['expr1'].setValue('abs(Ag-Bg)')
merge['expr2'].setValue('abs(Ab-Bb)')
yellow = nuke.createNode("Constant")
yellow['color'].setValue([1,1,0,1])
merge.connectInput(0, yellow)
nuke.toNode('MergeExpression1').setSelected(True)
nukescripts.connect_selected_to_viewer(0)
# Auto-alignment in Node Graph
for everyNode in nuke.allNodes():
everyNode.autoplace()
Consider! MergeExpression node is much slower that regular Merge(difference) node.

How to iterate over and download each image in an image collection from the Google Earth Engine python api

I am new to google earth engine and was trying to understand how to use the Google Earth Engine python api. I can create an image collection, but apparently the getdownloadurl() method operates only on individual images. So I am trying to understand how to iterate over and download all of the images in the collection.
Here is my basic code. I broke it out in great detail for some other work I am doing.
import ee
ee.Initialize()
col = ee.ImageCollection('LANDSAT/LC08/C01/T1')
col.filterDate('1/1/2015', '4/30/2015')
pt = ee.Geometry.Point([-2.40986111110000012, 26.76033333330000019])
buff = pt.buffer(300)
region = ee.Feature.bounds(buff)
col.filterBounds(region)
So I pulled the Landsat collection, filtered by date and a buffer geometry. So I should have something like 7-8 images in the collection (with all bands).
However, I could not seem to get iteration to work over the collection.
for example:
for i in col:
print(i)
The error indicates TypeError: 'ImageCollection' object is not iterable
So if the collection is not iterable, how can I access the individual images?
Once I have an image, I should be able to use the usual
path = col[i].getDownloadUrl({
'scale': 30,
'crs': 'EPSG:4326',
'region': region
})

It's a good idea to use ee.batch.Export for this. Also, it's good practice to avoid mixing client and server functions (reference). For that reason, a for-loop can be used, since Export is a client function. Here's a simple example to get you started:
import ee
ee.Initialize()
rectangle = ee.Geometry.Rectangle([-1, -1, 1, 1])
sillyCollection = ee.ImageCollection([ee.Image(1), ee.Image(2), ee.Image(3)])
# This is OK for small collections
collectionList = sillyCollection.toList(sillyCollection.size())
collectionSize = collectionList.size().getInfo()
for i in xrange(collectionSize):
ee.batch.Export.image.toDrive(
image = ee.Image(collectionList.get(i)).clip(rectangle),
fileNamePrefix = 'foo' + str(i + 1),
dimensions = '128x128').start()
Note that converting a collection to a list in this manner is also dangerous for large collections (reference). However, this is probably the most scalable method if you really need to download.

Here is my solution:
import ee
ee.Initialize()
pt = ee.Geometry.Point([-2.40986111110000012, 26.76033333330000019])
region = pt.buffer(10)
col = ee.ImageCollection('LANDSAT/LC08/C01/T1')\
.filterDate('2015-01-01','2015-04-30')\
.filterBounds(region)
bands = ['B4','B5'] #Change it!
def accumulate(image,img):
name_image = image.get('system:index')
image = image.select([0],[name_image])
cumm = ee.Image(img).addBands(image)
return cumm
for band in bands:
col_band = col.map(lambda img: img.select(band)\
.set('system:time_start', img.get('system:time_start'))\
.set('system:index', img.get('system:index')))
# ImageCollection to List
col_list = col_band.toList(col_band.size())
# Define the initial value for iterate.
base = ee.Image(col_list.get(0))
base_name = base.get('system:index')
base = base.select([0], [base_name])
# Eliminate the image 'base'.
new_col = ee.ImageCollection(col_list.splice(0,1))
img_cummulative = ee.Image(new_col.iterate(accumulate,base))
task = ee.batch.Export.image.toDrive(
image = img_cummulative.clip(region),
folder = 'landsat',
fileNamePrefix = band,
scale = 30).start()
print('Export Image '+ band+ ' was submitted, please wait ...')
img_cummulative.bandNames().getInfo()
A reproducible example can you found it here: https://colab.research.google.com/drive/1Nv8-l20l82nIQ946WR1iOkr-4b_QhISu

You could possibly use ee.ImageCollection.iterate() with a function that gets the image and adds it to a list.
import ee
def accumluate_images(image, images):
images.append(image)
return images
for img in col.iterate(accumulate_images, []):
url = img.getDownloadURL(dict(scale=30, crs='EPSG:4326', region=region))
Unfortunately I am not able to test this code as I do not have access to the API, but it might help you arrive at a solution.

I have a similar problem and was not able o solve with presented solutions. Then I have elaborated a sample code for this purpose. It iterates over an image collection in client side, then it is not affected by limitations (server side only) of .map() or .iterate().
It is possible to download the code and see its explanation here
It basically transform the ImageCollection into a list (ic.toList()). Then it performs a standard loop, and for each individual image it is possible to convert it back to ee.Image(list.get(i)), and then process one by one taking all images in the collection.
In your particular case, to download each image, the function to be called within the loop could be: getDOwnloadURL() or getThumbURL():
var url = imgNew.getDownloadURL({
region: geometry,
});
var thumbURL = imgNew.getThumbURL({region: geometry,dimensions: 512, format: 'png'});

im.getcolors() returns None

I am using a simple code to compare an image to a desktop screenshot through the function getcolors() from PIL. When I open an image, it works:
im = Image.open('sprites\Bowser\BowserOriginal.png')
current_sprite = im.getcolors()
print current_sprite
However, using both pyautogui.screenshot() and ImageGrab.grab() for the screenshot, my code returns none. I have tried using the RGB conversion as shown here: Cannot use im.getcolors.
Additionally, even when I save a screenshot to a .png, it STILL returns none.
i = pyautogui.screenshot('screenshot.png')
f = Image.open('screenshot.png')
im = f.convert('RGB')
search_image = im.getcolors()
print search_image
First time posting, help is much appreciated.

Pretty old question but for those who sees this now:
Image.getcolors() takes as a parameter "maxcolors – Maximum number of colors." (from the docs here).
The maximum number of colors an image can have, equals to the number of pixels it contains.
For example, an image of 50*60px will have maximum 3,000 colors.
To translate it into code, try this:
# Open the image.
img = Image.open("test.jpg")
# Set the maxcolors number to the image's pixels number.
colors = img.getcolors(img.size[0]*img.size[1])

If you'd check the docs, getcolors returns None if the number of colors in the image is greater than the default parameter, which is set to 256.

Python Image Library: How to combine 4 images into a 2 x 2 grid?

I have 4 directories with images for an animation. I would like to take the set of images and generate a single image with the 4 images arranged into a 2x2 grid for each frame of the animation.
My code so far is:
import Image
fluid64 = "Fluid64_half_size/00"
fluid128 = "Fluid128_half_size/00"
fluid512 = "Fluid512_half_size/00"
fluid1024 = "Fluid1024_half_size/00"
out_image = "Fluid_all/00"
for pic in range(1, 26):
blank_image = Image.open("blank.jpg")
if pic < 10:
image_num = "0"+str(pic)
else:
image_num = str(pic)
image64 = Image.open(fluid64+image_num+".jpg")
image128 = Image.open(fluid128+image_num+".jpg")
image512 = Image.open(fluid512+image_num+".jpg")
image1024 = Image.open(fluid1024+image_num+".jpg")
out = out_image + image_num + ".jpg"
blank_image.paste(image64, (0,0)).paste(fluid128, (400,0)).paste(fluid512, (0,300)).paste(fluid1024, (400,300)).save(out)
Not sure why it's not working. I'm getting the error:
Traceback (most recent call last):
File "C:\Users\Casey\Desktop\Image_composite.py", line 24, in <module>
blank_image.paste(image64, (0,0)).paste(fluid128, (400,0)).paste(fluid512, (
ste(fluid1024, (400,300)).save(out)
AttributeError: 'NoneType' object has no attribute 'paste'
shell returned 1
Any help would be awesome. Thanks!

The only problem there is that "paste" does not return an image object - it rather modifies the "blank" image inplace.
So, when the second paste is called (the one that uses the fuild128 image), it tries to be applied on "None" - which is the return value of the first image.
If that is the only problem you are having, just make one paste call per line, like this:
blank_image.paste(image64, (0,0))
blank_image.paste(fluid128, (400,0))
blank_image.paste(fluid512, (0,300))
blank_image.paste(fluid1024, (400,300))
blank_image.save(out)
Although it looks likely you'd need to scale each image so that their format match as well.
And your code for the "image_num" variable is unecessary. Python is really good with strings - just do something like this:
image64 = Image.open(fluid64 + "%02d.jpg" % pic)

You may want to be using something along the lines of :
blank_image = Image.new("RGB", (800, 600))
This will create a new area in memory in which you can generate your image. You should then be able to paste you images into that.
Then you'll need to save it out again later on with:
blank_image.save("blank.jpg")

Read the error message:
AttributeError: 'NoneType' object has no attribute 'paste'
This means you tried to call .paste on something that was of type NoneType, i.e. on the None object.
Image.paste returns None. You can't "chain" together calls like that except when the functions are specifically designed to support it, and Image.paste is not. (Support for this sort of thing is accomplished by having the function return self. You get an error that talks about NoneType because the function is written not to return anything, and everything in Python returns None by default if nothing else is returned explicitly.) This is considered Pythonic: methods either return a new value, or modify self and return None. Thus, so-called "fluent interfaces" are not used when the functions have side effects - Pythonistas consider that harmful. Returning None is a warning that the function has side effects. :)
Just do four separate .paste calls.

Tiling figures in a 2-by-2 grid would be easy to achieve with the append_images function defined in this reply
https://stackoverflow.com/a/46623632/8738113
For example:
img1 = append_images([image64, image128], direction='horizontal')
img2 = append_images([image512, image1024], direction='horizontal')
final = append_images([img1, img2], direction='vertical')
final.save("Fluid_all/00.jpg")

Unlike PIL APIs copy, crop, resize or rotate which return an Image object, paste returns None which prevents chained method calls. Not so convenient API design.