I have a Geotiff that I display on a tile map, but it's slightly off to the south. For example, on this screenshot the edge of the image should be where the country border is, but it's a bit to the south:
Here's the relevant part of the code:
tiff_rio_500 = rioxarray.open_rasterio('/content/mw/mw_dist_to_light_at_all_from_light_mask_mw_cut_s3_500.tif')
dataarray_500 = tiff_rio_500[0]
dataarray_500_meters = dataarray_500.copy()
dataarray_500_meters['x'], dataarray_500_meters['y'] = ds.utils.lnglat_to_meters(dataarray_500.x, dataarray_500.y)
hv_dataset_500_meters = hv.Dataset(dataarray_500_meters, name='nightlights', vdims='cumulative_cost')
hv_tiles_osm_bokeh = hv.element.tiles.OSM().opts(width=1000, height=800)
hv_image_500_meters_bokeh = hv.Image(hv_dataset_500_meters, kdims=['x', 'y'], vdims=['cumulative_cost'], rtol=1).opts(cmap='inferno_r')
hv_combined_osm_500_meters_bokeh = hv_tiles_osm_bokeh * hv_image_500_meters_bokeh
hv_combined_osm_500_meters_bokeh
You can see the live notebook on google colab.
Now this is not the usual "everything is way off" problem that occurs when one doesn't convert the map to Web Mercator. It is almost perfect, it just isn't.
The Geotiff is an Earth Engine export. This is how it looked originally in Earth Engine (see live code):
As you can see, the image follows the borders everywhere.
At first, I suspected that maybe the export went wrong, or the google map tileset is somewhat different, but no, if I open the same exported Tiff in the QGis application on my windows laptop and view it on the same OSM tilemap as I do in the colab notebook, it looks fine:
Okay, the image does not follow the borders perfectly, but I know why and that's unrelated (I oversimplified the country border geometry). The point is, that it is projected to the correct location. So based on that, the tiff contains the correct information, it can be displayed at the same location as the borders are in the OSM tilemap, but still in my Holoviews-Datashader-Bokeh project it is slightly off.
Any idea why this happens?
I've got the answer on the Holoviz Discourse from one of the developers. Seeing how the recommended function is practically undocumented, I copy it here in case somebody looks for an easy way to load a geotiff and add to a tilemap in Holoviews/Geoviews:
https://discourse.holoviz.org/t/geotiff-overlay-position-is-slightly-off-on-holoviews-bokeh-tilemap/2071
philippjfr
I wouldn’t expect manually transforming the coordinates to work
particularly well. While it’s a much heavier weight dependency for
accurate coordinate transforms I’d recommend using GeoViews.
img = gv.util.load_tiff( '/content/mw/mw_dist_to_light_at_all_from_light_mask_mw_cut_s3_500.tif' )
gv.tile_sources.OSM() * img.opts(cmap='inferno_r')
Edit: Now it is possible one doesn't want to use Geoviews as it has a pretty heavy dependency chain that requires a lot of patience and luck to set it up right. Fortunately rioxarray (through rasterio) has a tool to reproject, just append ".rio.reproject('EPSG:3857')" to the first line and then you don't have to use the lnglat_to_meters which is not intended for this purpose.
So the corrected code becomes:
tiff_rio_500 = rioxarray.open_rasterio('/content/mw/mw_dist_to_light_at_all_from_light_mask_mw_cut_s3_500.tif').rio.reproject('EPSG:3857')
hv_dataset_500_meters = hv.Dataset(tiff_rio_500[0], name='nightlights', vdims='cumulative_cost')
hv_tiles_osm_bokeh = hv.element.tiles.OSM().opts(width=1000, height=800)
hv_image_500_meters_bokeh = hv.Image(hv_dataset_500_meters, kdims=['x', 'y'], vdims=['cumulative_cost'], rtol=1).opts(cmap='inferno_r')
hv_combined_osm_500_meters_bokeh = hv_tiles_osm_bokeh * hv_image_500_meters_bokeh
hv_combined_osm_500_meters_bokeh
Now compared to the Geoviews solution (that supposedly handles everything automatically), this solution has a downside that if you use a Hover Tooltip to display the values and coordinates under the mouse cursor, the coordinates are showing up in the newly projected web mercator system in millions of meters instead of the expected degrees. The solution for that is outside the scope of this answer, but I'm just finishing a detailed step by step guide that contains a solution for that too, and I will link that here as soon as it is published. Of course if you don't use Hover Tooltip, the code above will be perfect for you without any more tinkering.
Related
I have some time series plots I've done a long time ago that I would like to improve in terms of graphics and style. However, I didn't save the raw data and I cannot recover them.
So I was wondering, is there a way to retrieve data points from a chart image (e.g. png file)? like I input an image and I get a csv/dataframe/array with pairs of x,y coordinates?
To give an idea, that's the kind of images I would like to convert:
I've seen this GRABIT could potentially work but I'm not familiar with MatLab. Is there anything python-based or possibly some web tools?
Preferences:
work on linux systems (in particular ubuntu)
doesn't require installation
there is a little and simple software develop here in Brazil that can do this kind of thing.
You have to download in the following link:
http://paginapessoal.utfpr.edu.br/lasouza/analise-nao-linear-de-estruturas/Pega%20Ponto%201.0.exe/view
You can load the image, specify the origin and set the x and y labels. After you can retrieve the points that you clicked on the image.
So I have a shapefile that is 3GB in size and as you can imagine my browser doesn't like it. How can I compress the data I have which is either in lon/lat coordinates or points on an X,Y grid?
I saw a video on Computerphile about Discreet Cosine Transforms for reducing high dimesionality data but being a programmer and not a mathematician I don't know if this is even possible. I have tried to take a point every 10 steps in the file like so: map[0:100000:10] but this had an udesireable and very lossy effect.
I would ideally like to have my data so it would work like Google Earth in which the resolution adjusts to your viewport altitude. So when you zoom in to the map higher freqency data is presented in the viewport, limiting the amount of points but I don't know how they do this and Google return nothing of value.
Last point is that since these are just vectors is there any type of vector compression I could use? I'm not to great at math so as you can imagine when I look into this I just get confused fairly quickly. I uderstand SciPy has some DCT built in and I know it has a whole bunch of other features which I don't understand, perhaps I could use this?
I can answer the "level of detail" part: you can experiment with leaflet (a javascript mapping library). You could then define a "coarse" layer wich is displayed for low zoom levels and "high detail" layers that are only displayed at higher zoom levels. You probably need to capture the map zoomend event and load/unload your layers from there.
One solution to this problem is to use a Web Map Server (WMS) like GeoServer or MapServer that stores your ShapeFile (though a spatial database like PostGIS would be better) on the server and sends a rendered image (often broken down into cacheable tiles) to the browser.
Question:
Using graphicsmagick, what is a good way to find the coordinates of a small image inside a bigger image?
Explnation:
To explain further, I have a large screen shot that I am working with and would like to find the pixel coordinates of a known icon that is expected to be found somewhere within the screen shot.
Also, if this is not a good library to be using for this purpose, would love to hear suggestions for alternatives that will preferably be compatible with Python.
Thanks so much!
I use "gm display" to do that.
gm display &
Click on the image. Select Transform, then Crop. Put the cursor at the
top left of the small image. Read the coordinates from the small
information window. Select "dismiss"
Note that this is a manual method, which is OK if you are really "working
with the image" on screen. If you are looking for a batch method, it'll
be a little more complex.
It's a long one so you might want to get that cup of tea/coffee you've been holding off on ;)
I run a game called World of Arl, it's a turn based strategy game akin to Risk or Diplomacy. Each player has a set of cities, armies and whatnot. The question revolves around the display of these things. Currently the map is created using a background image with CSS positioning of team icons on top of that to represent cities. You can see how it looks here: WoA Map
The background image for the map is located here: Map background and created in Omnigraffle. It's not designed to draw maps but I'm hopelessly incompetent with photoshop and this works for my purposes just fine.
The problem comes that I want to perform such fun things as pathfinding and for that I need to have the map somehow stored in code. I have tried using PIL, I have looked at incorporating it with Blender, I tried going "old school" and creating tiles as from many older games and finally I tried to use SVG. I say this so you can see clearly that it's not through lack of trying that I have this problem ;)
I want to be able to store the map layout in code and both create an image from it and use it for things such as pathfinding. I'm using Python but I suspect that most answers will be generic. The cities other such things are stored already and easily drawn on, I want to store the layout of the landmass and features on the landmass.
As for pathfinding, each type of terrain has a movement cost and when the map is stored as just an image I can't access the terrain of a given area. In addition to pathfinding I wish to be able to know the terrain for various things related to the game, cities in mountains produce stone for example.
Is there a good way to do this and what terms should I have used in Google because the terms I tried all came up with unrelated stuff (mapping being something completely different most of the time).
Edit 2:
Armies can be placed anywhere on the map as can cities, well, anywhere but in the water where they'd sink, drown and probably complain (in that order).
After chatting to somebody on MSN who made me go over the really minute details and who has a better understanding of the game (owing to the fact that he's played it) it's occurring to me that tiles are the way to go but not the way I had initially thought. I put the bitmap down as it is now but also have a data layer of tiles, each tile has a given terrain type and thus pathfinding and suchlike can be done on it yet at the same time I still render using Omnigraffle which works pretty well.
I will be making an editor for this as suggested by Adam Smith. I don't know that graphs will be relevant Xynth but I've not had a chance to look into them fully yet.
I really appreciate all those that answered my question, thanks.
I'd store a game map in code as a graph.
Each node would represent a country/city and each edge would represent adjacency. Once you have a map like that, I'm sure you can find many resources on AI (pathfinding, strategy, etc.) online.
If you want to be able to build an image of the map programattically, consider adding an (x, y) coordinate and an image for each node. That way you can display all of the images at the given coordinates to build up a map view.
The key thing to realize here is that you don't have to use just one map. You can use two maps:
The one you already have which is drawn on screen
A hidden map which isn't drawn but which is used for path finding, collision detection etc.
The natural next question then is where does this second map come from? Easy, you create your own tool which can load your first map, and display it. Your tool will then let you draw boundaries around you islands and place markers at your cities. These markers and boundaries (simple polygons e.g.) are stored as your second map and is used in your code to do path finding etc.
In fact you can have your tool emit python code which creates the graphs and polygons so that you don't have to load any data yourself.
I am just basically telling you to make a level editor. It isn't very hard to do. You just need some buttons to click on to define what you are adding. e.g. if you are adding a polygon. Then you can just add each mouse coordinate to an array each time you click on your mouse if you have toggled your add polygon button. You can have another button for adding cities so that each time you click on the map you will record that coordinate for the city and possibly a corresponding name that you can provide in a text box.
You're going to have to translate your map into an abstract representation of some kind. Either a grid (hex or square) or a graph as xynth suggests. That's the only way you're going to be able to apply things like pathfinding algorithms to it.
IMO, the map should be rendered in the first place instead of being a bitmap. What you should be doing is to have separate objects each knowing its dimensions clearly such as a generic Area class and classes like City, Town etc derived from this class. Your objects should have all the information about their location, their terrain etc and should be rendered/painted etc. This way you will have exact knowledge of where everything lies.
Another option is to keep the bitmap as it is and keep this information in your objects as their data. By doing this the objects won't have a draw function but they will have precise information of their placement etc. This is sort of duplicating the data but if you want to go with the bitmap option, I can't think of any other way.
If you just want to do e.g. 2D hit-testing on the map, then storing it yourself is fine. There are a few possibilities for how you can store the information:
A polygon per island
Representing each island as union of a list rectangles (commonly used by windowing systems)
Creating a special (maybe greyscale) bitmap of the map which uses a unique solid colour for each island
Something more complex (perhaps whatever Omnigiraffe's internal representation is)
Asuming the map is fixed (not created on the fly) its "correct" to use a bitmap as graphical representation - you want to make it as pretty as possible.
For any game related features such as pathfinding or whatever fancy stuff you want to add you should add adequate data structures, even if that means some data is redundant.
E.g. describe the boundaries of the isles as polygon splines (either manually or automatically created from the bitmap, thats up to you and how much effort you want to spend and is needed to get the functionality you want).
To sum it up: create data structures matching the problems you have to solve, the bitmap is fine for looks but avoid doing pathfining or other stuff on it.
Background
I have been asked by a client to create a picture of the world which has animated arrows/rays that come from one part of the world to another.
The rays will be randomized, will represent a transaction, will fade out after they happen and will increase in frequency as time goes on. The rays will start in one country's boundary and end in another's. As each animated transaction happens a continuously updating sum of the amounts of all the transactions will be shown at the bottom of the image. The amounts of the individual transactions will be randomized. There will also be a year showing on the image that will increment every n seconds.
The randomization, summation and incrementing are not a problem for me, but I am at a loss as to how to approach the animation of the arrows/rays.
My question is what is the best way to do this? What frameworks/libraries are best suited for this job?
I am most fluent in python so python suggestions are most easy for me, but I am open to any elegant way to do this.
The client will present this as a slide in a presentation in a windows machine.
The client will present this as a slide in a presentation in a windows machine
I think this is the key to your answer. Before going to a 3d implementation and writing all the code in the world to create this feature, you need to look at the presentation software. Chances are, your options will boil down to two things:
Animated Gif
Custom Presentation Scripts
Obviously, an animated gif is not ideal due to the fact that it repeats when it is done rendering, and to make it last a long time would make a large gif.
Custom Presentation Scripts would probably be the other way to allow him to bring it up in a presentation without running any side-programs, or doing anything strange. I'm not sure which presentation application is the target, but this could be valuable information.
He sounds like he's more non-technical and requesting something he doesn't realize will be difficult. I think you should come up with some options, explain the difficulty in implementing them, and suggest another solution that falls into the 'bang for your buck' range.
If you are adventurous use OpenGL :)
You can draw bezier curves in 3d space on top of a textured plane (earth map), you can specify a thickness for them and you can draw a point (small cone) at the end. It's easy and it looks nice, problem is learning the basics of OpenGL if you haven't used it before but that would be fun and probably useful if your in to programing graphics.
You can use OpenGL from python either with pyopengl or pyglet.
If you make the animation this way you can capture it to an avi file (using camtasia or something similar) that can be put onto a presentation slide.
It depends largely on the effort you want to expend on this, but the basic outline of an easy way. Would be to load an image of an arrow, and use a drawing library to color and rotate it in the direction you want to point(or draw it using shapes/curves).
Finally to actually animate it interpolate between the coordinates based on time.
If its just for a presentation though, I would use Macromedia Flash, or a similar animation program.(would do the same as above but you don't need to program anything)