pyosmium - Build a Geojson linestring based on OSM Relation - python

I have a python script to analyse OSM data, and the objective is to build a GeoJson with specific data issued from OSM relation.
I'm currently focusing on OSM relation that represents 'hiking' trail like this one.
According to the document:
members
(read-only) Ordered list of relation members. See osmium.osm.RelationMemberList.
the relation object has an attribute members which collects all members of the relation.
Hence The first part of the script manages to extract all relation that have a tag sac_scale=hiking and collects all its ways.
The following script is on purpose focusing only on 1 specific relation : r104369
class HikingWaysfromRelations(osmium.SimpleHandler):
def __init__(self):
super(HikingWaysfromRelations, self).__init__()
self.dict = {}
def _getWays(self, elem, elem_type):
# tag
if 'sac_scale' in elem.tags and elem.tags['sac_scale']=='hiking' and elem.id==104369:
list=[]
for mem in elem.members:
if mem.type=="w":
list.append(str(mem.ref))
self.dict["r"+str(elem.id)]=list
else:
pass
def relation(self,r):
self._getWays(r, "relation")
ml = HikingWaysfromRelations()
ml.apply_file('../pbf/new-zealand.osm.pbf')
The result is a dictionary containing the expected relation as the only key, and its ways:
{"r104369": ["191668175", "765285136", "765285135", "765285138", "765285139", "191668225", "765542429", "765542430", "765542432", "765542431", "765542435", "765542436", "765542434", "765542433", "765542437", "765542438", "765542439", "765542440", "765542441", "765542442", "765548983", "271277386", "765548985", "765548984", "684295241", "684295239", "464603363", "464603364", "464607430", "299788481", "178920047", "155711655", "155711646", "684294192", "259362037", "684294189", "259362038", "259362041", "259362036", "259362043", "259362039", "259362040"]}
Now the question is: How to build a GeoJson containing a single Feature MultiLineString that connects all those ways and rebuild the expected hiking trail?
Based on what I've found on the net, I should re-run a simpleHander on the full .pbf file, and each time I encounter a way I'm looking for - based on the values of the above dictionary - I could reconstruct a LineString with:
import shapely.wkb as wkblib
wkbfab = osmium.geom.WKBFactory()
def getRelationGeometry(elem):
wkb=wkbfab.create_linestring(elem)
return wkb
The issue is that it looks like some ways have only 1 node, hence triggering following error:
RuntimeError: need at least two points for linestring (way_id=155711655)
So what would be the solution to re-build a GeoJson feature - multiLineString - of multiple ways, to be able to plot the result on https://geojson.io/#map=2/20.0/0.0 ?
How for instance openstreetmap manages to re-build the track of a relation when I hit link if not by connecting all nodes (from all ways) issued from the relation ?
Thanks a lot for your help
I know there is way with bash, where you first filter the initial pbf by keeping only relation with the tag sac_scale=hiking, and then transforming this filtered result to GeoJson - but I really want to be able to generate the same with python to understand how OSM data are stored. I just can't figure out an easy way to do so, knowing pyosmium is equivalent (supposedly) to osmium, I believe there should be an easy way there too
osmium export output/output_food-drinks.pbf -f geojson

Looking at the way with the id shown in the error in your post (155711655), it has two nodes, not one. Visible here as of the time of this answer.
Knowing that, I can think of two reasons why you would get that error:
You're not passing in the argument location=True to the apply_file method as suggested by the documentation:
Because of the way that OSM data is structured, osmium needs to internally cache node geometries, when the handler wants to process the geometries of ways and areas. The apply_file() method cannot deduce by itself if this cache is needed. Therefore locations need to be explicitly enabled by setting the locations parameter to True:
h.apply_file("test.osm.pbf", locations=True, idx='flex_mem')
Looking at your code above, the apply_file method only has the input file as an argument, so I think this is likely your problem.
The way may be referencing a node that is missing in your pbf extract. This is simple to verify with the osmium cli tool:
osmium check-refs <your pbf file>
This is the result I get from running that on a valid pbf of my own
There are 6702885 nodes, 431747 ways, and 2474 relations in this file.
Nodes in ways missing: 0
Note the Nodes in ways missing: 0.

Related

Retrieving node locations from pydotplus (or any layered graph drawing engine)

I'm preparing a layered graph drawing using a dataframe containing node data:
type label
0 Class Insurance Product
1 Class Person
2 Class Address
3 Class Insurance Policy
And another containing relationship data:
froml tol rel fromcard tocard
0 Insurance Policy Insurance Product ConveysProduct One One
1 Person Insurance Policy hasPolicy One Many
2 Person Address ResidesAt None None
I populate a pydotplus dot graph with the content, which I can then use to generate a rendering:
pdp_graph = pydotplus.graphviz.Dot(graph_name="pdp_graph", graph_type='digraph', prog="dot")
for i,e in b_rels_df.iterrows():
edge = pydotplus.graphviz.Edge(src=e['froml'], dst=e['tol'], label=e['rel'])#, set_fromcard=e['fromcard'], set_tocard=e['tocard'])
pdp_graph.add_edge(edge)
for i,n in ents_df.iterrows():
node = pydotplus.graphviz.Node(name=n['label'], set_type=n['type'], set_label=n['label'])
pdp_graph.add_node(node)
png = pdp_graph.create_png()
display(Image(png))
So far so good - but now I want to retrieve the node positions for use in my own interactive layout (the png is a nice example/diagram, but I want to build upon it), so am attempting to retrieve the node locations calculated via:
[n.get_pos() for n in pdp_graph.get_nodes()]
But this only returns:
> [None, None, None, None]
I've tried lots of different methods, graphviz/dot are installed fine - as proven by the image of the layout - how can I extract the positions of the nodes as data from any type of dot-style layout?
There is a way I can do this via the pygraphviz library via networkx, but the installation-overhead restricts me (pygraphviz needs to be recompiled to cinch with the graphviz install) from being able to use that for the target installations where I've less control over the base environments, hence my attempt to use pydotplus, which appears less demanding in terms of install requirements.
How do I retrieve the layout data from a layered graph drawing using this setup (or one similar), such that I can use it elsewhere? I'm looking for x,y values that I can map back to the nodes that they belong to.
I know nothing about your python setup. ( As usual with python it seems awkward and restrictive )
I suggest using Graphviz directly. In particular the 'attributed DOT format' which is a plain text file containing the layout ( resulting from the layout engine ) produced when the engine is run with the command option -Tdot. The text file is easily parsed to get exactly what you need.
Here is a screenshot of the first paragraph of the relevant documentation
The graphviz.org website contains all the additional details you may need.
You are creating an output file of png format. All position data is lost in this process. Instead, create output format="dot". Then, read that back in & modify as desired.

How do I pull specific property values from an injected object when adding a batch of edges in a Gremlin/TinkerPop traversal?

I want to add batches of edges to a JanusGraph db that already contains nodes. I want my edges to support setting dynamic/optional properties.
I've cobbled together the following traversal (based on this SO question) that I believe illustrates what I want to do:
1..inject() a batch of edges
2. Pull to/from vertex ids from the objects in the injected edge batch
3. Set all fields in edge batch objects as edge properties with .sideEffect()
uuid_1 = "89079f8fa3ee849a61a45e0b3e6d28cd"
uuid_2 = "00a9ae430dc812f483b0660212264190"
edge_batch = [
{
"from_uuid": uuid_1,
"to_uuid": uuid_2,
"posted_at": 1650012568000,
"test_property_2": "I was here"
},
{
"from_uuid": uuid_2,
"to_uuid": uuid_1,
"posted_at": 1650012568888,
"test_property_3": "I'M STILL HERE"
}
]
new_edges = (
g
.inject(edge_batch)
.unfold()
.as_("edge_batch")
.V()
.has("uuid", __.select("edge_batch").select("to_uuid"))
.as_("to_v")
.V()
.has("uuid", __.select("edge_batch").select("from_uuid"))
.addE("MY_EDGE_TYPE")
.to("to_v")
.as_("new_edge")
.sideEffect(
__.select("edge_batch")
.unfold()
.as_("kvp")
.select("new_edge")
.property(
__.select("kvp").by(Column.keys), __.select("kvp").by(Column.values)
)
)
.iterate()
)
As written, the code above results in a traversal timeout when the referenced vertices exist. If I replace the first two __.select("edge_batch")... expressions above with references to the uuid_1 and uuid_2 variables, the code works. I think my problem is I just can't figure out how I'm supposed to reference properties of the injected, unfolded edge batch objects.
I'm using gremlin-python v3.6.0, JanusGraph v0.6.1, TinkerPop v3.5.1.
Your code runs just fine with a small graph. Only, the two unfolds from a list with two elements makes your code run four times, unintentionally I guess.
As to why the code does not run on your janusgraph installation:
Be sure uuid is an indexed property if your graph is large
Maybe you were confused by vertices with uuid_1 and uuid_2 being present in the janusgraph cache, because .has("uuid", __.select("edge_batch").select("to_uuid")) and .has("uuid", uuid_1) really do the same.

Dynamo Revit set formula for a parameter in a family

I am trying to add a formula to a parameter within a Revit Family.
Currently I have multiple families in a project. I run Dynamo from within that project then I extract the families that I want to modify using Dynamo standard nodes.
Then I use a python script node that goes through every selected family and find the parameter I am interested in, and assign a formula for it.
That seemed fine until I noticed that it is not assigning the formula, but it is entering it as a string — as in it is in quotes. And sure enough, the code i am using will only work with Text type parameters.
Can someone shed the light on how to assign a formula to a parameter using dynamo?
see line 32 in code below
Thanks
for family in families:
TransactionManager.Instance.ForceCloseTransaction()
famdoc = doc.EditFamily(family)
FamilyMan = famdoc.FamilyManager
found.append(family.Name)
TransactionManager.Instance.EnsureInTransaction(famdoc)
check = 0
# Loop thru the list of parameters to assign formula values to them... these are given as imput
for r in range(len(param_name_lst)):
# Loop thru the list of parameters in the current family per the families outter loop above.
for param in FamilyMan.Parameters:
#for param in FamilyMan.get_Parameter(param_name_lst[r]):
# for each of the parameters get their name and store in paramName.
paramName = param.Definition.Name
# Check if we have a match in parameter name.
if param_name_lst[r] in paramName:
if param.CanAssignFormula:
canassignformula.append(param_name_lst[r])
else:
cannotassignformula.append(param_name_lst[r])
try:
# Make sure that the parameter is not locked.
if FamilyMan.IsParameterLocked(param):
FamilyMan.SetParameterLocked(param,False)
locked.append(paraName)
# Enter formula value to parameter.
FamilyMan.SetFormula(param, param_value_lst[r])
check += 1
except:
failed.append(paramName)
else:
continue
Actually, you can access the family from the main project, and you can assign a formula automatically.... That's what i currently do, i load all the families i want in one project and run the script.
After a lot of work, i was able to figure out what i was doing wrong, and in it is not in my code... my code was fine.
The main problem is that i need to have all of my formula's dependencies lined up.... just like in manual mode.
so if my formula is:
size_lookup(MY_ID_tbl, "MY_VAR", "MY_DefaultValue", ND1,ND2)
then i need to have the following:
MY_ID_tbl should exist and be assigned a valid value, in this case it should have a csv filename. Moreover, that file should be also loaded. This is important for the next steps.
MY_VAR should be defined in that csv file, so Does ND1, ND2
The default value (My_Default_Value) should match what that csv file says about that variable...in this case, it is a text.
Needless to say, i did not have all of the above lined up as it should be, once i fixed that, my setFormula code did its job. And i had to change my process altogether, cause i have to first create the MY_ID_tbl and load the csv file which i also do using dynamo, then i go and enter the formulas using dynamo.
Revit parameters can only be assigned to a formula inside the family editor only, that is the first point, so you should run your dynamo script inside the family editor for each family which will be a waste of time and you just edit the parameter's formula manually inside each family.
and the second point, I don't even think that it is possible to set a certain parameter's formula automatically, it must be done manually ( I haven't seen anything for it in the Revit API docs).

How to identify node set names in an assembly in abaqus with Python scripting?

I am analyzing a model with various node sets in Abaqus, from which I want to extract different data.
I have been introducing node set names by hand for each analysis, and this can be quite tedious if there is plenty of node sets in the model.
I was wondering if there is any way of obtaining a list with all the node set names in it. Is it possible?
Example of the result I am expecting:
NSETS=['NSET-1',''NSET-2'...]
It depends on whether you're interacting with a model database or output database and where the set resides. Once you identify where you're accessing the sets dictionary you can use the keys method to get a list of set names. For example, to get the set names in an instance in the assembly:
NSETS = mdb.models['Model-1'].rootAssembly.instances['PART-1-1'].sets.keys()
You can use the same approach for getting the set names elsewhere. In the Abaqus Scripting Reference Manual there are two relevant sections that show where you can access sets. The first is for accessing sets in the model database:
Abaqus > Scripting Reference > Python commands > Region commands > Set object
import part
mdb.models[name].parts[name].allInternalSets[name]
mdb.models[name].parts[name].allSets[name]
mdb.models[name].parts[name].sets[name]
import assembly
mdb.models[name].rootAssembly.allinstances.sets[name]
mdb.models[name].rootAssembly.allInternalSets[name]
mdb.models[name].rootAssembly.allSets[name]
mdb.models[name].rootAssembly.instances[name].sets[name]
mdb.models[name].rootAssembly.modelInstances[i].sets[name]
mdb.models[name].rootAssembly.sets[name]
And for accessing sets from the output database:
Abaqus > Scripting Reference > Python commands > Odb commands > OdbSet object
import odbAccess session.odbs[name].parts[name].elementSets[name]
session.odbs[name].parts[name].nodeSets[name]
session.odbs[name].parts[name].surfaces[name]
session.odbs[name].rootAssembly.elementSets[name]
session.odbs[name].rootAssembly.instances[name].elementSets[name]
session.odbs[name].rootAssembly.instances[name].nodeSets[name]
session.odbs[name].rootAssembly.instances[name].surfaces[name]
session.odbs[name].rootAssembly.nodeSets[name]
session.odbs[name].rootAssembly.surfaces[name]
session.odbs[name].steps[name].frames[i].fieldOutputs[name].values[i].instance.elementSets[name]
session.odbs[name].steps[name].frames[i].fieldOutputs[name].values[i].instance.nodeSets[name]
session.odbs[name].steps[name].frames[i].fieldOutputs[name].values[i].instance.surfaces[name]

Django finding paths between two vertexes in a graph

This is mostly a logical question, but the context is done in Django.
In our Database we have Vertex and Line Classes, these form a (neural)network, but it is unordered and I can't change it, it's a Legacy Database
class Vertex(models.Model)
code = models.AutoField(primary_key=True)
lines = models.ManyToManyField('Line', through='Vertex_Line')
class Line(models.Model)
code = models.AutoField(primary_key=True)
class Vertex_Line(models.Model)
line = models.ForeignKey(Line, on_delete=models.CASCADE)
vertex = models.ForeignKey(Vertex, on_delete=models.CASCADE)
Now, in the application, the user will be able to visually select TWO vertexes (the green circles below)
The javascript will then send the pk of these two Vertexes to Django, and it has to find the Line classes that satisfy a route between them, in this case, the following 4 red Lines :
Business Logic:
A Vertex can have 1-4 Lines related to it
A Line can have 1-2 Vertexes related to it
There will only be one possible route between two Vertexes
What I have so far:
I understand that the answer probably includes recursion
The path must be found by trying every path from one Vertex untill the other is find, it can't be directly found
Since there are four and three-way junctions, all the routes being tried must be saved throughout the recursion(unsure of this one)
I know the basic logic is looping through all the lines of each Vertex, and then get the other Vertex of these lines, and keep walking recursively, but I really don't know where to start on this one.
This is as far as I could get, but it probably does not help (views.py) :
def findRoute(request):
data = json.loads(request.body.decode("utf-8"))
v1 = Vertex.objects.get(pk=data.get('v1_pk'))
v2 = Vertex.objects.get(pk=data.get('v2_pk'))
lines = v1.lines.all()
routes = []
for line in lines:
starting_line = line
#Trying a new route
this_route_index = len(routes)
routes[this_route_index] = [starting_line.pk]
other_vertex = line.vertex__set.all().exclude(pk=v1.pk)
#There are cases with dead-ends
if other_vertex.length > 0:
#Mind block...
As you has pointed, this is not a Django/Python related question, but a logical/algorithmic matter.
To find paths between two vertexes in a graph you can use lot of algorithms: Dijkstra, A*, DFS, BFS, Floyd–Warshall etc.. You can choose depending on what you need: shortest/minimum path, all paths...
How to implement this in Django? I suggest to don't apply the algorithm over the models itself, since this could be expensive (in term of time, db queries, etc...) specially for large graphs; instead, I'd rather to map the graph in an in-memory data structure and execute the algorithm over it.
You can take a look to this Networkx, which is a very complete (data structure + algorithms) and well documented library; python-graph, which provides a suitable data structure and a whole set of important algorithms (including some of the mentioned above). More options at Python Graph Library

Categories