I am trying to triangulate a large amount of massive data using Delaunay to scipy.spatial for triangulation and networkx to get the node adjacency relations. My code works very well on small data sets but when I try to introduce volumes of about 2 miollion points I always get the following error:
raise NetworkXError(f"The node {n} is not in the graph.") from e
NetworkXError: The node 1 is not in the graph.
It seems like my graph store the first node and nothing more. When I did my research I found that networkx is well adapted to massive data
Here is my code :
import numpy as np
import networkx as nx
import scipy.spatial
points = np.genfromtxt('las1.xyz', delimiter = ';')
xy= points[:,0:2]
z= points[:,2]
delTri = scipy.spatial.Delaunay(xy)
edges = set()
for n in range(delTri.nsimplex):
edge = sorted([delTri.vertices[n,0], delTri.vertices[n,1]])
edges.add((edge[0], edge[1]))
edge = sorted([delTri.vertices[n,0], delTri.vertices[n,2]])
edges.add((edge[0], edge[1]))
edge = sorted([delTri.vertices[n,1], delTri.vertices[n,2]])
edges.add((edge[0], edge[1]))
pts_neigh = {}
graph = nx.Graph(list(edges))
for i in range(len(xy)):
pts_neigh[i] = list(graph.neighbors(i))
I still get the edges list from my networkx graph but it seems like it fails at the level of constructing the nodes.
I will be so grateful for your help.
Although it's possible to instantiate graph with specific data, the syntax can be a bit complex. An easier option is to explicitly add edges from a list:
graph = nx.Graph()
graph.add_edges_from(list(edges))
Related
I have gotten sensor location data from Highway England. I want to add these sensor locations to OSM multidigraph. How to do that?
import numpy as np
import pandas as pd
import networkx as nx
from shapely.geometry import Point, Polygon, LineString
import geopandas as gpd
import osmnx as ox
Graph data is
graph = ox.graph.graph_from_bbox(52.2, 51.85, -.6, -0.9, network_type='drive', simplify=False)
I want to add sensor = Point(-0.6116768, 51.8508765) on the edge nearest to it. Nearest edges to this sensor is n_edge = osmnx.distance.nearest_edges(graph, -0.6116768, 51.8508765, return_dist=False). Now, I need to bend this n_edge such that it passes through the given sensor point.
I found a way to solve this issue by creating a new node in graph, graph.add_node('sensor25', y= 51.8508765, x= -0.6116768, street_count = 2) then graph.add_edges_from([(n_edge[0], 'sensor25'), ('sensor25', n_edge[1)]). However, the node created by me (sensor25) is not identical to other nodes. How to make this node similar to existing nodes?
I have went through following questions
add attribute to node
add new node to existing edge in networkx
add random nodes on edges manually.
I'm not 100% certain what you need, what I understand: You want to add new edges with attributes: speed_limit, length, street number one way, copied from the edge you delete?
I assume that some of these attributes can be copied 1:1, like one way, while others will have to be recalulated. For simplicity, let's assume we have a function d(a, b) that takes (graph) nodes a and b, extracts their position, and calculates the air distance between them. Define other functions as required.
Then you could e.g. define the new edge like this:
# Get from/to id of closest edge
f, t = osmnx.distance.nearest_edges(graph, -0.6116768, 51.8508765, return_dist=False)[0]
c = 'sensor25' # Id of new node, c as in 'center'
edge_attrs = g[f][t] # Copy edge attributes
g.remove_edge(f, t) # Remove edge from graph
graph.add_node(c, y= 51.8508765, x= -0.6116768, street_count = 2)
# Add new edges, recalculating atttributes as required
g.add_edge(f, c, **{**edge_attrs, 'length': d(f, c)})
g.add_edge(c, t, **{**edge_attrs, 'length': d(c, t)})
Hope the syntax is clear, otherwise ask. It copies edge_attrs 1:1, except for attributes you specify after, like lenght. Probably you will have to define multiple functions like d, that also calculate the geometry etc.
The code isn't tested.
Is it possible to generate the original OSMnx graph from the simplified graph (which has the edge geometries preserved)?
For instance:
import osmnx as ox
place = 'Piedmont, California, USA'
G = ox.graph_from_place(place, network_type='drive', simplify=False)
G_simple = ox.simplify_graph(G)
G_simple has the original edge geometries of G stored as "geometry" on the simplified edges:
simple_nodes, simple_edges = ox.graph_to_gdfs(G_simple)
print(simple_edges.iloc[10].geometry)
# LINESTRING (-122.2429303 37.8205234, -122.2426591 37.8207235, -122.2424827 37.820899, -122.2421775 37.8212363, -122.2420372 37.8214758, -122.2420254 37.8215051, -122.2419343 37.8217305, -122.2418551 37.8218894, -122.2415415 37.8222826)
Would it be possible to generate the original graph G from the simplified one? I have many simplified graphs stored on disk, but unfortunately cannot regenerate the unsimplified graphs, so I need to find a way to "unsimplify" them.
This is a one-way destruction of information built into OSMnx. You could try to write your own script, but it would be nontrivial. You'd have to identify each vertex in each geometry of each edge, create a new node there, and break the edge into two at that vertex.
I have a large graph object with many nodes that I am trying to graph. Due to the large number of nodes, many are being drawn one over another. This in itself is not a problem. However, a small percentage of nodes have node attributes which dictate their colour.
Ideally I would be able to draw the graph in such a way that nodes with this property are drawn last, on top of the other nodes, so that it is possible to see their distribution across the graph.
The code I have so far used to generate the graph is shown below:
import networkx as nx
import matplotlib.pyplot as plt
import numpy as np
import os
import pickle
from pathlib import Path
def openFileAtPath(filePath):
print('Opening file at: ' + filePath)
with open(filePath, 'rb') as input:
file = pickle.load(input)
return file
# Pre manipulation path
g = openFileAtPath('../initialGraphs/wordNetadj_dictionary1.11.pkl')
# Post manipulation path
# g = openFileAtPath('../manipulatedGraphs/wordNetadj_dictionary1.11.pkl')
print('Fetching SO scores')
scores = list()
for node in g.nodes:
scores.append(g.node[node]['weight'])
print('Drawing network')
nx.draw(g,
with_labels=False,
cmap=plt.get_cmap('RdBu'),
node_color=scores,
node_size=40,
font_size=8)
plt.show()
And currently the output is as shown:
This graph object itself has taken a relatively long time to generate and is computationally intensive, so ideally I wouldn't have to remake the graph from scratch.
However, I am fairly sure that the graph is drawn in the same order that the nodes were added to the graph object. I have searched for a way of changing the order that the nodes are stored within the graph object, but given directional graphs actually have an order, my searches always end up with answers showing me how to reverse the direction of a graph.
So, is there a way to dictate the order in which nodes are drawn, or alternatively, change the order that nodes are stored inside some graph object.
Potentially worthy of a second question, but the edges are also blocked out by the large number of nodes. Is there a way to draw the edges above the nodes behind them?
Piggybacking off Paul Brodersen's answer, if you want different nodes to be in the foreground and background, I think you should do the following:
For all nodes that belong in the same layer, draw the subgraph corresponding to the nodes, and set the , as follows:
pos = {...} # some dictionary of node positions, required for the function below
H = G.subgraph(nbunch)
collection = nx.draw_networkx_nodes(H, pos)
collection.set_zorder(zorder)
Do this for every group of nodes that belong in the same level. It's tedious, but it will do the trick. Here is a toy example that I created based on looking up this question as part of my own research
import matplotlib as mpl
mpl.use('agg')
import pylab
import networkx as nx
G = nx.Graph()
G.add_path([1, 2, 3, 4])
pos = {1 : (0, 0), 2 : (0.5, 0), 3 : (1, 0), 4 : (1.5, 0)}
for node in G.nodes():
H = G.subgraph([node])
collection = nx.draw_networkx_nodes(H, pos)
collection.set_zorder(node)
pylab.plot([0, 2], [0, 0], zorder=2.5)
pylab.savefig('nodes_zorder.pdf', format='pdf')
pylab.close()
This makes a graph, and then puts the each node at a successively higher level going from left to right, so the leftmost node is farthest in the background and the rightmost node is farthest in the foreground. It then draws a straight line whose zorder is 2. As a result, it comes in front of the two left nodes, and behind the two right nodes. Here is the result.
draw is a wrapper around draw_networkx_nodes and draw_networkx_edges.
Unlike draw, the two functions return their respective artists ( PathCollection and LineCollection, IIRC). These are your standard matplotlib artists, and as as such their relative draw order can be controlled via their zorder attribute.
How can I draw a graph with it's communities using python networkx like this image :
image url
The documentation for networkx.draw_networkx_nodes and networkx.draw_networkx_edges explains how to set the node and edge colors. The patches bounding the communities can be made by finding the positions of the nodes for each community and then drawing a patch (e.g. matplotlib.patches.Circle) that contains all positions (and then some).
The hard bit is the graph layout / setting the node positions.
AFAIK, there is no routine in networkx to achieve the desired graph layout "out of the box". What you want to do is the following:
Position the communities with respect to each other: create a new, weighted graph, where each node corresponds to a community, and the weights correspond to the number of edges between communities. Get a decent layout with your favourite graph layout algorithm (e.g.spring_layout).
Position the nodes within each community: for each community, create a new graph. Find a layout for the subgraph.
Combine node positions in 1) and 3). E.g. scale community positions calculated in 1) by a factor of 10; add those values to the positions of all nodes (as computed in 2)) within that community.
I have been wanting to implement this for a while. I might do it later today or over the weekend.
EDIT:
Voila. Now you just need to draw your favourite patch around (behind) the nodes.
import numpy as np
import matplotlib.pyplot as plt
import networkx as nx
def community_layout(g, partition):
"""
Compute the layout for a modular graph.
Arguments:
----------
g -- networkx.Graph or networkx.DiGraph instance
graph to plot
partition -- dict mapping int node -> int community
graph partitions
Returns:
--------
pos -- dict mapping int node -> (float x, float y)
node positions
"""
pos_communities = _position_communities(g, partition, scale=3.)
pos_nodes = _position_nodes(g, partition, scale=1.)
# combine positions
pos = dict()
for node in g.nodes():
pos[node] = pos_communities[node] + pos_nodes[node]
return pos
def _position_communities(g, partition, **kwargs):
# create a weighted graph, in which each node corresponds to a community,
# and each edge weight to the number of edges between communities
between_community_edges = _find_between_community_edges(g, partition)
communities = set(partition.values())
hypergraph = nx.DiGraph()
hypergraph.add_nodes_from(communities)
for (ci, cj), edges in between_community_edges.items():
hypergraph.add_edge(ci, cj, weight=len(edges))
# find layout for communities
pos_communities = nx.spring_layout(hypergraph, **kwargs)
# set node positions to position of community
pos = dict()
for node, community in partition.items():
pos[node] = pos_communities[community]
return pos
def _find_between_community_edges(g, partition):
edges = dict()
for (ni, nj) in g.edges():
ci = partition[ni]
cj = partition[nj]
if ci != cj:
try:
edges[(ci, cj)] += [(ni, nj)]
except KeyError:
edges[(ci, cj)] = [(ni, nj)]
return edges
def _position_nodes(g, partition, **kwargs):
"""
Positions nodes within communities.
"""
communities = dict()
for node, community in partition.items():
try:
communities[community] += [node]
except KeyError:
communities[community] = [node]
pos = dict()
for ci, nodes in communities.items():
subgraph = g.subgraph(nodes)
pos_subgraph = nx.spring_layout(subgraph, **kwargs)
pos.update(pos_subgraph)
return pos
def test():
# to install networkx 2.0 compatible version of python-louvain use:
# pip install -U git+https://github.com/taynaud/python-louvain.git#networkx2
from community import community_louvain
g = nx.karate_club_graph()
partition = community_louvain.best_partition(g)
pos = community_layout(g, partition)
nx.draw(g, pos, node_color=list(partition.values())); plt.show()
return
Addendum
Although the general idea is sound, my old implementation above has a few issues. Most importantly, the implementation doesn't work very well for unevenly sized communities. Specifically, _position_communities gives each community the same amount of real estate on the canvas. If some of the communities are much larger than others, these communities end up being compressed into the same amount of space as the small communities. Obviously, this does not reflect the structure of the graph very well.
I have written a library for visualizing networks, which is called netgraph. It includes an improved version of the community layout routine outlined above, which also considers the sizes of the communities when arranging them. It is fully compatible with networkx and igraph Graph objects, so it should be easy and fast to make great looking graphs (at least that is the idea).
import matplotlib.pyplot as plt
import networkx as nx
# installation easiest via pip:
# pip install netgraph
from netgraph import Graph
# create a modular graph
partition_sizes = [10, 20, 30, 40]
g = nx.random_partition_graph(partition_sizes, 0.5, 0.1)
# since we created the graph, we know the best partition:
node_to_community = dict()
node = 0
for community_id, size in enumerate(partition_sizes):
for _ in range(size):
node_to_community[node] = community_id
node += 1
# # alternatively, we can infer the best partition using Louvain:
# from community import community_louvain
# node_to_community = community_louvain.best_partition(g)
community_to_color = {
0 : 'tab:blue',
1 : 'tab:orange',
2 : 'tab:green',
3 : 'tab:red',
}
node_color = {node: community_to_color[community_id] for node, community_id in node_to_community.items()}
Graph(g,
node_color=node_color, node_edge_width=0, edge_alpha=0.1,
node_layout='community', node_layout_kwargs=dict(node_to_community=node_to_community),
edge_layout='bundled', edge_layout_kwargs=dict(k=2000),
)
plt.show()
I wanna draw something like this :
The closest thing to this I could find was NetworkX Edge Colormap:
http://networkx.github.io/documentation/latest/examples/drawing/edge_colormap.html
and here is the source code:
#!/usr/bin/env python
"""
Draw a graph with matplotlib, color edges.
You must have matplotlib>=87.7 for this to work.
"""
__author__ = """Aric Hagberg (hagberg#lanl.gov)"""
try:
import matplotlib.pyplot as plt
except:
raise
import networkx as nx
G=nx.star_graph(20)
pos=nx.spring_layout(G)
colors=range(20)
nx.draw(G,pos,node_color='#A0CBE2',edge_color=colors,width=4,edge_cmap=plt.cm.Blues,with_labels=False)
plt.savefig("edge_colormap.png") # save as png
plt.show() # display
After playing around with their source code, I can't figure out how to hardcode distance of the edge circles from the centre. Right now its random.
Also how do I label the edge circles and their distance from the centre?
I know for position comes from pos=nx.spring_layout(G). So I looked at the spring_layout attribute and found that position can be specified by using a pos variable which is a dictionary with nodes as keys and values as a list. (https://networkx.github.io/documentation/latest/reference/generated/networkx.drawing.layout.spring_layout.html)
But even when I do the following result is random edges :
ap = {'uniwide':[55,34,1],'eduram':[34],'uniwide_webauth':[20,55,39],'uniwide_guest':[55,34],'tele9751_lab':[100],'HomeSDN':[100],'TP-LINK':[39]}
pos=nx.spring_layout(G,pos=ap)
You can set the node positions explicitly with the pos dictionary.
For example
import networkx as nx
import matplotlib.pyplot as plt
G = nx.Graph()
G.add_edge('center',1)
G.add_edge('center',2)
G.add_edge('center',3)
G.add_edge('center',4)
pos = {'center':(0,0),
1:(1,0),
2:(0,1),
3:(-1,0),
4:(0,-1)
}
nx.draw(G, pos=pos, with_labels=True)
plt.show()
I'm trying to be as helpful as I can. I wouldn't try to keep them static. You'll want to add and remove things, and the algorithm's automatic placement is something you don't want to lose. According to the docs, you should probably tweak k. It looks like n is 20, so multiply k times some factor to increase the distance.
n = 20
nx.spring_layout(G, k=(1.0/pow(n, .5))) # what it currently is
should maybe be this:
nx.spring_layout(G, k=(1.0/pow(n, .5))*1.5) # play around with this factor