So, I'm playing around with the Spatialite Virtualknn to try my hand at some routing using open street maps data and Python.
After downloading the Australian osm data, I've extracted a roads layer and created a separate Spatialite database.
spatialite_osm_net.exe -o australia-latest.osm.pbf -d aus_road.sqlite -T roads
I then ran this query from Spatialite_gui
select ST_AsText(ref_geometry)
from knn where f_table_name = 'roads'
and ref_geometry = ST_POINT(145.61249, -38.333801)
And got these 3 results
POINT(145.61249 -38.333801)
POINT(145.61249 -38.333801)
POINT(145.61249 -38.333801)
So I then created this quick and nasty Python script (Python 3.9.0 but I have tried other versions)
import spatialite
file = "./aus_road.sqlite"
db = spatialite.connect(file)
allme = db.execute("select ST_AsText(ref_geometry) from knn where f_table_name = 'roads' and ref_geometry = ST_POINT(145.61249, -38.333801)")
allme.fetchall()
And all I get back is
[]
Anybody have any idea why Virtualknn wont work in Python?
Related
After building and installing the Python engine shipped with Matlab 2019b in Anaconda
(TestEnvironment) PS C:\Program Files\MATLAB\R2019b\extern\engines\python> C:\Users\USER\Anaconda3\envs\TestEnvironment\python.exe .\setup.py build -b C:\Users\USER\MATLAB\build_temp install
for Python 3.7 I wrote a simple script to test a couple of features I'm interested in:
import matlab.engine as ml_e
# Start Matlab engine
eng = ml_e.start_matlab()
# Load MAT file into engine. The result is a dictionary
mat_file = "samples/lena.mat"
lenaMat = eng.load("samples/lena.mat")
print("Variables found in \"" + mat_file + "\"")
for key in lenaMat.keys():
print(key)
# print(lenaMat["lena512"])
# Use the variable from the MAT file to display it as an image
eng.imshow(lenaMat["lena512"], [])
I have a problem with imshow() (or any similar function that displays a figure in the Matlab GUI on the screen) namely that it shows quickly and then disappears, which - I guess - at least confirms that it is possible to use it. The only possibility to keep it on the screen is to add an infinite loop at the end:
while True:
continue
For obvious reasons this is not a good solution. I am not looking for a conversion of Matlab data to NumPy or similar and displaying it using matplotlib or similar third party libraries (I am aware that SciPy can load MAT files for example). The reason is simple - I would like to use Matlab (including loading whole environments) and for debugging purposes I'd like to be able to show this and that result without having to go through loops and hoops of converting the data manually.
I'm trying to use cartopy to plot several maps and I want to use them offline. Cartopy has a data directory,
import cartopy.config
cartopy.config
{'data_dir': '/home/user/.local/share/cartopy',
'downloaders': {('shapefiles',
'gshhs'): <cartopy.io.shapereader.GSHHSShpDownloader at 0x7f3ee33ee7d0>,
('shapefiles',
'natural_earth'): <cartopy.io.shapereader.NEShpDownloader at 0x7f3ee33ee710>},
'pre_existing_data_dir': '',
'repo_data_dir': '/home/user/bin/virtualenvs/mobi/local/lib/python2.7/site-packages/cartopy/data'}
So I believe that i can download the maps from Natural Earth site. How can I structure this data on this directory so cartopy would not use the internet to plot? And how can I do the same for OpenStreetMap data?
(Partial answer only)
At Natural Earth web site, http://www.naturalearthdata.com/downloads/, you can find all the downloadable files.
For example, this link provides low resolution data: http://www.naturalearthdata.com/downloads/110m-physical-vectors/
One of the data files on that page has this link address:
http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/physical/ne_110m_coastline.zip
This piece of code will download that file (if it is not readily available on the computer):-
import cartopy
fname = cartopy.io.shapereader.natural_earth( \
resolution='110m', \
category='physical', \
name='110m_coastline')
fname is full pathname of the downloaded file.
You dont need to arrange the download location for cartopy. It already has default location that you can find by:
cartopy.config['data_dir'] # usually 'C:\\Users\\username\\.local\\share\\cartopy'
You can check out the files you downloaded and see how they are structured in that location.
Next time when you use cartopy function cartopy.io.shapereader.natural_earth (with default config) it will use the local files if they are available.
I had Faced similar issue wherein, with cartopy, the plt.gca().coastlines() was triggering the download of a zip file from external server, but the download was failing as internet connectivity was absent.
/home/apps/CARTOPY/0.16.0/lib64/python2.7/site-packages/Cartopy-0.16.0-py2.7-linux-x86_64.egg/cartopy/io/__init__.py:260: DownloadWarning: Downloading: http://naciscdn.org/naturalearth/110m/physical/ne_110m_coastline.zip
warnings.warn('Downloading: {}'.format(url), DownloadWarning)
I manually downloaded the zip file , and extracted under - ~/.local/share/cartopy/shapefiles/natural_earth/physical.
~/.local/share/cartopy/shapefiles/natural_earth/physical> ls
ne_110m_coastline.README.html ne_110m_coastline.cpg ne_110m_coastline.prj ne_110m_coastline.shx
ne_110m_coastline.VERSION.txt ne_110m_coastline.dbf ne_110m_coastline.shp
then after renaming/removing "ne_" prefix from some files, i was able to solve this issue.
~/PLOT_TEST> ls ~/.local/share/cartopy/shapefiles/natural_earth/physical/
110m_coastline.cpg 110m_coastline.dbf 110m_coastline.prj 110m_coastline.shp 110m_coastline.shx ne_110m_coastline.README.html ne_110m_coastline.VERSION.txt
I have prepared a code in where you can download the the shapefiles from natural earth and then convert them into a dataframe. Pay attention, the country coordinates in natural earth are in polygon and multi-polygon format. In the case of dealing with Rivers which are linestring you need to modify the code.
You might need to manipulate the "name" with your desired filename like "coastlines". Find more information in the following link:
https://www.naturalearthdata.com/downloads/
import cartopy.io.shapereader as shpreader
ne_earth_countries = shpreader.natural_earth(resolution = '10m',
category = 'cultural',
name='admin_0_countries')
countries = shpreader.Reader(ne_earth_countries).records()
def extract_geom_meta(country):
coords = np.empty(shape=[0,2])
for geom in country.geometry:
coords = np.append(coords, geom.exterior.coords, axis=0)
country_name = country.attributes["ADMIN"]
return [country_name, coords]
WorldDF = pd.DataFrame([extract_geom_meta(country) for country in countries],
columns=['countries','coordinates'])
CountryDF = pd.concat([pd.DataFrame(WorldDF['coordinates'][country_idx])
for country_idx in range(len(WorldDF))]).reset_index()
CountryDF['Label'] = CountryDF.apply(lambda row: 1 if row['index'] == 0
else 0,axis=1).cumsum()-1
CountryDF['Country'] = CountryDF['Label'].apply(lambda row: WorldDF.countries[row])
CountryDF.rename(columns={0:'Longitude', 1:'Latitude'},inplace=True)
print(CountryDF.head())
We are creating openstack Images nightly with Packer. We want to keep the last 5 that we've created and automatically delete the rest. (Ideally delete them only if they have no instance running that is based on them, but indiscriminately is fine if that is not possible).
I have been looking through the documentation of the various Openstack APIs and I can't seem to find a neat way to do this.
What I was thinking of was running nova image-list and parsing that with bash, but this seems a bit finicky and a potentially dangerous way to do things when we're talking about auto-deletion.
The bash script would look something like this :
cross reference this:
nova image-list | grep "CentOS-7-x86_64 " | cut -d'|' -f 2
with this:
nova list | grep "ACTIVE" | cut -d'|' -f 2 | while read -r line ; do nova show "$line" ; done
But already this is stupidly unwieldy and it doesn't even do anything yet....
My question is, is there a better way to do this that I've missed?
Thanks.
I would solve this using the Python API rather than trying to solve this using the CLI tools. Here is some code that will get you the necessary keystone, nova, and glance clients:
#!/usr/bin/python
import os
import keystoneclient.auth.identity as keystone_identity
import keystoneclient.session as keystone_session
import keystoneclient.client as keystone_client
import novaclient.client as nova_client
import glanceclient.client as glance_client
auth = keystone_identity.v2.Password(auth_url=os.environ['OS_AUTH_URL'],
username=os.environ['OS_USERNAME'],
password=os.environ['OS_PASSWORD'],
tenant_name=os.environ['OS_TENANT_NAME'])
# establish a keystone session
sess = keystone_session.Session(auth=auth)
# get a keystone client
kc = keystone_client.Client('2',
session=sess,
auth_url=sess.auth.auth_url)
# and authenticate it
kc.authenticate(token=sess.get_token(),
project_id=sess.get_project_id(),
tenant_id=sess.get_project_id())
# get a nova client
nc = nova_client.Client('2', session=sess)
# get a glance client
gc = glance_client.Client('2',
endpoint=kc.service_catalog.url_for(
service_type='image',
endpoint_type='publicURL'),
token=sess.get_token())
Assuming that you have that available, you would first get a list of available images matching your criteria:
images = [img for img in gc.images.list()
if 'CentOS-7-x86_64 'in img.name]
And then get the list of images you want to delete:
to_delete = images[5:]
Then get a list of images that are in use:
in_use = []
for server in nc.servers.list():
in_use.append(server.image['id'])
And then delete everything that's not in use:
for img in to_delete:
if img['id'] not in in_use:
gc.images.delete(img['id'])
I am running a query to select a polygon from a set of polygons. Then I input that polygon into a feature dataset in a geodatabase. I then use this polygon(or set of polygons) to dissolve to get the boundary of the polygons and the centroid of the polygon(s), also entered into separate feature datasets in a geodatabase.
import arcpy, os
#Specify the drive you have stored the NCT_GIS foler on
drive = arcpy.GetParameterAsText(0)
arcpy.env.workspace = (drive + ":\\NCT_GIS\\DATA\\RF_Properties.gdb")
arcpy.env.overwriteOutput = True
lot_DP = arcpy.GetParameterAsText(1).split(';')
PropertyName = arcpy.GetParameterAsText(2)
queryList= []
for i in range(0,len(lot_DP)):
if i % 2 == 0:
lt = lot_DP[i]
DP = lot_DP[i+1]
query_line = """( "LOTNUMBER" = '{0}' AND "PLANNUMBER" = {1} )""".format(lt, DP)
queryList.append(query_line)
if i < (len(lot_DP)):
queryList.append(" OR ")
del queryList[len(queryList)-1]
query = ''.join(queryList)
#Feature dataset for lot file
RF_Prop = drive + ":\\NCT_GIS\\DATA\\RF_Properties.gdb\\Lots\\"
#Feature dataset for the property boundary
RF_Bound = drive + ":\\NCT_GIS\\DATA\\RF_Properties.gdb\\Boundary\\"
#Feature dataset for the property centroid
RF_Centroid = drive + ":\\NCT_GIS\\DATA\\RF_Properties.gdb\\Centroid\\"
lotFile = drive + ":\\NCT_GIS\\DATA\\NSWData.gdb\\Admin\\cadastre"
arcpy.MakeFeatureLayer_management(lotFile, "lot_lyr")
arcpy.SelectLayerByAttribute_management("lot_lyr", "NEW_SELECTION", query)
#Create lot polygons in feature dataset
arcpy.CopyFeatures_management("lot_lyr", RF_Prop + PropertyName)
#Create property boundary in feature dataset
arcpy.
arcpy.Dissolve_management(RF_Prop + PropertyName , RF_Bound + PropertyName, "", "", "SINGLE_PART", "DISSOLVE_LINES")
#Create property centroid in feature dataset
arcpy.FeatureToPoint_management(RF_Bound + PropertyName, RF_Centroid + PropertyName, "CENTROID")
Every time I run this I get an error when trying to add anything to the geodatabase EXCEPT when copying the lot layer into the geodatabase.
I have tried not copying the lots into the geodatabase and copying it into a shapefile and then using that but still it the boundary and centroid will not import into the geodatabase. I tried outputing the boundaries into shapefiles then using the FeatureClassToGeodatabase tool but still I get error after error.
If anyone can shed light on this It would be grateful
In my experience, I found out that if I had recently opened then closed ArcMap or ArcCatalog it would leave two ArcGIS services running (check task manager) even though I had closed ArcMap and ArcCatalog. If I tried running a script while these two services were running I would get this error. Finding these services in Windows task manager and ending them fixed this error for me. The two services were
ArcGIS cache manager
ArcGIS online services
I've also heard that your computer's security/anti-virus software may possibly interfere with scripts running. So adding your working directory as an exception to your security software may also help.
If in the rare occasions that this didn't work I just had to restart the computer.
I have 5 PDF files, each of which have links to different pages in another PDF file. The files are each tables of contents for large PDFs (~1000 pages each), making manual extraction possible, but very painful. So far I have tried to open the file in Acrobat Pro, and I can right click on each link and see what page it points to, but I need to extract all the links in some manner. I am not opposed to having to do a good amount of further parsing of the links, but I can't seem to pull them out by any means. I tried to export the PDF from Acrobat Pro as HTML or Word, but both methods didn't maintain the links.
I'm at my wits end, and any help would be great. I'm comfortable working with Python, or a range of other languages
Looking for URIs using pyPdf,
import pyPdf
f = open('TMR-Issue6.pdf','rb')
pdf = pyPdf.PdfFileReader(f)
pgs = pdf.getNumPages()
key = '/Annots'
uri = '/URI'
ank = '/A'
for pg in range(pgs):
p = pdf.getPage(pg)
o = p.getObject()
if o.has_key(key):
ann = o[key]
for a in ann:
u = a.getObject()
if u[ank].has_key(uri):
print u[ank][uri]
gives,
http://www.augustsson.net/Darcs/Djinn/
http://plato.stanford.edu/entries/logic-intuitionistic/
http://citeseer.ist.psu.edu/ishihara98note.html
etc...
I couldn't find a file that had links to another pdf, but I suspect that the URI field should contain URIs of the form file:///myfiles
I've just made a small Python tool for exactly this, to list/download all referenced PDFs from a given PDF: https://www.metachris.com/pdfx/ (also: https://github.com/metachris/pdfx)
$ ./pdfx.py https://weakdh.org/imperfect-forward-secrecy.pdf -d ./
Reading url 'https://weakdh.org/imperfect-forward-secrecy.pdf'...
Saved pdf as './imperfect-forward-secrecy.pdf'
Document infos:
- CreationDate = D:20150821110623-04'00'
- Creator = LaTeX with hyperref package
- ModDate = D:20150821110805-04'00'
- PTEX.Fullbanner = This is pdfTeX, Version 3.1415926-2.5-1.40.14 (TeX Live 2013/Debian) kpathsea version 6.1.1
- Producer = pdfTeX-1.40.14
- Title = Imperfect Forward Secrecy: How Diffie-Hellman Fails in Practice
- Trapped = False
- Pages = 13
Analyzing text...
- URLs: 49
- URLs to PDFs: 17
JSON summary saved as './imperfect-forward-secrecy.pdf.infos.json'
Downloading 17 referenced pdfs...
Created directory './imperfect-forward-secrecy.pdf-referenced-pdfs'
Downloaded 'http://cr.yp.to/factorization/smoothparts-20040510.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/smoothparts-20040510.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35517.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35517.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35514.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35514.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35519.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35519.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35522.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35522.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35509.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35509.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35528.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35528.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35513.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35513.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35533.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35533.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35551.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35551.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35527.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35527.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35520.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35520.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35526.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35526.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35515.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35515.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35529.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35529.pdf'...
Downloaded 'http://cryptome.org/2013/08/spy-budget-fy13.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/spy-budget-fy13.pdf'...
Downloaded 'http://www.spiegel.de/media/media-35671.pdf' to './imperfect-forward-secrecy.pdf-referenced-pdfs/media-35671.pdf'...
The tool uses PyPDF2, the de-facto Python standard library to read PDF content, a regular expression to match all urls, and starts a download thread for each PDF if you run it with the -d option (for --download-pdfs).
In case you cannot use Python, but have a method of decompressing the PDF's internal object streams like qpdf, you can grep for URIs:
qpdf --qdf --object-streams=disable input.pdf - | grep -Poa '(?<=/URI \().*(?=\))'