I am importing Polygon shapes into a PostGIS database, using Python (GeoPandas, SQLAlchemy, GeoAlchemy2). I followed the instructions mentioned here.
I have a database with a table named maps_region with a column/field called geom.
I am able to get the Polygon field (named geom) to import into the PostGIS database table in text format (WKT, WKB, and WKB Hex), but, I am unable to successfully convert this text column into a proper Polygon format in the database.
I tried importing with the geom field in several different formats-- in Well-Known Text (WKT) format, WKB format, and WKB Hex format-- but could not convert to Polygon from any of the three formats.
For instance, I imported the shapes into the geom field as WKT format, and then converted to WKB Hex format, using the following command, which worked fine:
database=> UPDATE maps_region SET geom = ST_GeomFromText(geom, 4326);
UPDATE 28
However, when I then try to convert the geom field from a text format into a Polygon type, I get the following errors:
database=> ALTER TABLE maps_region ALTER COLUMN geom TYPE Geometry(POLYGON, 4326);
ERROR: Geometry type (MultiPolygon) does not match column type (Polygon)
database=> ALTER TABLE maps_region ALTER COLUMN geom TYPE Geometry(MULTIPOLYGON, 4326);
ERROR: Geometry type (Polygon) does not match column type (MultiPolygon)
I tried both ways: converting to Polygon, and converting to MultiPolygon-- and neither worked. Instead, the error messages were just reversed!
Any help would be greatly, greatly appreciated.
Thanks in advance!
I realized that the shapes were being registered in mixed format: all but one were in Polygon format, while one was in MultiPolygon format -- see here. Looks like this sufficiently explains the issue/invalid conversion.
Related
I have a FITS file with a BinTableHDU that has many entries that have been converted from digital numbers various units like Volts and Currents. I would like to turn off this conversion and access the original digital number value that was stored in the table.
This table makes use of TTYPE, TSCAL, TZERO, and TUNIT keys in the header to accomplish the conversions. So I could use these header keys to undo the conversion manually.
undo = (converted - tzero ) / tscal
Since the astropy.table.Table and astropy.table.QTable class automatically interpret these fields I can't help but think I'm overlooking a way to use astropy functions to undo the conversion. Is there an attribute or function in the astropy QTable or Table class that could let me undo the conversion automatically. (or perhaps another easier way) to undo this conversion?
EDIT: Thanks to the answer by Tom Aldcroft below.
In my situation a FITS binary table can be grabbed and put into a pandas.DataFrame either converted or unconverted using the following code:
import pandas as pd
from astropy.table import Table
from astropy.io import fits
# grab converted data in various scaled values and units volts / current
converted = Table(fits.getdata(file, 2)).to_pandas()
# grab unconverted data in its original raw form
unconverted = Table(fits.getdata(file, 2)._get_raw_data()).to_pandas()
You'll need to be using the direct astropy.io.fits interface instead of the high-level Table interface. From there something like this might work:
from astropy.io import fits
with fits.open('data.fits') as hdus:
hdu1 = hdus[1].data
raw = hdu1._get_raw_data()
See https://github.com/astropy/astropy/blob/645a6f96b86238ee28a7057a4a82adae14885414/astropy/io/fits/fitsrec.py#L1022
I would like to convert an Oracle DB frame including a 'SHAPE' column that hold the 'oracle.sql.STRUCT#' information into something more accessible, either a geojson/shapefile/dataframe either using Python/R or SQL.
Any ideas?
Create your frame with a query using one of the SDO_UTIL functions to convert the shape (sdo_geometry type) to a type easily consumed by Python/R, i.e. wkb, wkt, geojson. For example SDO_UTIL.TO_WKTGEOMETRY(shape). See info on the conversion functions here; https://docs.oracle.com/en/database/oracle/oracle-database/19/spatl/SDO_UTIL-reference.html
I'm reading out an Oracle DB with geospatial geometries which I save in a pandas dataframe, say df having a geometric object of format <cx_Oracle.Object MDSYS.SDO_GEOMETRY at 0x7f28 in a column named 'geometry'. Let's store it as:
g = df.geometry[0]
What I want to do:
Transform the data stored g to present it on a folium map as a PolyLine from shapely. I know that it consists of a bunch of points representing a line object.
What I can do:
I can read out the SDO_GTYPE, i.e. g.SDO_GTYPE gives 2002.
I can read out SDO_ORDINATES, but it won't show me the coordinates, saying: <cx_Oracle.Object MDSYS.SDO_ORDINATE_ARRAY at 0x7f287848e4f0>.
What I cannot do:
Transform geometric information with shapely and asShape:
from shapely.geometry import asShape
shape = asShape(g)
gives error: 'Context does not provide geo interface'.
Use Get_WKT() or any other functions in SQL-Statements
There are colleagues reading the data with a GIS-tool, i.e. why I doubt that the data is corrupt. I would be happy about any suggestions regarding this issue.
Thanks a lot.
Don't have any experience with Oracle DB, but this SO question seems similar to yours.
This sample may be of help to you.
I am attempting to insert a dataframe into my postgres database using the pscycopg2 module used with sqlalchemy. The process is loading an excel file into a pandas dataframe and then inserting the dataframe into the database via the predefined table schema.
I believe these are the relevant lines of code:
post_meta.reflect(schema="users")
df = pd.read_excel(path)
table = sql.Table(table_name, post_meta, schema="users")
dict_items = df.to_dict(orient='records')
connection.execute(table.insert().values(dict_items))
I'm getting the following error:
<class 'sqlalchemy.exc.ProgrammingError'>, ProgrammingError("(psycopg2.ProgrammingError) can't adapt type 'numpy.int64'",)
All data field types in the dataframe are int64.
I can't seem to find a similar question or information regarding why this error is and what it means.
Any direction would be great.
Thanks
Looks like you're trying to insert numpy integers, and psycopg2 doesn't know how to handle those objects. You need to convert them to normal python integers first. Maybe try calling the int() function on each value... Please provide more context with code if that fails.
I also ran into this error, and then realized that I was trying to insert integer data into a SqlAlchemy Numeric column, which maps to float, not int. Changing the offending DataFrame column to float did the trick for me:
df[col] = df[col].astype(float)
Perhaps you are also trying to insert integer data into a non-integer column?
I found this regarding Point type in Postgres: http://www.postgresql.org/docs/current/interactive/datatype-geometric.html
Is there the SQLAlchemy version of this?
I am storing values in this manner: (40.721959482, -73.878993913)
You can use geoalchemy2 whis is an extension to sqlalchemy and can be used with flask-sqlalchemy too.
from sqlalchemy import Column
from geoalchemy2 import Geometry
# and import others
class Shop(db.Model):
# other fields
coordinates = Column(Geometry('POINT'))
You can extend UserDefinedType to achieve what you want.
Here's an example I found that gets pretty close to what you want subclassing UserDefinedType
Note that Mohammad Amin's answer is valid only if your point is intended to be a geographic point (latitude and longitude constraints). It doesn't apply if you want to represent any point on a plane. Also, in that case you would need to install the PostGIS extension, which I encourage if you are working with geography points as it provides a lot of utlities and extra functions.
I found out it's a slight modification of Mohammad's answer.
Yes I need to add a point column via geo/sqlalchemy
from sqlalchemy import Column
from geoalchemy2 import Geometry
# and import others
class Shop(db.Model):
# other fields
coordinates = Column(Geometry('POINT'))
On my PostGIS/Gres and PGloader side, since I'm loading from a csv that formats my latitude and longitude points as: "(40.721959482, -73.878993913)" I needed to do a macro in vim for all of my entries (there's a lot) in order to force that column to adhere to how points are created in PostGIS, so I turned the csv column into point(40.721959482 -73.878993913), then upon table creation set the location column with the datatype as geometry(point) location geometry(point).