IF statements with psycopg2 and PostGIS in python - python

I am trying to run a SQL query using psycopg2 in python. The query is supposed to create a set of points (in the {nome1} identifier) for each polygon (in the {nome2} identifier, which was created from the segmentos.json I'm still using to run the for loop) they intersect and save the intersection to a new table (the {nome0} identifier). This bit works fine.
The problem is I want the new {nome0} table to remain droped in case there are no points that intersect the polygon I'm iterating. When I run this code, I get the following error:
SyntaxError: syntax error at or near "IF"
Should I be using a function to run IF statements in psycopg2? What am I missing here?
Thanks in advance!
intersecta="""DROP TABLE IF EXISTS {nome0};
IF ST_Intersection({nome2}.geom,{nome1}.geom)=true THEN
SELECT {nome1}.*, {nome2}.nome INTO {nome0}
FROM {nome2} INNER JOIN {nome1} ON ST_Intersects({nome2}.geom, {nome1}.geom) AND {nome2}.nome=%s;
END IF;
"""
for feature in segmentos.json()['features']:
nomedosegmento=feature['properties']['nome']
nomecompleto=str(feature['properties']['nome']+'_'+tabelagerada)
cur.execute(sql.SQL(intersecta)
.format(nome0=sql.Identifier(nomecompleto),nome1=sql.Identifier(tabelagerada),nome2=sql.Identifier(segmentnome)),[nomedosegmento,])
#print(cur.query)
conn.commit()

Related

Entering data into specific field using "WHERE" in "INSERT" command in SQL? (Python)

I'm currently writing a program for a parents evening system. I have two tables, a bookings table and a teacher table - set up with the following column headings: TeacherSubject | 15:30 | 15:35 | 15:40 etc... When people make a booking, they select a teacher from a drop-down menu and also a time. Therefore, I need the bookingID added into the booking table where the teacher selected = to the same teacher in the table and where time selected = time in the database.
At the moment, my code only attempts to match the teacher, but this doesn't work as I'm getting the error of: (line 5)
TypeError: 'str' object is not callable
Am I doing the whole thing wrong and is this actually possible with the way I have set the table up?
def insert(parent_name, parent_email, student_name,student_form,teacher,app_time,comments):
conn=sqlite3.connect("parentsevening.db")
cur=conn.cursor()
cur.execute("INSERT INTO bookings VALUES (NULL,?,?,?,?,?,?,?)",(parent_name,parent_email,student_name,student_form,teacher,app_time,comments))
cur.execute("INSERT INTO teachers VALUES (?) WHERE teachers = (?)" (id,teacherName,))
conn.commit()
conn.close()
This SQL Query is invalid.
INSERT INTO teachers VALUES (?) WHERE teachers = (?)
It should be
INSERT INTO teachers (id, name) VALUES(?, ?)
Note that I'm guessing the teachers columns (id, name) WHERE on the insert isn't valid because it's used to find data (SELECT, UPDATE, DELETE)
OK, let's take out the comments and make this into an answer.
Python error
I think your error comes from WHERE teachers = (?) have you tried WHERE teachers = ? instead.
But...
bad sql syntax
Also that command as a whole doesnt make much sense, SQL syntax wise - you seem to be trying to insert where a teacher that doesn't exist (if you are inserting them) and values on an insert does not go with where and where needs a from. i.e. once you've solved your python error, sqlite is going to have a fit as well.
That's already covered by another answer.
But...
probably not what you should be doing
If you have an existing teacher, you only need to insert their teacherid into table bookings. You don't have to, and in fact, you can't insert into table teachers at this point, you'd get a duplicate data error.
So, rather than fixing your second query, just get rid of it entirely.
If you can get a command line or GUI SQL tool up, try running these queries by hardcoding them by hand before coding them in Python. the sqlite command should be able to do that for you.
(recommendation) don't use insert table values
Try being explicit with insert into table (<column list>) values .... The reason is that, as soon as the table changes in some way that affects column order (possibly an alter column) the values won't line up with the implied insert list. hard to debug, hard to know what was intended at time of writing. Been there, done that. And had to debug buncha folks' code who took this shortcut, it's never fun

Dynamic SQL query Psycopg2 values problem

Using Python and psycopg2 I am trying to build a dynamic SQL query to insert rows into tables.
The variables are:
1. Table name
2. Variable list of column names
3. Variable list of values, ideally entering multiple rows in one statement
The problems I have come across are the treatment of string literals from Python to SQL and psycopg2 trying to avoid you exposing your code to SQL injection attacks.
Using the SQL module from psycopg2, I have resolved dynamically adding the Table name and List of columns. However I am really struggling with adding the VALUES. Firstly the values are put into the query as %(val)s and seem to be passed literally like this to the database, causing an error.
Secondly, I would then like to be able to add multiple rows at once.
Code below. All help much appreciated :)
import psycopg2 as pg2
from psycopg2 import sql
conn = pg2.connect(database='my_dbo',user='***',password='***')
cols = ['Col1','Col2','Col3']
vals = ['val1','val2','val3']
#Build query
q2 = sql.SQL("insert into my_table ({}) values ({})") \
.format(sql.SQL(',').join(map(sql.Identifier, cols)), \
sql.SQL(',').join(map(sql.Placeholder,vals)))
When I print this string as print(q2.as_string(conn)) I get:
insert into my_table ("Col1","Col2","Col3") values %(val1)s,%(val2)s,%(val3)s
And then when i try and a execute such a string I get the following error:
ProgrammingError: syntax error at or near "%"
LINE 1: ... ("Col1","Col2","Col3") values (%(val1)s...
^
Ok I solved this. Firstly use Literal rather than Placeholder, secondly put your row values together as tuples within a tuple, loop through adding each tuple to a list as literals and then drop in at the end when building the query.

psycopg2 each execute yields different table and cursor not scrolling to the beginning

I am querying a postgresql database through python's psycopg2 package.
In short: The problem is psycopg2.fetchmany() yields a different table everytime I run a psydopg2.cursor.execute() command.
import psycopg2 as ps
conn = ps.connect(database='database', user='user')
nlines = 1000
tab = "some_table"
statement= """ SELECT * FROM """ + tab + " LIMIT %d;" %(nlines)
crs.execute(statement)
then I fetch the data in pieces. Running the following executes just fine and each time when I scroll back to the beginning I get the same results.
rec=crs.fetchmany(10)
crs.scroll(0, mode='absolute')
print rec[-1][-2]
However, if I run the crs.execute(statement) again and then fetch the data, it yields a completely different output. I tried running ps.connect again, do conn.rollback(), conn.reset(), crs.close() and nothing ever resulted in consisted output from the table. I also tried a named cursor with scrollable enabled
crs= conn.cursor(name="cur1")
crs.scrollable=1
...
crs.scroll(0, mode= 'absolute')
still no luck.
You don't have any ORDER BY clause in your query, and Postgres does not guarantee any particular ordering without one. It's particularly likely to change ordering for tables which have lots of churn (i.e. lots of inserts/updates/deletes).
See the Postgres SELECT doc for more details, but the most salient snippet here is this:
If the ORDER BY clause is specified, the returned rows are sorted in
the specified order. If ORDER BY is not given, the rows are returned
in whatever order the system finds fastest to produce.
I wouldn't expect any query, regardless of the type of cursor used, to necessarily return the exact same result set given a query of this type.
What happens when you add an explicit ORDER BY?

Sql query causes python to crash

The following query causes python to crash ('python.exe has encountered a problem ...'
Process terminated with an exit code of -1073741819
The query is:
create temp table if not exists MM_lookup2 as
select lower(Album) || lower(SongTitle) as concat, ID
from MM.songs
where artist like '%;%' collate nocase
If I change from "like" to = it runs as expected, eg
create temp table if not exists MM_lookup2 as
select lower(Album) || lower(SongTitle) as concat, ID
from MM.songs
where artist = '%;%' collate nocase
I am running python v2.7.2, with whatever version of sqlite that ships in there.
The problem query runs without problem outside python.
You didn't write the database system/driver you are using. I suspect that your SQL is the problem. The % characters needs to be escaped. Possibly the db driver module tries to interpret %, and %) as format chars, and it cannot convert non-existing parameter value into a format that is acceptable by the database backend.
Can you please give us concrete Python code? Can you please try to run the same query but putting the value of '%,%' into a parameter, and pass it to the cursor as a parameter?

OperationalError creating an index in sqlite

EDIT: TL;DR version
I typed this
CREATE INDEX IF NOT EXISTS IDX_FILE_SIZE table_name (file_size);
instead of this
CREATE INDEX IF NOT EXISTS IDX_FILE_SIZE ON table_name (file_size);
Don't do that.
Some silly questions:
Is it a concidence that the offending statement is missing the word ON?
CREATE INDEX IF NOT EXISTS IDX_FILE_FULLPATH_FILE_PARENT_DIR ON table_name (file_fullpath, file_parent_dir);
CREATE INDEX IF NOT EXISTS IDX_FILE_SIZE table_name (file_size); -- missing ON
CREATE INDEX IF NOT EXISTS IDX_TAG_TITLE ON table_name (tag_title);
Somewhere in all the verbiage in your question, did I see the phrase "syntax error"?
Did you try the simple step of running the SQL statements in the sqlite3 command interpreter and seeing which syntax error you were actually getting?
E.g.
SQLite version 3.6.14
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> create table foo (bar int, zot int);
sqlite> create index barx on foo(bar);
sqlite> create index zotx foo(zot);
SQL error: near "foo": syntax error
sqlite>
Have you considered perusing TFRRD (The Fantastic Rail-Road Diagram) in the docs?
You wrote: """when I run that command in the smaller script (verifyIndexSmaller), it gives no error. If I then try to run the larger script again, even though the index has been created by the smaller script, I still get the error""".
Have you considered the possibility that you didn't run that command in the smaller script, but actually ran another (fixed!) version of that command?
Do you now understand why S.Lott was trying to get you to cut the waffle and focus on the piece of SQL that was causing the error?

Categories