We Keep Coding

mixing placeholders, executemany, and table names

I can iterate through a python object with te following code, however I would like to be able to use placeholders for the schema and table name, normally I do this with {}.{} ad the .format() methods, but how do you combine the two?
cur.executemany("INSERT INTO schema.table_name (x,y,z) "
"values (%s, %s, %s)", top_sample)

Depends on which python you use you can try use f-string
schema = "schema"
table_name = "table_name"
cur.executemany(f"INSERT INTO {schema}.{table_name} (x,y,z) values (%s, %s, %s)", top_sample)
check PEP 498 -- Literal String Interpolation
another option is a simple format
cur.executemany("INSERT INTO {schema}.{table_name} (x,y,z) values (%s, %s, %s)".format(schema=schema, table_name=table_name), top_sample)
but I find the first option shorter and cleaner

I'm not sure what the issue is. You can very well use format like this:
cur.executemany("INSERT INTO {}.{} (x,y,z) values (%s, %s, %s)".format('hello', 'world'), top_sample)

cur.executemany(
"""INSERT INTO schema.{table_name} (x,y,z) values (%s, %s, %s)""".format(table_name=your_table_name),
top_sample
)
place your table name in place of your_table_name

python database insert formatting

I am trying to insert into a postgresql database in python 3.6 and currently am trying to execute this line
cur.execute("INSERT INTO "+table_name+"(price, buy, sell, timestamp) VALUES (%s, %s, %s, %s)",(exchange_rate, buy_rate, sell_rate, date))
but every time it tries to run the table name has ' ' around it so it turns out like INSERT INTO table_name('12', ..., ..., ...) ... instead of
INSERT INTO table_name(12, ..., ..., ...) ... how can I make the string formatter leave the quotes out or remove them or something? It is causing a syntax error around the 12 because it doesn't need the single quotes.

Use it with triple quotes. Also you may pass table_name as a element of second parameter, too.
cur.execute("""INSERT INTO %s (price, buy, sell, timestamp) VALUES (%s, %s, %s, %s)""",(table_name, exchange_rate, buy_rate, sell_rate, date))
More detailed approach;
Triple qoutes give developers a change to write SQL query as multi-lines.
Also it allows you to use single and double qoutes without escaping from them. (It is beneficiary for complex SQL Queries but you don't need that on your case)

Use the new string formatting to have a clean representation. %s is explicitly converting to a string, you don't want that. Format chooses the most fitting type for you.
table_name = "myTable"
exchange_rate = 1
buy_rate = 2
sell_rate = 3
date = 123
x = "INSERT INTO {0} (price, buy, sell, timestamp) VALUES ({1}, {2}, {2}, {4})".format(
table_name, exchange_rate, buy_rate, sell_rate, date)
print x
>INSERT INTO myTable (price, buy, sell, timestamp) VALUES (1, 2, 2, 123)

POSTGIS inserts become slow after some time

I have a location table with following structure:
CREATE TABLE location
(
id BIGINT,
location GEOMETRY,
CONSTRAINT location_pkey PRIMARY KEY (id, location),
CONSTRAINT enforce_dims_geom CHECK (st_ndims(location) = 2),
CONSTRAINT enforce_geotype_geom CHECK (geometrytype(location) = 'POINT'::TEXT OR location IS NULL),
CONSTRAINT enforce_srid_geom CHECK (st_srid(location) = 4326)
)
WITH (
OIDS=FALSE
);
CREATE INDEX location_geom_gist ON location
USING
GIST (location);
I run the following query to insert data:
def insert_location_data(msisdn, lat, lon):
if not (lat and lon):
return
query = "INSERT INTO location (id, location) VALUES ('%s', ST_GeomFromText('POINT(%s %s)', 4326))"%(str(id), str(lat), str(lon))
try:
cur = get_cursor()
cur.execute(query)
conn.commit()
except:
tb = traceback.format_exc()
Logger.get_logger().error("Error while inserting location in sql: %s", str(tb))
return False
return True
I run this block of code 10,000,000 times in a loop but somewhere after 1 million inserts the inserting speed drops drastically. The speed returns to normal when I restart the script but it again drops around a million documents and the same trend continues. I cannot figure out why?
Any help.

Here's a few tips.
Watch out for str(id), which would always return a string '<built-in function id>', since id is not shown to be a variable in the question, and is a built-in id() function.
The correct axis order for PostGIS is (X Y) or (lon lat).
There are more efficient ways to insert points.
Don't format a string to insert
This is how to insert one point:
cur.execute(
"INSERT INTO location (id, location) "
"VALUES (%s, ST_SetSRID(ST_MakePoint(%s, %s), 4326))",
(msisdn, lon, lat))
And see executemany if you want to insert more records at a time, where you would prepare a list of parameters to insert (i.e. [(msisdn, lon, lat), (msisdn, lon, lat), ..., (msisdn, lon, lat)]).

Unrecognized token in SQLite statement

I am inserting data from one table to another, however for some reason I get "unrecognized token". This is the code:
cur.execute("INSERT INTO db.{table} SELECT distinct latitude, longitude, port FROM MessageType1 WHERE latitude>={minlat} AND latitude<={maxlat} AND longitude>= {minlong} AND longitude<= {maxlong}".format(minlat = bottomlat, maxlat = toplat, minlong = bottomlong, maxlong = toplong, table=tablename))
This translates to the following, with values:
INSERT INTO db.Vardo SELECT distinct latitude, longitude, port FROM MessageType1 WHERE latitude>=69.41 AND latitude<=70.948 AND longitude>= 27.72 AND longitude<= 28.416
The error code is the following:
sqlite3.OperationalError: unrecognized token: "70.948 AND"
Is the problem that there is three decimal points?
This is the create statement for the table:
cur.execute("CREATE TABLE {site} (latitude, longitude, port)".format(site = site))

Don't make your SQL queries via string formatting, use the driver's ability to prepare SQL queries and pass parameters into the query - this way you would avoid SQL injections and it would make handling of passing parameters of different types transparent:
query = """
INSERT INTO
db.{table}
SELECT DISTINCT
latitude, longitude, port
FROM
MessageType1
WHERE
latitude >= ? AND
latitude <= ? AND
longitude >= ? AND
longitude <= ?
""".format(table=tablename)
cur.execute(query, (bottomlat, toplat, bottomlong, toplong))

Try using ? for your parameters:
cur.execute("INSERT INTO db.? SELECT distinct latitude, longitude, port FROM MessageType1 WHERE latitude>=? AND latitude<=? AND longitude>= ? AND longitude<= ?",(bottomlat, toplat, bottomlong, toplong, tablename))

MySQL statement in Python: Variables don't work

I have a problem with MySQL and Python's MySQLdb when I try to INSERT more than one variable.
I have a table wordurl with three fields. The first one is an auto_increment ID, second and third should hold the values. Second and third fields are named word_id and url_id.
This is the code.
cursor.execute("INSERT INTO wordurl (word_id, url_id) VALUES (%s, %s)", (word_temp_id, url_temp_id))
When I try to INSERT only one value the code works, two not.
Error message:
(1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '),), (('2',),))' at line 1")
I also tried it with tripple ticks around the statement, the variables with and without bracket, without the field names and with the first id field included. I also tried with C-style-printf-stuff % (which isn't clever!). Nothing worked.
And you, glorious people on stackoverflow, you are my last hope :)
MySQL-Server is 5.5.9 on FreeBSD 8.2. Python version is 2.7:82508 on OSX Lion
Thanks in advance!
Steffen
UPDATE: I use cursor.fetchall() and cursor.fetchone() to get the IDs. Maybe this information is important.

Regarding your update:
AFAIK:
fetchall() returns a tuple with tuples inside. Those tuples inside are what you would get from fetchone(). You can image it like
fetchall() = (fetchone(), fetchtwo(), ...)
And fetchone also returns a tuple which has the actual variable values. This is why you can't simply insert the return value of fetchall().
Just to clarify: the insert statement you pasted, look like this when you fill in the value for max_price:
c.execute("""SELECT spam, eggs, sausage FROM breakfast WHERE price < %s""", (5,))
so the first %s is replaced with the first element of the tuple (5). Your insert statement looks like this:
cursor.execute("INSERT INTO wordurl (word_id, url_id) VALUES (%s, %s)", (((233L,),), ((3L,),)))
I can't make it more clear.

Did you try this:
cursor.execute("INSERT INTO wordurl (word_id, url_id) VALUES ('%s', '%s')" %(word_temp_id, url_temp_id))
It depends on what variables you're trying to insert, but on strings this is needed.
Notice the difference:
>>> word_temp_id = "bla1"
>>> url_temp_id = "bla2"
>>> query = "INSERT INTO wordurl (word_id, url_id) VALUES (%s, %s)" %(word_temp_id, url_temp_id)
>>> print query
INSERT INTO wordurl (word_id, url_id) VALUES (bla1, bla2)
>>> query = "INSERT INTO wordurl (word_id, url_id) VALUES ('%s', '%s')" %(word_temp_id, url_temp_id)
>>> print query
INSERT INTO wordurl (word_id, url_id) VALUES ('bla1', 'bla2')