As mentioned above I need to query several databases with all the same schema using pandas.read_sql_query. I've tried to create loop over all the dbs and run on the fly the sql statement that would result in something like this:
USE db_test_1; SELECT * from test
That's what I've done so far:
cursor = conn.cursor()
cursor.execute("SHOW DATABASES LIKE '%test_%'")
cursor.close()
dbs = [v.replace("'", "") for (v, ) in cursor]
for db in dbs[:100]:
temp = "USE " + db + ";"
fd = open('my_query.sql')
query = fd.read()
fd.close
sql = temp + query
data = pd.read_sql_query(sql, conn)
print(data)
Gives an error saying that the mysql syntax is wrong. Do you have any idea how to handle it or point me to the error?
Many thanks
Your problem lies with your my_query.sql file.
SELECT (SELECT * from tab1), (SELECT * from tab2)
The above is not valid SQL; a subselect can only return a single column. To do this, you would need to join the two subselects in the FROM clause. Which columns you do this on will be entirely dependent on your schema and the needed relation.
Update:
Okay, so the problem here seems to be more about how you're dealing with the query. Your cursor object is connected to a single database. Not the entire database server.
That means that your cursor object cannot use the use keyword here. You need to create a new connection and cursor object for each database you want to connect to.
Related
I'm trying to transfer a user input from a python code to a table in postgresql
What I want to do is place an input() in this code and make it's value go to the comment (#) in the code.
conn = psycopg2.connect(
host="localhost",
database="Twitterzuil",
user="postgres",
password="")
cur = conn.cursor()
cur.execute("INSERT INTO Bericht2 (name) VALUES (#THIS IS WHERE I WANT THE INPUT TO GO)");
conn.commit()
I have no idea how, I'm really a beginner in all this so any help is appreciated
I believe what you are asking about is called string interpolation. Using f-style format, this might look like
new_name = "'bob'" # need single quotes for SQL strings
sql = f"INSERT INTO Bericht2 (name) VALUES ({new_name})" # => sql == "INSERT INTO Bericht2 (name) VALUES ('bob')"
cur.execute(sql)
Note the f at the start of the string, when you do this expressions inside {} pairs get replaced with their python values (tutorial). There are also string formatting approaches involving % substitution and the .format method on strings.
If you are doing anything beyond the basics you should look into using the SQLAlchemy package; here's the link to their insert api. Using SQLAlchemy will help reduce the risks that can come with manually constructing SQL queries.
Example from "Inserting Rows with SQLAlchemy"
from sqlalchemy import insert
stmt = insert(user_table).values(name='spongebob', fullname="Spongebob Squarepants")
with engine.connect() as conn:
result = conn.execute(stmt)
conn.commit()
Now, I have a study about python sqlite3 database. I think it is very simple problem but not allow next step. Could help me?
There is print OK on vscode terminal, but not revised to DB file. I'm searching several times but I can not fix it.
If I execute the code, it not sorting on DB files.
import sqlite3
conn = sqlite3.connect('sqliteDB1.db')
cursor = conn.cursor()
cursor.execute("SELECT * FROM member")
temp123 = cursor. fetchall()
print(temp123)
cursor.execute("SELECT * FROM member ORDER BY -code")
temp321 = cursor.fetchall()
conn.commit
print(temp321)
conn.close()
A select statement just returns data from a database, it will not modify it. Moreover, tables in SQL databases are inherently unordered sets. They have no intrinsic value, and you should never rely on the order of the rows that happens to be returned unless you explicitly sort it with an order by clause.
I'm trying to do something extremely simple that works, but not the way I expect it to. I have a database with various tables and for each of those tables, I'm trying to extract the column names from the information schema. I'm using the code below and everything works like a charm (python):
import psycopg2 as pgsql
# code to connect and generate cursor
table = 'some_table_name'
query = 'SELECT column_name FROM information_schema.columns WHERE table_name = %s'
cursor.execute(query, (table,))
result = pd.DataFrame(cursor.fetchall())
print(result)
So far, so good. The problem arises when I replace the query variable with the following:
import psycopg2 as pgsql
# code to connect and generate cursor
table = 'some_table_name'
**query = 'SELECT column_name FROM information_schema.columns WHERE table_name='+table
cursor.execute(query)**
result = pd.DataFrame(cursor.fetchall())
print(result)
If I print the statement, it's correct:
SELECT column_name FROM information_schema.columns WHERE table_name=some_table_name
However, when I run the query, I'm getting this error message:
UndefinedColumn: column "some_table_name" does not exist
LINE 1: ... FROM information_schema.columns WHERE table_name=some_tabl...
some_table_name is a table name as a parameter to the WHERE clause, not a column name. How is this even possible?
Thanks!
Your problem is that you haven't put some_table_name in quotes so it is treated as a column name, not a string literal. Why not stick with the first method which both worked and is in line with the psycopg documentation?
I'm a postgres newbie and am having some issues querying a text field in postgresql using Python. What is the correct syntax that will allow me to search the content of column "body" from table "jivemessage" out of database "postgres"?
try:
conn = psycopg2.connect("dbname='postgres' user='postgres' host='localhost' password='<password>'")
except:
print "cannot connect"
i = 'test'
cur = conn.cursor()
cur.execute('SELECT * from jivemessage WHERE body LIKE "%'+i+'%"')
Keep getting the following error:
ProgrammingError: column "%test%" does not exist
Thanks for any help.
You are not quoting the query properly. Don't use string concatenation here, use SQL parameters instead:
cur.execute('SELECT * from jivemessage WHERE body LIKE %s', ("%{}%".format(i),))
Here, the %s placeholder signals to the database driver that the first value of the second argument should be placed there when querying.
This leaves the interpolation up to the database driver, giving the database the opportunity to optimize for the query once, even if you were to reuse the same query.
It also prevents SQL injection attacks better than you could yourself, and most of all, guarantees that the correct quoting rules are followed.
So My problem is this, I have a query that uses Mysql User-defined variable like:
#x:=0 SELECT #X:=#X+1 from some_table and this code returns a column from 1-1000.
However, this query doesn't work if I sent it through mySQLdb in Python.
connection =MySQLdb.Connect(host='xxx',user='xxx',passwd='xxx',db = 'xxx')
cursor = connection.cursor
cursor.execute("""SET #X:=0;SELECT #X:=#X+1 FROM some_table""")
rows = cursor.fetchall()
print rows
It prints a empty tuple.
How can I solve this?
Thanks
Try to execute one query at a time:
cursor.execute("SET #X:=0;");
cursor.execute("SELECT #X:=#X+1 FROM some_table");
Try it as two queries.
If you want it to be one query, the examples in the comments to the MySQL User Variables documentation look like this:
SELECT #rownum:=#rownum+1 rownum, t.* FROM (SELECT #rownum:=1) r, mytable t;
or
SELECT if(#a, #a:=#a+1, #a:=1) as rownum
See http://dev.mysql.com/doc/refman/5.1/en/user-variables.html