I have a series of identical mysql tables in a list of databases as follows
A_1.table
A_2.table
A_3.table
the tables are accesssed by means of a series of ip addresses as follows
IP1, IP2, IP3.
The mysql database table names are stored as tablelist and the ip addresses are stored in list called hostlist
These tables are stored in mysql database with the following credentials
username=x
password=pw
port=1234
I have created the following code
def proc(db, hostname):
con = mysql.connector.connect(user=x, password=pw,
host=hostname,
database=db, port=1234)
db_cursor = con.cursor()
db_cursor.execute('SELECT * FROM table')
table_rows = db_cursor.fetchall()
df = pd.DataFrame(table_rows)
print(df.head(1))
con.close()
Next I have applied the above code to both lists as follows.
a=map(proc, tablelist, hostlist)
The code seems to work as there are no errors. How do I test if the map command works. In R We get True as an output to let us know if map has worked
Related
I want a Python code that reads a database using pyodbc, and then reads the curser into a list of dictionaries that I want to iterate and find the entry with the key I want.
for example I have this db with this header: ['id', 'user_name', 'user_lname', 'user_username'].
What I want is to print out the entry in the db that has 'user_username'='JOHNN'
This is how the list looks like :
[{'id': 105, 'user_name': 'John', 'user_lname': 'Smith', 'user_username': 'JOHNN'}]
Below is the python code:
from flask import Flask
import pyodbc
app = Flask(__name__)
conn = pyodbc.connect('Driver={SQL Server};'
'Server=DESKTOP-9JKA425;'
'Database=PD;'
'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('SELECT * FROM Users')
columns = [column[0] for column in cursor.description]
results = []
for row in cursor.fetchall():
results.append(dict(zip(columns, row)))
# for x in results:
print(results)
Please advise
When you want to query a database with specific criteria, it is better to do this by narrowing your SQL query, rather than querying for * and then filtering in Python.
For example:
query = "SELECT * FROM Users WHERE username = 'JOHNN'"
cursor.execute(query)
Using SQL to narrow down your result set will also reduce the amount of data transported over the network and in many cases, SQL databases will do the necessary conditionals faster than Python.
See https://github.com/mkleehammer/pyodbc/wiki/Getting-started#parameters for how to add dynamic parameters safely for SQL queries when using pyodbc, if you simply use parameters directly, you run the risk of introducing SQL injections into your code.
Written safely, the above becomes:
query = "SELECT * FROM Users WHERE username = ?"
cursor.execute(query,'JOHNN')
I have parameterized queries with f strings such that the queries will select some data from a series of tables and joins, and I want to insert the resulting set of data into another pre-created table (tables been designed to house these results).
Python executes the code but the query results never show up in my table.
Assuming target_table is already created in singlestore database:
qry_load = 'insert into target_table select * from some_tables'
conn = engine.connect()
trans = conn.begin()
try:
conn.execute(qry_load)
trans.commit()
except:
trans.rollback()
raise
The code executes and acts as if all is ok, but the data never shows up in the target table.
How do I see what singlestore is passing back to better debug what is happening within the database?
Just replace begin() with cursor() function:
conn = engine.connect()
trans = conn.cursor()
If not resolved
1- Verify structure of source and destination tables if they are same or not.
2- remove try ,except and rollback() block so you can know the actual error.
Ex.
qry_load = 'insert into target_table select * from some_tables'
conn = engine.connect()
trans = conn.cursor()
conn.execute(qry_load)
trans.commit()
I have a column called REQUIREDCOLUMNS in a SQL database which contains the columns which I need to select in my Python script below.
Excerpt of Current Code:
db = mongo_client.get_database(asqldb_row.SCHEMA_NAME)
coll = db.get_collection(asqldb_row.TABLE_NAME)
table = list(coll.find())
root = json_normalize(table)
The REQUIREDCOLUMNSin SQL contains values reportId, siteId, price, location
So instead of explicitly typing:
print(root[["reportId","siteId","price","location"]])
Is there a way to do print(root[REQUIREDCOLUMNS])?
Note: (I'm already connected to the SQL database in my python script)
You will have to use cursors if you are using mysql or pymysql , both the syntax are almost similar below i will mention for mysql
import mysql
import mysql.connector
db = mysql.connector.connect(
host = "localhost",
user = "root",
passwd = " ",
database = " "
)
cursor = db.cursor()
sql="select REQUIREDCOLUMNS from table_name"
cursor.execute(sql)
required_cols = cursor.fetchall()#this wll give ["reportId","siteId","price","location"]
cols_as_string=','.join(required_cols)
new_sql='select '+cols_as_string+' from table_name'
cursor.execute(new_sql)
result=cursor.fetchall()
This should probably work, i intentionally split many lines into several lines for understanding.
syntax could be slightly different for pymysql
I'm running a series of complex sql queries in python and it involves temp tables. My auto-commit method doesn't seem to be working to retrieve the data from the temp table. The code snippet I'm using below and this is the output I'm getting:
testQuery="""
Select top 10 *
INTO #Temp1
FROM Table1 t1
JOIN Table2 t2
on t1.key=t2.key
"""
cnxn=pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=DB;UID=UID;PWD=PWD')
cnxn.autocommit=True
cursor=cnxn.cursor()
cursor.execute(testQuery)
cursor.execute("""Select top 10 * from #Temp1""")
<pyodbc.Cursor at 0x8f78930>
cnxn=pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=DB;UID=UID;PWD=PWD')
cnxn.autocommit=True
cursor=cnxn.cursor()
cursor.execute(testQuery)
cursor.execute("""Select top 10 * from #Temp1""")
Even though this question has a "solution", i.e., using global temp table instead of a local temp table, future readers might benefit from understanding why the problem happened in the first place.
A temporary table is automatically dropped when the last connection using said table is closed. The difference between a local temp table (#Temp1) and a global temp table (##Temp1) is that the local temp table is only visible to the connection that created it, while an existing global temp table is available to any connection.
So the following code using a local temp table will fail ...
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT 1 AS foo, 2 AS bar INTO #Temp1
"""
crsr.execute(sql)
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT foo, bar FROM #Temp1
"""
crsr.execute(sql)
row = crsr.fetchone()
print(row)
... while the exact same code using a global temp table will succeed ...
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT 1 AS foo, 2 AS bar INTO ##Temp1
"""
crsr.execute(sql)
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT foo, bar FROM ##Temp1
"""
crsr.execute(sql)
row = crsr.fetchone()
print(row)
... because the second pyodbc.connect call opens a separate second connection to the SQL Server without closing the first one.
The second connection cannot see the local temp table created by the first connection. Note that the local temp table still exists because the first connection was never closed, but the second connection cannot see it.
However, the second connection can see the global temp table because the first connection was never closed and therefore the global temp table continued to exist.
This type of behaviour has implications for ORMs and other mechanisms that may implicitly open and close connections to the server for each SQL statement that it executes.
I asked a colleague about this live and his suggestions worked. So I went and changed the testQuery to create a global temp table instead of a local (##Temp1 instead of #Temp1). And went to sql server to test whether the temp table was actually being created-it was. So I isolated that the problem was the second cursor.execute statement. I modified the code to use pandas read_sql_query instead and it all worked out! Below is the code I used:
testQuery="""
Select top 10 *
INTO ##Temp1
FROM Table1 t1
JOIN Table2 t2
on t1.key=t2.key
"""
cnxn=pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=DB;UID=UID;PWD=PWD')
cnxn.autocommit=True
cursor=cnxn.cursor()
cursor.execute(testQuery)
cnxn.commit()
query1="Select top 10 * from ##Temp1"
data1=pd.read_sql_query(query1, cnxn)
data1[:10]
Best way to go about this is to start your SQL query with:
"SET NOCOUNT ON"
This will output the desired data
The SET NOCOUNT ON is what worked for me.
execute Method () - JDBC Driver for SQL Server | Microsoft Docs
Return Value
true, if the statement returns a result set.
false, if it returns an update count or no result.
If you want a result set, then setting SET NOCOUNT ON s the setting you need in your statements.
I am using SQL Alchemy to connect to my Oracle 11g database residing in a Linux machine. I am writing a script in Python to connect to the database and retrieve values from a particular table.
The code i wrote is:
from sqlalchemy import *
def db_connection():
config = ConfigParser.RawConfigParser()
config.read('CMDC_Analyser.cfg')
USER = config.get('DB_Connector','db.user_name' )
PASSWORD = config.get('DB_Connector','db.password' )
SID = config.get('DB_Connector','db.sid' )
IP = config.get('DB_Connector','db.ip')
PORT = config.get('DB_Connector','db.port')
engine = create_engine('oracle://{user}:{pwd}#{ip}:{port}/{sid}'.format(user=USER, pwd=PASSWORD, ip=IP, port=PORT, sid=SID), echo=False)
global connection
connection = engine.connect()
p = connection.execute("select * from ssr.bouquet")
for columns in p:
print columns
connection.close()
The complete values from the table is printed out here. I wanted to select value from a particular column only. Hence i used the following code:
for columns in p:
print columns['BOUQUET_ID']
But here i am getting the following error.
sqlalchemy.exc.NoSuchColumnError: "Could not locate column in row for column 'BOUQUET_ID'"
How to fix this?