Thanks for reading this. I’m learning python and have run into a bit of a blocker.
I’m trying to set up a basic database to join some data together and query it.
I’ve entered the code below but get this error when I run it
no such table: customer
There is more to the error message but I think if I solve this the rest will go away…Hopefully
import pandas as pd
from sqlalchemy import create_engine
engine = create_engine(f"sqlite:///HFSS.db")
conn = engine.connect()
customer = pd.read_excel('customer_orders.xlsx',index_col = 'CU_GTIN' * 1)
customer ## running this shows me a table
When I try to run a simple query in SQLite I get the error.
query = '''SELECT EccCustomerLevel3Name
FROM customer'''
pd.read_sql_query(query, conn)
Thanks for your time
D
Related
I have an SQLAlchemy database that is having data added to it in real time. I want a way to be able to send a keyword or message in Teams and run an SQL call that will return in teams. It is the reverse of this post Send automated messages to Microsoft Teams using Python_
Does anyone know if this is possible? Here is my current SQL call:
from sqlalchemy import create_engine
import pandas as pd
link = "WEBHOOK_LINK"
myTeamsMessage = pymsteams.connectorcard(link)
pwd = "path_to_working_directory"
engine = create_engine(f'sqlite:///{pwd}\\database.db', echo=False)
pd.DataFrame(engine.execute("""
SELECT *
FROM California_table
ORDER BY Page_Num DESC
LIMIT 1
"""))
We have data in a Snowflake cloud database that we would like to move into an Oracle database. As we would like to work toward refreshing the Oracle database regularly, I am trying to use SQLAlchemy to automate this.
I would like to do this using Core because my team is all experienced with SQL, but I am the only one with Python experience. I think it would be easier to tweak the data pulls if we just pass SQL strings. Plus the Snowflake db has some columns with JSON that seems easier to parse using direct SQL since I do not see JSON in the SnowflakeDialect.
I have established connections to both databases and am able to do select queries from both. I have also manually created the tables in our Oracle db so that the keys and datatypes match what I am pulling from Snowflake. When I try to insert, though, my Jupyter notebook just continuously says "Executing Cell" and hangs. Any thoughts on how to proceed or how to get the notebook to tell me where the hangup is?
from sqlalchemy import create_engine,pool,MetaData,text
from snowflake.sqlalchemy import URL
import pandas as pd
eng_sf = create_engine(URL( #engine for snowflake
account = 'account'
user = 'user'
password = 'password'
database = 'database'
schema = 'schema'
warehouse = 'warehouse'
role = 'role'
timezone = 'timezone'
))
eng_o = create_engine("oracle+cx_oracle://{}[{}]:{}#{}".format('user','proxy','password','database'),poolclass=pool.NullPool) #engine for oracle
meta_o = MetaData()
meta_o.reflect(bind=eng_o)
person_o = meta_o['bb_lms_person'] # other oracle tables follow this example
meta_sf = MetaData()
meta_sf.reflect(bind=eng_sf,only=['person']) # other snowflake tables as well, but for simplicity, let's look at one
person_sf = meta_sf.tables['person']
person_query = """
SELECT ID
,EMAIL
,STAGE:student_id::STRING as STUDENT_ID
,ROW_INSERTED_TIME
,ROW_UPDATED_TIME
,ROW_DELETED_TIME
FROM cdm_lms.PERSON
"""
with eng_sf.begin() as connection:
result = connection.execute(text(person_query)).fetchall() # this snippet runs and returns result as expected
with eng_o.begin() as connection:
connection.execute(person_o.insert(),result) # this is a coinflip, sometimes it runs, sometimes it just hangs 5ever
eng_sf.dispose()
eng_o.dispose()
I've checked the typical offenders. The keys for both person_o and the result are all lowercase and match. Any guidance would be appreciated.
use the metadata for the table. the fTable_Stage update or inserted as fluent functions and assign values to lambda variables. This is very safe because only metadata field variables can be used in the lambda. I am updating three fields:LateProbabilityDNN, Sentiment_Polarity, Sentiment_Subjectivity
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
connection=engine.connect()
metadata=MetaData()
Session = sessionmaker(bind = engine)
session = Session()
fTable_Stage=Table('fTable_Stage', metadata,autoload=True,autoload_with=engine)
stmt=fTable_Stage.update().where(fTable_Stage.c.KeyID==keyID).values(\
LateProbabilityDNN=round(float(late_proba),2),\
Sentiment_Polarity=round(my_valance.sentiment.polarity,2),\
Sentiment_Subjectivity= round(my_valance.sentiment.subjectivity,2)\
)
connection.execute(stmt)
I'm trying to retrieve a table into a dataframe but I'm getting a "ValueError: hour must be in 0..23".
Here is my code :
from sqlalchemy import create_engine
import pyodbc
import pandas as pd
SERVER = '(local)'
DATABASE = 'Projects'
DRIVER = 'SQL Server'
DATABASE_CONNECTION = f'mssql://#{SERVER}/{DATABASE}?driver={DRIVER}'
engine = create_engine(DATABASE_CONNECTION)
connection = engine.connect()
data = pd.read_sql_query('select TOP 1 * from PRODSYNTHESIS',connection)
#ValueError: hour must be in 0..23
connection.close()
engine.dispose()
table schema
My understanding is that I should probably be using python's datetime module to handle this datatype but not sure exactly how to get there. All other solutions I've found relate to the dateframe itself not the data type coming through from sql.
Not sure, but you can always woraround by converting the date to an ISO8601 string in the query itself. EG:
data = pd.read_sql_query('select TOP 1 Project_id, Project_name, convert(varchar(23), Date_results, 126) Date_results, P_Injecte from PRODSYNTHESIS',connection)
Unable to understand why my sql query is throwing an exception of [Oracle][ODBC][Ora]ORA-00936: missing expression.
The case is that the code seems to be working fine when I'm using
select* from reports.ORDERS_NOW.
So it's letting me pull all the data, but for my case, I want only specific columns for which I'm writing the query. Please look at the code below and let me know what's wrong with it.
import pyodbc
import pandas as pd
conn = conn = pyodbc.connect('DSN=abcd;UID=xxxxxx;PWD=xxxxxx')
if conn:
print("Connection is successful")
db query
sql = '''
select [QUANTITY] from reports.ORDERS_NOW
'''
df = pd.read_sql(sql,conn)
i think [] is not allowed in oracle so remove it
select QUANTITY from reports.ORDERS_NOW
I am trying to use 'pandas.read_sql_query' to copy data from MS SQL Server into a pandas DataFrame. I need to do multiple joins in my SQL query. The tables being joined are on the same server but in different databases. The query I am passing to pandas works fine inside MS SQL Server Management Studio. In a Jupyter Notebook I tried to query data like so (to make things readable the query itself is simplified to just 2 joins and generic names are used):
import pandas as pd
import sqlalchemy as sql
import pyodbc
server = '100.10.10.10'
driver = 'SQL+Server+Native+Client+11.0'
myQuery = '''SELECT first.Field1, second.Field2
FROM db1.schema.Table1 AS first
JOIN db2.schema.Table2 AS second
ON first.Id = second.FirstId
'''
engine = sql.create_engine('mssql+pyodbc://{}?driver={}'.format(server, driver))
df = pd.read_sql_query(myQuery, engine)
This does not work and returns an error:
DBAPIError: (pyodbc.Error) ('IM010', '[IM010] [Microsoft][��������� ��������� ODBC] ������� ������� ��� ��������� ������ (0) (SQLDriverConnect)')
It seems that the problem is in the engine which does not include information about the database, because everything works fine with the next kind of code, where I include database in the engine:
myQuery = 'select Field1 from schema.Table1'
db = 'db1'
engine = sql.create_engine('mssql+pyodbc://{}/{}?driver={}'.format(server, db, driver))
df = pd.read_sql_query(myQuery, engine)
but breaks like the code with joins above if I don't include database in the engine, but add it to the query like so:
myQuery = 'select Field1 from db1.schema.Table1'
engine = sql.create_engine('mssql+pyodbc://{}?driver={}'.format(server,
driver))
df = pd.read_sql_query(myQuery, engine)
So how should I specify the pandas.read_sql_query 'sql' and 'con' parameters in
this case when I need to join tables from different databases but the same server?
P.S. I only have read access to this server I am connecting to. I can not create new tables or views or anything like that.
Update:
The MS SQL Server version is 2008 R2.
Update 2: I am using Python 3.6 and Windows 10.
So I have found a workaround: use pymssql instead of pyodbc (both in the import statement and in the engine). It lets you build your joins using database names and without specifying them in the engine. And there is no need to specify a driver in this case.
There might be a problem if you are using Python 3.6 which is not supported by pymssql oficially yet, but you can find unofficial wheels for your Python 3.6 here. It works as is supposed to with my queries.
Here is the original code with joins, rebuilt to work with pymssql:
import pandas as pd
import sqlalchemy as sql
import pymssql
server = '100.10.10.10'
myQuery = '''SELECT first.Field1, second.Field2
FROM db1.schema.Table1 AS first
JOIN db2.schema.Table2 AS second
ON first.Id = second.FirstId'''
engine = sql.create_engine('mssql+pymssql://{}'.format(server))
df = pd.read_sql_query(myQuery, engine)
As for the unofficial wheels, you need to download the file for Python 3.6 from the link I gave above, then cd to the download folder and run pip install wheels where 'wheels' is the name of the wheels file.
UPDATE:
Actually, it is possible to use pyodbc too. I am not sure if this should work for any SQL Server setup, but everything worked for me after I had set 'master' as my database in the engine. The resulting code would look like this:
import pandas as pd
import sqlalchemy as sql
import pyodbc
server = '100.10.10.10'
driver = 'SQL+Server'
db = 'master'
myQuery = '''SELECT first.Field1, second.Field2
FROM db1.schema.Table1 AS first
JOIN db2.schema.Table2 AS second
ON first.Id = second.FirstId'''
engine = sql.create_engine('mssql+pyodbc://{}/{}?driver={}'.format(server, db, driver))
df = pd.read_sql_query(myQuery, engine)
The following code is working for me. I am using SQL server with SQLAlchemy
import pyodbc
import pandas as pd
cnxn = pyodbc.connect('DRIVER=ODBC Driver 17 for SQL Server;SERVER=your_db_server_id,your_db_server_port;DATABASE=pangard;UID=your_db_username;PWD=your_db_password')
query = "SELECT * FROM database.tablename;"
df = pd.read_sql(query, cnxn)
print(df)