I am trying to pull a query using pyodbc and it keeps returning a reset by peer error, I can run the query no issue 100% of the time in Teradata SQL Assistant.
This happens with one other query I use but everything else I have done works using the same code.
I have tried running it multiple times, always works in Teradata SQL Assistant but not in python. Hundreds of other queries have worked with no issue, only two have given issues. I have tried slightly changing the queries with no luck.
I also tried in R with RODBC and it worked there.
I asked our DBA team and they said they have no process that would automatically boot that process and there is no issues with the query.
import pyodbc
import pandas.io.sql as psql
connection_info = 'DSN=xxxxxx'
conn = pyodbc.connect(connection_info)
sql1 = '''
QUERY HERE
'''
df = psql.read_sql_query(sql1, conn)
expect df to = resuls, instead get the following error:
10054 WSA E ConnReset: Connection reset by peer
Related
I am trying to connect a jupyter notebook I'm running in a conda environment to a Hadoop cluster through Apache Hive on cloudera. I understand from this post that I should install/set up the cloudera odbc driver and use pydobc and with a connection as follows:
import pyodbc
import pandas as pd
with pyodbc.connect("DSN=<replace DSN name>", autocommit=True) as conn:
df = pd.read_sql("<Hive Query>", conn)
My question is about the autocommit parameter. I see in the pyodbc connection documentation that setting autocommit to True will make it so that I don't have to explicitly commit transactions, but it doesn't specify what that actually means.
What exactly is a transaction ?
I want to select data from the hive server using pd.read_sql_query() but I don't want to make any changes to the actual data on the server.
Apologies if this question is formatted incorrectly or if there are (seemingly simple) details I'm overlooking in my question - this is my first time posting on stackoverflow and I'm new to working with cloudera / Hive.
I haven't tried connecting yet or running any queries yet because I don't want to mess up anything on the server.
Hive do not have concept of commit and starting transactions like RDBMS systems.
You should not worry about autocommit.
I am trying to move from SQLite to MySQL - and I am almost there. However, I all the time encounter one problem, that plagues me regardless if I am connecting to my local MySQL db, or the one hosted on Google Cloud. Have anyone had the same issue.
This is the code I use to append pandas dataframe to the table
import pandas as pd
from sqlalchemy import create_engine
connection = create_engine(f"mysql+mysqlconnector://{user}:{pw}#{host}/{db}")
tablename = 'TABLENAME'
df.to_sql(tablename, connection, if_exists='append', index=False)
The pandas table is not very large, a few tens of rows at a time.
Every now and then I get this error
sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) 2055: Lost connection to MySQL server at 'serverip:port', system error: 32 Broken pipe
So far I have been using sqlite3 package for SQLite - do you think it is sqlalchemy or MySQL which is the problem?
I'm trying to connect to an oldschool jTDS ms server for a variety of different analysis tasks. Firstly just using Python with SQL alchemy, as well as using Tableau and Presto.
Focusing on SQL Alchemy first at the moment I'm getting an error of:
Data source name not found and no default driver specified
With this, based on this thread here Connecting to SQL Server 2012 using sqlalchemy and pyodbc
i.e,
import urllib
params = urllib.parse.quote_plus("DRIVER={FreeTDS};"
"SERVER=x-y.x.com;"
"DATABASE=;"
"UID=user;"
"PWD=password")
engine = sa.create_engine("mssql+pyodbc:///?odbc_connect={FreeTDS}".format(params))
Connecting works fine through Dbeaver, using a jTDS SQL Server (MSSQL) driver (which is labelled as legacy).
Curious as to how to resolve this issue, I'll keep researching away, but would appreciate any help.
I imagine there is an old drive on the internet I need to integrate into SQL Alchemy to begin with, and then perhaps migrating this data to something newer.
Appreciate your time
I can connect via pyODBC to an unsupported database over ODBC. Queries appear to execute correctly. If I try to connect using mssql+pyodbc, I can't connect properly (image not found).
I've tried "base:///", "base+pyodbc:///", or "pyodbc:///".
Do I need to write my own "dialect" that doesn't make any changes to base, and are there any useful (up to date) guides on how to do this?
EDIT:
import pyodbc
conn = pyodbc.connect(DSN = "ODBCCONNECTIONNAME", UID = "ODBCUSER", PWD="PASSWORD")
cursor = conn.cursor()
Also works with this line replacing the connection above:
conn = pyodbc.connect("DSN=fmp_production;UID=odbc_user;PWD=Pwd222")
I can then run selects, and modifications fine, using standard SQL.
EDIT:
Okay, so I'm getting an error:
"Abort trap: 6" basically my python process is crashing out. I've tried testing with a functioning ODBC connection to a MySQL database, using:
engine = create_engine('mysql+pyodbc://root:rootpwd#testenvironment')
If I include the name of the database, then I get an image not found error instead. But I think the actual problem is whatever is crashing my python.
I have a daemon that uses sqlalchemy to interact with MySQL database. Since interaction is seldom, the connections are prone to timing out. I've tried to fix the problem by setting various flags when creating the database engine, e.g. pool_recycle=3600, but nothing seems to help.
To help me debug the problem, I set the timeout of my local mysql server to 10 seconds, and tried the following program.
import time
import sqlalchemy
engine = sqlalchemy.engine.create_engine("mysql://localhost")
while True:
connection = engine.connect()
result = connection.execute("SELECT 1")
print result.fetchone()
connection.close()
time.sleep(15)
Surprisingly, I continue to get exceptions like the following:
sqlalchemy.exc.OperationalError: (OperationalError) (2006, 'MySQL server has gone away')
However, if I remove the call to connection.close(), the problem goes away. What is going on here? Why isn't sqlalchemy trying to establish a new connection each time I call connect()?
I'm using Python 2.7.3 with sqlalchemy 0.9.8 and MySQL 5.5.40.
the document mention:
MySQL features an automatic connection close behavior,
for connections that have been idle for eight hours or more.
To circumvent having this issue, use the pool_recycle option
which controls the maximum age of any connection:
engine = create_engine('mysql+mysqldb://...', pool_recycle=3600)
You can just put "pool_recycle" parameter when the create_engine is invoked.
I'm not 100% sure if this is going to fix your problem or not, but I ran into a similar issue when dealing with mysql. It was only fixed when I used pymysql to connect to the database.
You can install like this:
pip install pymysql
Which will work for both linux and windows (if you have it installed)
Then give your connection string like this:
import time
import sqlalchemy
engine = sqlalchemy.engine.create_engine("mysql+pymysql://localhost")
while True:
connection = engine.connect()
result = connection.execute("SELECT 1")
print result.fetchone()
connection.close()
time.sleep(15)
I get the following output when I run it:
(1,)
(1,)
(1,)
(1,)
On another note, I have found certain queries to break with SQLAlchemy 0.9.8. I had to install version 0.9.3 in order for my applications to not break anymore.