I'm trying to connect to an oldschool jTDS ms server for a variety of different analysis tasks. Firstly just using Python with SQL alchemy, as well as using Tableau and Presto.
Focusing on SQL Alchemy first at the moment I'm getting an error of:
Data source name not found and no default driver specified
With this, based on this thread here Connecting to SQL Server 2012 using sqlalchemy and pyodbc
i.e,
import urllib
params = urllib.parse.quote_plus("DRIVER={FreeTDS};"
"SERVER=x-y.x.com;"
"DATABASE=;"
"UID=user;"
"PWD=password")
engine = sa.create_engine("mssql+pyodbc:///?odbc_connect={FreeTDS}".format(params))
Connecting works fine through Dbeaver, using a jTDS SQL Server (MSSQL) driver (which is labelled as legacy).
Curious as to how to resolve this issue, I'll keep researching away, but would appreciate any help.
I imagine there is an old drive on the internet I need to integrate into SQL Alchemy to begin with, and then perhaps migrating this data to something newer.
Appreciate your time
Related
I created an Azure Function with Python and want to write some data into an Azure SQL DB.
If I run the code on my local machine via AZ Function Debugger, everything is working. But when I deploy everything to Azure, I only get a message that there is an error (no additional specific information).
I think this is related to the ODBC Driver?
I'm using the following code to connect and insert data:
with pyodbc.connect('DRIVER='+driver+';SERVER=tcp:'+server+';PORT='+port+';DATABASE='+database+';UID='+username+';PWD='+ password + ";Authentication=ActiveDirectoryPassword", timeout=120) as conn:
with conn.cursor() as cursor:
try:
cursor.execute(data)
except:
logging.error("Can't execute SQL Query!")
I use driver= '{ODBC Driver 17 for SQL Server}' as driver.
I assume that this is missing in Azure? How can this issue be fixed? What is the right approach to connect from Azure Functions to an Azure SQL DB via Python?
It seems the ODBC driver is included, it was just poorly documnented:
https://github.com/MicrosoftDocs/azure-docs/issues/54423
There is an example project here:
https://github.com/kevin808/azure-function-pyodbc-MI
The full tutorial including creating the system assigned identity can be found here:
https://techcommunity.microsoft.com/t5/apps-on-azure-blog/how-to-connect-azure-sql-database-from-python-function-app-using/ba-p/3035595
There is currently a SQL Extension under development but it only supports C# at the moment. Python has been requested as an ehancement so you could add your 👍 to the issue so that you could use bindings
https://github.com/Azure/azure-functions-sql-extension/issues/172
I am trying to switch a pyodbc connection to sqlalchemy engine. My working pyodbc connection is:
con = pyodbc.connect('DSN=SageLine50v23;UID=#####;PWD=#####;')
This is what I've tried.
con = create_engine('pyodbc://'+username+':'+password+'#'+url+'/'+db_name+'?driver=SageLine50v23')
I am trying to connect to my Sage 50 accounting data but just can't work out how to build the connection string. This is where I downloaded the odbc driver https://my.sage.co.uk/public/help/askarticle.aspx?articleid=19136.
I got some orginal help for the pyodbc connection using this website (which is working) https://www.cdata.com/kb/tech/sageuk-odbc-python-linux.rst but would like to use SQLAlchemy for it connection with pandas. Any ideas? Assume the issue is with this part pyodbc://
According to this thread Sage 50 uses MySQL to store its data. However, Sage also provides its own ODBC driver which may or may not use the same SQL dialect as MySQL itself.
SQLAlchemy needs to know which SQL dialect to use, so you could try using the mysql+pyodbc://... prefix for your connection URI. If that doesn't work (presumably because "Sage SQL" is too different from "MySQL SQL") then you may want to ask Sage support if they know of a SQLAlchemy dialect for their product.
I have a program, in which I have been using the phoenixdb package developed by Lukas Lalinsky but during the past few days it seems to have become very unstable. I think this is due to the size of the database (as it is constantly growing). By unstable I mean that around half my queries are failing with a runtime exception.
So I have moved on and tried to find a more stable way to connect with my Phoenix "server". Therefore I want to try out a JDBC connection. As far as I have understood Phoenix should have great integration with JDBC.
I do however have problems with understanding how to set up the initial connection.
I read the following Usage section of the JayDeBeApi package, but I don't know what the Driver Class is or where it is located? If I have to download it myself? How to set it up? And so forth.
I was hoping someone in here would know and hopefully explain it in detail.
Thanks!
EDIT:
I've managed to figure out that my connect statement should be something along this:
import jaybedeapi as jdbc
conn = jdbc.connect('org.apache.phoenix.jdbc.PhoenixDriver', ['jdbc:phoenix:<ip>:<port>:', '', ''], '<location-of-phoenix-client.jar>')
However I still don't know where to get my hands on that phoenix-client.jar file and how to reference to it.
I managed to find the solution after having set up a Java project and testing out JDBC in that development environment and getting a successful connection.
To get the JDBC connection working in Java I used the JDBC driver found in the Phoenix distribution from Apache here. I used the driver that matched my Phoenix and HBase versions - phoenix-4.9.0-HBase-1.2-client.jar
Once that setup was completed and I could connect to Phoenix using Java I started trying to set it up using Python. I started a connection to Phoenix with the following:
import jaydebeapi as jdbc
import os
cwd = os.getcwd()
jar = cwd + '/phoenix-4.9.0-HBase-1.2-client.jar'
drivername = 'org.apache.phoenix.jdbc.PhoenixDriver'
url = 'jdbc:phoenix:<ip>:<port>/'
conn = jdbc.connect(drivername, url, jar)
Now I had a successful connection through JDBC to Phoenix using Python. Hope someone else out there can use this question in the future.
I created a cursor using the following and could issue commands like in the following:
cursor = conn.cursor()
sql = """SELECT ...."""
cursor.execute(sql)
resp = cursor.fetchone() # could use .fetchall() or .fetchmany() if needed
I hope this helps someone out there!
I am currently using python with adodb api for MS Sql database connection. This is working very well for smaller databases. But for large databases when the query is returning huge data, I get MS Sql cursor out of memory error. This works fine with Server side cursor. But then it consumes lot of memory on my server.
Question 1: How can I continue to use client side cursor? Can reading data in chunks be an option?
Question 2: Is there any alternative to python ado db API?
I searched for other options to connect to MS Sql. Found this link about comparison - pymssql versus pyodbc versus adodbapi versus.... This looks old information. I have started my own investigation with pyodbc and pymssql.
Can anybody recommend which is better out of these?
As suggested by Gord, I tested my code with pyodbc and its working 3 times faster than adodb api.
I have a django application, which is making use of SQLAlchemy to connect to a SQL Server instance on Windows Azure. The app has worked perfectly for 3 months on a local SQL Server instance, and for over a month on an Azure instance. The issues appeared this monday, after a week without any code changes.
The site uses:
Python 2.7
Django 1.6
Apache/Nginx
SQLAlchemy 0.9.3
pyODBC 3.0.7
FreeTDS
The application appears to lock up right after a connection is pulled out of the Pool (I have setup verbose logging at every point in the workflow). I assumed this had something to do with the connections going stale. So we tried making the pool_recycle incredibly short (5 secs), all the way up to an hour. That did not help.
We also tried using the NullPool to force a new connection on every page view. However that does not help either. After about 15 minutes the site will completely lock up again (meaning no pages that use the database are viewable).
The weird thing is, half the computers that experience the "hang", will end up loading the page about 15 minutes later.
Has anyone had any experience with SQL Azure and SQLAlchemy?
I found a workaround for this issue. Please note that this is definitely not a fix, since the site worked perfectly fine before. We could not determine what the actual issue is because SQL Azure has no error log (one of the 100 reasons I would suggest never considering SQL Azure over a real database server).
I got around the problem by turning off all Connection Pooling, at the application level, AND at the driver level.
Things started consistently working after making my /etc/odbcinst.ini look like:
[FreeTDS]
Description = TDS driver (Sybase/MS SQL)
# Some installations may differ in the paths
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
CPReuse =
CPTimeout = 0
FileUsage = 1
Pooling = No
The key being setting CPTimeout (Connection Pool Timeout) to 0, and Pooling to No. Just turning pooling off at the application level (in SQL Alchemy) did not work, only after setting it at the driver level did things start working smoothly.
I am now at 4 days without a problem after changing that setting.