There is a multi-million record database in MS SQL Server 2008 R2 under Windows XP SP3.
My co-worker has written a .Net application, which directly connects to this db and runs a reasonable amount of queries. I dont know .Net, but I'm sure this app does not connect to DB using ODBC.
On the other hand I've written a command-line python (CPython version 2.7.5) application, which connects to this database and runs simple queries on it to send the data over internet to somewhere else. Database connection is made using pyodbc 3.0.7 (installer from http://www.lfd.uci.edu/~gohlke/pythonlibs/) and a DSN which uses SQL Server Native Client 10.0 driver. I've tried both disabling and enabling connection pools for this driver in Connection Pooling tab of windows Data Sources (ODBC) applet. The script sends 100 records from the db and then closes the connection and sleeps for 2 minutes and then runs again.
Both of these programs run constantly on the same machine as db.
The question is the .Net app runs nicely when I remove the defined DSN ( and of course the python script is not running). When I define the DSN again and start the python script to run side by side .Net app, there is no problem for roughly 5 hours. but then gradually while python script is mostly fine, the .Net app starts to get timeouts from the db.
What could go wrong that this happens?
EDIT:
python script (which connects using ODBC) runs fine, all the time. but .Net app falls behind of usual performance after couple of hours. When I close the python script, .Net app still stays behind. but when I remove the ODBC DSN which I defined for python script, .Net app goes back to normal performance. It's very strange. As I said I don't know anything about .Net, so this could be result of non-standard code on the part of .Net app, maybe open transactions, locks, too many connections, etc. To make the case stranger, cutting database size to half by deleting records and rebuilding indexes, seems to have solved the .Net app issue so far.
EDIT 2:
The only two queries that python script runs, are:
SELECT TOP 100 FROM tbl_data WHERE id > ? ORDER BY id
and
SELECT * FROM tbl_data WHERE id = ?
The first query usually only runs once in each run of the python script. The second one runs at most 100 times. id is primary key, and so is indexed. As you can see the queries could not be simpler. For the first query I read the entire result set in the program to not keep a cursor open on DB server. Also I've turned off connection pooling in ODBC applet for the driver I'm using, so after each run of the script, DB connection should have been disposed and all resources on DB server should have been freed. The script sleeps for 2 minutes and then repeats this.
The queries that .Net app runs are far more complex, combined with some triggers on the database. And the strange thing is it runs mostly fine, on itself. but when DSN is defined, it starts to get long waits on a single insert statement, which sometimes results in timeout.
Also I should have said that Windows and MSSQL are not updated with latest patches from microsoft, so if it's a bug in ODBC driver or MSSQL itself, it could have been already solved for others.
EDIT 3
Table is clustered on PK index. the data table contains about 1.5M records now. DB size is about 160GB. Server does not have a high spec. Intel Core i7 2600, 4GB RAM, plain 1TB SATA disk drive.
There are a number of things that can be affecting performance. There could be resource contention, sub-optimal SQL Server configuration, slow disks, missing indexes etc...
I would start by monitoring the system resources when both processes are running. You can use perfmon to monitor the OS and SQL counters. I would start by looking at
ProcessorInformation/%ProcessorTime
LogicalDisk/Avg Disk sec/read
LogicalDisk/Avg Disk sec/write
Memory/Available MBytes
SQLServerBufferManager/BufferCacheHitRatio
SQLServerBufferManager/PageLifeExpectency
Here is a great article on using perfmon, http://www.brentozar.com/archive/2006/12/dba-101-using-perfmon-for-sql-performance-tuning/
The next step is optimizing query and SQL Server performance.
Query Performance, http://technet.microsoft.com/en-us/magazine/2007.11.sqlquery.aspx
SQL Server Performance,
-Separate data and log files on different volumes (raid 10 is preferable)
-tempdb (http://technet.microsoft.com/en-us/library/ms175527(v=SQL.105).aspx)
-configure min/max memory
-configure lock pages in memory and instant file initialization
-Check out this link, http://www.brentozar.com/sql/sql-server-performance-tuning/
The point is that there are so many possibilities that could be affecting your database, that it is nearly impossible to identify your problem based on the provided information. I recommend that you go through each of these items to identify why your system is not running optimally.
Related
We are using the Snowflake ODBC Driver to connect to our warehouse from a series of different, completely unrelated backends.
We have Alteryx workflows and Python applications connecting to the warehouse through ODBC.
We can successfully establish a connection and access the information in our databases, however, there is a hard limit of 1000000 when selecting records from a table. It never pulls more than that. No error or warning messages either, it simply does not pull more than a million record, regardless of the source where we run the process from.
The applications and Alteryx run on completely different servers as well, so it's unlikely that it is a server issue.
I have not been able to find anything about this on the Snowflake documentation, and I'm at a lost as to where to search.
What's even more odd: We can upload more than a million records from Alteryx into Snowflake. The issue is just for when we receive the data FROM snowflake.
Any ideas on where/what to check?
We're developing a website which requires database access. Accessing such a page works fine; accessing multiple in a row is also fine. However, if you wait for a long amount of time (15 minutes seems to be enough), accessing another page will hang for a long time (10-20 minutes has been observed). Afterwards, this error will print.
Here's the relevant code:
if __name__ == "__main__":
conf = load_conf(sys.argv[1])
engine = create_engine('postgresql://%s:%s#%s:%s/%s' %
(conf['db']['user'], conf['db']['pw'], conf['db']['address'],
conf['db']['port'], conf['db']['database']), echo=False)
Session = sessionmaker(bind=engine)
session = Session()
app = make_app(session, conf)
app.listen(8888)
tornado.ioloop.IOLoop.current().start()
The database is on a different server. My personal machine is in the Netherlands, while the database is in a server in Germany. My partner's personal machine is in Italy.
Most notably, this issue is only present on my machine, running Arch Linux. We've tested this on two other machines, running Windows and some other Linux (I assume Ubuntu, can check if needed). At this point, we have no clue on how to continue debugging.
Of course, I will provide any extra needed information on request.
It's unclear where this 15-minute timeout is coming from, although as other commenters have indicated it's likely coming from something in the network between your computer and the server. However, wherever it is coming from, there are a couple of options to work around it in SQLAlchemy.
The pool_pre_ping=True option will issue a simple test query before any attempt to reuse a connection, allowing it to detect this problem and reconnect transparently (at a small performance cost)
The pool_recycle=600 option tells sqlalchemy not to reuse a connection that has been idle for longer than 10 minutes. This is a more efficient solution to the problem since it doesn't add any new queries, but it requires you to work out the best recycle timeout to use.
I am creating a Python application that uses embedded SQLite databases. The programme creates the db files and they are on a shared network drive. At this point there will be no more than 5 computers on the network running the programme.
My initial thought was to ask the user on startup if they are the server or client. If they are the server then they create the database. If they are the client they must find a server instance on the network. The one way I suppose is to send all db commands from client to server and server implements in the database. Will that solve the shared db issue?
Alternatively, is there some way to create a SQLite "server". I presume this would be the quicker option if available?
Note: I can't use a server engine such as MySQL or PostgreSQL at this point but I am implementing using ORM and so when this becomes viable, it should be easy to change over.
Here's a "SQLite Server", http://sqliteserver.xhost.ro/, but it looks like not in maintain for years.
SQLite supports concurrency itself, multiple processes can read data at one time and only one can write data into it. Also, When some process is writing, it'll lock the whole database file for a few seconds and others have to wait in the mean time according official document.
I guess this is sufficient for 5 processes as yor scenario. Just you need to write codes to handle the waiting.
I have a django application, which is making use of SQLAlchemy to connect to a SQL Server instance on Windows Azure. The app has worked perfectly for 3 months on a local SQL Server instance, and for over a month on an Azure instance. The issues appeared this monday, after a week without any code changes.
The site uses:
Python 2.7
Django 1.6
Apache/Nginx
SQLAlchemy 0.9.3
pyODBC 3.0.7
FreeTDS
The application appears to lock up right after a connection is pulled out of the Pool (I have setup verbose logging at every point in the workflow). I assumed this had something to do with the connections going stale. So we tried making the pool_recycle incredibly short (5 secs), all the way up to an hour. That did not help.
We also tried using the NullPool to force a new connection on every page view. However that does not help either. After about 15 minutes the site will completely lock up again (meaning no pages that use the database are viewable).
The weird thing is, half the computers that experience the "hang", will end up loading the page about 15 minutes later.
Has anyone had any experience with SQL Azure and SQLAlchemy?
I found a workaround for this issue. Please note that this is definitely not a fix, since the site worked perfectly fine before. We could not determine what the actual issue is because SQL Azure has no error log (one of the 100 reasons I would suggest never considering SQL Azure over a real database server).
I got around the problem by turning off all Connection Pooling, at the application level, AND at the driver level.
Things started consistently working after making my /etc/odbcinst.ini look like:
[FreeTDS]
Description = TDS driver (Sybase/MS SQL)
# Some installations may differ in the paths
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
CPReuse =
CPTimeout = 0
FileUsage = 1
Pooling = No
The key being setting CPTimeout (Connection Pool Timeout) to 0, and Pooling to No. Just turning pooling off at the application level (in SQL Alchemy) did not work, only after setting it at the driver level did things start working smoothly.
I am now at 4 days without a problem after changing that setting.
I am working on a program to automate parsing data from XML files and storing it into several databases. (Specifically the USGS realtime water quality service, if anyone's interested, at http://waterservices.usgs.gov/rest/WaterML-Interim-REST-Service.html) It's written in Python 2.5.1 using LXML and PYODBC. The databases are in Microsoft Access 2000.
The connection function is as follows:
def get_AccessConnection(db):
connString = 'DRIVER={Microsoft Access Driver (*.mdb)};DBQ=' + db
cnxn = pyodbc.connect(connString, autocommit=False)
cursor = cnxn.cursor()
return cnxn, cursor
where db is the filepath to the database.
The program:
a) opens the connection to the database
b) parses 2 to 8 XML files for that database and builds the values from them into a series of records to insert into the database (using a nested dictionary structure, not a user-defined type)
c) loops through the series of records, cursor.execute()-ing an SQL query for each one
d) commits and closes the database connection
If the cursor.execute() call throws an error, it writes the traceback and the query to the log file and moves on.
When my coworker runs it on his machine, for one particular database, specific records will simply not be there, with no errors recorded. When I run the exact same code on the exact same copy of the database over the exact same network path from my machine, all the data that should be there is there.
My coworker and I are both on Windows XP computers with Microsoft Access 2000 and the same versions of Python, lxml, and pyodbc installed. I have no idea how to check whether we have the same version of the Microsoft ODBC drivers. I haven't been able to find any difference between the records that are there and the records that aren't. I'm in the process of testing whether the same problem happens with the other databases, and whether it happens on a third coworker's computer as well.
What I'd really like to know is ANYTHING anyone can think of that would cause this, because it doesn't make sense to me. To summarize: Python code executing SQL queries will silently fail half of them on one computer and work perfectly on another.
Edit:
No more problem. I just had my coworker run it again, and the database was updated completely with no missing records. Still no idea why it failed in the first place, nor whether or not it will happen again, but "problem solved."
I have no idea how to check whether
we have the same version of the
Microsoft ODBC drivers.
I think you're looking for Control Panel | Administrative Tools | Data Sources (ODBC). Click the "Drivers" tab.
I think either Access 2000 or Office 2000 shipped with a desktop edition of SQL Server called "MSDE". Might be worth installing that for testing. (Or production, for that matter.)