I am using Scrapy 2.4.x pipeline.py to write data sets to a remote MySQL 5.7.32 server. In some cases errors happen and the script throws an exception - which is OK.
for selector in selectors:
unit = unitItem()
try:
unit['test'] = selector.xpath('form/text()').extract_first()
if not unit['test']:
self.logger.error('Extraction failed on %s', response.url)
continue
else:
unit['test'] = str(unit['test'].strip()
except Exception as e:
self.logger.error('Exception: %s', e)
continue
# more code
yield unit
There are 2 problems:
RAM usage is climbing up constantly. Do I somehow need to destroy the item?
There are many MySQL abborted connections errors. I believe this is due to MySQL connection is not closed
mysql error Log:
Aborted connection 63182018 to db: 'mydb' user: 'test' host: 'myhost' (Got an error reading communication packets)
The connection got opened at the very beginning of process_item and get closed at the very end of the method.
Would it help to close the connection upon exception? If so, is there a recommended routine?
I believe it would be more effective to open SQL connection on spider_opened()
and close it on spider_closed()
The only thing to keep in mind is that spider_closed() signal is fired when spider is closed gracefully.
I have a celery task which takes about 6 hours. At the end of it, Django (or possibly Celery) raises an exception "MySQL server has gone away".
After doing some reading, it appears that this is a known issue with long tasks. I don't (think I have) control over pinging or otherwise mid-task; but the exception is raised after the call which takes time has finished (but still within the task function).
Is there a call I can make within the function to re-establish the connection?
(I have run this task "locally" with the same RDS MySQL DB and not had the issue, but I am getting it when running on an AWS instance.)
Eventually found what appears to have worked:
from django.db import close_old_connections
import time
def check_and_retry_django_db_connection():
close_old_connections()
db_conn = False
while not db_conn:
try:
connection.ensure_connection()
db_conn = True
except OperationalError:
print('Database unavailable, waiting 1 second...')
time.sleep(1)
print('Database available')
The key is the close_old_connections call - ensure_connection will not work otherwise.
Ian
I have a huge problem:
There seems to be some hardware problems on the router of the server my python software runs on. The connection to the database only is successfull about every third time. So a psycopg2.connect() can take up to 5 minutes before I get an timeout exception.
2014-12-23 15:03:12,461 - ERROR - could not connect to server: Connection timed out
Is the server running on host "172.20.19.1" and accepting
That's the code I'm using.
# Connection to the DB
try:
db = psycopg2.connect(host=dhost, database=ddatabase,
user=duser, password=dpassword)
cursor = db.cursor(cursor_factory=psycopg2.extras.DictCursor)
except psycopg2.DatabaseError, err:
print(str(err))
logging.error(str(err))
logging.info('program terminated')
sys.exit(1)
I tried some timeout additions for the query, but that didn't helped, since the connection didn't got established at all.
Is there a way, I can stop the program immediately, when the connection couldn't be established?
When using the keyword arguments syntax to the connect function it is possible to use any of the libpd supported connection parameters. Among those there is connect_timeout in seconds:
db = psycopg2.connect (
host=dhost, database=ddatabase,
user=duser, password=dpassword,
connect_timeout=3
)
http://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-PARAMKEYWORDS
http://initd.org/psycopg/docs/module.html
A connection time out raises an OperationalError exception.
We have a python application running with uwsgi, nginx.
We have a fallback mechanism for DBs. ie., if one server refuses to connect, we connect to the other server. But the issue is that the connection takes more than 60s to timeout.
As nginx times out in 60s, it displays the nginx error page. Where can we change the timeout for connecting to mysql servers so that we can make three attempts of connection to mysql in the given 60s nginx timeout period?
We use Web2py and default DAL object with pymysql adapter
you're talking about the option connect_timeout?
conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', passwd='', db='mysql', connect_timeout=20)
in DAL terms this option will be something about this (not tested)
db = DAL('mysql://username:password#localhost/test', driver_args={connect_timeout=20})
I'm having an issue connecting to my local MySQL database using Python's MySQLdb library. The script has been working well previously, but I will occasionally get the MySQL error in the title. There seems to be no explanation for when the error occurs, and the script is always run from the same machine with the same arguments.
The MySQL server is running as a service on Windows XP SP3 using port 3306 (locally hosted phpMyAdmin works), and the script is run from an Ubuntu 10.04 guest operating system in Oracle VM VirtualBox.
I am currently working around this issue by opening a command prompt and executing 'net stop MySQL' then 'net start MySQL'. This allows me to run the script a few times again before resulting in the error, which I've been fixing by restarting the MySQL service.
As I am still making changes to the script, there are occasions when the script raises an exception and doesn't exit gracefully, though I do catch the exception and close the cursor and connection.
The code to connect to the database:
def __init__(self):
try:
print "Connecting to the MySQL database..."
self.conn = MySQLdb.connect( host = "192.168.56.1",
user = "guestos",
passwd = "guestpw",
db = "testdb")
self.cursor = self.conn.cursor(MySQLdb.cursors.DictCursor)
print "MySQL Connection OK"
except MySQLdb.Error, e:
print "MySQLdb error %d: %s" % (e.args[0],e.args[1])
raise
The full error generated when this happens is as follows:
MySQLdb error 2013: Lost connection to MySQL server at 'reading initial communication packet', system error: 0
Traceback (most recent call last):
File "search.py", line 45, in <module>
dataHandler = DataHandler()
File "/home/guestos_user/workspace/Search/src/data_handler.py", line 25, in __init__
db = "testdb")
File "/usr/lib/pymodules/python2.6/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 170, in __init__
super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0")
sudo vi /etc/mysql/my.cnf
delete
bind-address = 127.0.0.1
then
sudo reboot now
That's it. Be aware that this will make your mysql server less secure as you are exposing it.
I have seen this happen when child processes try to share the same mysql connection id (solution = create new connections for each child process). I'm not sure if this is also possible when sharing connection objects with multiple threads.
However, that's only one of the many possible causes. See VVS's answer in MySQL Error 2013 for a list of troubleshooting resources.
Do you have in your MySQL server an acount called guestos#YOURIPADDRESS?
You must have an account to access to your MySQL server from YOURIPADDRESS!
For example:
Your IP address is 192.168.56.2; then you must create and account if not exist to access.
mysql> create user guestos#192.168.56.2 identified by 'guestpw';
The problem fixed for me just by restarting my mac. Though there might be a more specific fix for it.
I received a similar error when attempting to connect to my MySQL server remotely through a user with the sufficient permissions.
After editing the /etc/mysql/my.cnf file to include
[mysqld]
bind-address=xx.xx.xxx.xxx
where xx.xx.xxx.xxx is my local IP address, I began experiencing the exact same error as you. From there, I found an answer regarding this issue (answered by Coffee Converter) which worked for me, and can be found here: Lost connection to MySQL server at 'reading initial communication packet', system error: 0 on a windows machine
All I did to fix the issue for myself was edit the /etc/hosts.allow to include
mysqld: ALL: allow
Works great now! I hope this helped :)
Could you change the bind-address=localhost and restart MySQL server? Seems like this issue is related to yours: http://forums.mysql.com/read.php?152,355740,355742#msg-355742
Also this-
If MySQL port is wrong result is MySQL client error 2013 "Lost
connection ...". Note that this error also occurs if port forwarding
is disabled in SSH configuration (the configuration parameter
'AllowTcpForwarding' is set to 'no' in the 'sshd_config' file). It
(here) simply tells that there is no connection from SSH to MySQL for
some reason. But the mySQL client API 'thinks' there was one
connection and that is why is says 'Lost connection ...' and not
'Can’t connect...'. There was one successful connection - but not to
the MySQL server - to the SSH daemon only! But the MySQL client API is
not designed to 'see' the difference!
Refer this.
I run a windows server and from time to time the php-win.exe will load and stay in the processes list on the windows task manager.
If you know the host file is correct, then kill the php-win.exe process and restart iis iisreset
If you are running windows then your problem should be solved.
I've had the exact same mysql error (ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0=) and have resolved it by adding a newline to /etc/hosts.deny.
Possibility: your database is corrupted.
I encountered this situation when I was running an UPDATE statement on a specific row of a specific table. (Specifically, I was editing an item in a Django Admin site.) Most of the time the database worked just fine.
I finally resolved the problem by running:
OPTIMIZE TABLE `your_table`
After that everything was OK, no connection lost.
Conclusion:
The problem "Lost connection to MySQL server at 'reading initial communication packet'", sometimes "Can't connect to MySQL server on '127.0.0.1'", could possibly be resolved by running a full database optimization if the database is corrupted. For more info, read this.
Just to further extend the list of possible causes: it could also be as banal as wrong connection data/credentials. I encountered this error in conjunction with sqlalchemy:
sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
In my code I connect to several different databases and once in a while it happens that I don't get the mapping between the db connections and their credentials (e.g. ip address of server, db-name, password etc.) right, which then also results in the 2013-error (in this case wrapped into an sqlalchemy operational error).
setting.py file set like:
'ENGINE': 'django.db.backends.mysql',
'NAME': 'test2',
'USER': 'root',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '3308',
This bug report might be of interest to you. Don't know if this will help you, but some were able to solve it by using the name of the server rather than the ip address in the connection properties.