we are trying to make an api server for our project.
we are using mongodb with pymongo on debian boxes. everyhting is up to
date.
but we are having a really weird connection problem. there is
generally more than 15k-32k connections to mongodb port when i check
with
root#webserver1:/# netstat -na | grep mongo_db_ip | wc -l
i got 15363
connections are at TIME_WAIT state...
but when i check mongo, i only see 5-6 connections at the moment...
we wrote a mongodb class, that creates an instance and makes
connection. we tried call conn.disconnect() or conn.end_request()
everytime query end but it is not stoped that high connection
number...
is there anybody can tell what should be my mistake, or is there any
written python class for mongodb to examine how others make such as
stuff...
thanks for help, and information...
TIME_WAIT is not an open connection. It's an Operating System state for a socket so that it can make sure all data has come through. AFAIK, the default length for this on Linux is a minute. Have a look at http://antmeetspenguin.blogspot.com/2008/10/timewait-in-netstat.html, it has a good explanation. You can tell the kernel to reuse the TIME_WAIT sockets though:
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
reduces it to 30 seconds.
However, you should be checking why you are making so many connections. You're saying you're using the Debian packages for mongod and pymongo, and they tend to be out of date. You really want to be running mongod 2.0.2 and pymongo 2.1.1.
Related
I am running a python2.7 application that performs "inserts" into a single mysql/mariadb instance on a multi-core 64 bit CentOS(or ubuntu) machine. as soon as the parallel processes/cores exceed 4 or maybe 6, I see this error. (at different points in the execution)
2003: Can't connect to MySQL server on '127.0.0.1:3306' (99 Cannot assign requested address)
I am running the application on CentOS6.5, mariadb 10.1
I have also tried with Ubuntu 14.04 (64 bit), mysql resulting in the same problem.
I tried making the following changes:
In my.cnf file:
[mysqld]
interactive_timeout=1
wait-timeout = 1
thread_cache_size = 800
max_connections = 5000
#max_user_connections = 5000
max_connect_errors = 150
In sysctl.conf file:
fs.file-max = 65536
In limits.confg file:
* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535
I am inclined to think that this is a configuration issue, because the code runs just fine on 2 core Mac. Can someone suggest some configuration tweaks or any easy way to reuse connections?
You're probably connecting/disconnecting mysqld on a pretty high rate?
And when hitting error 99 you're probably seeing a lot of connections in TIME_WAIT state in netstat -nt output?
Problem most likely is that you are running out of client ports pretty quick due to frequent reconnects and the TIME_WAIT delay. This would also explain why you are more likely to run into this the higher your number of parallel clients is.
The TL;DR solution may be to set net.ipv4.tcp_tw_reuse to 1, e.g. using
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
or, as you have clients and mysql server on the same machine anyway, you could use UNIX domain socket connections instead of TCP. This may be as simple as connecting to verbatim "localhost" host name instead of 127.0.0.1, but I don't know about the various Python connectors and how these handle this ...
For more detailed tips and explanations see
http://www.fromdual.com/huge-amount-of-time-wait-connections
and
http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html
In a more recent environment, I had the same error message when trying to access MariaDB in a Docker container behind a Traefik 2.0-beta reverse proxy.
The error message has no sense compared to what occurred for real that's why I add my 2 cents.
And the solution I found was to use the name of the Docker service as host instead of the local domain name I gave to my MariaDB server with Traefik.
Some new inputs may be available here https://github.com/jclaveau/docker-standalone-webdev-ci-stack/tree/master/examples/docker-example-lamp
you can use sleep mechanism to solve this issue.
import time
time.sleep(2) ## sleep for two seconds.
this will reduce the number of processes in wait state.
I wonder how does Postgres sever determine to close a DB connection, if I forgot at the Python source code side.
Does the Postgres server send a ping to the source code? From my understanding, this is not possible.
PostgreSQL indeed does something like that, although it is not a ping.
PostgreSQL uses a TCP feature called keepalive. Once enabled for a socket, the operating system kernel will regularly send keepalive messages to the other party (the peer), and if it doesn't get an answer after a couple of tries, it closes the connection.
The default timeouts for keepalive are pretty long, in the vicinity of two hours. You can configure the settings in PostgreSQL, see the documentation for details.
The default values and possible values vary according to the operating system used.
There is a similar feature available for the client side, but it is less useful and not enabled by default.
When your script quits your connection will close and the server will clean it up accordingly. Likewise, it's often the case in garbage collected languages like Python that when you stop using the connection and it falls out of scope it will be closed and cleaned up.
It is possible to write code that never releases these resources properly, that just perpetually creates new handles, something that can be problematic if you don't have something server-side that handles killing these after some period of idle time. Postgres doesn't do this by default, though it can be configured to, but MySQL does.
In short Postgres will keep a database connection open until you kill it either explicitly, such as via a close call, or implicitly, such as the handle falling out of scope and being deleted by the garbage collector.
The Python elasticsearch client in my applicaiton is having connectivity issues (refused connections) because idle TCP connections timeout due to a firewall (I have no way to prevent this).
The easiest way for me to fix this would be if I could prevent the connection from going idle by sending some data over it periodically, the sniffing options in the elasticsearch client seem ideal for this, however they're not very well documented:
sniff_on_start – flag indicating whether to obtain a list of nodes
from the cluser at startup time
sniffer_timeout – number of seconds
between automatic sniffs
sniff_on_connection_fail – flag controlling
if connection failure triggers a sniff
sniff_timeout – timeout used for the sniff request - it should be a fast api call and we are talking potentially to more nodes so we want to fail quickly. Not used during initial sniffing (if sniff_on_start is on) when the connection still isn’t initialized.
What I would like is for the client to sniff every (say) 5 minutes, should I be using the sniff_timeout or sniffer_timeout option? Also, should the sniff_on_start parameter be set to True?
I used the suggestion from #val and found that these settings solved my problem:
sniff_on_start=True
sniffer_timeout=60
sniff_on_connection_fail=True
The sniffing puts enough traffic on the TCP connections so that they are never idle for long enough for our firewall to kill the conneciton.
I have a python client that needs to talk to a remote server I manage. They communicate using zeromq. When I tested the client/server locally everything worked. But now I have the client and server deployed on the cloud, each using a different provider. My question is, what's the simplest way (that is safe) to make the connection? I'm assuming I can't pass the password over, and even if I could I'm guessing there are safer alternatives.
I know how to set an ssh connection without a password using ssh-keygen. Would that work? Would the client need to make an ssh connection with the server before sending the tcp req? If there's a python library that helps with this it'd be a big help.
Thanks!
Update:
So more than 24 hours passed and no one replied/answered. I think I'm getting closer to solve this, but not quite there yet. I added my client's key to .ssh/authorized_key on the server, and now I can ssh from the client to the server without a password. Next, I followed this post about "Tunneling PyZMQ Connections with SSH". Here's what I have in my client code:
1 context = zmq.Context()
2 socket = context.socket(zmq.REQ)
3 socket.connect("tcp://localhost:5555")
4 ssh.tunnel_connection(socket, "tcp://locahost:5555", "myuser#remote-server-ip:5555")
5 socket.send_string(some_string)
6 reply = socket.recv()
This doesn't work. I don't really understand lines 3 & 4 and I assume I do something wrong there. Also, my server (hosted on linode) has a "Default Gateway" IP and a "Public IP" -- in the tunnel connection I only specify the public ip, which is also the ip I use to ssh to the machine.
Indeed, ZMQ way is - tunnelling connection with the SSH. Your example is exactly what needs to be done, except that one should either use connect or tunnel_connection, not both.
Also, when specifying server to connect to, make sure to define the SSH port, not the ZMQ REP socket port. That is, instead of myuser#remote-server-ip:5555 you might try myuser#remote-server-ip or myuser#remote-server-ip:22.
import zmq
import zmq.ssh
context = zmq.Context()
socket = context.socket(zmq.REQ)
zmq.ssh.tunnel_connection(socket, "tcp://locahost:5555", "myuser#remote-server-ip")
socket.send(b"Hello")
reply = socket.recv()
Finally, make sure you've installed either pexpect or paramiko - they will do the tunnelling actually. Note that if you're using Windows, paramiko is the only solution which will work - pexpect openssh tunnelling won't work on Windows.
If you use paramiko instead of pexpect, make sure to set paramiko=True in the tunnel_connection arguments.
I have found ssh in Python to be iffy at best, even with paramiko and fabric libraries, so to debug, you might try setting up a tunnel separately, just to see if that's the issue with the broken connection.
For example:
ssh myuser#remote-server-ip -L 5050:localhost:5555 -N
This says: connect to myuser#remote-server-ip, and whenever I request a connection to localhost:5050 on my machine, forward it across the ssh connection so that the server at remote-server-ip thinks it's receiving a connection from localhost:5555.
-L constructs the tunnel, and -N means don't do anything else on the connection.
With that running in another shell, e.g., a different Terminal window, on your local development machine, try to connect to a zeromq server at localhost:5050, which will actually be the zeromq running on the remote server.
You could use 5555:localhost:5555 in the ssh command above, but I find that can be confusing and often conflicts with a local copy of the same service.
After accessing my web app using:
- Python 2.7
- the Bottle micro framework v. 0.10.6
- Apache 2.2.22
- mod_wsgi
- on Ubuntu Server 12.04 64bit; I'm receiving this error after several hours:
OperationalError: (2006, 'MySQL server has gone away')
I'm using MySQL - the native one included in Python. It usually happens when I don't access the server. I've tried closing all the connections, which I do, using this:
cursor.close()
db.close()
where db is the standard MySQLdb.Connection() call.
The my.cnf file looks something like this:
key_buffer = 16M
max_allowed_packet = 128M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
It is the default configuration file except max_allowed_packet is 128M instead of 16M.
The queries to the database are quite simple, at most they retrieve approximately 100 records.
Can anyone help me fix this? One idea I did have was use try/except but I'm not sure if that would actually work.
Thanks in advance,
Jamie
Update: try/except calls didn't work.
This is MySQL error, not Python's.
The list of possible causes and possible solutions is here: MySQL 5.5 Reference Manual: C.5.2.9. MySQL server has gone away.
Possible causes include:
You tried to run a query after closing the connection to the server. This indicates a logic error in the application that should be corrected.
A client application running on a different host does not have the necessary privileges to connect to the MySQL server from that host.
You have encountered a timeout on the server side and the automatic reconnection in the client is disabled (the reconnect flag in the MYSQL structure is equal to 0).
You can also get these errors if you send a query to the server that is incorrect or too large. If mysqld receives a packet that is too large or out of order, it assumes that something has gone wrong with the client and closes the connection. If you need big queries (for example, if you are working with big BLOB columns), you can increase the query limit by setting the server's max_allowed_packet variable, which has a default value of 1MB. You may also need to increase the maximum packet size on the client end. More information on setting the packet size is given in Section C.5.2.10, “Packet too large”.
You also get a lost connection if you are sending a packet 16MB or larger if your client is older than 4.0.8 and your server is 4.0.8 and above, or the other way around.
and so on...
In other words, there are plenty of possible causes. Go through that list and check every possible cause.
Make sure you are not trying to commit to a closed MySqldb object
An answer to a (very closely related) question has been posted here: https://stackoverflow.com/a/982873/209532
It relates directly to the MySQLdb driver (MySQL-python (unmaintained) and mysqlclient (maintained fork)), but the approach is the the same for other driver the does not support automatic reconnect.
For me this was fixed using
MySQLdb.connect("127.0.0.1","root","","db" )
instead of
MySQLdb.connect("localhost","root","","db" )
and then
df.to_sql('df',sql_cnxn,flavor='mysql',if_exists='replace', chunksize=100)