We Keep Coding

ZMQError: Address already in use while using sockets in loops

I am trying to "simulate message passing between 6 nodes in a distributed environment" using Zero MQ in Python, in particular with the classic client/server architecture with REQ and REP. My idea is, while using a TCP/IP connection between these nodes, in the first iteration node-1 must be the server and the clients are the other nodes. In the next one, node-2 will be server and the rest (including node-1) should be clients and so on. At every iteration, server tells that it has established itself and clients send requests to the server to which an acknowledgement is sent back. Once the ACK has been received, the clients send their "MESSAGE" to the server (which is of course viewed as output) and we move to the next iteration.
The problem now is that I am facing the well known ZMQError: Address already in use
I'm not sure if it's due to the socket binding. I have added a socket.close() and context.term() into both client and server functions, but in vain.
And when I somehow try to run the code the VM goes into deadlock and I'm unable to recover unless I perform a hard reboot. Here is a snippet of my code -
#staticmethod
def server(node_id):
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:%s" % port_s)
print "Running server node %s on port: %s and value of server = __temp__" % (node_id, port_s)
message = socket.recv()
print "Received request : %s from c_node %s and value (temp): __value__" % (message, c_node)
socket.send("Acknowledged - from %s" % port_s)
time.sleep(1)
socket.close()
context.term()
#staticmethod
def client(c_node):
context = zmq.Context()
# print "Server node __num__ with port %s" % port_s
socket = context.socket(zmq.REQ)
#for port in ports:
socket.connect ("tcp://localhost:%s" % port_c)
#for request in range(20):
print "c_node %s Sending request to server node __num__" % c_node
socket.send ("Hello")
message = socket.recv()
print "Received ack from server %s and message %s" % (node_id, message)
time.sleep (1)
socket.close()
context.term()
def node(self, node_id):
#global node_id
# global key
# ser_p = Process(target=self.server, args=(node_id,))
print 'Memory content of node %d\n' % node_id
for key in nodes_memory[node_id]:
print 'Neighbor={%s}, Temp={%s}\n' % (key, nodes_memory[node_id][key])
#return key
global c_node
#key1 = key
# cli_p = Process(target=self.client, args=(c_node,))
with open("Book5.csv","r+b") as input:
has_header = csv.Sniffer().has_header(input.read(1024))
input.seek(0) # rewind
incsv = csv.reader(input)
if has_header:
next(incsv) # skip header
csv_dict = csv.DictReader(input, skipinitialspace=True, delimiter=",")
node_id = 0
for row in csv_dict:
for i in row:
#print(row[i])
if type(row[i]) is str:
g.add_edge(node_id, int(i), conn_prob=(float(row[i])))
max_wg_ngs = sorted(g[node_id].items(), key=lambda e: e[1]["conn_prob"], reverse=True)[:2]
#maxim = max_wg_ngs.values.tolist()
#sarr = [str(a) for a in max_wg_ngs]
print "\nNeighbours of Node %d are:" % node_id
#print(max_wg_ngs)
ser_p = multiprocessing.Process(target=self.server, args=(node_id,))
ser_p.start()
for c_node, data in max_wg_ngs:
for key in nodes_memory[node_id]: #print ''.join(str(item))[1:-1]
#if type(key1) == node_id:
cli_p = multiprocessing.Process(target=self.client, args=(c_node,))
cli_p.start()
print('Node {a} with Connection Rate = {w}'.format(a=c_node, w=data['conn_prob']))
print('Temperature of Node {a} = {b}'.format(a=c_node, b=nodes_memory[node_id][key]))
node_id += 1
pos=nx.spring_layout(g, scale=100.)
nx.draw_networkx_nodes(g, pos)
nx.draw_networkx_edges(g,pos)
nx.draw_networkx_labels(g,pos)
#plt.axis('off')
#plt.show()
The "message" is "temperature" (file not shown in the code snippet, but not needed at the moment) and for reference the values of Book5.csv are -
0,1,2,3,4,5
0,0.257905291,0.775104118,0.239086843,0.002313744,0.416936603
0.346100279,0,0.438892758,0.598885794,0.002263231,0.406685237
0.753358102,0.222349243,0,0.407830809,0.001714776,0.507573592
0.185342928,0.571302688,0.51784403,0,0.003231018,0.295197533
0,0,0,0,0,0
0.478164621,0.418192795,0.646810223,0.410746629,0.002414973,0
ser_p and cli_p are objects for the server and client functions which are called in the node function, i.e ser_p is called in the loop for row in csv_dict and cli_p is called even further in for c_node, data in max_wg_ngs. I'm using Networkx Python library here as well (Only to find 2 nearest neighbours out of the clients using the connection probability values from Book5.csv).
Does anyone know where I might be going wrong? Why is it that it shows address is already in use even though the socket is closed at every iteration?
Thanks a lot in advance :) (Using Ubuntu 14.04 32-bit VM)

StackOverflow has agreed to use an MCVE-based question asking:
Would you mind to reach that level and complete the missing part of the MCVE -- neither port_c, nor port_s are lexically correct ( never defined, elsewhere ).
If the code presented here works with a file, always kindly prepare such Minimum-version of the file, that will ensure the MCVE-code work as your expect it to work with the file. Statements like: "file not shown in the code snippet, but not needed at the moment" are not compatible with StackOverflow MCVE-rules.
Next, analyse the logic.
If multiple processes try to .bind() onto a same port# set via port_s, they simply will ( and have to ) collide and fall into an exception with ZMQError: Address already in use. First reboot the O/S, next pre-scan the already used IP:port#-s, next setup the non-colliding server-side .bind() ( there could be still a hanging .context() with non-terminated .socket() instance(s) ( typically from manual prototyping or from unhandled exceptions ), that keep the IP:port# without making it free. So the reboot + port scan is a way ahead.
Use any deterministic, principally non-colliding server-2-<transport-class://address:port> mapping ( .bind()-s using wildcards, locking on all IP-adresses, is a bit dangerous habit ) & your code will work smooth.
Always use <socket>.setsockopt( zmq.LINGER, 0 ) so as to prevent infinite deadlocks.
Always use try: {...} except: {...} finally: {...} formal expressions, so as to avoid any unhandled exceptions to orphan any .context() instance outside of your control, perhaps without a graceful .term()-ination and release ( even if new API tells you, that this is not necessary - it is professional to explicitly handle these situations and remain in control, so no exceptions, no excuse ).

List appending during loop seems to be optimized out in python?

I have a client-server application, with the client gathering some information, and having it json serialized and sent to the server, who will later store it in a database. In the server's tcp serving module, I have code that loads json objects from strings in a list and then appends them to another list like this (look at the loop inside bind_and_listen, which begins with for client in clients):
""" TCP Server before the main database. """
import socket
import sys
import logging
import json
import structs.client_data as cl_data
import db.register as register
cursor = register.cursor
client_list = []
domain_list = []
# connection constants
HOST = None
PORT = None
def config_server(host, port):
""" Configure the server's address and port. """
if not host:
logging.warning("HOST not supplied in the configuration file.")
sys.exit(-1)
if not port:
logging.warning("PORT not supplied in the configuration file.")
sys.exit(-1)
global HOST
global PORT
HOST = host
PORT = int(port)
def handle_files_question(request):
""" Read the file policy from the database for the server that sent it.
Params:
#request: The request as sent from the server.
"""
index = request.find(':')
hostname = request[index + 1:]
return register.query_file_policy(hostname)
def handle_size_question(request):
""" Read the size policy from the database for the server that sent it.
Params:
#request: The request as sent from the server.
"""
index = request.find(':')
hostname = request[index + 1:]
return register.query_size_policy(hostname)
def bind_and_listen():
""" Bind the server to the socket, and listen for incoming connections. """
serv = None
# create the socket
try:
serv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, v:
message = "Failed to create a socket: Message is {} and error code is {}"
errno, errormsg = v # is a (errno, message) tuple
message = message.format(errormsg, errno)
logging.error(message)
print message
sys.exit(-1)
# bind out socket to the address
try:
serv.bind((HOST, PORT)) # the double parens are to create a tuple with one element
except socket.error, msg:
message = "Binding of the socket failed. Error Code : {} Message: {}."
message = message.format(msg[0], msg[1])
logging.error(message)
print message
sys.exit(-2)
serv.listen(5) # 5 is the maximum number of queued connections we will allow
logging.info("Server is now listening on port %d" % (PORT,))
# we have the try block here only to facilitate the finally block
# that follows, so that we can reliably close the database everytime
# we have to close the program, but we don't have an exception raised
# everytime a client disconnects, the db is closed, and the networking
# module attempts to use the cursor of the expired connection.
try:
while True:
conn, addr = serv.accept()
hostname = None
output = str()
# this is the code handling the first policy question, regarding files
# 256 bytes should be enough to specify the question and hostname
res = conn.recv(256)
output = res
if output.startswith('files'):
files = handle_files_question(output)
conn.send(str(files))
# reset the output string
output = str()
# repeat the above procedure for the size policy question
res = conn.recv(256)
output = res
if output.startswith('size'):
size = handle_size_question(output)
conn.send(str(size))
# reset the output string
output = str()
# handle the rest of the data that the client sends
# (mainly, the clients and domains lists)
while True:
res = conn.recv(1024)
if not res:
break
output += res
# at this point, we assume that we have the full output string
# so we will split it to a list of strings representing the json
# representation of our clients
output = output.split('---')
domains_index = output.index('Domains_list')
clients = output[0:domains_index - 1] # -1 to compensate for a newline in the list
print clients
domains = output[domains_index + 1:-1]
hostname = output[0].split()[1]
clients = clients[2:-1] # ignoring the hostname and the Clients_list delimiter
for client in clients:
jason = json.loads(client)
client = cl_data.Client(jason['client_id'], jason['login'], jason['pname'], jason['email'])
client.files = jason['files']
client.size = jason['size']
client.size_human = jason['size_human']
client.domains = jason['domains']
client.domain_num = jason['domain_num']
client_list.append(client)
# the code below seems to prevent the problem?
#for client in client_list:
# print client.pname
for domain in domains:
jason = json.loads(domain)
domain = cl_data.DomainName(jason['domain_id'], jason['name'], jason['client_id'])
domain_list.append(domain)
conn.close()
register.update_clients(hostname, client_list)
register.update_domains(hostname, domain_list)
logging.info("Connection from %s is terminated." % (addr,))
except KeyboardInterrupt:
message = "\nCtrl + C was pressed, and an interruption signal has been registered."
message += " Terminating the server now."
logging.info(message)
print message
finally:
register.close_connection()
All well and good at this point. After I am finished with the loop I use the client_list to update an SQL database with this code:
register.update_clients(hostname, client_list)
The thing is, if I add something like print len(client_list) before the above code, I am getting 1 as a result, unless I do something like:
for client in client_list:
print client.email
in which case, I get the list with the whole number of objects that should be in it (about 180 or so).
From the above, I am gathering that the list appending gets optimized out for some reason that I can not comprehend, unless I force the list to be iterated immediately after the first loop has ended. Does anyone else have any insight on this? Is this standard behavior and I am doing something wrong? Is this a bug or something?
This is python 2.7.9 on debian 8 (jessie) according to the interpreter and /etc/os-release.
(I know I could use the dis module to investigate it further, but the way the module is written presents me with some ImportErrors when I try to import it to a python interpreter)

Python: socket.recv() doesn't receive push messages

Python: socket.recv() doesn't receive push messages
Hello,
I'm coding a socket based IMAP client in Python3 which successfully establishes a connection to the server, succussfully transmits the IDLE command but then fails to receive incoming data from the server.
If you are wondering why I do not use libimap or sth., the answer is easy: I just want to implement an IDLE command-supporting python client which must be written without that library.
An extract:
import socket
def runIMAPPeek():
#socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(29 * 60)
#connection
s.connect((IMAP_SERVER , 443))
#login
data = b"a1 LOGIN " + USER + b" " + PASSWORD + b"\n"
s.sendall(data)
reply = read(s)
#Idle loop
#As in RFC 3501 connection will be reinitialized every 29 minutes
while True:
# Idle command
print("#Sending IDLE...")
data = b"a2 IDLE\n"
s.sendall(data)
reply = read(s)
if reply.startswith("+ idling"):
print(" #idling.")
else:
print(" #Unexpected answer: {}".format(reply))
#sys.exit()
# waiting for incoming mails ----------------------------------
try:
push_msg = read(s)
# got push message = new message arrived
getNewEnvelope(s, push_msg)
except socket.timeout:
# timeout
print(" #timeout. Reinitializing IDLE...")
#TODO: except (KeyboardInterrupt, SystemExit)
# Quit Idle
data = b"DONE\n"
write(s, data)
reply = read(s)
if reply.startswith(prefix_str + " OK"):
print(" #quit idling.")
else:
print(" #Unexpected answer: {}".format(reply))
#sys.exit()
def read(s):
"""Read socket data, print it, convert to string, replace new lines
and return it.
"""
print("#Receiving...", end=" ")
reply = s.recv(4096)
reply = str(reply)[2:-1] #convert and remove byte indicators
reply = reply.replace("\\r\\n", "\n")
print(reply)
return reply
The problem is marked with the "----". Although messages are received in the mailbox, python does not react but remains in the idling/receiving state. In fact, the print line above the s.recv() command isn't even printed.
I tried everything successfully with Telnet, so there is no server problem.

In addition to my comment above, you have never selected INBOX. You will not receive any push messages, because you haven't told it what folder you want. Technically, IDLE is not valid in the unselected state.

Constructs like this one:
if reply.startswith("+ idling"):
are completely non-compliant. The IDLE RFC specifies that the client shall expect a continuation request, not this particular string (which also happens to be a continuation request).

Killing a thread with a socket in it (python)

I am pretty new at this.I am trying to build a server(chat server)
Sorry for presenting such a messing code.
There are alot of things that i am going to change about this code.
but as of now i just need help with one thing:
when i start let say more then one cleints on this ...and then just close the client i get this message:
Unhandled exception in thread started by
i have tryed to kill the thread as you can see in many places in this code. but i don't know what i am doing wrong ..
i am new at this.
any syggestions on what i should do ?
#encoding: utf-8
import socket, random, time, thread, errno, traceback
print socket.gethostname()
print "current machines IP address: "+socket.gethostbyname(socket.gethostname())
host ="10.0.0.1"# raw_input("IP: ")
# = socket.gethostbyname(socket.gethostname())
port = 12345
print host
conn_list =[None]*10
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((host, port))
sock.listen(10)
print "connected..\n\n\n\n"
def recv(conn):
while True:
try:
message = conn.recv(1024)
if "MESG" == message[1:5]:
message = message[6:].split(':')
name = str(conn)
conn_number = conn_list.index(conn)
conn_name = str(conn_number)
message = message[2]
reciever = message[0:1]
reciever = int(reciever)
for conn in conn_list:
if reciever == conn_list.index(conn):
conn.send(message)
print "Connection "+conn_name+" -----> "+str(reciever)+" :"+message+"\n"
#conn = findTheRightConnection(conn_list, conn_number)
break
else:
pass
except ValueError:
print "ValueError by %s" % (str(conn))
print conn.send("\nOpps you are not typing the correct connection number infront your message!")
except IOError:
bye(conn,conn_list)
print"Going to try to kill the thread here "
thread.quit()
thread.isAlive()
print "Still alive..."
except socket.error, v:
errorcode=v[0]
bye(conn,conn_list)
print"Going to try to kill the thread or here"
thread.quit()
thread.isAlive()
print "Still alive..."
except Exception, e:
traceback.print_exc()
finally:
thread.isAlive()
print"\nChanging the conn back to what it was... "
conn = findTheRightConnection(conn_list, conn_number)
def handle_connection(conn):
try:
recv(conn)
except socket.error, v:
errorcode=v[104]
bye(conn)
def bye(conn,conn_list):
i= 0
print "bye"
connectionName = str(conn_list.index(conn))
conn.close
conn_list = conn_list
print conn_list
for element in conn_list:
if element == conn:
conn_list[i] = None
break
i += i
print "Connection number "+connectionName+" is terminated"
print conn_list
return "Connection Terminated"
def welcome(conn,conn_list):
i = 0
for element in conn_list:
if element == None:
conn_list[i] = conn
print "Connection added in the conn_list on the slot %d" % (i)
print conn_list
return conn_list
else:
i = i+1
pass
print "The server if full! No more space left"
return conn_list
def findTheRightConnection(conn_list, number):
for conn in conn_list:
if number == conn_list.index(conn):
return conn
else:
pass
print "\nSomthing went wrong while trying to find the right connection in the method findTheRightConnection()"
return
while True:
conn, addr = sock.accept()
conn_list = welcome(conn,conn_list)
print "Got connection from : "+str(addr[0])+" by connection number: "+str(conn_list.index(conn))+"\n\n\n\n"
msg = "Welcome to the server!"
conn.send(":INFO:"+str(int(time.time()))+":"+str(len(msg))+":"+msg)
thread.start_new_thread(handle_connection, (conn,))

If you are still having trouble creating a instant messaging program in Python, you might be interested in this answer to another question.
Simple_Server.py is a minimal server implementation. A far more complex server with a variety of features can be provided on request. The complex server supports authentication, friends, private messaging, channels, filters, math evaluation, and admin controls.
MultichatClient.py is a port of a Java program written by a teacher from a networking class. The program must be run from the command line, and it must be given the server as an argument. You can use either the server's name on the network or its IP address.
Simple_Client.pyw is a more complicated client that does not require being started from the command line. When it starts, it will ask for server's name and try connecting to it while showing a progress dialog. The program will automatically try logging any errors to a file.
affinity.py is required for threadbox.py to run. (runs code on a specific thread regardless of origin)
threadbox.py is required for safetkinter.py to run. (metaclass clones classes to run using affinity)
safetkinter.py is required for Simple_Client.pyw to run. (makes tkinter safe to use with threads)

Assist with Python script

I have written a code in Python, and I want to change it.
I use it when I am performing a penetration tests in my organization and I want to make my script better.
The script gets a username that I entering and it connect to the SMTP server over port 25 and check if the user exists or not.
Here is the script:
#!/usr/bin/python
import socket
import sys
if len(sys.argv) != 2:
print "Usage: vrfy.py <username>"
sys.exit(0)
# Create a Socket
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Connect to the Server
connect=s.connect(('192.168.0.10',25))
# Recieve the banner
banner=s.recv(1024)
print banner
# VRFY a user
s.send('VRFY ' + sys.argv[1] + '\r\n')
result=s.recv(1024)
print result
# Close the socket
s.close()
The changes that I want to perform are the following:
Instead of entering only one username to check, I want to mention a location of a txt file and the script will read all the usernames it contains.
Also, I what that the IP address at the script wont be hard coded and I will need to insert it every time I am using the script.
Many thanks in advance,
Bar Aviv

You're not really supposed to use the low-level socket send() and recv() directly like that. They're not guaranteed to send/receive any particular amount of data. It might just happen to work the way you want talking to one particular mail server, but as soon as commands/responses don't fit one-to-one into IP packets, you're going to get weird results.
If you want a simple blocking conversational stream, try the file interface you get from socket.makefile.
You should probably implement a little more of SMTP as well... at least say helo!
The below reads usernames from standard input, so you can direct a file of usernames to it by saying:
python vrfy.py 127.0.0.1 < usernames.txt
though you could change it to read sys.argv[2] for another filename parameter if you wanted.
users= []
for line in sys.stdin:
line= line.strip()
if line!='':
users.append(line)
s= socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((sys.argv[1], 25))
fp= s.makefile('rwb')
fp.readline() # ignore banner
fp.write('HELO test.example.com\r\n')
fp.flush()
fp.readline() # ignore response
for user in users:
fp.write('VRFY %s\r\n' % user)
fp.flush()
print '%s: %s' % (user, fp.readline().strip())
fp.write('QUIT\r\n')
fp.flush()
s.close()

Use something like the following:
with open(sys.argv[1]) as f:
for line in f:
username = line.replace('\n', '')
print 'Testing with user: %s' % user
For the IP address, you can pass it as a second argument -- verify.py username-list-file server-ip-address -- and reference as sys.argv[2].
You can also do something like:
try:
userlist = sys.argv[1]
server_address = sys.argv[2]
except IndexError:
print 'Usage: verify.py username-list-file server-ip-address'
sys.exit(-1)