I trying to write a client server application in Python, but I faced a problem, in the client side I'm not getting all the sent data. First I tried to send the numbers 1 to 10 and I received 1,2,5,6,10, so there are missing a lot of numbers.
Server side:
def __init__( self ):
super( MCCommunication, self ).__init__()
HOST, PORT = socket.gethostbyname( socket.gethostname() ), 31000
self.server = SocketServer.ThreadingTCPServer( ( HOST, PORT ), MCRequestHandler )
ip, port = self.server.server_address
# Start a thread with the server
# Future task: Make the server a QT-Thread...
self.server_thread = threading.Thread( target = self.server.serve_forever )
# Exit the server thread when the main thread terminates
self.server_thread.setDaemon( True )
self.textUpdated.emit( 'Server Started!' )
print( 'Server Started!' )
self.server_thread.start()
def handle( self ):
#self.request.setblocking( 0 )
i = 10;
while True:
if( self.clientname == 'MasterClient' ):
try:
#ans = self.request.recv( 4096 )
#print( 'after recv' )
""" Sendign data, testing purpose """
while i:
mess = str( i );
postbox['MasterClient'].put( self.creatMessage( 0, 0 , mess ) )
i = i - 1
while( postbox['MasterClient'].empty() != True ):
sendData = postbox['MasterClient'].get_nowait()
a = self.request.send( sendData )
print( a );
#dic = self.getMessage( sendData )
#print 'Sent:%s\n' % str( dic )
except:
mess = str( sys.exc_info()[0] )
postbox['MasterClient'].put( self.creatMessage( 1, 0 , mess ) )
pass
def creatMessage( self, type1 = 0, type2 = 0, message = ' ', extra = 0 ):
return pickle.dumps( {"type1":type1, "type2":type2, "message":message, "extra":extra} );
Where the postbox['MasterClient'] is a Queue with the serialized message.
And this is the client:
def run( self ):
sock = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
addr = ( self.ip, self.port )
#print addr
sock.connect( addr )
#sock.setblocking( 0 )
while True:
try:
ans = sock.recv( 4096 )
dic = self.getMessage( ans )
self.recvMessageHandler( dic )
print 'Received:%s\n' % str( dic )
except:
pass
The server may have sent multiple messages by the time the client attempts to read them, and if these fit within the same 4k buffer, the recv() call will obtain both of them.
You don't show the getMessage code, but I'd guess you're doing something like pickle.loads(msg), but this will only give you the first message and discard the rest of the string, hence the dropped messages. You'll also get another issue if more than 4096 bytes are buffered by the time you read, as you could end up getting a fragment of a message and thus an unpickling error.
You'll need to break up the string you get back into seperate messages, or better, just treat the socket as a stream and let pickle.load pull a single message from it.
Related
I'm trying to create a simple multiplayer game in Python using pygame and socket modules. It just consists in two circles that are controlled with the W, A, S, D keys from two different computers.
At first I created a client with the recv() in the middle of the pygame loop. It worked well but the recv() blocked the loop, so the movement of the circles wasn't smooth and I had to increase the velocity to 6 (before it was set to 0.6) in order to have a normal speed. This is the code (I've summarised it a bit):
Client 1st version
#import modules
def main(sock):
pygame.init()
#Display screen and set initial me_x and me_y
vel = 0.6
while True:
keys_pressed = pygame.key.get_pressed()
#Change me_x and me_y (with A, D, W, S keys)
#Make sure me_x and me_y don't get off the screen
screen.fill(color)
if other_x and other_y:
pygame.draw.circle(screen, colorMe, (other_x, other_y), radi)
pygame.draw.circle(screen, colorOther, (int(me_x), int(me_y)), radi)
pygame.display.flip()
sock.send(int(me_x).to_bytes(3, byteorder = 'big') + int(me_y).to_bytes(3, byteorder = 'big'))
otherPos = sock.recv(BUFSIZ)
other_x = int.from_bytes(otherPos[:3], byteorder = 'big')
other_y = int.from_bytes(otherPos[3:], byteorder = 'big')
print(other_x, other_y)
#CONNECT TO TCP SOCKET
other_x = None
other_y = None
main(client_socket)
Then, I tried to put the recv() in a thread to stop blocking the loop:
Client 2nd version
#import modules
def main(sock):
pygame.init()
#Display screen and set initial me_x and me_y
vel = 0.6
while True:
for i in range(30):
keys_pressed = pygame.key.get_pressed()
#Change me_x and me_y (with A, D, W, S keys)
#Make sure me_x and me_y don't get off the screen
screen.fill(color)
if other_x and other_y:
pygame.draw.circle(screen, colorOther, (other_x, other_y), 15)
pygame.draw.circle(screen, colorMe, (int(me_x), int(me_y)), 15)
pygame.display.flip()
msg = int(me_x).to_bytes(3, byteorder = 'big') + int(me_y).to_bytes(3, byteorder = 'big')
sock.send(msg)
def recv_pos(sock):
while True:
other_pos = sock.recv(BUFSIZ)
other_x = int.from_bytes(other_pos[:3], byteorder = 'big')
other_y = int.from_bytes(other_pos[3:], byteorder = 'big')
print(other_x, other_y)
#CONNECT TO TCP SOCKET
other_x = None
other_y = None
receive_thread = threading.Thread(target = recv_pos, args = (client_socket,))
receive_thread.daemon = True
receive_thread.start()
main(client_socket)
However, when I start 2 instances of the client 2 in two different computers, it gives me an OverflowError:
OverflowError: Python int too large to convert to C long
I added the for i in range(30): because I thought that the server was getting collapsed because there were too much messages being sent at the same time. The output is the same: after 3 seconds more or less, the program crashes giving an OverflowError.
I added a print() statement in both versions just after the recv() to see which values of x and y I was receiving. In the 1st version, all of them were in the width and height range. However, in the 2nd version, 1/5 of the received messages was a big number such as 124381473265. If this number is even bigger, it gives the OverflowError. I don't understand why this is happening: I'm encoding and decoding the messages in the same way in both versions, but one works and the other doesn't.
I'm not including the server code because I don't think it's necessary. It's just just transmits the messages between both clients without modifying them. The server is not the error, as in the 1st client version the messages are sent and received properly.
Any help will be apprecieted. If you need more information about any point just tell me.
Thanks in advance!
Put your socket code into a thread, or use select.select() for non-blocking socket-reads. Then post custom event-messages back to the main loop when a data-gram arrives from the server.
import pygame
import enum
class NetworkEvents( enum.IntEnum ):
EVENT_HANGUP = pygame.USEREVENT + 1
EVENT_MESSAGE = pygame.USEREVENT + 2
Your socket-reading code should handle partial-packets, hang-ups and delays too. I think your error above is caused by trying to unpack a partial (or somehow junk) packet. Commonly people use the pickle module to encapsulate this data, but to begin with, I would do pre-testing with simple string data. It's easier to debug. Then once the transmission code is all bedded-down and tested, change to binary - if necessary.
I like to use select() in my socket code no matter what. It gives fine-grained control over what's happening with the socket.
Here's a simple threaded socket-conversation handler. It really only supports socket-hangup and incoming message. But it could be easily extended to handle further message types.
import threading
import pygame
import random
import enum
import socket
import select
import time
class ConversationHandlerThread( threading.Thread ):
""" A thread that handles a conversation with a single remote server.
Accepts commands of 'close', 'red', 'green' or 'blue', and posts messages
to the main PyGame thread for processing """
def __init__( self, server_address, server_port ):
threading.Thread.__init__(self)
self.server_address = server_address
self.server_port = server_port
self.server_socket = None
self.data_buffer = ''
self.daemon = True # exit with parent
self.done = False
def stop( self ):
self.done = True
def connect( self ):
self.server_socket = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
while True:
try:
self.server_socket.connect( ( self.server_address, self.server_port ) )
print( "Connected to %s:%d" % ( self.server_address, self.server_port ) )
break;
except:
print( "Failed to connect %s:%d" % ( self.server_address, self.server_port ) )
time.sleep( 12 )
print( "Retrying..." )
def run( self ):
""" Connects to Server, then Loops until the server hangs-up """
self.connect()
# Now we're connected, start reading commands
read_events_on = [ self.server_socket ]
while ( not self.done ):
# Wait for incoming data, or errors, or 0.3 seconds
(read_list, write_list, except_list) = select.select( read_events_on, [], [], 0.5 )
if ( len( read_list ) > 0 ):
# New data arrived, read it
incoming = self.server_socket.recv( 8192 )
if ( len(incoming) == 0):
# Socket has closed
new_event = pygame.event.Event( NetworkEvents.EVENT_HANGUP, { "address" : self.server_address } )
pygame.event.post( new_event )
self.server_socket.close()
self.done = True
else:
# Data has arrived
try:
new_str = incoming.decode('utf-8')
self.data_buffer += new_str
except:
pass # don't understand buffer
# Parse incoming message (trivial parser, not high quality)
# commands are '\n' separated
if (self.data_buffer.find('\n') != -1 ):
for line in self.data_buffer.split('\n'):
line = line.strip()
# client disconnect command
if ( line == 'close' ):
new_event = pygame.event.Event( NetworkEvents.EVENT_HANGUP, { "address" : self.server_address } )
pygame.event.post( new_event )
self.server_socket.close()
self.done = True
# only make events for valid commands
elif ( line in ( 'red', 'green', 'blue' ) ):
new_event = pygame.event.Event( NetworkEvents.EVENT_MESSAGE, { "address" : self.server_address, "message" : line } )
pygame.event.post( new_event )
self.data_buffer = '' # all used-up
The beauty of putting all this complexity into a thread, is that once it's started, you can forget about it.
# Start the network-handler thread
thread1 = ConversationHandlerThread( '127.0.0.1', 5555 )
thread1.start()
In your main loop, process the events like any others:
for event in pygame.event.get():
if ( event.type == pygame.QUIT ):
done = True
elif ( event.type == NetworkEvents.EVENT_HANGUP ):
print(" CLIENT DISCONNECTED %s " % ( str(event.address) ) )
elif ( event.type == NetworkEvents.EVENT_MESSAGE ):
print(" CLIENT MESSAGE FROM %s - %s " % ( str(event.address), event.message ) )
if ( event.message == 'red' ):
new_sprite = AlienSprite( RED )
SPRITES.add( new_sprite )
I have created simple client/server using pyzmq.
One thing I am not sure is the .recv() does not receive the message even though it has been sent from server. It just ignores it and throws an error which I find to be strange.
Client.py
try:
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:2222")
print("Sending request")
socket.send(b"send the message")
message = socket.recv(flags=zmq.NOBLOCK)
print("Received reply %s " % message)
except Exception as e:
print(str(e))
Server.py
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:2222")
while True:
message = socket.recv()
socket.send(b"Ack")
I think the client should receive the Ack and print it instead of throwing the exception.
The document says,
With flags=NOBLOCK, this raises ZMQError if no messages have arrived
Clearly the server is responding with "Ack" as soon as it receives the message.
The Error message is,
Resource temporarily unavailable
Remember that in concurrent environments there are no guarantees about the order of execution of independent processes. Even though you are responding immediately to the message in the server.py, the response may not get to the receiving socket before you call socket.recv. When you call socket.send the message needs to go over the network to your server, the server needs to create the message and respond and then the message needs to go back over the network to your client code. The time to send the messages over the network will be quite long and you are calling socket.recv immediately after socket.send.
So in fact when you call message = socket.recv(flags=zmq.NOBLOCK) the client socket will not have received the Ack from the server yet, and since you are using NOBLOCK an error is thrown since no messages have been received on the socket.
NOBLOCK is likely not appropriate in this scenario. You can experiment with this by adding a sleep call in between send and recv to show that the time delay of waiting for the response from the server is indeed the issue but that's not a good solution for your client code long-term.
If you want to exit after waiting for a certain amount of time you should use socket.poll instead.
event = socket.poll(timeout=3000) # wait 3 seconds
if event == 0:
# timeout reached before any events were queued
pass
else:
# events queued within our time limit
msg = socket.recv()
Pyzmq Doc for socket.poll()
Q : say the server is not up in that case the recv() in the client will be blocked forever which I don't want.
ZeroMQ is a fabulous framework for doing smart signaling/messaging in distributed-systems
Let's sketch a demo of a principally non-blocking modus-operandi, with some inspirations of how the resources ought be both acquired and also gracefully released before process termination.
Maybe a bit of reading about the main conceptual differences in ZeroMQ hierarchy in less than a five seconds will also help.
Server.py
aContext = zmq.Context()
aLightHouse = aContext.socket( zmq.PUB )
aRepSocket = aContext.socket( zmq.REP )
aRepSocket.setsockopt( zmq.LINGER, 0 )
aRepSocket.setsockopt( zmq.COMPLETE, 1 )
aRepSocket.bind( "tcp://*:2222" )
aLightHouse.bind( "tcp://*:3333" )
aLightHouse.setsockopt( zmq.LINGER, 0 )
aLightHouse.setsockopt( zmq.CONFLATE, 1 )
aLightHouse_counter = 0
#------------------------------------------------------------
print( "INF: Server InS: ZeroMQ({0:}) going RTO:".format( zmq.zmq_version() ) )
#------------------------------------------------------------
while True:
try:
aLightHouse_counter += 1
aLightHouse.send( "INF: server-RTO blink {0:}".format( repr( aLightHouse_counter ) ),
zmq.NOBLOCK
)
if ( 0 < aRepSocket.poll( 0, zmq.POLLIN ) ):
try:
message = aRepSocket.recv( zmq.NOBLOCK ); print( "INF: .recv()ed {0:}".format( message ) )
pass; aRepSocket.send( b"Ack", zmq.NOBLOCK ); print( "INF: .sent() ACK" )
except:
# handle EXC: based on ...
print( "EXC: reported as Errno == {0:}".format( zmq.zmq_errno() ) )
else:
# NOP / Sleep / do other system work-units to get processed during the infinite-loop
except:
# handle EXC:
print( "EXC: will break ... and terminate OoS ..." )
break
#------------------------------------------------------------
print( "INF: will soft-SIG Server going-OoS..." )
aLightHouse.send( "INF: server goes OoS ... " )
#------------------------------------------------------------
print( "INF: will .close() and .term() resources on clean & graceful exit..." )
Sleep( 0.987654321 )
aRepSocket.unbind( "tcp://*:2222" )
aRepSocket.close()
aLightHouse.unbind( "tcp://*:3333" )
aLightHouse.close()
aContext.term()
#------------------------------------------------------------
print( "INF: over and out" )
Client.py
try:
aContext = zmq.Context()
aReqSocket = aContext.socket( zmq.REQ )
aBeeper = aContext.socket( zmq.SUB )
aReqSocket.setsockopt( zmq.LINGER, 0 )
aReqSocket.setsockopt( zmq.COMPLETE, 1 )
aReqSocket.connect( "tcp://localhost:2222" )
aBeeper.connect( "tcp://localhost:3333" )
aBeeper.setsockopt( zmq.SUBSCRIBE, "" )
aBeeper.setsockopt( zmq.CONFLATE, 1 )
#------------------------------------------------------------
print( "INF: Client InS: ZeroMQ({0:}) going RTO.".format( zmq.zmq_version() ) )
#------------------------------------------------------------
try:
while True:
if ( 0 == aBeeper.poll( 1234 ) ):
print( "INF: Server OoS or no beep visible within a LoS for the last 1234 [ms] ... " )
else:
print( "INF: Server InS-beep[{0:}]".format( aBeeper.recv( zmq.NOBLOCK ) ) )
try:
print( "INF: Going to sending a request" )
aReqSocket.send( b"send the message", zmq.NOBLOCK )
print( "INF: Sent. Going to poll for a response to arrive..." )
while ( 0 == aReqSocket.poll( 123, zmq.POLLIN ) ):
print( "INF: .poll( 123 ) = 0, will wait longer ... " )
message = socket.recv( flags = zmq.NOBLOCK )
print( "INF: Received a reply %s " % message )
except Exception as e:
print( "EXC: {0:}".format( str( e ) ) )
print( "INF: ZeroMQ Errno == {0:}".format( zmq.zmq_errno() ) )
print( "INF: will break and terminate" )
break
except Exception as e:
print( "EXC: {0:}".format( str( e ) ) )
finally:
#------------------------------------------------------------
print( "INF: will .close() and .term() resources on clean & graceful exit..." )
aBeeper.close()
aReqSocket.close()
aContext.term()
#------------------------------------------------------------
print( "INF: over and out" )
you are using non-blocking mode which mean it will raise an error to inform you, that there's nothing that could be done with the message and you should try again later, but if you are using blocking mode it blocks until the peers connects.
this answer is from here
basically if you remove flags=zmq.NOBLOCK it will work.
Update
if you want to use non-blocking mode you should have a look to this
I've tried so many ways the past week or so to try to get this to work.
I've managed to get Skype2IRC to work in another script, and not spam IRC or anything. It was good. Just on message status, send message to IRC. That part I can get down easily. However, the part going from IRC to Skype is where I'm having issues. First, I figured I'd try to get it to work with multi-threading. Didn't go out well. Then, I tried this;
import os,sys,time,Skype4Py,socket,re,string
from random import choice
s = Skype4Py.Skype()
s.Attach() #Attach to Skype. Make sure to accept permissions from Skype client!
name = "Ibex"
network = "irc.myserver.net"
channel = "#Skype"
port = 6667
irc = socket.socket ( socket.AF_INET, socket.SOCK_STREAM )
irc.connect ( ( network, port ) )
print irc.recv ( 4096 )
irc.send ( 'NICK '+name+'\r\n' )
irc.send ( 'USER '+name+' '+name+' '+name+' : IRC Bot\r\n' )
irc.send ( 'JOIN '+channel+'\r\n' )
ircusername = '' #referencing variable so it can be used before, data will change in while loop
ircmessage = '' #same as above
def Ibex(Message, Status):
chat = Message.Chat
members = chat.MemberObjects
msg = Message.Body
send = Message.Chat.SendMessage
sAlias = Message.FromDisplayName
sUsername = Message.FromHandle
if Status == 'RECEIVED':
irc.send ( 'PRIVMSG '+channel+' :<Skype - '+sUsername+'>: '+msg+'\r\n' )
if ircusername != '':
try:
ircmessage = data.split(channel+" :",1)[1]
send("<IRC - "+ircusername+">: "+ircmessage)
except:
print "Error getting new IRC message."
s.OnMessageStatus = Ibex
while True:
data = irc.recv ( 4096 )
print data
if data.find ( 'PING' ) != -1:
irc.send ( 'PONG ' + data.split() [ 1 ] + '\r\n' )
if data.find ( 'KICK' ) != -1:
irc.send ( 'JOIN '+channel+'\r\n' )
try:
ircusername = re.search(':(.*)!', data)
ircusername = ircusername.group(1)
except:
pass
After tons of trial and error. And, a LOT of spam in IRC and Skype, I managed to get this to send IRC messages to skype.
HOWEVER, it spams.
I'll send a message in IRC, and nothing will happen.
I'll send a message in Skype (hence s.OnMessageStatus = Ibex) and it sends the IRC message, as expected. However, it won't stop sending it. It spams it like crazy. I tried doing something like setting a sendMSG = True when a message is received, and setting sendMSG to false after sending it, then if sendMSG = true, then send. That didn't seem to work out well.
Anyone have any suggestions? Am I just missing something small causing this error? Thanks to anyone in advanced, this has been bothering me for a while now.
I managed to get an old IRC/Skype bridge to work a while ago, but that script is long gone and I can't remember exactly how I did it.
Well the bot I made in python and it is connecting to random servers. I cant figure out why. so far this code is mishmash of other projects so maybe I'm overlooking something. basicly I want it to connect to irc.rizon.net join #brook_nise then lurk there.
I conect to these servers when I run my script:
irc.rizon.io
irc.sxci.net
irc.broke-it.com
irc.rizon.sexy
.
import socket
network = 'irc.rizon.net'
network = network.encode(encoding='UTF-8',errors='strict')
port = 6667
irc = socket.socket ( socket.AF_INET, socket.SOCK_STREAM )
USR = "USER boxxxy boxxxy boxxxy :boxxxy\r\n"
PAS = '/msg NickServ IDENTIFY pass\r\n'
JOI = 'JOIN #brook_nise\r\n'
pi = 'PING'
po = 'PONG'
PING = pi.encode(encoding='UTF-8',errors='strict')
PONG = po.encode(encoding='UTF-8',errors='strict')
USER = USR.encode(encoding='UTF-8',errors='strict')
PASS = PAS.encode(encoding='UTF-8',errors='strict')
JOIN = JOI.encode(encoding='UTF-8',errors='strict')
irc.connect ( ( network, port ) )
print (irc.recv ( 4096 ))
irc.send (USER)
irc.send (PASS)
irc.send (JOIN)
while True:
data = irc.recv ( 4096 )
if data.find ( PING ) != -1:
irc.send ( PONG + data.split() [ 1 ] + '\r\n' )
print (data)
This happens because irc.rizon.net is a geobalanced DNS record. It checks where your bot is coming from and then automatically assigns it a server to connect to.
Basically there is no such server as 'irc.rizon.net', if you always want the same one (you don't) then just specify one of the servers you have listed.
I am trying to get an acknowledgement from my subscriber back to my publisher using ZeroMQ in Python.
I tried a few code examples, using a zmq.PUSH and zmq.PULL code sequence, but to no avail.
My code as it stands:
PUB_SERVER.PY
import zmq
import random
import sys
import time
port = "5556"
if len( sys.argv ) > 1:
port = sys.argv[1]
int( port )
context = zmq.Context()
socket = context.socket( zmq.PUB )
socket.bind( "tcp://*:%s" % port )
while True:
# topic = random.randrange( 9999, 10005 )
topic = 10000
messagedata = random.randrange( 1, 215 ) - 80
print "%d %d" % ( topic, messagedata )
socket.send( "%d %d" % ( topic, messagedata ) )
time.sleep( 1 )
SUB_CLIENT.PY
import sys
import zmq
port = "5556"
if len( sys.argv ) > 1:
port = sys.argv[1]
int( port )
if len( sys.argv ) > 2:
port1 = sys.argv[2]
int( port1 )
# Socket to talk to server
context = zmq.Context()
socket = context.socket( zmq.SUB )
print "Collecting updates from weather server..."
socket.connect( "tcp://192.168.0.21:%s" % port )
if len( sys.argv ) > 2:
socket.connect( "tcp://192.168.0.21:%s" % port1 )
# Subscribe to zipcode, default is NYC, 10001
topicfilter = "10000"
socket.setsockopt( zmq.SUBSCRIBE, topicfilter )
# Process 5 updates
total_value = 0
for update_nbr in range( 5 ):
string = socket.recv()
topic, messagedata = string.split()
total_value += int( messagedata )
print topic, messagedata
print "Average messagedata value for topic '%s' was %dF" % ( topicfilter, total_value / update_nbr )
That code gives me the output of the server in one SSH window (on a Parallella), and the received filtered messages of the client in another SSH window (on a RaspberryPi) which is working great.
Where I am lost is, once the client has gotten a filtered message from the server, how would it acknowledge that filtered message being received, and then have the server log those acknowledged messages?
Eventually, I'd want to do some intelligent decision making of sending a file to the subscriber who acknowledges.
How to acknowledge?
May create a parallel messaging structure for a soft-signalling for that purpose.
Extend PUB_SERVER.PY with a .SUB Rx access point:
anAckRxSOCKET = context.socket( zmq.SUB ) # create SUB side
anAckRxSOCKET.bind( "tcp://*:%s" % aServerAckPORT ) ) # .bind()
anAckRxSOCKET.setsockopt( zmq.SUBSCRIBE, "" ) # SUB to *-anything
# ...
anAckRxSTRING = anAckRxSOCKET.recv() # .recv()
Extend SUB_CLIENT.PY with a .PUB Tx socket to the Server side access point:
anAckTxSOCKET = context.socket( zmq.PUB ) # create PUB side(s)
anAckTxSOCKET.connect( "tcp://192.168.0.21:%s" % aServerAckPORT ) )
and
send ACK(s) with "a-proxy-ID" for any server-side processing you may want or need
anAckTxSOCKET.send( topicfilter ) # ACK with an "identity"-proxy