Is it possible to send an array through UDP Sockets using Python? I am using Python 2.5 and trying to send a simple array but it's not working. It can send the array successfully but when I try to print it with an item of the array the program crashes. I'm not sure what the error is as I take the precaution of converting the data into an array but it's not working. Hope I explained the problem as clearly as possible. I would appreciate the help!
# Client program
from socket import *
import numpy
from array import*
# Set the socket parameters
host = "localhost"
port = 21567
buf = 4096
addr = (host,port)
# Create socket
UDPSock = socket(AF_INET,SOCK_DGRAM)
def_msg = "===Enter message to send to server===";
print "\n",def_msg
a = array('i',[1,3,2])
# Send messages
while (1):
data = raw_input('yes or now')
if data!= "yes":
break
else:
if(UDPSock.sendto(a,addr)):
print "Sending message"
# Close socket
UDPSock.close()
# Server program
from socket import *
# Set the socket parameters
host = "localhost"
port = 21567
buf = 4096
addr = (host,port)
# Create socket and bind to address
UDPSock = socket(AF_INET,SOCK_DGRAM)
UDPSock.bind(addr)
# Receive messages
while 1:
data,addr = UDPSock.recvfrom(buf)
L = eval(data)
if not data:
print "Client has exited!"
break
else:
print "\nReceived message '", L[1],"'"
# Close socket
UDPSock.close()
eval is doing something completely different than what you think.
To send data over network, you need to serialize it into an array of bytes, then deserialize it back. In Python, serialization of most objects can be done via pickle module:
if (UDPSock.sendto( pickle.dumps(a), addr)):
Deserialization:
data,addr = UDPSock.recvfrom(buf)
L = pickle.loads(data)
print repr(L) # prints array('i', [1, 3, 2])
I would personally use tostring and fromstring since the built-in serialization methods are many times faster and pickle may not support NaN, Inf and other undefined values.
You're trying to send a python object through a socket, it is normal that it doesn't work, you can't send objects in a socket, objects are not data, they are the representation of some data in a given programming language. You need to "translate" your object to data and re-create the object from the data on the other socket's side. One way to do this would be with the pickle module.
On the client side, you "pickle" the object:
data = pickle.dumps(my_array)
And on the server side, you "unpickle" the received data:
my_array = pickle.loads(received_data)
You could try to pickle the array. Pickle is a python library to en- and decode python objects. It is able to do much more, but it is definitely sufficient to fulfill your task:
on the sender side you pickle the object to a string:
pickled_string = pickle.dumps(a)
on the receiver side you unpickle the object:
a = pickle.loads(received_string)
# a is now your sent array
It has been a while since this question was asked, but I thought it's worth sharing the jsonsocket library. It makes it really easy to send strings, lists and dictionaries over sockets. It can handle big amounts of data efficiently. And you don't need to do any manual serialization/deserialization. Under the hood, it serializes the data as JSON strings on the client, and deserializes it on the server.
If you don't need UDP specifically, try zmqObjectExchanger (https://github.com/ZdenekM/zmq_object_exchanger). It wraps pickle and zmq to transfer python objects over TCP.
Related
I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1
As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.
One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.
You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()
I am experimenting with python sockets to try to understand the whole concept better, but I have run into a problem. I have a simple server and a client, where the client sends a list to the server, and then waits for the server to send a string signaling the process is complete.
This is the client file:
import socket
import json
host = '192.168.1.102'
port = 14314
def request():
print 'Connecting'
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect((host, port))
print 'Sending request'
clientsocket.sendall(json.dumps([1, 2, 3, 4, 5, 6, 7, 8, 9]))
print 'Receiving data'
data = clientsocket.recv(512)
print 'Received: {}'.format(data)
request()
and here is the server file:
import socket
import json
host = '192.168.1.102'
port = 14314
def run():
print 'Binding socket'
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serversocket.bind((host, port))
print 'Waiting for client'
serversocket.listen(1)
clientsocket, addr = serversocket.accept()
print 'Receiving data'
raw_data = ''
while True:
tmp = clientsocket.recv(1024)
if not tmp:
break
raw_data += tmp
data = json.loads(raw_data)
print 'Received: {}'.format(data)
print 'Sending data'
clientsocket.sendall('done')
run()
The problem is that while the client is done sending the list, the server is stuck in the recv loop, waiting for nothing. The whole data has been received in the first iteration and in the second iteration there is nothing to be received because the client has moved on to the receiving part.
The weird part is that if I comment out the receive part from the client and the send part from the server, the process completes successfully. So, what am I doing wrong? Why is this not working?
Thanks.
The Docs for socket.recv talk about additional flags being able to be passed in to the recv function described in the unix documentation. So turning to that documentation, I found the following message:
If no messages are available at the socket, the receive calls wait for
a message to arrive, unless the socket is nonblocking (see fcntl(2)),
in which case the value -1 is returned
So once again, we're directed to another page. The documentation for fcntl says
Performs one of the operations described below on the open file
descriptor
So, normally the socket.recv function is blocking (it will wait indefinitely for new data), unless we use a file descriptor. How do we do that? Well there is a socket.makefile function that gives us a file descriptor attached to the socket. Cool. This SO question gives us an example of how we can read and write to a socket, using a file descriptor.
Well what if we don't want to use a file descriptor. Reading further into the unix documentation for the recv function, I see that I can use the MSG_DONTWAIT flag. This doesn't work in Windows, but I did find out that we can use socket.setbocking(False) to permamently change the socket to non-blocking mode. You would then need to ignore any "A non-blocking socket operation could not be completed immediately" errors. Those are normal and non-fatal(error #10035 of this page mentions it is non-fatal).
Another possible implementation would be to multi-thread your program, you can implement a receiving and a sending thread for your socket. This might give you the best performance, but it would be a lot of work to setup.
Python is awesome. I just found some libraries Python has that does asynchronous sockets too. There's asyncore, asynchat which have both been deprecated in favor of asyncio if that is available in the version of Python you are using.
Sorry for throwing so much out there. I don't know a whole lot about sockets. I used them once with the Paramiko library, and that was it. But it looks like there are a lot of ways of implementing them.
I am using the following python script for raw socket packet transfer. Packet transfer is fine, but I am not able to print the incoming packet from the other end.
from socket import socket, AF_PACKET, SOCK_RAW
s = socket(AF_PACKET, SOCK_RAW)
s.bind(("eth0", 0))
src_addr = "\x54\xbe\xf7\x40\xf5\x82"
dst_addr = "\xff\xff\xff\xff\xff\xff"
payload = ("[("*30)+"Hello"+("]"*30)
checksum = "\x1a\x2b\x3c\x4d"
data = payload+checksum
s.send(dst_addr+src_addr+data)
#for receive function
response=s.recv(4096)
print response
s.close()
There is a third argument to the socket function: protocol. If not given, it's defaulting to 0. For AF_PACKET / SOCK_RAW, the protocol argument specifies what kind of packets you're interested in receiving. The values are documented in the packet(7) man page: http://man7.org/linux/man-pages/man7/packet.7.html
I don't think the values are actually defined anywhere in the core python2 modules. Some of them can be found in scapy (http://www.secdev.org/projects/scapy/), or you can just hunt up the linux header file where they are defined (/usr/include/linux/if_ether.h).
So, to fix this, change your code to:
from socket import socket, AF_PACKET, SOCK_RAW, htons
ETH_P_ALL = 3
ETH_P_IP = 0x800 # Alternatively using this will receive the next IP packet
s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))
...
Other comments on your code:
As written, the packet you're sending is unlikely to be comprehensible by anyone. You've got a dst and src MAC address, but then you're not providing an EtherType. Instead the first "[(" will be seen as the EtherType. That probably won't make sense to any receiver of the packet so it will just be discarded.
Also, you should understand that with a raw socket, you're going to receive the next packet of the type you've specified in the protocol. That isn't necessarily (and in fact probably won't be) a response to the packet you just sent.
Thanks everyone now I am able receive the packet with the following the script. But still I am facing issue with printing multiple response packets(While doing in loop).
from socket import socket, AF_PACKET, SOCK_RAW, htons
from struct import *
import select
import time
ETH_P_ALL = 3
ETH_P_IP = 0x800
s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))
s.bind(("eth0", 0))
src_addr = "\x54\xbe\xf7\x40\xf7\x82"
dst_addr = "\xff\xff\xff\xff\xff\xff"
l = "\x00\x21"
ethertype = "\x08\x01"
a ="\x00\x10\x00\x10\x00\x10"
b = "\x00\x11\x00\x11\x00\x11"
payload = ethertype + l + a + b
for i in range(5):
time.sleep(5)
s.send(dst_addr+src_addr+ payload)
message=s.recv(4096)
print message
I want to send an array using a UDP connection. When I use the sendto function, it complains that it must be a string. Is there any way around this?
Thanks
You must serialize your data (in this case is an array) before sending it. Then in receiver, you will deserialize to get the original data.
You can do it in Python, using pickle or cPickle module:
import cPickle as p
# Sender
data_str = p.dumps(array)
sock.sendto(data_str, addr)
# Receiver
data,addr = sock.recvfrom(buf)
data_origin = p.loads(data)
I am writing a client-sever program based on Python socket.
The client sends a command to the server and the server responds.
But now, some client can broadcast a message to other clients, so the client can receive more than one response at the same time.
data = s.recv(1024)
the line of code above will retrieve only one response from the server.
but if I use a while loop like this
while True:
data = s.recv(1024)
if not data: break
actually, data=s.recv(1024) will block the program when there is no data left.
I don't want to block the program and want to retrieve all the responses available in the connection at one time. Can anyone find a solution? Thank you.
You can use the select module to wait until the socket is readable or until a timeout has elapsed; you can then perform other processing. For example:
while True:
# If data can be received without blocking (timeout=0), read it now
ready = select.select([s], [], [], 0)
if s in ready[0]:
data = s.recv(1024)
# Process data
else:
# No data is available, perform other tasks
You could make the socket (s) non-blocking. This way, it will retrieve all the received responses and when there is none, it will return back. Of course, with non-blocking, you will have to periodically retry.
You could make the socket (s) non-blocking using the setblocking() method:
s.setblocking(0)
The other option is to use another thread to handle the receive part. This way, your main thread can continue doing its main task and act upon the message only if it receives one.
You can use socket.setblocking or socket.settimeout:
import socket
import sys
HOST = 'www.google.com'
PORT = 80
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.setblocking(0)
s.sendall('Hello, world')
try:
data = s.recv(1024)
except:
print 'Oh noes! %s' % sys.exc_info()[0]
s.close()
socket.recv takes two parameters, the second is a set of flags. If you're on a Linux system, you can do man recv for a list of flags you can supply, and their corresponding errors.
Lastly, in general, you can't really know that the other side is done with sending you data (unless you're controlling both sides), even if you're both following a protocol. I believe the right way to go about it is to use timeouts, and quit after sending a reset (how you do this will depend upon what protocol you're using).