I have simple tcp socket program and I would like to send strings in chunks of 10 bytes. The server will join the chunks.
However I'm not sure how to split a string into binary and how to send the chunks of binaries. Instead of sending 512 bytes at one time I want to send 10 byte several times.
I have found a module Pickle that can serialize data into bytestrings (?) but how do I apply socket.send() on this?
Server:
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("", my_port))
server_socket.listen(5)
client_socket, address = server_socket.accept()
data = client_socket.recv(512)
Client:
message = "some string I want to send in chunks"
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect((host, my_port))
client_socket.send(message)
client_socket.close()
First of all, your code is not actually sending 512 bytes at a time, it's receiving 512 bytes at a time.
Now, I think you're really asking two questions.
How do you send 10 bytes at a time over a TCP connection (you are using socket.SOCK_STREAM)
How do you send raw bytes using a Python socket.
Let's answer 2. first: If you call socket.send on a byte string, it should be sent out in binary as TCP payload.
For 1., the simplest approach would be to split the data into chunks (now that you know you're operating on strings, you can simply do that using the slice operations (see the Python tutorial on strings - e.g. s[0:10], s[10:20] etc). Next, you need to ensure these slices are sent individually. This could be done by calling socket.send, but the problem is, that your TCP/IP stack may group these into packets even if you don't want it to - you have after all asked it to provide you with a stream socket, socket.SOCK_STREAM. If you were writing to a file, what you'd do in this case is you'd flush the file, but this does not appear to be easy for Python sockets (see this question).
Now, I think the answer to that question, saying it's impossible, is not quite right. It appears that Scapy will let you send 10 byte TCP chunks (I got the chunks() function from here). I checked it in wireshark and tried it multiple times with consistent results, but I didn't check the implementation to make sure this is guaranteed to happen.
You should probably ask yourself why you want to send data in chunks of 10 bytes, rather than let TCP deal with what it was designed for, and consider using delimiters.
Anyway, here's the code using Scapy (fun fact: it looks like running this does not require root privileges):
client:
from scapy.all import *
import socket
host = '192.168.0.x' #replace with your IP
my_port = 3002
message = "some string I want to send in chunks"
def chunks(lst, n):
"Yield successive n-sized chunks from lst"
for i in xrange(0, len(lst), n):
yield lst[i:i+n]
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect((host, my_port))
ss=StreamSocket(client_socket,Raw)
for chunk in chunks(message, 10):
print "sending: " + chunk
ss.send(Raw(chunk) )
client_socket.close()
server:
import socket
my_port=3002
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("", my_port))
server_socket.listen(5)
client_socket, address = server_socket.accept()
while (True):
data = client_socket.recv(512)
if (data):
print data
else:
break
Related
I'm trying to receive several megabytes of data using a Unix domain socket. My problem is the received data always cuts off at 219,264 bytes. After receiving that amount, I can keep sending, however I cannot receive anything afterwards until I completely reset the connection. My code&minimal examples(given that the socket regularly gets sent data to) is below.
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(f"{socketpath}")
data = b""
while True:
received = s.recv(1024)
data += received
print("Received some data")
while True:
sock.send(b"{a JSON-RPC string}")
Server side is go-ethereum. There is no exception with receiving(simply a hang upon calling recv), with sending an in s.send(...) OSError: [Errno 32] Broken pipe gets thrown once 219264 bytes are sent in total. Also, once I begin getting Broken pipe, recv begins to respond but always returns an empty result (b'').
*Seems to be happening with 36544 bytes as well now. Completely randomly one of the two.
I tried to create a UDP socket and was successful in creating one.
Now when I tried to send some raw data, socket.send wants me to convert it into bytes.
But this raw data is a command to change the time of my application which I am working on, so i wanted to send the data as it is.
Is there a way to send this without converting it into bytes?
here's the code i used:
Socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
bufferSize = 1024
recvBuf = 4096
new_pkt= x00\x1c\x7fb\xb5\xfd\x00PV\xb8\x08\x9f\x08\x00E\x00\x000B\xad\x00\x00\x80\x11\x00\x00\n\xe7\xa0\xc6\n\xe7\x922\xc0\xb8\x05\xdc\x00\x1cH\xf4\t\x8d\x01\x00\x01\x01\x00\x10\x00\x00\x01,\x00\x00\x00\x004\xe6\xc0
S = Socket.connect_ex(("10.231.146.50",port))
Socket.settimeout(10)
Res = Socket.sendall(new_pkt)
Socket.close
obtained an error to convert the packet into bytes, while trying to send it
I get no error with Python2.7, for send or sendall, if I put double quotes around your data string
new_pkt= "x00\x1c\x7fb\xb5\xfd\x00PV\xb8\x08\x9f\x08\x00E\x00\x000B\xad\x00\x00\x80\x11\x00\x00\n\xe7\xa0\xc6\n\xe7\x922\xc0\xb8\x05\xdc\x00\x1cH\xf4\t\x8d\x01\x00\x01\x01\x00\x10\x00\x00\x01,\x00\x00\x00\x004\xe6\xc0"
Give it a try ;-)
If you use Python3.x, you need bytes. So you just prefix your string with a b:
new_pkt_for_py3 = b"x00\x1c\x7fb\xb5\xfd...."
my code for connecting to the serial port
ser = serial.Serial(
port='COM4',
baudrate=9600,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS,timeout=0.5
)
if (ser.isOpen):
ser.close()
ser.open()
buf=b''
out = ''
while 1:
while ser.inWaiting() > 0:
sleep(1)
#if ser.outWaiting()==0:
out = ser.read(ser.inWaiting())
if out != '':
handle_command(out, ser)
print ">>" + out.encode('hex')
out = ''
This part is where I write the data to the port
elif(some condition):
if(ser.outWaiting()==0):
out=ser.write(pSportTotals_p1)
sleep(0.001)
ser.flush()
if(ser.outWaiting()==0):
ser.write(pSportTotals_p2)
sleep(0.001)
ser.flush()
But the output is erratic, and pSportsTotals_p1 is sent in two parts and pSportTotals_p2 is sometimes sent along with the rest of the previous packet. In the attached screenshot, 0a signifies the start of each packet and in the second image you can see that it is attached with the remaining chunk of the previous packet. I am new to serial programming and will appreciate any help in making me understand my errors.first packet
second packet
I don't have specific knowledge about pyserial, but having dealt with serial communication before in different languages, I've come to a conclusion you can't know for sure how your data is going to be sent/received.
You have multiple options :
Always send the same amount of N bytes, and on the receiving end, wait until the buffer has more than N data before reading a chunk of N bytes.
Implement a protocol with delimiters (for example '\n') and parse the received data to reconstruct your data by looking for the delimiters.
The following code is for a python server that can receive a string.
import socket
TCP_IP = '127.0.0.1'
TCP_PORT = 8001
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connection address:', addr
while 1:
length = conn.recv(1027)
data = conn.recv(int(length))
import StringIO
buff = StringIO.StringIO()
buff.write(data)
if not data: break
print "received data:", data
conn.send('Thanks') # echo
get_result(buff)
conn.close()
Can anyone help me to edit this code or create a similar one to be able to receive images instead of string?
First, your code actually can't receive a string. Sockets are byte streams, not message streams.
This line:
length = conn.recv(1027)
… will receive anywhere from 1 to 1027 bytes.
You need to loop around each recv and accumulate a buffer, like this:
def recvall(conn, length):
buf = b''
while len(buf) < length:
data = conn.recv(length - len(buf))
if not data:
return data
buf += data
return buf
Now you can make it work like this:
while True:
length = recvall(conn, 1027)
if not length: break
data = recvall(conn, int(length))
if not data: break
print "received data:", data
conn.send('Thanks') # echo
You can use StringIO or other techniques instead of concatenation for performance reasons, but I left that out because it's simpler and more concise this way, and understanding the code is more important than performance.
Meanwhile, it's worth pointing out that 1027 bytes is a ridiculous huge amount of space to use for a length prefix. Also, your sending code has to make sure to actually send 1027 bytes, no matter what. And your responses have to always be exactly 6 bytes long for this to work.
def send_string(conn, msg):
conn.sendall(str(len(msg)).ljust(1027))
conn.sendall(msg)
response = recvall(conn, 6)
return response
But at least now it is workable.
So, why did you think it worked?
TCP is a stream of bytes, not a stream of messages. There's no guarantee that a single send from one side will match up with the next recv on the other side. However, when you're running both sides on the same computer, sending relatively small buffers, and aren't loading the computer down too badly, they will often happen to match up 1-to-1. After all, each time you call recv, the other side has probably only had time to send one message, which is sitting in the OS's buffers all by itself, so the OS just gives you the whole thing. So, your code will appear to work in initial testing.
But if you send the message through a router to another computer, or if you wait long enough for the other side to make multiple send calls, or if your message is too big to fit into a single buffer, or if you just get unlucky, there could be 2-1/2 messages waiting in the buffer, and the OS will give you the whole 2-1/2 messages. And then your next recv will get the leftover 1/2 message.
So, how do you make this work for images? Well, it depends on what you mean by that.
You can read an image file into memory as a sequence of bytes, and call send_string on that sequence, and it will work fine. Then the other side can save that file, or interpret it as an image file and display it, or whatever it wants.
Alternatively, you can use something like PIL to parse and decompress an image file into a bitmap. Then, you encode the header data (width, height, pixel format, etc.) in some way (e.g., pickle it), send_string the header, then send_string the bitmap.
If the header has a fixed size (e.g., it's a simple structure that you can serialize with struct.pack), and contains enough information for the other side to figure out the length of the bitmap in bytes, you don't need to send_string each one; just use conn.sendall(serialized_header) then conn.sendall(bitmap).
I'm trying to send a file between a client and a server in my home network. I just want to test with a simple file, client.txt.
The client program should read X bytes and send it over the tcp socket I've created, but I cant wrap my head around how to do the sending part:
f = open("client.txt", "rb")
while 1:
// should read X bytes and send to the socket
I think I need to check if the data I want to send is valid, if a file for instance is smaller then the amount (1024 for instance) I'm sending in each batch.... or does it not work that way?
Since you mentioned you have problems setting up the server part, I'll rip this out from Python documentation and edit it slightly:
import socket
HOST = ''
PORT = 50007
s = socket.socket()
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
f = open("client.txt", "rb")
while 1:
data = f.read(1024)
if not data: break
conn.send(data)
conn.close()
The relevant document can be found here
read() takes an optional parameter that specifies the number of bytes to read in.
Documentation
To read a file’s contents, call
f.read(size), which reads some
quantity of data and returns it as a
string. size is an optional numeric
argument. When size is omitted or
negative, the entire contents of the
file will be read and returned; it’s
your problem if the file is twice as
large as your machine’s memory.
Otherwise, at most size bytes are read
and returned. If the end of the file
has been reached, f.read() will return
an empty string ("").