I recently wrote a code for a file transfer in Python. Sockets connect fine when I connect them from different terminals on the same system. But the same doesn't seem to work when I connect them from different computers which are connected over the same Wifi network.
Here's the server code:
import os
import socket
# Creating a socket
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.bind(("192.164.X.X",2222))
sock.listen(5)
print("Host Name: " , sock.getsockname())
# Accepting the connection
client , addr = sock.accept()
# Getting file details
file_name = input("File Name:")
file_size = os.path.getsize(file_name)
# Sending file name and details
client.send(file_name.encode())
client.send(str(file_size).encode())
# Opening file and sending data
with open(file_name,"rb") as file:
c = 0
while c <= file_size:
data = file.read(1024)
if not (data):
break
client.sendall(data)
c += len(data)
# closing the socket
sock.close()
Here's my client code:
import os
import socket
host = input("Host Name: " )
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Trying to connect to socket
sock.connect((host,2222))
print("Connected Successfully")
# send file details
file_name = sock.recv(100).decode()
file_size = sock.recv(100).decode()
with open("./rec/" + file_name , "wb") as file:
c = 0
while c <= int(file_size):
data = sock.recv(1024)
if not (data):
break
file.write(data)
c += len(data)
sock.close()
When I try to connect The client From a different computer I get this error :
while c <= int(file_size):
ValueError: invalid literal for int() with base 10: '3hi\n'
The file I am trying to transfer has a single word 'hi'.
File transfer works correctly from different terminals on same machine. But the same doesn't work on different computers which are connected over the same wifi network.
I understand the error (trying to convert string to int) but I don't WHY it's happening and how to fix it.
Your server code is sending a single TCP packet containing the content of multiple client.send() calls. This is commonly known as "corking", and can usually be disabled (depending on your OS) using the socket.TCP_NODELAY socket option after accepting the connection.
client, addr = sock.accept()
client.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
This is however not guaranteed to work, and depends on your OS and OS settings.
The real solution would be to create a more robust protocol and avoid relying on data being sent in different packets. In fact, this is the only sane way of implementing any protocol based on TCP. Never rely on data being split in packets in a specific way.
Decide a fixed size for encoding and sending lengths, then do the following on the server:
Send a length (of fixed size, for example 8 characters or 8 bytes, or whatever you would like) for the file name.
Send the filename.
Send the file size (again of fixed size).
Send the file contents.
While on the client:
Receive exactly 8 bytes and decode the length.
Receive exactly length bytes for the filename.
Receive exactly 8 bytes and decode the file size.
Receive exactly size bytes for the file contents.
Most importantly, note that the .recv() method of sockets can return less than the requested amount (you seem to already know that), so whatever kind of receiving operation you need to do, you will need to accumulate data in a loop until you have received the expected amount, for example:
expected = 100
data = b''
while len(data) < expected:
data += sock.recv(expected - len(data))
Related
I am trying to send TCP packets using socket in Python, and I can successfully send packets outside, I can see them using WireShark. However, as you can see that inside the while loop the code wait for ENTER key to be entered in order to send a packet. When I press ENTER one by one, I can see that my data (18 byte starts with 0xEFEB000000 in bytearray format) which is read from file, but not given here because I think it unnecessary and confusing, is sent packet by packet as I wanted. However, when I press and hold the ENTER button, it consecutively sends packets way faster than one by one. In this case, I can see in WireShark that some of the data are combined into a single TCP packet. I do not want this, and this is very strict unfortunately. How can I force socket to send the payload I gave packet by packet separately?
from socket import socket, AF_INET, SOCK_STREAM
if __name__ == "__main__":
CLIENT_SOCK = socket(AF_INET, SOCK_STREAM)
CLIENT_SOCK.connect(("192.168.1.10", 42000))
while True:
input("Press ENTER to send a TCP packet...")
# some file IO here to construct data variable,
# 18 byte bytearray format. starts with 0xEFEB000000
CLIENT_SOCK.sendall(data)
Single Packet with single payload
Single Packet with 5 payloads combined
TCP isn't message-based. It just returns the bytes written into the socket in the order sent, but due to network buffer can break the bytes up in whatever packets it chooses. Wrap the receiver in a socket.makefile object which will buffer read data, then .read(n) will read exactly n bytes unless the socket closes early. Example:
server.py
import socket
s = socket.socket()
s.bind(('', 5000))
s.listen()
c, a = s.accept()
# 'with' will close the client socket and makefile object
# when it exits.
with c, c.makefile('rb') as f:
while True:
data = f.read(18)
if not data:
break
print(data)
client.py
import socket
s = socket.socket()
s.connect(('localhost', 5000))
s.sendall(b'abcdefghijklmnopqr')
s.sendall(b'abc')
s.sendall(b'defghijklm')
s.sendall(b'nopqr')
s.sendall(b'abcdefghijklm')
s.sendall(b'nopqrabcdefghijklm')
s.sendall(b'nopqr')
s.close()
Server output:
b'abcdefghijklmnopqr'
b'abcdefghijklmnopqr'
b'abcdefghijklmnopqr'
b'abcdefghijklmnopqr'
Note: This may appear to work with a direct c.recv(18) but in a busy/complex network environment can fail to receive 18 bytes every time.
I've set up a python client and server with socket in Python, that allows the server to send text to the client and I've been trying to extend it so that images can be sent to the client.
Server code:
import socket
#setup and bind server socket
s_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)#setup socket
s_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)#reuses same port (allows reconnection)
s_socket.bind(('192.168.178.52', 9001))
s_socket.listen(1)
#connects and prints clients data and send message
clientsocket, address = s_socket.accept()
print('Connection from {}'.format(address))
clientsocket.send(bytes('Welcome to the server', 'utf-8'))
#Loops for server to sent text data to client
while True:
m = input('Enter')
try:
file = open(m, 'rb')
b = file.read(2048)
clientsocket.send(b)
except:
clientsocket.send(bytes(m, 'utf-8'))
Client code:
import socket
import webbrowser
import os
import pyautogui
#setup and bind client socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.connect(('----------', 9001))#ip swapped for post
while True:
message = s.recv(2048)#recieves all messages sent with buffer size
if message:
txt = str(message)
with open('plswork.png', 'wb') as file:
file.write(message)
file.close()
The problem I'm having is that it will send the file over and create it perfectly fine, but only part of the image will load in when i open it (see image) I am pretty sure this is something to do with the buffer size however when I increase it, it wont recognise the file at all and I'll get an error trying to open the photo (preferably you would be able to send most photos). New to python sockets so any help would be appreciated!
(at the moment trying to send a pic of tux...)
https://i.stack.imgur.com/lBblq.png
I don't know the size of the file, but shouldn't you read the file until it is read completely and send data in chunks?
while True:
m = input('Enter')
try:
file = open(m, 'rb')
while True:
b = file.read(2048)
if not b:
break
clientsocket.send(b)
except:
clientsocket.send(bytes(m, 'utf-8'))
Client side had to be adapted as well.
Most network protocols add more information to simplify reception.
It could for example be a good idea, if you first send the number of bytes, that the welcome message contains, then the welcome message, then some indicator, that you will send a file, then some information, how many bytes you will send for the image and only then the bytes of the image
You will also find out, that it is complicated for the client to know what is the text message and what is part of the png file.
In fact if you would remove the input() command from the server and hard code a file name you might probably notice. that the welcome message and the png file could arrive combined at the client side. and it would have difficulties separating the two.
So to make your code robust, there's a lot of work to do.
I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1
As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.
One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.
You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()
I'm trying to send a file over a socket in Python 2.7.x . My client and server code is below. It's working, but only after the client connection kills the socket. So I added raw_input('finished') to the end of both for debugging.
So if I start the server, then run the client... It looks like all but the last bit of the file sends, until I forcefully kill the client and then it's all there. So the problem is definitely in the server loop... I just don't know how to fix it. if not data: break isn't being triggered. But, if I do something like if len(data) < 1024: break it won't work for bigger files.
Any help is appreciated!
# client.py
import socket
conn = socket.socket()
conn.connect(('localhost', 1337))
f = open('test.jpg', 'rb')
data = f.read(1024)
while data:
conn.send(data)
data = f.read(1024)
f.close()
raw_input('finished')
# server.py
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('localhost', 1337))
s.listen(5)
conn, addr = s.accept()
f = open('test.jpg', 'wb')
while True:
data = conn.recv(1024)
if not data:
break
f.write(data)
f.close()
raw_input('finished')
From your posted code:
while data:
conn.send(data)
data = f.read(1024)
From the Python socket documentation:
socket.send(string[, flags])
[...]
Returns the number of bytes sent. Applications are responsible for checking
that all data has been sent; if only some of the data was transmitted, the
application needs to attempt delivery of the remaining data.
That should tell you what the problem is, but just to be explicit about it: send() may or may not accept all of the bytes you asked it to send before returning, and it's up to you to handle it correctly in the case where it only accepts the first (n) bytes rather than the entire buffer. If you don't check send()'s return value, then you will sometimes drop some of the bytes of your file without knowing it. You need to check send()'s return value and if it is less than len(data), call send() again (as many times as necessary) with the remaining bytes. Alternatively you could call conn.sendall() instead of conn.send(), since sendall() will perform that logic for you.
I have a central data feed that I want to redistribute to many clients. The data feed produces approx. 1.8 kB/s. Currently I'm writing the feed to a file and each client reads off the end of the file. Something about this just seems wrong. Here is pseudo code for what I have now...
The feed:
o = open('feed.txt','a',0) #no buffering, maybe line buffer would be better
while 1:
data = feed.read(8192)
data = parse_data(data)
o.write(data)
time.sleep(0.01)
The server (each client connects in a new thread):
feed = open('feed.txt','r')
feed.seek(-1024,2)
while 1:
dat = feed.read(1024)
if len(dat)==0:
# For some reason if the end of the file is reached
# i can't read any more data, even there is more.
# some how backing off seems to fix the problem.
self.feed.seek(-1024,2)
self.feed.read(1024)
buffer += dat
idx = buffer.rfind('\n')
if idx>0:
data = buffer[:idx]
buffer = buffer[idx+1:]
for msg in data.split('\n'):
client.send(msg)
time.sleep(0.01)
What I'd like to do is just replace the file with a socket and write the messages directly to multicast packets. Any time a new client connects to the server I just spin up a new thread and start listening for the multicast packets. Are there any standard design patterns to handle this case?
Even simpler, just have all clients multicast on the same port. Then your server doesn't even need to track pseudo-connections.
We use a similar scheme for some of the software on our internal network, based on the fact that multicasting is "mostly reliable" on our networking infrastructure. We've stress tested the load and don't start dropping packets until there's over 30K messages/sec.
#!/usr/bin/python
import sys
import socket
ADDR = "239.239.239.9"
PORT = 7999
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((ADDR,PORT))
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
while True:
data, addr = sock.recvfrom(2048)
print data
sys.stdout.flush()