We Keep Coding

Python Sockets multiple file transfers

So I have a python program that basically let's the client connect to the server and send it an excel file which is used as an input for an optimization problem. I then want the server to send the output of this optimization (also an excel file) back to the client. The model itself takes about a minute to solve, and I think this is causing some issues with the client trying to receive the output 'too early'.
The client code:
SEPARATOR = "<SEPARATOR>"
BUFFER_SIZE = 4096
HEADER = 64
HEADERSIZE = 10
port = 1234
FORMAT = 'utf-8'
DISCONNECT_MESSAGE = "!DISCONNECT"
host = "123.45.678.910"
s = socket.socket()
s.connect((host, port))
filename = "input/Model_Input.xlsx"
filesize = os.path.getsize(filename)
s.send(f"{filename}{SEPARATOR}{filesize}".encode())
with open(filename, "rb") as f:
while True:
bytes_read = f.read(BUFFER_SIZE)
if not bytes_read:
break
s.sendall(bytes_read)
out_received = s.recv(BUFFER_SIZE).decode()
out_filename, out_filesize = out_received.split(SEPARATOR)
out_filename = os.path.basename(out_filename)
out_filesize = int(out_filesize)
with open(out_filename, "wb") as h:
while True:
out_bytes_read = s.recv(BUFFER_SIZE)
if not out_bytes_read:
break
h.write(out_bytes_read)
And the server code:
SERVER_PORT = 1234
SERVER_HOST = socket.gethostbyname(socket.gethostname())
BUFFER_SIZE = 4096
SEPARATOR = "<SEPARATOR>"
s = socket.socket()
s.bind((SERVER_HOST, SERVER_PORT))
s.listen(5)
client_socket, address = s.accept()
received = client_socket.recv(BUFFER_SIZE).decode()
filename, filesize = received.split(SEPARATOR)
filename = os.path.basename(filename)
filesize = int(filesize)
with open(filename, "wb") as f:
while True:
bytes_read = client_socket.recv(BUFFER_SIZE)
if not bytes_read:
break
f.write(bytes_read)
##################
## MODEL CODE ##
##################
outfilename = 'Model_Output.xlsx'
outfilesize = os.path.getsize(filename)
client_socket.send(f"{outfilename}{SEPARATOR}{outfilesize}".encode())
with open(outfilename, "rb") as h:
while True:
# read the bytes from the file
bytes_readed = h.read(BUFFER_SIZE)
if not bytes_readed:
break
client_socket.sendall(bytes_readed)
I am able to send the input file to the server and get the model to run, and save the output to storage. However, as soon as I add in the part to try send it back to the client, it gets stuck. It still sends and receives the input file successfully, but then the model never runs. Neither the client nor the server disconnect, they both just seem to be stuck.
Thank you

I can be very difficult for someone (i.e. me) to remotely debug this type of code, so I can't really point to a particular line of code that is necessarily the problem. If, however, your client and server are running on the same machine, then there is a potential problem in the client code that begins:
with open(out_filename, "wb") as h:
while True:
out_bytes_read = s.recv(BUFFER_SIZE)
if not out_bytes_read:
break
h.write(out_bytes_read)
When the open is executed, this sets the file size to 0. The server, meanwhile is reading this file to transmit it to the file and can possibly find that there are now only 0 bytes. But it has already sent a "header" saying that there are N bytes where N is non-zero. But that is a different problem than the one you describe. But it could be happening in the other direction, also. That is, when the client is sending the file and the server is opening the file for output, it is now zeroing out the file that the client is still reading. The code below solves that problem in both directions. Of course, if your client and server are on different computers not accessing the same files concurrently, then what I have described is not an issue. Not yet, anyway.
I can, however, offer a slightly different approach, which does seem to work:
See the Using a Socket section from the Socket Programming HOWTO article in the Python 3 Manual. I have adopted the suggestion to use fixed length messages. It's a little more laborious, perhaps, but also a bit more fullproof. That means that if you want to transmit the filename, you have to first transmit the length of the encoded filename as fixed length length message (3 bytes can handle encoded filenames up to 999 bytes in length) and then you can transmit the encode filename. Similarly, we transmit the length of a file as a 9 byte length (left-padded with zeroes), which can handle file sizes up to 999,999,999 bytes (I set the width of 9 arbitrarily). I have two functions, receive_msg and send_msg that will robustly send and receive complete byte messages and can be used by both the client and server. These are modeled on the MySocket.mysend and MySocket.myreceive methods from the article.
I assumed that the server should be able to handle more than one request before terminating. In fact, it should be able to handle requests concurrently. To that end the server passes a request to a thread pool worker function, process_request, for processing. It wasn't clear what the nature of the so-called "Model Code" was. Assuming the function that performs this computation, process_model, is CPU-intensive, process_requestis passed a multiprocessing pool instance that can be used to perform the process_model processing so that the CPU-intensive portion of processing will not be limited by the Global Interpreter Lock. If there is no real CPU-intensive processing involved, then remove the code that created the multiprocessing pool and then call process_model as a regular function.
Server Code
import socket
from multiprocessing.pool import Pool, ThreadPool
import os.path
BUFFER_SIZE = 4096
SERVER_PORT = 1234
SERVER_HOST = socket.gethostbyname(socket.gethostname())
def receive_msg(sock, msg_length):
chunks = []
bytes_recd = 0
while bytes_recd < msg_length:
chunk = sock.recv(min(msg_length - bytes_recd, BUFFER_SIZE))
if chunk == b'':
raise RuntimeError("socket connection broken")
chunks.append(chunk)
bytes_recd = bytes_recd + len(chunk)
return b''.join(chunks)
def send_msg(sock, msg):
msg_length = len(msg)
totalsent = 0
while totalsent < msg_length:
sent = sock.send(msg[totalsent:])
if sent == 0:
raise RuntimeError("socket connection broken")
totalsent = totalsent + sent
def server():
s = socket.socket()
s.bind((SERVER_HOST, SERVER_PORT))
s.listen(5)
process_pool = Pool(5)
thread_pool = ThreadPool(5)
while True:
client_socket, address = s.accept()
thread_pool.apply_async(process_request, args=(process_pool, client_socket))
def process_request(process_pool, s):
# Fixed length fields:
# width 3 for filename length, followed by filename, width 9 for filesize
filename_size = int(receive_msg(s, 3).decode())
filename = receive_msg(s, filename_size).decode()
filename = os.path.basename(filename)
filesize = int(receive_msg(s, 9).decode())
msg = receive_msg(s, filesize)
with open(filename, "wb") as f:
f.write(msg)
# Assuming processing the model is CPU-intensive,
# we use a process pool for doing that:
out_filename = process_pool.apply(process_model, args=(filename,))
out_filesize = os.path.getsize(out_filename)
encoded_filename = out_filename.encode()
msg1 = b"%03d%s%09dfilesize" % (len(encoded_filename), encoded_filename, out_filesize)
with open(out_filename, "rb") as h:
msg2 = h.read()
send_msg(s, msg1)
send_msg(s, msg2)
def process_model(filename):
...
# Returned filename should probably be a function of the passed filename
return 'Model_Output.xlsx' # name of the output file
if __name__ == '__main__':
server()
Client Code
import socket
import os.path
BUFFER_SIZE = 4096
port = 1234
host = "123.45.678.910"
def receive_msg(sock, msg_length):
chunks = []
bytes_recd = 0
while bytes_recd < msg_length:
chunk = sock.recv(min(msg_length - bytes_recd, BUFFER_SIZE))
if chunk == b'':
raise RuntimeError("socket connection broken")
chunks.append(chunk)
bytes_recd = bytes_recd + len(chunk)
return b''.join(chunks)
def send_msg(sock, msg):
msg_length = len(msg)
totalsent = 0
while totalsent < msg_length:
sent = sock.send(msg[totalsent:])
if sent == 0:
raise RuntimeError("socket connection broken")
totalsent = totalsent + sent
def client():
s = socket.socket()
s.connect((host, port))
filename = "input/Model_Input.xlsx"
filesize = os.path.getsize(filename)
# Fixed length fields:
# width 3 for filename length, followed by filename, width 9 for filesize
encoded_filename = filename.encode()
msg1 = b"%03d%s%09dfilesize" % (len(encoded_filename), encoded_filename, filesize)
with open(filename, "rb") as f:
msg2 = f.read()
send_msg(s, msg1)
send_msg(s, msg2)
out_filename_size = int(receive_msg(s, 3).decode())
out_filename = receive_msg(s, out_filename_size).decode()
out_filename = os.path.basename(out_filename)
out_filesize = int(receive_msg(s, 9).decode())
msg = receive_msg(s, out_filesize)
with open(out_filename, "wb") as h:
h.write(msg)
if __name__ == '__main__':
client()
Update
The entire programming can greatly be simplified by implementing the service as a Remote Procedure Call. The code is based on Python Cookbook, 3rd Edition:
Server
import socket
import pickle
from multiprocessing.connection import Listener
from threading import Thread
from multiprocessing.pool import Pool
import os.path
SERVER_PORT = 1234
SERVER_HOST = socket.gethostbyname(socket.gethostname())
class RPCHandler:
def __init__(self):
self._functions = { }
def register_function(self, func):
self._functions[func.__name__] = func
def handle_connection(self, connection):
try:
while True:
# Receive a message
func_name, args, kwargs = pickle.loads(connection.recv())
# Run the RPC and send a response
try:
r = self._functions[func_name](*args,**kwargs)
connection.send(pickle.dumps(r))
except Exception as e:
connection.send(pickle.dumps(e))
except EOFError:
pass
def server():
global process_pool
handler = RPCHandler()
handler.register_function(process_request)
sock = Listener((SERVER_HOST, SERVER_PORT))
process_pool = Pool(5)
while True:
client = sock.accept()
t = Thread(target=handler.handle_connection, args=(client,))
t.daemon = True
t.start()
def process_request(filename, contents):
filename = os.path.basename(filename)
with open(filename, "wb") as f:
f.write(contents)
# Assuming processing the model is CPU-intensive,
# we use a process pool for doing that:
out_filename = process_pool.apply(process_model, args=(filename,))
with open(out_filename, "rb") as h:
out_contents = h.read()
return (out_filename, out_contents)
def process_model(filename):
...
# Returned filename should probably be a function of the passed filename
return 'Model_Output.xlsx' # name of the output file
if __name__ == '__main__':
server()
Client
import pickle
import socket
import os.path
from multiprocessing.connection import Client
port = 1234
host = "123.45.678.910"
class RPCProxy:
def __init__(self, connection):
self._connection = connection
def __getattr__(self, name):
def do_rpc(*args, **kwargs):
self._connection.send(pickle.dumps((name, args, kwargs)))
result = pickle.loads(self._connection.recv())
if isinstance(result, Exception):
raise result
return result
return do_rpc
def client():
c = Client((host, port))
proxy = RPCProxy(c)
filename = "input/Model_Input.xlsx"
with open(filename, "rb") as f:
contents = f.read()
(out_filename, out_contents) = proxy.process_request(filename, contents)
out_filename = os.path.basename(out_filename)
with open(out_filename, "wb") as h:
h.write(out_contents)
if __name__ == '__main__':
client()

Pickle data truncated

While trying to send a list to a client process, I pickle it, but when I receive this on the client side it always gives me the error pickle data was truncated, and I don't know how to fix it.
sv
def handle_client(connection):
connection.send(str.encode('welcome to sv'))
stock = random.sample(output, 1)
order = pickle.dumps(stock)
while True:
data = connection.recv(2048)
if not data:
break
if data.decode('utf-8') == 'wanna trade!':
print("trade order received")
tcp.send(order)
reply = connection.recv(2048)
if reply.decode('utf-8') == 'Y':
tcp.send(order)
output.remove(order)
elif reply.decode('utf-8') == 'N':
print("doesn't wish to buy.")
connection.close()
client
while True:
Cliente.send(str.encode('wanna trade!'))
recv = Cliente.recv(2048)
if not recv:
break
if recv:
Response = pickle.loads(recv)
print(Response)
Check = input('Y/N: ')
Cliente.send(str.encode(Check))
recv2 = Cliente.recv(2048)
if recv2:
final = pickle.load(recv2)
purchases.append(final.decode('utf-8'))
print(purchases)
Cliente.close()

I can't test it but data can be longer than 2048 bytes and when you get from socket only 2048 bytes then you get truncated data.
Socket doesn't know how many data to get in client so you have to send size before data. And size should have always the same length (so client will know if it get full size) so sending it as string may not work (or you would have to read size char after char until you get some spcial char - ie. new line which server would have to send after string with size.)
Server should first send data size (as ie. 4 bytes converted with struct.pack()) and later send data.
And client should first read 4 bytes and convert it to integer with struct.unpack() and later use this value to read all data.
Server:
import struct
stock = ...
data = pickle.dumps(stock)
size = len(data)
size_in_4_bytes = struct.pack('I', size)
print(size, size_in_4_bytes)
tcp.send(size_in_4_bytes)
tcp.send(data)
Client:
import struct
size_in_4_bytes = Cliente.recv(4) # get only 4 bytes
size = struct.unpack('I', size_in_4_bytes)
size = size[0]
print(size, size_in_4_bytes)
data = Cliente.recv(size)
stock = pickle.loads(data)
EDIT:
If you put code in functions then you could use it many times in simply way. You could use it also to send different object: pickle, normal string, data as JSON string, image, any file.
import struct
def send_data(conn, data):
size = len(data)
size_in_4_bytes = struct.pack('I', size)
conn.send(size_in_4_bytes)
conn.send(data)
def recv_data(conn):
size_in_4_bytes = conn.recv(4)
size = struct.unpack('I', size_in_4_bytes)
size = size[0]
data = conn.recv(size)
return data
# -------------------------------------
# --- pickle ---
# send pickle
data = pickle.dumps(stock)
send_data(Client, data)
# recv pickle
data = recv_data(Client)
stock = pickle.loads(data)
# --- text ---
# send normal string
data = text.encode()
send_data(Client, data)
# recv normal string
data = recv_data(Client)
text = data.decode()
# --- JSON ---
# send data as JSON
stock = {'open': 12, 'close': 15}
text = json.dumps(stock)
data = text.encode()
send_data(Client, data)
# recv data as JSON
data = recv_data(Client)
text = data.decode()
stock = json.loads(text)
print(stock) # {'open': 12, 'close': 15}
# --- image (or any other file) ---
# send image
with open('image.jpg', 'rb') as image
data = image.read()
send_data(Client, data)
# recv image
with open('image.jpg', 'wb') as image
data = recv_data(Client)
image.write(data)
EDIT:
Full working example.
Client first sends text and receives text, next it sends directory converted to JSON, and it receives JSON with other directory.
Server uses threads to run with many clients at the same time. There is sleep() to have time to start another client.
I use my code from answer for question:
How to handle multithreading with sockets in Python?
Server:
import socket
import threading
import time
import struct
import json
# --- functions ---
def send_data(conn, data):
size = len(data)
size_in_4_bytes = struct.pack('I', size)
conn.send(size_in_4_bytes)
conn.send(data)
def recv_data(conn):
size_in_4_bytes = conn.recv(4)
size = struct.unpack('I', size_in_4_bytes)
size = size[0]
data = conn.recv(size)
return data
def handle_client(conn, addr):
print("[thread] starting")
# ---
# recv message
data = recv_data(conn)
text = data.decode()
print("[thread] client:", addr, 'recv:', text)
# simulate longer work - to start next client at the same time
time.sleep(5)
# send message
text = "Bye!"
print("[thread] client:", addr, 'send:', text)
data = text.encode()
send_data(conn, data)
# ---
# recv JSON
data = recv_data(conn)
text = data.decode()
stock = json.loads(text)
print("[thread] client:", addr, 'recv:', stock)
# send JSON
stock = {'diff': stock['close'] - stock['open']}
print("[thread] client:", addr, 'send:', stock)
text = json.dumps(stock)
data = text.encode()
send_data(conn, data)
# ---
conn.close()
print("[thread] ending")
# --- main ---
host = '0.0.0.0'
port = 8080
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) # solution for "[Error 89] Address already in use". Use before bind()
s.bind((host, port))
s.listen(1)
all_threads = []
try:
while True:
print("Waiting for client")
conn, addr = s.accept()
print("Client:", addr)
t = threading.Thread(target=handle_client, args=(conn, addr))
t.start()
all_threads.append(t)
except KeyboardInterrupt:
print("Stopped by Ctrl+C")
finally:
if s:
s.close()
for t in all_threads:
t.join()
Client:
import socket
import struct
import json
# --- functions ---
def send_data(conn, data):
size = len(data)
size_in_4_bytes = struct.pack('I', size)
conn.send(size_in_4_bytes)
conn.send(data)
def recv_data(conn):
size_in_4_bytes = conn.recv(4)
size = struct.unpack('I', size_in_4_bytes)
size = size[0]
data = conn.recv(size)
return data
# --- main ---
host = '0.0.0.0'
port = 8080
s = socket.socket()
s.connect((host, port))
print("Connected to the server")
# ---
# send message
text = "Hello"
print('send:', text)
data = text.encode()
send_data(s, data)
# recv message
data = recv_data(s)
text = data.decode()
print('recv:', text)
# ---
# send JSON
stock = {'open': 12, 'close': 15}
print('send:', stock)
text = json.dumps(stock)
data = text.encode()
send_data(s, data)
# recv JSON
data = recv_data(s)
text = data.decode()
stock = json.loads(text)
print('recv:', stock)
# ---
s.close()
Similar way client could send filename and server could send back image data. But for files it may need receiving in chunks because socket has limited buffer. It may need also to send extra iformation if server found image or not.

What's the best way to reassemble bytes sent from client to server?

What's the best way to stream bytes from client to server in chunks of determined size?
Right now I'm encoding an audio file with base64, then compressing it with zlib and sending through the socket connection. My problem is trying to rebuild the original within the server.
I thought and tested using an empty string that is added with all the bytes the server is receiving. Seemed alright, but the " b' " in the beginning was being kept, which left it unable to recover the original audio file.
I've just tried to decode the bytes and deleting the " b' " from the beginning and " " " from the end (data[2:-1]) of each set of strings received by the server, but this cut a few characters from the original.
client side:
with open(arquivo, 'rb') as input_file:
abc = base64.b64encode(input_file.read())
try1 = zlib.compress(abc)
n = 338
result = [try1[i:i+n] for i in range(0, len(try1), n)]
HOST = ''
PORT = 9999
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.connect((HOST,PORT))
i = 0
for item in result:
item = str(item)
print(item)
s.send(item.encode())
i += 1
print('i = ', i)
time.sleep(2)
Server side:
HOST = ''
PORT = 9999
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
print('Servidor Inicializado')
s.bind((HOST,PORT))
s.listen()
audiofile = ''
i = 0
conn, addr = s.accept()
while True:
data1 = conn.recv(2048)
print('data1 undecoded = ', data1)
text = data1.decode()
data = text[2:-1]
print('data EDITADO = ', data)
audiofile = audiofile + data
i += 1
print('i = ', i)
print('audiofile = ', audiofile)
if not data:
print('No Data Received!')
print('Recebeu tratado :', data)
No idea how to proceed, any help is appreciated. Thanks!

Here is an example of how I send and receive data with sockets.
Typically I'll pickle them. If you're not familiar with pickle it's used to serialize python objects to store or send over connections such as sockets.
Client Side:
import pickle
with open(arquivo, 'rb') as input_file:
abc = base64.b64encode(input_file.read())
# I haven't used these libraries so I'm assuming you know how to unpack it from here
try1 = zlib.compress(abc)
HOST = ''
PORT = 9999
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.connect((HOST,PORT))
# serialize the python object
message = pickle.dumps(try1)
# get the length of the pickled object
length = len(message)
# convert into a fixed width string
length = str(length).rjust(8, '0')
# send the length of the object we will send
s.sendall(bytes(length, 'utf-8'))
# send the object
self.client.sendall(message)
Server Side:
import pickle
HOST = ''
PORT = 9999
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST,PORT))
s.listen()
conn, addr = s.accept()
# get the length of the object we are about to receive
length = conn.recv(8)
# turn it back into an int
length = int(length.decode('utf-8'))
# I keep this to determine if we've received everything or not
full_length = length
message = None
# loop until we've zeroed out our length
while length > 0:
# only receive what we need
# at a maximum of 128 bit chunks
chunk_len = min(128, length)
length -= chunk_len
chunk = conn.recv(chunk_len)
if message is None:
message = chunk
else:
message = message + chunk
# Edit: I've had issues with slow connections not receiving the full data
# for those cases adding something like this works
while len(message) < full_length:
chunk_len = min(128, full_length - len(message))
chunk = conn.recv(chunk_len)
message = message + chunk
# now that we've received everything, we turn it back into a python object
try1 = pickle.loads(message)
# this should be the same try1 you sent
Disclaimer: I did not test any of this, nor do I know what the try1 object is or what you want to do with it. this is just getting it from point a to point b.

Sending Multiple Files Python Using Socket

I currently am trying to create a client-server application in which the client can send multiple files to the server using TCP protocol. The server will eventually create a hash-algorithm and send it back to the client but I am running into issues sending multiple files from the client to the server. In it's current form, the first file sends correctly but the files after encounter an error where the information is merged together. IE the file size is listed as the second file's name. I am a javascript dude and very new to python so an explanation to how I can make this happen would be much appreciated. I believe threading is the answer but with my limited understanding of python, I do not know how to make this work. Currently I can send one file at a time and the server stays open. However, I would like to enter several file names from my current directory and have them processed. I eventually will convert the entire client side into C but I am struggling to get the server to work correctly in python. Any advice would be much appreciated!
Server.py
import socket
import hashlib
import threading
import struct
HOST = '127.0.0.1'
PORT = 2345
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(10)
print("Waiting for a connection.....")
conn, addr = s.accept()
print("Got a connection from ", addr)
while True:
hash_type = conn.recv(1024)
print('hash type: ', hash_type)
if not hash_type:
break
file_name = conn.recv(1024)
print('file name: ', file_name)
file_size = conn.recv(1024)
file_size = int(file_size, 2)
print('file size: ', file_size )
f = open(file_name, 'wb')
chunk_size = 4096
while file_size > 0:
if file_size < chunk_size:
chuk_size = file_size
data = conn.recv(chunk_size)
f.write(data)
file_size -= len(data)
f.close()
print('File received successfully')
s.close()
Client.py
import socket
import threading
import os
HOST = '127.0.0.1'
PORT = 2345
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
hash_type = input('Enter hash type: ')
files = input('Enter file(s) to send: ')
files_to_send = files.split()
for file_name in files_to_send:
s.send(hash_type.encode())
print(file_name)
s.send(file_name.encode())
file_size = os.path.getsize(file_name)
file_size = bin(file_size)
print(file_size)
s.send(file_size.encode())
f = open(file_name, 'rb')
l = f.read()
while(l):
s.send(l)
l = f.read()
f.close()
print('File Sent')
s.close()

One way to handle what you're doing is to buffer your socket data. Below is a class that buffers data and knows how to send and receive null-terminated, UTF-8-encoded strings, and raw chunks of bytes:
buffer.py:
class Buffer:
def __init__(self,s):
'''Buffer a pre-created socket.
'''
self.sock = s
self.buffer = b''
def get_bytes(self,n):
'''Read exactly n bytes from the buffered socket.
Return remaining buffer if <n bytes remain and socket closes.
'''
while len(self.buffer) < n:
data = self.sock.recv(1024)
if not data:
data = self.buffer
self.buffer = b''
return data
self.buffer += data
# split off the message bytes from the buffer.
data,self.buffer = self.buffer[:n],self.buffer[n:]
return data
def put_bytes(self,data):
self.sock.sendall(data)
def get_utf8(self):
'''Read a null-terminated UTF8 data string and decode it.
Return an empty string if the socket closes before receiving a null.
'''
while b'\x00' not in self.buffer:
data = self.sock.recv(1024)
if not data:
return ''
self.buffer += data
# split off the string from the buffer.
data,_,self.buffer = self.buffer.partition(b'\x00')
return data.decode()
def put_utf8(self,s):
if '\x00' in s:
raise ValueError('string contains delimiter(null)')
self.sock.sendall(s.encode() + b'\x00')
With this class, your client and server become:
client.py:
import socket
import threading
import os
import buffer
HOST = '127.0.0.1'
PORT = 2345
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
with s:
sbuf = buffer.Buffer(s)
hash_type = input('Enter hash type: ')
files = input('Enter file(s) to send: ')
files_to_send = files.split()
for file_name in files_to_send:
print(file_name)
sbuf.put_utf8(hash_type)
sbuf.put_utf8(file_name)
file_size = os.path.getsize(file_name)
sbuf.put_utf8(str(file_size))
with open(file_name, 'rb') as f:
sbuf.put_bytes(f.read())
print('File Sent')
server.py:
import socket
import os
import buffer
HOST = ''
PORT = 2345
# If server and client run in same local directory,
# need a separate place to store the uploads.
try:
os.mkdir('uploads')
except FileExistsError:
pass
s = socket.socket()
s.bind((HOST, PORT))
s.listen(10)
print("Waiting for a connection.....")
while True:
conn, addr = s.accept()
print("Got a connection from ", addr)
connbuf = buffer.Buffer(conn)
while True:
hash_type = connbuf.get_utf8()
if not hash_type:
break
print('hash type: ', hash_type)
file_name = connbuf.get_utf8()
if not file_name:
break
file_name = os.path.join('uploads',file_name)
print('file name: ', file_name)
file_size = int(connbuf.get_utf8())
print('file size: ', file_size )
with open(file_name, 'wb') as f:
remaining = file_size
while remaining:
chunk_size = 4096 if remaining >= 4096 else remaining
chunk = connbuf.get_bytes(chunk_size)
if not chunk: break
f.write(chunk)
remaining -= len(chunk)
if remaining:
print('File incomplete. Missing',remaining,'bytes.')
else:
print('File received successfully.')
print('Connection closed.')
conn.close()
Demo
client:
Enter hash type: abc
Enter file(s) to send: demo1.dat demo2.dat
demo1.dat
File Sent
demo2.dat
File Sent
server:
Waiting for a connection.....
Got a connection from ('127.0.0.1', 22126)
hash type: abc
file name: uploads\demo1.dat
file size: 488892
File received successfully.
hash type: abc
file name: uploads\demo2.dat
file size: 212992
File received successfully.
Connection closed.

1.
file_size = conn.recv(1024)
In your server code you read 1024 bytes as your file_size, file_size is only 4 or 8 bytes long
2.
file_name = conn.recv(1024) Your server don't know how long the filename/hashtype is.
-> Use a long for both sizes and read only sizeof(long) bytes from the stream.
You can use https://docs.python.org/2/library/struct.html for packing/encoding of these numbers
-> Or just go the easy way and use https://docs.python.org/3/library/pickle.html for serialization

copy data from one block device and write to another block device using python sockets

To achive this, i have written socket_client.py and socket_server.py. I put all my affort, but i could not achieve the result.
In the current code, i want that a socket_client.py should read block device data and send to socket_server.py with the offest value from where data has been read. In socket program i just want to unpack the data contained in struct and display on the screen.
Note:
Data returned after unpack will be returned in tuple.
My code contains lots of debugging lines. Please let me know if you
need any other details.
Please let me know if there is any better way to do this. I am new to pyhon. So as per my understanding i tried to write this code to achive the result.
Need help from all of you.
socket_client.py:
import socket
import sys
import os,pickle
import struct as st
dev_read_data = os.open("/dev/sdc1",os.O_RDONLY)
buffer_size = 230400 #1048576
offset = 0
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server_address = ('localhost',10000)
print ('connecting to {} port {}'.format(server_address[0],server_address[1]))
sock.connect(server_address)
data_list = []
sent_data_size = 0
try:
while True:
data = os.pread(dev_read_data,buffer_size,offset)
#Data should be binary as its string
packed_data = st.pack("%dsi" %(len(data),),data,offset)
st_size = st.calcsize("%dsi" %(len(data)))
print ('st_size {}'.format(st_size))
packed_st_size = st.pack("i",st_size)
sock.send(packed_st_size)
#Sending data
while sent_data_size < st_size: #Send data till sent_data_size < st_size i.e. struct size
print ('{} size expected to recev'.format(st_size))
print ('value of sent_data_size before send command {}'.format(sent_data_size))
sock.send(packed_data)
sent_data_size = sock.recv(16) # rec the amount of data sent from socket_client.py to socket_server.py. This info is sent by socket_server.py
print ('value of sent_data_size sock.send command {}'.format(sent_data_size))
sent_data_size = st.unpack("Q",sent_data_size) #unpacking sent_data_size so that it can be compared in while loop condition
print ('value of sent_data_size sock.send command {}'.format(sent_data_size))
print ('Inside while loop')
#print ('Sent from offset {} data size {}'.format(offset,buffer_size))
print ('Outside while loop')
offset += buffer_size # Changing the offset to read new data using pread.
if offset == 10036977152:
break
finally:
print ('Closing socket')
sock.close()
socket_server.py:
import sys
import socket
import os
import pickle
import struct as st
#create a tcp/ip socket
dev_write_data = os.open("/dev/sdb1",os.O_WRONLY)
buffer_size = 230400 #1048576
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server_address = ('localhost',10000)
print ('Starting up on {} port {}'.format(server_address[0],server_address[1]))
sock.bind(server_address)
sock.listen(5)
data_size = 0
while True:
print ('waiting for a connection')
connection,client_address = sock.accept()
try:
print ('connection from {}'.format(client_address))
packed_st_size = connection.recv(16) #rec size of struct sent from socket_client.py
print ('Struct size {}'.format(packed_st_size))
st_size = st.unpack("i",packed_st_size)[0] # After unpacking data returned is tuple. So this stmt takes 0th index data from tuple and initializing to st_size
print ('Outside while loop struct size {}'.format(st_size))
print ('Outside while Data_Size {}'.format(data_size))
while data_size <= st_size: #Keep looping till rec data_size <= st_size i.e size of struct which was actually sent.
print ('Inside while loop struct size {}'.format(st_size))
print ('Outside while Data_Size {}'.format(data_size))
if data_size == 0: # if data_size is 0, data variable is initialized with data send by socket_client.py
data = connection.recv(st_size)
else:
data += connection.recv(st_size) #Otherwise byte stream data is concatinated with sequentially sent data by socket_client.py
x = len(data) #get data len.
data_len_rec = st.pack("Q",x) #get the data len rec in bytes using struct pack method
connection.send(data_len_rec) #Sending the len of data recv to socket_client.py
print ('{} bytes received.'.format(x))
data_size += len(data) # Adding length of rec data to data_size for iteration
#final_data = st.unpack("%dsi" %(len(data)),data)
#print ('Final Data recv {}'.format(final_data))
finally:
connection.close()