We Keep Coding

Can't figure out how to send a 2D array over a socket - OSError: [WinError 10022]

I'm trying to send a 2D array across a socket, it needs to be made out of multiple socket.recv() otherwise I will get a _pickle.UnpicklingError: pickle data was truncated error.
I've tried to do this with a while loop that receives packets and appends them to a list until all data is received:
def receive(self):
packets = []
while True:
packet = self.socket.recv(1024)
if not packet:
break
packets.append(packet)
data = b"".join(packets)
data = pickle.loads(data)
return data
Just doing:
def receive(self):
data = self.socket.recv(1024)
data = pickle.loads(data)
return data
Works if I send something smaller than the 2D array e.g. a coordinate pair tuple (2, 3). But not with the 2D array.
I get a OSError: [WinError 10022] when attempting the while loop approach.
I have seen that a OSError: [WinError 10022] could be an issue with not binding the socket but I think I have done that.
I can't figure out where connections are closing and am very confused.
Rest of the code:
Server:
import socket
from _thread import start_new_thread
clients = []
def threaded(client):
while True:
data = client.recv(1024)
if not data:
print('Not Data.')
break
# Send the data received from one client to all the other clients.
for c in clients:
if c != client:
c.send(data)
client.close()
def Main():
host = ""
port = 5555
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
print("Socket binded to port:", port)
s.listen(2)
print("Socket is listening.")
while True:
c, addr = s.accept()
clients.append(c)
print(f"Connected to: {addr[0]}:{addr[1]}")
start_new_thread(threaded, (c,))
s.close()
Client:
import socket
import pickle
class Client:
def __init__(self):
self.host = 'localhost'
self.port = 5555
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((self.host, self.port))
def send(self, data):
message = pickle.dumps(data)
try:
self.socket.send(message)
except socket.error as e:
return str(e)
def receive(self):
packets = []
while True:
packet = self.socket.recv(1024)
if not packet:
break
packets.append(packet)
data = b"".join(packets)
data = pickle.loads(data)
return data
The client-server I want to make will have 2 clients and each client will send data to the other constantly. How can I correct my code and properly implement this?

After doing more research and figuring out what was going wrong I got a solution.
The code never got out of this while loop - because data was constantly being sent and some packet was always incoming.
def receive(self):
packets = []
while True:
packet = self.socket.recv(1024)
if not packet:
break
packets.append(packet)
A solution I found was to send a message indicating how big the message being received should be, so I can tell when all the bytes for a particular message have arrived. Links: Sockets Python 3.5: Socket server hangs forever on file receive, Python Socket Receive Large Amount of Data
def send(self, data):
message = pickle.dumps(data)
msg_len = len(message)
try:
# Send what the total length of the message to be sent is in bytes.
self.socket.send(msg_len.to_bytes(4, 'big'))
self.socket.sendall(message)
except socket.error as e:
return str(e)
def receive(self):
remaining = int.from_bytes(self.socket.recv(4), 'big')
chunks = []
while remaining:
# until there are bytes left...
# fetch remaining bytes or 4096 (whatever smaller)
chunk = self.socket.recv(min(remaining, 4096))
remaining -= len(chunk)
# write to file
chunks.append(chunk)
chunks = b"".join(chunks)
data = pickle.loads(chunks)
return data

How can I receive multiple messages from one connection?

I have a server and I need it to receive multiple connections and messages.
The server receives new connections without problems but it doesn't get multiple messages from one connection.
import socket
import select
HEADER_LENGTH = 1024
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
try:
server_socket.bind((HOST, PORT))
except socket.error as e:
print(str(e))
print("Server is connected")
server_socket.listen(5)
sockets_list = [server_socket]
clients = {}
print("Server is listening")
def receive_message(conn):
try:
data = conn.recv(HEADER_LENGTH)
if not len(data):
return False
strdata = data.decode('utf-8')
print(strdata)
return strdata
except Exception as e:
print(e)
return False
def handle_client():
conn, addr = server_socket.accept()
print(f"Accepted new connection from {addr[0]}:{addr[1]}")
sockets_list.append(conn)
while True:
read_sockets, _, exception_sockets = select.select(sockets_list, [], [], 0)
for i in read_sockets:
if i == server_socket:
handle_client()
else:
print("received message")
message = receive_message(i)
if message is False:
sockets_list.remove(i)
try:
del clients[i]
except KeyError:
pass
continue
if message is not None:
clients[i] = message
if message is not None:
for client_socket in clients:
if client_socket != i:
client_socket.send(str.encode(message))
print("sent to all players")
What happens it that after receiving the first message, the server stops receiving messages from that connection.
And of course there is a lot more code but I showed you the relevant code.
I'll be very happy if someone helps me with that, I've surfed the web so much but haven't seen a solution for my problem.
updates:
I've tried to put socket.close() on my client side(written in Java) and then server gets maximum 2 messages and the problems with it are:
1. The server gets maximum 2 messages.
2. the connection changes(I need that the connection will stay static if possible)

try this code block
#-*- coding:utf-8 -*-
import socket
import sys
#get machine ip address
server_ip = socket.gethostbyname(socket.gethostname())
#create socket object
s = socket.socket()
#define port number
port = 6666
#bind ip and port to server
s.bind((server_ip,port))
#now waiting for clinet to connect
s.listen(5)
print("Enter this ip to connect your clinet")
print(server_ip)
clients = []
flag = True
recv_data = ""
if not clients:
c, addr = s.accept()
print("this is c ",c," this is Addr ",addr)
clients.append(c)
recv_data = c.recv(1024)
print(recv_data.decode("utf-8"))
if flag == True:
while recv_data.decode("utf-8") != "EX":
recv_data = c.recv(1024)
recv_data.decode("utf-8")
if recv_data.decode("utf-8") == "EX":
s.close()
print("check false")
break
s.close()

Python TCP Server using select()

this is my first entry #stackoverflow;-))
I need a Python3 TCP server, which sends all inputs of the clients to the other participants. The server works, but unfortunately the message is only returned to the sending client. As soon as the second client sends something, it gets all entries.The entries are therefore available in the output queue.
Why are the entries not recognized by select ()?
Thx for the support.
`
import select
import socket
import sys
import queue
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setblocking(0)
server_address = ('192.168.100.28', 8888)
print ("starting up on %s port %s" % server_address)
server.bind(server_address)
server.listen(5)
inputs = [ server ]
outputs = [ ]
message_queues = {}
while inputs:
print ("\nwaiting for the next event")
readable, writable, exceptional = select.select(inputs, outputs, inputs)
Handle inputs
for s in readable:
if s is server:
# A "readable" server socket is ready to accept a connection
connection, client_address = s.accept()
print ("new connection from", client_address)
connection.setblocking(0)
inputs.append(connection)
# Give the connection a queue for data we want to send
message_queues[connection] = queue.Queue()
else:
data = s.recv(1024)
if data:
# A readable client socket has data
print ('received "%s" from %s' % (data, s.getpeername()))
# Add output channel for response
#message_queues[s].put(data)
if s not in outputs:
outputs.append(s)
for aQueue in message_queues:
message_queues[aQueue].put(data)
# AQueue.send ("Test".encode())
else:
# Interpret empty result as closed connection
print ("closing", client_address, "after reading no data")
# Stop listening for input on the connection
if s in outputs:
outputs.remove(s)
inputs.remove(s)
s.close()
# Remove message queue
del message_queues[s]
Handle outputs
for s in writable:
try:
next_msg = message_queues[s].get_nowait()
except queue.Empty:
# No messages waiting so stop checking for writability.
print ("output queue for", s.getpeername(), "is empty")
outputs.remove(s)
else:
print ('sending "%s" to %s' % (next_msg, s.getpeername()))
s.send(next_msg)
Handle "exceptional conditions"
for s in exceptional:
print ("handling exceptional condition for", s.getpeername())
# Stop listening for input on the connection
inputs.remove(s)
if s in outputs:
outputs.remove(s)
s.close()
# Remove message queue
del message_queues[s]
`

Your problem is that you only add a peer to the output list when you read something from it. You should instead do it when you write somthing to its message queue.
The receiving part should become:
...
data = s.recv(1024)
if data:
# A readable client socket has data
print ('received "%s" from %s' % (data, s.getpeername()))
# Add output channel for response
#message_queues[s].put(data)
# if s not in outputs:
# outputs.append(s)
for aQueue in message_queues:
message_queues[aQueue].put(data)
if aQueue not in outputs: # enqueue the msg
outputs.append(aQueue) # and ask select to warn when it can be sent
# AQueue.send ("Test".encode())
else:
...
According to your own requirements, you should only enqueue messages for other participants:
for aQueue in message_queues:
if aQueue != s: # only send to other participants
...
Finally, you often test existence of a socket in outputs. That let think that a set would be more appropriate than a list.

How to send stream trough socket when using select()?

After long hours of research and testing I finally ask here.
My script has to handle multiple client connections and in the same time has to get and send a stream from another socket.
Finally I've been able to make it work but only for one user. That user connects to the socket, the script connects to the other socket, then return the stream to the client.
The script works pretty well but has a some hard limitations :
- it send the stream to the client but,
- even if the socket is in non-blocking mode I think that calling a socket inside another one is the main reason why it reacts like it was in blocking mode (because one ot these is continuously sending datas ?)
By the way I think that the select() method could allow me to do what I want, but I don't clearly understand how.
Here is the server code taht works for one client, but is blocking
#!/usr/bin/env python
# coding: utf-8
from __future__ import print_function
import sys, time, base64, socket
server_ip = 'XX.XX.XX.XX'
def caster_connect(connected_client, address):
username = 'XXXXXXX'
password = 'XXXXXXXXX'
host = 'XX.XX.XX.XX'
port = 2102
pwd = base64.b64encode("{}:{}".format(username, password).encode('ascii'))
pwd = pwd.decode('ascii')
u_message = ''
stream_type = 'CMRp'
header = \
"GET /" + str(stream_type) + " HTTP/1.1\r\n" +\
"Host " + str(host) + "\r\n" +\
"Ntrip-Version: Ntrip/1.0\r\n" +\
"User-Agent: my_script.py/0.1\r\n" +\
"Accept: */*\r\n" +\
"Authorization: Basic {}\r\n\r\n".format(pwd) +\
"Connection: close\r\n"
print("Connecting to caster...\n")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host,int(port)))
s.send(header.encode('ascii'))
print("Waiting answer from caster...\n")
while True:
try:
data = s.recv(2048)
connected_client.send(data)
print("Sending data from caster at %s" % time.time())
sys.stdout.flush()
# On any error, close sockets
except socket.error, e:
print("No data received from caster : %s" % e)
print("Close client connection at %s" % format(address))
s.close()
break
return
#----------------
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((server_ip, 5680))
sock.settimeout(3)
try:
while True:
try:
sock.listen(5)
client, address = sock.accept()
print ("%s connected" % format(address) )
msg = client.recv(4096)
except socket.timeout, e:
err = e.args[0]
if err == 'timed out':
print("Timed out, retry later")
continue
else:
print(socket.error)
sock.close()
except socket.error:
print(socket.error)
sock.close()
else:
if len(msg) == 0:
print("Shutdown on client end")
sock.close()
else:
print(msg)
caster_response = caster_connect(client, address)
sys.stdout.flush()
print("Close")
client.close()
sock.close()`enter code here`
except KeyboardInterrupt:
print("W: Keyboard interrupt, closing socket")
finally:
sock.close()
And this is the code I found to handle select()
#!/usr/bin/env python
# coding: utf-8
import select, socket, sys, Queue
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setblocking(0)
server.bind(('XX.XX.XX.XX', 64000))
server.listen(5)
inputs = [server]
outputs = []
message_queues = {}
while inputs:
readable, writable, exceptional = select.select(
inputs, outputs, inputs)
for s in readable:
if s is server:
connection, client_address = s.accept()
print("New connection from %s" % client_address)
connection.setblocking(0)
inputs.append(connection)
message_queues[connection] = Queue.Queue()
else:
data = s.recv(1024)
print("Data received : %s" % data)
if data:
message_queues[s].put(data)
if s not in outputs:
outputs.append(s)
else:
if s in outputs:
outputs.remove(s)
inputs.remove(s)
s.close()
del message_queues[s]
for s in writable:
try:
next_msg = message_queues[s].get_nowait()
print("Next msg : %s" % next_msg)
except Queue.Empty:
outputs.remove(s)
else:
s.send(next_msg)
for s in exceptional:
inputs.remove(s)
if s in outputs:
outputs.remove(s)
s.close()
del message_queues[s]
In this code (found at this page) I didn't make changes as I don't know how to handle this.
Maybe by creating another server script that would only handle the stream part, so the main script would act as a server for clients, but as client for the stream part ?

How to detect when a client disconnects from a UDS (Unix Domain Socket)

When a client connects to the pipe, and sends data I can receive this fine and I can keep receiving the data. Trouble comes when the client disconnects and the while loop is still active, connection.recv() doesn't block and therefore keeps looping frantically! So I need a way to detect if a client is still connected.
I have the following code:
pipe = './pipes/uds_defzone-lrecv'
try:
os.unlink(pipe)
except OSError:
if os.path.exists(pipe):
raise
self.logger.debug('Created UDS pipe: ' + pipe)
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.bind(pipe)
sock.listen(1)
self.logger.debug('Waiting for connection: ' + pipe)
connection, client_address = sock.accept()
self.logger.debug('Connection from: ' + client_address)
while True:
self.logger.debug('Waiting for data')
data = connection.recv(4096)
self.logger.debug('Received: ' + str(data))
For reference, the sender.py code:
# Create a UDS socket
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
# Connect the socket to the port where the server is listening
pipe = './pipes/uds_defzone-lrecv'
logger.debug('connecting to: ' + pipe)
try:
sock.connect(pipe)
except socket.error, msg:
logger.debug(msg)
sys.exit(1)
try:
message = 'THIS IS A TEST'
logger.debug('sending: ' + message)
sock.sendall(message)
time.sleep(2)
finally:
logger.debug('closing socket')
sock.close()
TIA!
UPDATE
I can slow it down with the following code I suppose, but not exactly what I want.
while True:
try:
self.logger.debug('Waiting for data')
data_present = select.select([sock], [], [], 30)
if data_present[0]:
data = connection.recv(4096)
self.logger.debug('Received: ' + data)
except select.timeout:
pass
UPDATE 2
For reference this is the code I came up with:
while True:
logger.debug('Waiting for data')
data = connection.recv(4096)
if not data == '':
logger.debug('Received: ' + data)
else:
logger.debug('Nothing received')
break
A hack I came up with in the process... Might be usable where it is legitimate that a client might send empty data, for signalling perhaps?
while True:
try:
logger.debug('Waiting for data')
data = connection.recv(4096)
# *** This throws an exception when client has disconnected
x = connection.getpeername()
logger.debug('Received: ' + data)
except:
logger.debug('Client disconnected')
break

connection.recv() doesn't block and therefore keeps looping frantically! So I need a way to detect if a client is still connected.
If the peer disconnects recv data will return empty data (''). You need to check this and exit the loop.