I have made 2 application:
The client extract data from a sql server (10k lines), and send every line pickled to a "collector" server via socket.
The server uses twisted (this is mandatory) and receive every line, unpikle it and store the data in another sql server.
Everytime i start sending data from client to server, in the first 200 line (everytime a different line) the server throws an exception:
SOMETIMES it something like:
Traceback (most recent call last):
File "collector2.py", line 81, in dataReceived
self.count,account = pickle.loads(data)
File "/usr/lib/python2.6/pickle.py", line 1374, in loads
return Unpickler(file).load()
File "/usr/lib/python2.6/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.6/pickle.py", line 1138, in load_pop
del self.stack[-1]
IndexError: list assignment index out of range
But it's NOT every time the same. Printing my exception i red:
Exception: pop from empty list
Exception: list index out of range
Exception: "'"
Exception: list assignment index out of range
Another strange errors is:
File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
import(module)
exceptions.ImportError: No module named ond'
for i in listaSAI:
crm={}
try:
crm['uid']=i[0]
except:
crm['uid']=None
try:
crm['type_cond']=i[01]
except:
crm['type_cond']=None
try:
crm['population_id']=i[2]
except:
crm['population_id']=None
try:
crm['product_id']=i[3]
except:
crm['product_id']=None
try:
crm['a_id']=i[4]
except:
crm['a_id']=None
try:
crm['status']=i[5]
except:
crm['status']=None
#time.sleep(0.001)
serialized = pickle.dumps((count,crm))
#print "sent num", count, crm
s.sendall(serialized)
count += 1
And my server:
def dataReceived(self, data):
try:
self.count,account = pickle.loads(data)
except Exception as e:
print "Eccezione:", e
print self.count+1
print "DISAGIO", data
print traceback.print_exc()
Printing the data in my client tells me that everything it's ok.
If i try to slow down the process of sending using time.sleep(0.01) in my client, EVERYTHING IS FINE, and no exception are raised.
What can i do to debug my code?
p.s.
I suspect that exceptions.ImportError: No module named ond' refers to "type_cond" key in crm.
Since you have no problem when adding a delay between writes, it is clear the problem is not related to pickle, but to the transport of the data.
One option is that the client keeps writing data to the socket faster than the server can process it, resulting in buffer overflow and packet loss.
Or, do you have more than one thread writing to the socket potentially at the same time?
Your protocol has no framing, and you're trying to use a SOCK_STREAM (TCP connection) as if it were datagram-oriented (ie, as a SOCK_DGRAM, like UDP).
So you send some bytes that make up a complete pickled dataset, but the server has no idea where a particular pickle dataset begins or ends. Instead, the receiving code tries to unpickle whatever random chunk of bytes happens to be delivered to it.
As you've discovered, this works sometimes. However, there is no guarantee that the bytes you pass to sock.sendall will be delivered as a single group to dataReceived. You have to add framing so the receiver can discover message boundaries. There are several approaches to this. One approach is to add a length prefix. Then the receiver knows how many bytes to wait for and can unpickle them when it has received the correct number.
Of course, you should never ever unpickle data received over the network. See the big red box on http://docs.python.org/2/library/pickle.html? It's there because if you write a server like the one you're showing here, then you've written a server with a remote arbitrary code vulnerability in it big enough to drive a semi through.
Since your data seems fairly simple and structured, I suggest taking a look at AMP instead.
Related
7 socket listener setup. It works great and keeps the connection open, non blocking, all that. From time to time a file will show up that I need to hand back to the client. That works to, but it only send the data in the file if the client sends a character first. I need to have it send the data when the file shows up and not wait. I am coming from php and know what I am doing there. Python is new to me so there are some nuances I don't understand about this code.
while True:
try:
#I want this bit here to fire without waiting for the client to send anything
#right now it works except the client has to send a character first
#check for stuff to send back
for fname in os.listdir('data/%s/in' % dirname):
print(fname)
f = open('data/%s/in/%s' % (dirname, fname), "r")
client.send(f.readline())
data = client.recv(size)
if data:
bucket=bucket+data
else:
raise error('Client disconnected')
except Exception as e:
client.close()
print(e)
return False
I am trying to run a mini web server on my ESP32 feather huzzah32, but I hit an error every time I try to open the console. I tried several times, updated everything, erased the flash also few time to start over and the same.
The line 37 seems to have a problem I cannot figure out.
When I am done with compiling and upload the code, I see this error:
(IOError #[0047:0013:0000:0048:0000:0000:0000:0000]
oops, something wrong while linkin:
(IOError at line 37 of main raised at line 97 of wireless.wifi.link
Here is my code.
import streams
import socket
from wireless import wifi
from espressif.esp32net import esp32wifi as wifi_driver
streams.serial()
wifi_driver.auto_init()
wifi.link("my-ssid",wifi.WIFI_WPA2,"my-password")
except Exception as e:
print("ooops, something wrong while linking :(", e)
while True:
sleep(1000)
print("Linked!")
info = wifi.link_info()
print("My IP is:",info[0])
sock = socket.socket()
sock.bind(80)
sock.listen()
while True:
try:
client = streams.SocketStream(clientsock)
line = client.readline()
while line!="\n" and line!="\r\n":
line = client.readline()
print("HTTP request received!")
print("HTTP/1.1 200 OK\r",stream=client)
print("Content-Type: text/html\r",stream=client)
print("Connection: close\r\n\r",stream=client)
print("<html><body>Hello Zerynth!",random(0,100),"</body></html>",stream=client)
client.close()
except Exception as e:
print("ooops, something wrong:",e)
I am using Zerynth.
Any ideas?
I found the issue. It was a bit silly but often errors are indeed silly. Istead of using routers SSID, I tried with the name of the WIFI network and it worked just fine. There is the code, line 37.
wifi.link("NAME_OF_THE_NETWORK",wifi.WIFI_WPA2,"NETWORK_PASSWORD")
In my Python program I have a socket listening to incoming connections and when I run nmap on it, it closes abruptly as it resets the connection when scanning the port.
I tried putting an except: pass block on the end of my code, along with the following:
if __name__ == "__main__":
try:
listen()
except KeyboardInterrupt:
pass
But upon receiving the connection reset, it doesn't pass. It just closes the program without printing anything.
How might I solve this?
The server-side code responsible for receiving and next to it, the handling of info. Should I put something after the while block, or just an except?
connection.listen(10)
while True:
current_connection, address = connection.accept()
current_connection.send('Input:')
while True:
data = current_connection.recv(2048)
The error:
Traceback (most recent call last):
File "server.py", line 41, in <module>
listen()
File "server.py", line 12, in listen
current_connection.send('Input:')
socket.error: [Errno 104] Connection reset by peer
The way your code is structured, if you put an except: pass at the top-level, then when the exception occurs, it's only trapped at the top level and there's nothing in your code to cause it go back into your listen..accept code. You should instead trap the exception at the right spot in the code so that your code continues the outer while True loop.
In general. it's a good idea to place all socket operations that might ordinarily fail inside a try-except block. It's considered bad practice in python to have a bare except though -- as a bare except tends to hide errors in your code that should not be hidden. Hence, you'll typically do something like this:
connection.listen(10)
while True:
current_connection, address = connection.accept()
try:
current_connection.sendall('Input:')
while True:
data = current_connection.recv(2048)
if data == '':
break # End of file (peer socket was closed)
except socket.error as sockerr:
print("Socket exception on send/recv", sockerr)
current_connection.close()
Also, note that I replaced send with sendall. This ensures that the entire string will get sent. (With send, it's valid in some circumstances for fewer bytes to be sent than the entire string.)
I'm working on assignment where I need to connect to a server (no details of server are disclosed), capture the reply, modify it and send it back for verification.
I have created following code which does what I need, but the issue is that after 1st correct reply, server sends another.
Code:
# -*- encoding: utf-8 -*-
import socket
from decryptmsg import decryptmsg
from cleanmsg import cleanmsg
#connection
ip="<IP>"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
def recvall(sock):
BUFFER = 8192
data = b''
while True:
part = sock.recv(BUFFER)
data += part
if len(part) < BUFFER:
break
return data
while True:
print "[+] start communication"
data = recvall(sock)
print data
data = cleanmsg(data)
if data != None:
valmis = decryptmsg(str(data))
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis)
continue
When I hit the second question, I get the input captured fine with this code and processed as expected, but when I try to send the 2nd reply back I get error:
Traceback (most recent call last):
File "challenge.py", line 28, in <module>
sock.send(valmis)
socket.error: [Errno 32] Broken pipe
If I do not close or shutdown the socket, no reply is ever sent to server.
How can I tell my client to send the message and wait for reply without socket.shutdown? Or if I need to open new socket for each loop, how should the loop be constructed? The reply from server changes each time so if I open new connection completely and request for data, I get new reply and the process starts from beginning again.
UPDATE:
the issue seems to be when trying to receive the second reply from server, only the first line of message is received by client.
How do you know it does not send anything? I modified your code a bit (there is something odd in the else: clause, I will come back to that later).
import socket
#connection
ip="localhost"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
while True:
data = sock.recv(8192)
if not data: break;
print data
if data != None:
valmis = data
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis) #this never gets sent without
continue
Basically this is a stripped version of your code - no decrypting or external functionality. It just sends back whatever it receives from the server.
Then I ran a "server" with ncat:
ncat -l 4000
start your program and start typing in lines (1, 2, 3, 4 etc) and this happens at "server". The client promptly echoes my messages back:
test#xyzzy:/tmp$ ncat -l 4000
1
1
2
2
3
3
And this happens at the client:
test#xyzzy:/tmp$ python so.py
1
[+] sending message...
2
[+] sending message...
3
[+] sending message...
To me it looks this code works fine. If the server does not receive your reply, it might be that there is a problem on the server side. It might for example expect a terminator character in the response. Does your cleanmsg clean the message too much and for example remove a trailing newline and the server expects to receive one?
There is a problem in your original else clause as you do another sock.recv() there. Which means after receiving a reply, you block there to wait for the next message from server and when you do receive one, you will continue your loop and hit sock.recv() again. The second message was consumed already in your else clause.
This may be intentional if your server somehow acknowledges your decryption. If your protocol is this:
server -> client (request)
client -> server (decrypted message)
server -> client (some kind of acknowledgement - unclear from your code)
server -> client (request 2)
etc.
Then you have probably hit the issue in Jason's comment. TCP sockets are completely agnostic to the concept of a message. They just transmit data. When your code hits sock.recv(), one of five things can happen:
There is nothing in the socket and the call blocks
There is a full "message" and only that in the socket and you receive that
There is a partial message and you will receive that. Either because the message is over 8192 bytes, or your code just decides to read when the server has only transmitted some of the message data.
There are two or more complete "messages" waiting and you will receive them all.
As four, but the last message is partial
Always when operating with TCP sockets, you must cater for scenarios 2-5. You must parse the data, make sure everything is there, and if not, wait for more. And if there was more than you expected, process them accordingly. If they are complete messages, process them. If the last message is partial, process everything else and wait for more.
If messages seem to "disappear" in a self-made communication protocol using TCP sockets, 99% of the problems are caused by making the assumption that sockets would know or care what is the structure of your "message". A very common mistake is to read your socket empty and ignore everything you received after your first message.
Hope this is helpful. Your code without the additional recv seems to work fine from socket communication perspective - no need to shut down the socket. It may be a server side issue, protocol issue or message parsing problem.
For this reason, always have only one recv call for your socket. Even if you are expecting some kind of an acknowledgement instead of a new message, have only one place where you process socket data. Then do some kind of a conditional there to detect what kind of a message you received and then decide what to do with it.
Questions
What is the reason for the exception?
Did the client cause any errors?
If at all possible, please explain other errors.
Background
I am creating a Python GUI socket Server. When a client connects to my server, the GUI window will open (I am still working on this). But, when a client does connect, I get an error:
Unhandled exception in thread started by <function clientthread at 0x10246c230>
Since the actual script is rather long, I have provided a pastebin link.
Here is the thread code. s is the name of my socket object.
def clientthread(s):
#Sending message to connected client
#This only takes strings (words
s.send("Welcome to the server. Type something and hit enter\n")
#loop so that function does not terminate and the thread does not end
while True:
#Receiving from client
data = s.recv(1024)
if not data:
break
s.sendall(data)
print data
s.close()
Traceback
Thanks for the suggestion Morten. Here is the traceback.
Socket Created
Socket Bind Complete
Socket now listening
Connected
Traceback (most recent call last):
File "/Users/BigKids/Desktop/Coding/Python 2/Sockets/Function/Server GUI Alpha Function.py", line 80, in clientthread
s.send("Welcome to the server. Type something and hit enter\n")
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 170, in _dummy
raise error(EBADF, 'Bad file descriptor')
error: [Errno 9] Bad file descriptor
Personally, I believe that many errors are due to the GUI.
Thanks!
For one, you could catch the exception, print it and see what it is :)
Do this, for instance by surrounding it all with a try/except clause and printing whatever exception occurs.
def clientthread(s):
try:
#Sending message to connected client
#This only takes strings (words
s.send("Welcome to the server. Type something and hit enter\n")
#loop so that function does not terminate and the thread does not end
while True:
#Receiving from client
data = s.recv(1024)
if not data:
break
s.sendall(data)
print data
s.close()
except Exception:
import traceback
print traceback.format_exc()
I'm guessing the reason for this is client disconnect. This will cause an exception and you should handle it appropriately. If a client can disconnect in many ways. By telling you, by timing out, by dropping the connection while you're trying to send something etc.
All these scenarios are plausible exception cases, and you should test for them and handle them. Hopefully this will help you move on, if not, please comment :)