I am trying to send queries coming from a client to a base server that forwards then to 2 more servers that the base server is connected to.
(Those 2 servers find a match with the query in their database, if it matches, they send the query back, if no match, they send nothing)
First I send the query from the base server to the 1st server called ts1, If I don't get a reply, I send it to 2nd server called ts2. If I don't get a reply from ts2 as well, I send a error host message to the client.
I've tried these while and inner while loops to accomplish this task but my program hangs after sending the first query. Because the ts1 server sends nothing on the first query.
How can I correct my code so it wont hang and the entire code executes? maybe its non-blocking or blocking of the sockets ? I don't know how that is implemented.
while True:
# receive a message from the client
data_from_client = csockid.recv(1024)
word = data_from_client.decode('utf-8').lower()
if word == '':
break
print('[Client]: ' + word)
print("[ls Server]: Sending " + word + " to ts1 & ts2 for Query....")
ts1.send(word.encode('utf-8'))
while True:
search_from_ts1 = ts1.recv(1024)
ts1_server_answer = search_from_ts1.decode('utf-8')
print("[ls Server]: sending reply from ts1 server " + ts1_server_answer + " to Client")
csockid.send(ts1_server_answer.encode('utf-8'))
if not search_from_ts1:
ts2.send(word.encode('utf-8'))
while True:
search_from_ts2 = ts2.recv(1024)
ts2_server_answer = search_from_ts2.decode('utf-8')
print("[ls Server]: sending reply from ts2 server " + ts2_server_answer + " to Client")
csockid.send(ts2_server_answer.encode('utf-8'))
if not search_from_ts2:
print("[ls Server]: No match found in both ts1 & ts2")
error = " - Error:HOST NOT FOUND"
csockid.send(error.encode('utf-8'))
break
break
if not data_from_client:
break
You might want to set a timeout, then try the other server, but that might make the code unacceptably slow. At this point you might want to explore async functions which give you the instruments for doing this properly (sending the request to both servers "simultaneously" and then take whatever is available but have a timeout on both request replies).
Have a look at the non-blocking sockets section in https://docs.python.org/2/howto/sockets.html for reference. It tells you how to set up non-blocking sockets and the use of select to poll more than one socket.
Related
I have a server-client code using TCP and Twisted. I want the first peer object that is created (by order of the first connected client) to serve (send messages) future upcoming clients as well. So I save the first peer (global list) and I use it for all upcoming connections but it only serves the first client (that it's connected to) while ignoring the others.
How can I make the peer to serve all connected clients simultaneously? (I'll test it for no more than 3 clients).
def connectionMade(self):
global connectedList
if self.pt == 'client':
self.connected = True
else:
print "Connected from", self.transport.client
try:
self.transport.write('<connection up>')
except Exception, e:
print e.args[0]
self.ts = time.time()
reactor.callLater(5, self.sendUpdate)
connectedList.append(self.transport) # add peer object
def sendUpdate(self):
global updateCounter, connectedList
print "Sending update"
try:
updateCounter += 1
print(connectedList[0])
# Send updates through first connected peer
connectedList[0].write('<update ' + str(updateCounter) + '>')
except Exception, ex1:
print "Exception trying to send: ", ex1.args[0]
if self.connected == True:
reactor.callLater(5, self.sendUpdate)
to serve (send messages) future upcoming clients as well
This sentence is difficult to understand. My interpretation is that you want sendUpdate to send messages to all of the clients except the first (ordered by when they connected).
but it only serves the first client
This is similarly difficult. My interpretation is that you observe a behavior in which only the first client (ordered by when they connected) receives any messages from the server.
Here is your code for sending messages to clients:
connectedList[0].write('<update ' + str(updateCounter) + '>')
Notice that this code always sends a message to connectedList[0]. That is, it only sends a message to one client - regardless of how many there are - and it always selects the first client in connectedList (which corresponds to the first client to connect to the server).
You may want something more like this:
for c in connectedList[1:]:
c.write('<update ' + str(updateCounter) + '>')
Notice how this sends a message to more than one client.
Also, unrelated to your question, you should eliminate your use of globals and you should avoid using a bare ITransport as your protocol interface.
I'm working on assignment where I need to connect to a server (no details of server are disclosed), capture the reply, modify it and send it back for verification.
I have created following code which does what I need, but the issue is that after 1st correct reply, server sends another.
Code:
# -*- encoding: utf-8 -*-
import socket
from decryptmsg import decryptmsg
from cleanmsg import cleanmsg
#connection
ip="<IP>"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
def recvall(sock):
BUFFER = 8192
data = b''
while True:
part = sock.recv(BUFFER)
data += part
if len(part) < BUFFER:
break
return data
while True:
print "[+] start communication"
data = recvall(sock)
print data
data = cleanmsg(data)
if data != None:
valmis = decryptmsg(str(data))
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis)
continue
When I hit the second question, I get the input captured fine with this code and processed as expected, but when I try to send the 2nd reply back I get error:
Traceback (most recent call last):
File "challenge.py", line 28, in <module>
sock.send(valmis)
socket.error: [Errno 32] Broken pipe
If I do not close or shutdown the socket, no reply is ever sent to server.
How can I tell my client to send the message and wait for reply without socket.shutdown? Or if I need to open new socket for each loop, how should the loop be constructed? The reply from server changes each time so if I open new connection completely and request for data, I get new reply and the process starts from beginning again.
UPDATE:
the issue seems to be when trying to receive the second reply from server, only the first line of message is received by client.
How do you know it does not send anything? I modified your code a bit (there is something odd in the else: clause, I will come back to that later).
import socket
#connection
ip="localhost"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
while True:
data = sock.recv(8192)
if not data: break;
print data
if data != None:
valmis = data
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis) #this never gets sent without
continue
Basically this is a stripped version of your code - no decrypting or external functionality. It just sends back whatever it receives from the server.
Then I ran a "server" with ncat:
ncat -l 4000
start your program and start typing in lines (1, 2, 3, 4 etc) and this happens at "server". The client promptly echoes my messages back:
test#xyzzy:/tmp$ ncat -l 4000
1
1
2
2
3
3
And this happens at the client:
test#xyzzy:/tmp$ python so.py
1
[+] sending message...
2
[+] sending message...
3
[+] sending message...
To me it looks this code works fine. If the server does not receive your reply, it might be that there is a problem on the server side. It might for example expect a terminator character in the response. Does your cleanmsg clean the message too much and for example remove a trailing newline and the server expects to receive one?
There is a problem in your original else clause as you do another sock.recv() there. Which means after receiving a reply, you block there to wait for the next message from server and when you do receive one, you will continue your loop and hit sock.recv() again. The second message was consumed already in your else clause.
This may be intentional if your server somehow acknowledges your decryption. If your protocol is this:
server -> client (request)
client -> server (decrypted message)
server -> client (some kind of acknowledgement - unclear from your code)
server -> client (request 2)
etc.
Then you have probably hit the issue in Jason's comment. TCP sockets are completely agnostic to the concept of a message. They just transmit data. When your code hits sock.recv(), one of five things can happen:
There is nothing in the socket and the call blocks
There is a full "message" and only that in the socket and you receive that
There is a partial message and you will receive that. Either because the message is over 8192 bytes, or your code just decides to read when the server has only transmitted some of the message data.
There are two or more complete "messages" waiting and you will receive them all.
As four, but the last message is partial
Always when operating with TCP sockets, you must cater for scenarios 2-5. You must parse the data, make sure everything is there, and if not, wait for more. And if there was more than you expected, process them accordingly. If they are complete messages, process them. If the last message is partial, process everything else and wait for more.
If messages seem to "disappear" in a self-made communication protocol using TCP sockets, 99% of the problems are caused by making the assumption that sockets would know or care what is the structure of your "message". A very common mistake is to read your socket empty and ignore everything you received after your first message.
Hope this is helpful. Your code without the additional recv seems to work fine from socket communication perspective - no need to shut down the socket. It may be a server side issue, protocol issue or message parsing problem.
For this reason, always have only one recv call for your socket. Even if you are expecting some kind of an acknowledgement instead of a new message, have only one place where you process socket data. Then do some kind of a conditional there to detect what kind of a message you received and then decide what to do with it.
I have created a client server setup from a tutorial, but the server is stuck in the while loop below and i can't figure out why.
I have searched through the questions on here and as far as I can see it should work.
I'm sure it will be something simple. I'm new to Python so any help would be appreciated.
Please know this is my first question so i hope it works ok.
I'm using python 3.5.2
# create a client thread
def clientthread(conn):
# sending message to the client
message = 'Welcome to the Server. Type something and hit Enter\n'
conn.send(message.encode()) # send only takes a string
while 1:
# receiving from client
data = conn.recv(1024)
if not data.decode(): break
reply = 'OK ' + data.decode()
conn.sendall(reply.encode())
# came out of loop
conn.close()
# keep server up
while 1:
# accept incomming connections
conn, addr = soc.accept()
# display client info
print('Connected with' + addr[0] + ':' + str(addr[1]))
# start a new thread takes function name and a tupal
start_new_thread(clientthread, (conn, ))
soc.close()
Figured out the issue, I was using telnet to test the server and it adds a \r\n to the end which the server see's as data so never shuts down. Added code to check for this and works fine now
this is my first post here!
My goal is to duplicate the payload of a unidirectional TCP stream and send this payload to multiple endpoints concurrently. I have a working prototype written in Python, however I am new to Python, and to Socket programming. Ideally the solution is capable of running in both Windows and *nix environments.
This prototype works, however it creates a new send TCP connection for each Buffer length (currently set to 4096 bytes). The main problem with this is I will eventually run out of local ports to send from, and ideally I would like the data to pass from each single incoming TCP stream to one single TCP stream out (for each endpoint). The incoming data can vary from less than 1024 bytes to hundreds of megabytes.
At the moment a new outgoing TCP stream is initiated for every 4096 bytes. I am not sure if the problem is in my implementation of threading, or if I have missed something else really obvious.
In my research I have found that select() could help, however I am not sure if it would be appropriate because I may need to process some of the incoming data and respond to the sending client for certain cases in the future.
Here is the code I have so far (some of the code variations I have tried are commented out):
#!/usr/bin/python
#One way TCP payload duplication
import sys
import threading
from socket import *
bufsize = 4096
host= ''
# Methods:
#handles sending the data to the endpoints
def send(endpoint,port,data):
sendSocket = socket(AF_INET, SOCK_STREAM)
#sendSocket.setblocking(1)
sendSocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
#sendport = sendSocket.getsockname
#print sendport
try:
sendSocket.connect((endpoint, port))
sendSocket.sendall(data)
except IOError as msg:
print "Send Failed. Error Code: " + str(msg[0]) + ' Message: ' + msg[1]
sys.exit()
#handles threading for sending data to endpoints
def forward(service, ENDPOINT_LIST, port, data):
#for each endpoint in the endpoint list start a new send thread
for endpoint in ENDPOINT_LIST:
print "Forwarding data for %s from %s:%s to %s:%s" % (service,host,port,endpoint,port)
#send(endpoint,port,data)
ethread = threading.Thread(target=send, args=(endpoint,port,data))
ethread.start()
#handles threading for incoming clients
def clientthread(conn,service,ENDPOINT_LIST,port):
while True:
#receive data form client
data = conn.recv(bufsize)
if not data:
break
cthread = threading.Thread(target=forward, args=(service, ENDPOINT_LIST, port, data))
cthread.start()
#no data? then close the connection
conn.close()
#handles listening to sockets for incoming connections
def listen(service, ENDPOINT_LIST, port):
#create the socket
listenSocket = socket(AF_INET, SOCK_STREAM)
#Allow reusing addresses - I think this is important to stop local ports getting eaten up by never-ending tcp streams that don't close
listenSocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
#try to bind the socket to host and port
try:
listenSocket.bind((host, port))
#display an error message if you can't
except IOError as msg:
print "Bind Failed. Error Code: " + str(msg[0]) + ' Message: ' + msg[1]
sys.exit()
#start listening on the socket
listenSocket.listen(10)
print "Service %s on port %s is listening" %(service,port)
while True:
#wait to accept a connection
conn, addr = listenSocket.accept()
print 'Connected to ' + addr[0] + ':' + str(addr[1]) + ' on port ' + str(port)
#start new thread for each connection
lthread = threading.Thread(target=clientthread , args=(conn,service,ENDPOINT_LIST,port))
lthread.start()
#If no data close the connection
listenSocket.close()
service = "Dumb-one-way-tcp-service-name1"
ENDPOINT_LIST = ["192.168.1.100","192.168.1.200"]
port = 55551
listen(service,ENDPOINT_LIST,port)
I have looked into other libraries to try to achieve my goal, including using:
Twisted
Asyncore
Scapy
However I found them quite complicated for my modest needs and programming skill level.
If anyone has any suggestions on how I could refine the approach I have, or any other ways this goal could be achieved, please let me know!
In short, your question is there are not enough ports, right? It seems you didn't close socket after sending. Try this in send():
...
except IOError as msg:
print "Send Failed. Error Code: " + str(msg[0]) + ' Message: ' + msg[1]
sys.exit()
finally:
sendSocket.close()
There are two ways, if you don't want to learn some more advanced framework like Twisted.
The closest to what you're doing: use threads, but you need to have one thread per outgoing connection --- and not per outgoing packet. Create 3 Queue.Queue objects, and create 3 threads, passing to each one one of the Queue objects and one of the destinations. Each thread opens a socket, and then in a loop, it gets the next string from its own Queue and sends it to the socket. The clientthread (which can be just the main thread, a priori) receives data as strings, and puts each of these strings into all the Queues. This way, the packets sent don't get out of order, as they could if you create one thread per packet.
The alternative is to avoid threads completely, and use select(). It's a bit more mind-bending. Basically you have only one big loop that starts with select(). It needs careful management to pass the correct list of sockets to select(): you want the call to select() to wake up either when there is incoming data from the inbound socket, or if an outbound socket is both ready to send more and there is something more to send. In this model you'd have 3 lists of strings; when you read incoming data you append it to all three lists; the select() call is passed the list of outbound sockets that have a non-empty list (so, more to send); and when sending, you must not use sendall() in this model but send(), and if less than the full string was sent, you must re-add the remainder to the beginning of the corresponding list.
I made a simple TCP fuzzer in Python. I need it to be able to receive some response and if I didn't get the response, break the loop. My code is this:
import socket
from time import sleep
import sys
ip = raw_input ("please insert host ip: ")
port = input ("please insert port to fuzz: ")
packet = raw_input ("what string would you like to fuzz with? : ")
multi = input ("in what jumps would you liike to multiply the string ? (10 = A*10) : ")
host = ip, port
s = socket.socket()
char = packet * multi
a = 1
try:
while a > 0:
s.connect((host))
s.send(packet)
sleep(1)
print 'fuzzing param %s' % (packet)
packet = char + packet
s.close()
except (Exception):
print "Connection lost for some reason"'
But when I run the program I get this error:
please insert host ip: 10.0.0.138
please insert port to fuzz: 80
what string would you like to fuzz with? : A
in what jumps would you liike to multiply the string ? (10 = A*10) : 2
fuzzing param A
Connection lost
which is weird because it just suppose to reconnect in an endless loop , (i know the server didn't crush)
The remote endpoint simply hung up, probably because the data you send doesn't match the format it expects.
You can either create a new connection every time the remote end hangs up, or send a data in the format that the remote end expects. For example, if the remote end is an HTTP server, you may want to send the request line first, and then the fuzzed part, like this:
GET / HTTP/1.0
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
When you fuzz testing (and in general) it is very important to handle errors. You should expect that something will get wrong when you are sending Gibberish to your server. So I suggest that you wrap the calls with try ... except ... finally: s.close() clause. And print debug messages to see when you are fail to send and start see why - You don't know how the server react to what you send, and you might just have killed the server after the first call...