I'm making a client-server program, and there is problem with client part.
Problem is in infinite receiving data. I've tested this particular class, listed below, in a python interpreter. I've succesfuly(maybe not) connected to google, but then program stoped in function recvData() in data = self.socket.recv(1024)
class client():
def __init__(self, host, port):
self.host = host
self.port = port
self.socket = self.connect()
self.command = commands()
def connect(self):
'''
Connect to a remote host.
'''
try:
import socket
return socket.create_connection((self.host, self.port))
except socket.error:
print(":: Failed to connect to a remote port : ")
def sendCommand(self, comm):
'''
Send command to remote host
Returns server output
'''
comman = comm.encode()
# for case in switch(comman):
# if case(self.command.RETRV_FILES_LIST.encode()):
# self.socket.send(b'1')
# return self.recvData()
# if case():
# print(":: Got wrong command")
if (comman == b'1'):
self.socket.send(b'1')
return self.recvData()
def recvData(self):
'''
Receives all the data
'''
i = 0
total_data = []
while(True):
data = self.socket.recv(1024)
if not data: break
total_data.append(data)
i += 1
if i > 9:
break
return total_data
about commented part :
I thought problem in Case realization, so used just if-then statement. But it's not.
Your problem is that self.socket.recv(1024) only returns an empty string when the socket has been shut down on the server side and all data has been received. The way you coded your client, it has no idea that the full message has been received and waits for more. How you deal with the problem depends very much on the protocol used by the server.
Consider a web server. It sends a line-delimited header including a content-length parameter telling the client exactly how many bytes it should read. The client scans for newlines until the header is complete and then uses that value to do recv(exact_size) (if large, it can read chunks instead) so that the recv won't block when the last byte comes in.
Even then, there a decisions to make. The client knows how large the web page is but may want to send a partial data to the caller so it can start painting the page before all the data is received. Of course, the caller needs to know that is what happens - there is a protocol or set of rules for the API itself.
You need to define how the client knows a message is complete and what exactly it passes back to its caller. A great way to deal with the problem is to let some other protocol such as [zeromq](http://zeromq.org/ do the work for you. A simple python client / server can be implemented with xmlrpc. And there are many other ways.
You said you are implementing a client/server program then you mentioned "connected to google" and telnet... These are all very different things and a single client strategy won't work with all of them.
Related
I have a client application that is set to receive data from a given UDP port, and two servers (let's call them "primary" and "secondary") that are broadcasting data over that port.
I've set up a UDP receiver thread that uses a lossy queue to update my frontend. Lossy is okay here because the data are just status info strings, e.g. 'on'/'off', that I'm picking up periodically.
My desired behavior is as follows:
If the primary server is active and broadcasting, the client will accept data from the primary server only (regardless of data coming in from the secondary server)
If the primary server stops broadcasting, the client will accept data from the secondary server
If the primary server resumes broadcasting, don't cede to the primary unless the secondary server goes down (to prevent bouncing back and forth in the event that the primary sever is going in and out of failure)
If neither server is broadcasting, raise a flag
Currently the problem is that if both servers are broadcasting (which they will be most of the time), my client happily receives data from both and bounces back and forth between the two. I understand why this is happening, but I'm unsure how to stop it / work around it.
How can I structure my client to disregard data coming in from the secondary server as long as it's also getting data from the primary server?
NB - I'm using threads and queues here to keep my UDP operations from blocking my GUI
# EXAMPLE CLIENT APP
import queue
import socket as skt
import tkinter as tk
from threading import Event, Thread
class App(tk.Tk):
def __init__(self):
super().__init__()
self.title('UDP Client Test')
# set up window close handler
self.protocol('WM_DELETE_WINDOW', self.on_close)
# display the received value
self.data_label_var = tk.StringVar(self, 'No Data')
self.data_label = ttk.Label(self, textvariable=self.data_label_var)
self.data_label.pack()
# server IP addresses (example)
self.primary = '10.0.42.1'
self.secondary = '10.0.42.2'
self.port = 5555
self.timeout = 2.0
self.client_socket = self.get_client_socket(self.port, self.timeout)
self.dq_loop = None # placeholder for dequeue loop 'after' ID
self.receiver_queue = Queue(maxsize=1)
self.stop_event = Event()
self.receiver_thread = Thread(
name='status_receiver',
target=self.receiver_worker,
args=(
self.client_socket,
(self.primary, self.secondary),
self.receiver_queue,
self.stop_event
)
)
def get_client_socket(self, port: int, timeout: float) -> skt.socket:
"""Set up a UDP socket bound to the given port"""
client_socket = skt.socket(skt.AF_INET, skt.SOCK_DGRAM)
client_socket.settimeout(timeout)
client_socket.bind('', port) # accept traffic on this port from any IP address
return client_socket
#staticmethod
def receiver_worker(
socket: skt.socket,
addresses: tuple[str, str],
queue: queue.Queue,
stop_event: Event,
) -> None:
"""Thread worker that receives data over UDP and puts it in a lossy queue"""
primary, secondary = addresses # server IPs
while not stop_event.is_set(): # loop until application exit...
try:
data, server = socket.recvfrom(1024)
# here's where I'm having trouble - if traffic is coming in from both servers,
# there's a good chance my frontend will just pick up data from both alternately
# (and yes, I know these conditions do the same thing...for now)
if server[0] == primary:
queue.put_nowait((data, server))
elif server[0] == secondary:
queue.put_nowait((data, server))
else: # inbound traffic on the correct port, but from some other server
print('disredard...')
except queue.Full:
print('Queue full...') # not a problem, just here in case...
except skt.timeout:
print('Timeout...') # TODO
def receiver_dequeue(self) -> None:
"""Periodically fetch data from the worker queue and update the UI"""
try:
data, server = self.receiver_queue.get_nowait()
except queue.Empty:
pass # nothing to do
else: # update the label
self.data_label_var.set(data.decode())
finally: # continue updating 10x / second
self.dq_loop = self.after(100, self.receiver_dequeue)
def on_close(self) -> None:
"""Perform cleanup tasks on application exit"""
if self.dq_loop:
self.after_cancel(self.dq_loop)
self.stop_event.set() # stop the receiver thread loop
self.receiver_thread.join()
self.client_socket.close()
self.quit()
if __name__ == '__main__':
app = App()
app.mainloop()
My actual application is only slightly more complex than this, but the basic operation is the same: get data from UDP, use data to update UI...rinse and repeat
I suspect the changes need to be made to my receiver_worker method, but I'm not sure where to go from here. Any help is very much welcome and appreciated! And thanks for taking the time to read this long question!
Addendum: FWIW I did some reading about Selectors but I'm not sure how to go about implementing them in my case - if anybody can point me to a relevant example, that would be amazing
The core of the problem is: how do you determine that a given server is really offline as opposed to just temporarily taking a break, e.g. due to a momentary network glitch?
All your client really knows is whether it has received any UDP packets from a given source IP address recently or not, for some well-chosen definition of "recently". So what you can do in your client is update a per-IP-address member-variable to the current timestamp, whenever you receive a UDP packet from a given server. Then you can have a helper method like this (pseudocode):
def HowManyMillisecondsSinceTheLastUDPPacketWasReceivedFromServer(self, packetSourceIP):
{
return current_timestamp_milliseconds() - self._lastPacketReceiveTimeStamp[packetSourceIP]
}
Then e.g. if you know that your servers will be sending out a UDP packet once per second, you can decree that a given server is officially considered "offline" if you haven't received any UDP packets from it within the last 5 seconds. (Choose your own numbers here to suit, of course)
Then after you receive a packet and update the corresponding server-timestamp-member-variable, you can also update a member-variable indicating which server is the now the "active server" (i.e. the server you should currently be listening to):
def UpdateActiveServer(self)
{
millisSincePrimary = HowManyMillisecondsSinceTheLastUDPPacketWasReceivedFromServer(self._primaryServerIP)
millisSinceSecondary = HowManyMillisecondsSinceTheLastUDPPacketWasReceivedFromServer(self, _secondaryServerIP)
serverOfflineMillis = 5*1000 // 5 seconds
primaryIsOffline = (millisSincePrimary >= serverOfflineMillis)
secondaryIsOffline = (millisSinceSecondary >= serverOfflineMillis)
if ((primaryIsOffline) and (not secondaryIsOffline)):
self._usePacketsFromSecondaryServer = true
if ((secondaryIsOffline) and (not primaryIsOffline)):
self._usePacketsFromSecondaryServer = false
}
.... then the rest of your code can check the current value of self._usePacketsFromSecondaryServer to decide which incoming UDP packets to listen to and which ones to ignore (pseudocode):
def PacketReceived(whichServer):
if ((whichServer == self._primaryServerIP) and (not self._usePacketsFromSecondaryServer)) or ((whichServer == self._secondaryServerIP) and (self._usePacketsFromSecondaryServer)):
# code to parse and use UDP packet goes here
I'm trying to print file catalogue from FTP server. I have two sockets: first to send commands to the server, second to get requested data. To parse data, I want to send LIST command multiple times, but I don't know how to do this properly. Here is my code:
import socket
HOST = '[IP address]'
USERNAME = '[login]'
PASSWORD = '[password]'
sock_1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock_1.connect((HOST, 21))
sock_1.recv(1024)
sock_1.send(f'USER {USERNAME}\r\n'.encode())
sock_1.recv(1024)
sock_1.send(f'PASS {PASSWORD}\r\n'.encode())
sock_1.recv(1024)
sock_1.send('PASV\r\n'.encode())
response_values = sock_1.recv(1024).decode('utf-8').split(',')
port_high = int(response_values[-2])
port_low = int(response_values[-1].replace(').\r\n', ''))
port = port_high*256 + port_low
sock_2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock_2.connect((HOST, port))
# Works as expected.
sock_1.send('LIST\r\n'.encode())
print(sock_1.recv(4096))
print(sock_2.recv(4096))
# Here comes a problem.
sock_1.send('LIST\r\n'.encode())
print(sock_1.recv(4096))
print(sock_2.recv(4096))
And the output I get:
b'150 Here comes the directory listing.\r\n'
b'[a totally expected array of bytes that fits into the buffer entirely]'
b'226 Directory send OK.\r\n'
b''
I feel like I'm doing something wrong on a theoretical level, but since I'm new to socket programming, I can't figure what exactly.
I think I can just recreate the connection every time I want to send another command, but there must be a better solution.
I think I can just recreate the connection every time I want to send another command, but there must be a better solution.
FTP has a control connection for all commands and a data connection for each command transferring data. So the control connection (sock_1 in your code) will be kept open, the data connection must be established new though for each new LIST command.
Creating a new data connection means sending the PASV command, parsing the response to get the new server side address and connecting to this address and then (in case of LIST) reading all data from this data connection until the other side closes the connection (recv returns 0).
None of this is actually specific to socket programming, instead it is how the FTP protocol is defined at the application layer - see RFC 959 for the details.
so I basically have just a Client & Server application going on here and the server sends out data in a loop that looks like this {"Message": "Status Check From Server"}
For some reason after the server sends this out around 3 times, the Client then stops receiving data for about 10 seconds then prints out what looks like multiple merged messages of the same data, it looks something like this {"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Serv I literally have no clue what this is, I've tried so hard to debug this and just can't figure it out, I don't know if it's my code or if this is simply how TCP works. I'll put some of the code down below.
SERVER SIDE | HAD TO CHANGE SOME OF THE CODE IN HERE TO MAKE SENSE
# Binding Of The Socket
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.bind(('0.0.0.0', 777))
SOCK.listen()
# This Is The Function I Have Written Up That Sends That Status Check Message To The Clients
def DeviceChecker():
global DEVICE_LIST
while True:
for DEVICE in DEVICE_LIST:
try:
DEVICE.send(json.dumps({'Message': 'Status Check From Server'}).encode())
DATA = DEVICE.recv(4096, socket.MSG_PEEK)
if len(DATA) == 0:
raise BrokenPipeError
except BrokenPipeError:
DEVICE_LIST.remove(DEVICE)
CLIENT SIDE
# This Is Where The Client Connects To The Server
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.connect(('0.0.0.0', 777))
# This Is Where The Client Is Receiving The Data
def DataListener(SOCK):
while True:
try:
DATA = SOCK.recv(4096).strip()
print(DATA)
except BrokenPipeError:
return
Because of the way TCP sockets work, individual write() calls do not result in individual "messages". TCP is designed to break large data streams into packets for transmission over the internet, and rejoin them on the receiving end. The stream of data from one socket to another, in TCP streams at least, is continuous, and individual write()'s are lumped into the same buffer. This is the same issue as in this answered question which describes creating a file-like object using socket.makefile(), then using write() and then flush() on that object to ensure all the data is sent. If you need to make sure your messages are received one at a time, you can have your client send an acknowlegement after each message it receives, and have your server wait for that acknowlegement before sending the next message. As a side note, the socket.send() method is not guaranteed send all the data you give it, and will return, as an int, the number of bytes it actually sent. If you need to make sure all the data is sent in one function call, use socket.sendall(). That will call send() as many times as it needs to.
A "direct-tcpip" request (commonly known as port-forwarding) occurs when you run SSH as ssh user#host -L <local port>:<remote host>:<remote port> and then try to connect over the local port.
I'm trying to implement direct-tcpip on a custom SSH server, and Paramiko offers the check_channel_direct_tcpip_request in the ServerInterface class in order to check if the "direct-tcpip" request should be allowed, which can be implemented as follows:
class Server(paramiko.ServerInterface):
# ...
def check_channel_direct_tcpip_request(self, chanid, origin, destination):
return paramiko.OPEN_SUCCEEDED
However, when I use the aforementioned SSH command, and connect over the local port, nothing happens, probably because I need to implement the connection handling myself.
Reading the documentation, it also appears that the channel is only opened after OPEN_SUCCEDED has been returned.
How can I handle the direct-tcpip request after returning OPEN_SUCCEEDED for the request?
You indeed do need to set up your own connection handler. This is a lengthy answer to explain the steps I took - some of it you will not need if your server code already works. The whole running server example in its entirety is here: https://controlc.com/25439153
I used the Paramiko example server code from here https://github.com/paramiko/paramiko/blob/master/demos/demo_server.py as a basis and implanted some socket code on that. This does not have any error handling, thread related niceties or anything else "proper" for that matter but it allows you to use the port forwarder.
This also has a lot of things you do not need as I did not want to start tidying up a dummy example code. Apologies for that.
To start with, we need the forwarder tools. This creates a thread to run the "tunnel" forwarder. This also answers to your question where you get your channel. You accept() it from the transport but you need to do that in the forwarder thread. As you stated in your OP, it is not there yet in the check_channel_direct_tcpip_request() function but it will be eventually available to the thread.
def tunnel(sock, chan, chunk_size=1024):
while True:
r, w, x = select.select([sock, chan], [], [])
if sock in r:
data = sock.recv(chunk_size)
if len(data) == 0:
break
chan.send(data)
if chan in r:
data = chan.recv(chunk_size)
if len(data) == 0:
break
sock.send(data)
chan.close()
sock.close()
class ForwardClient(threading.Thread):
daemon = True
# chanid = 0
def __init__(self, address, transport, chanid):
threading.Thread.__init__(self)
self.socket = socket.create_connection(address)
self.transport = transport
self.chanid = chanid
def run(self):
while True:
chan = self.transport.accept(10)
if chan == None:
continue
print("Got new channel (id: %i).", chan.get_id())
if chan.get_id() == self.chanid:
break
peer = self.socket.getpeername()
try:
tunnel(self.socket, chan)
except:
pass
Back to the example server code. Your server class needs to have transport as a parameter, unlike in the example code:
class Server(paramiko.ServerInterface):
# 'data' is the output of base64.b64encode(key)
# (using the "user_rsa_key" files)
data = (
b"AAAAB3NzaC1yc2EAAAABIwAAAIEAyO4it3fHlmGZWJaGrfeHOVY7RWO3P9M7hp"
b"fAu7jJ2d7eothvfeuoRFtJwhUmZDluRdFyhFY/hFAh76PJKGAusIqIQKlkJxMC"
b"KDqIexkgHAfID/6mqvmnSJf0b5W8v5h2pI/stOSwTQ+pxVhwJ9ctYDhRSlF0iT"
b"UWT10hcuO4Ks8="
)
good_pub_key = paramiko.RSAKey(data=decodebytes(data))
def __init__(self, transport):
self.transport = transport
self.event = threading.Event()
Then you will override the relevant method and create the forwarder there:
def check_channel_direct_tcpip_request(self, chanid, origin, destination):
print(chanid, origin, destination)
f = ForwardClient(destination, self.transport, chanid)
f.start()
return paramiko.OPEN_SUCCEEDED
You need to add transport parameter to the creation of the server class:
t.add_server_key(host_key)
server = Server(t)
This example server requires you to have a RSA private key in the directory named test_rsa.key. Create any RSA key there, you do not need it but I did not bother to strip the use of it off the code.
You can then run your server (runs on port 2200) and issue
ssh -p 2200 -L 2300:www.google.com:80 robey#localhost
(password is foo)
Now when you try
telnet localhost 2300
and type something there, you will get a response from Google.
I am writing a client-sever program based on Python socket.
The client sends a command to the server and the server responds.
But now, some client can broadcast a message to other clients, so the client can receive more than one response at the same time.
data = s.recv(1024)
the line of code above will retrieve only one response from the server.
but if I use a while loop like this
while True:
data = s.recv(1024)
if not data: break
actually, data=s.recv(1024) will block the program when there is no data left.
I don't want to block the program and want to retrieve all the responses available in the connection at one time. Can anyone find a solution? Thank you.
You can use the select module to wait until the socket is readable or until a timeout has elapsed; you can then perform other processing. For example:
while True:
# If data can be received without blocking (timeout=0), read it now
ready = select.select([s], [], [], 0)
if s in ready[0]:
data = s.recv(1024)
# Process data
else:
# No data is available, perform other tasks
You could make the socket (s) non-blocking. This way, it will retrieve all the received responses and when there is none, it will return back. Of course, with non-blocking, you will have to periodically retry.
You could make the socket (s) non-blocking using the setblocking() method:
s.setblocking(0)
The other option is to use another thread to handle the receive part. This way, your main thread can continue doing its main task and act upon the message only if it receives one.
You can use socket.setblocking or socket.settimeout:
import socket
import sys
HOST = 'www.google.com'
PORT = 80
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.setblocking(0)
s.sendall('Hello, world')
try:
data = s.recv(1024)
except:
print 'Oh noes! %s' % sys.exc_info()[0]
s.close()
socket.recv takes two parameters, the second is a set of flags. If you're on a Linux system, you can do man recv for a list of flags you can supply, and their corresponding errors.
Lastly, in general, you can't really know that the other side is done with sending you data (unless you're controlling both sides), even if you're both following a protocol. I believe the right way to go about it is to use timeouts, and quit after sending a reset (how you do this will depend upon what protocol you're using).